disable journalling?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
I would like to disable journalling on a Pi that is running as a server,
with reliable power.  I read that the journalling writes data to the
same place all the time, and thus causes the SD card to fail at some
point in time.  Turning off journalling should improve that.
(I know the risk)

It is easy to turn off journalling on ext4, but the directions for doing
that always mention that the filesystem should first be unmounted,
then the flag can be toggled, then the filesystem should be checked
and thereafter it can be mounted again.

This is a bit tricky when one wants to set the option for the root
filesystem, and especially on a (remotely located) Pi.
Is it possible to do this in any way?

Re: disable journalling?
On 03/03/14 09:57, Rob wrote:
Quoted text here. Click to load it

Plug the SD card into a USB card reader; plug that into another linux
box, and do your magic.

Re: disable journalling?
On 03/03/2014 11:12, Tony van der Hoff wrote:
Quoted text here. Click to load it

Dude.


Re: disable journalling?
On Mon, 03 Mar 2014 10:12:06 +0000, Tony van der Hoff

Quoted text here. Click to load it
 What about system crashes?
Quoted text here. Click to load it
The firmware in the card prevents the data being written to the same
place, so you only need to be concerned with the total write capacity
of the device.

There will be extra wear on the device with journalling and
also some delays.  

My choice would be to double the size of the memory device
if possible and leave journalling on.  Avoiding 1 filesystem failure
due to a system crash or failure of the reliable power over the life
of a dozen or so devices will pay for doubling all of the storage.  

Quoted text here. Click to load it

Re: disable journalling?
Quoted text here. Click to load it

They have not happened yet.
I am running Linux since 1992 and for many years I used ext2 and never
had any problem due to the (infrequent) system crashes.  I think it is
not an important issue.

Quoted text here. Click to load it

Unfortunately the scarce sources of information are not definite on this.
Some state that this only happens in the Sata SSD disks that are used
in PCs as system disk replacement, and not on SD cards.

Also the "fstrim" command, that issued OK results in previous kernels,
no longer works since the Jan 3 #622 kernel.  It is unclear to me if it
never worked and returned fake results, or it worked but no longer does.

I also see claims that "the reason SD cards fail so quickly in the Pi
is the journalling"  (but I have not yet had an SD card fail)

Quoted text here. Click to load it

My most important Pi has about 15% of used space on the card.
However, it is unclear to me if the card "knows" that, given that fstrim
apparently no longer works, and of course the card is written with an
image at install time (and so the card may consider all blocks to be
in use)

Re: disable journalling?
On Mon, 03 Mar 2014 17:32:22 +0000, Rob wrote:

Quoted text here. Click to load it
Well, AFAIK nothing apart from Flash devices use wear levelling  
algorithms and so you won't fine them in the drivers of any operating  
system. So, it follows that the wear levelling must be in the SD card's  
internal controllers - and its the wear levelling algorithms that make  
sure that write hotspots don't stay in one spot on the card.

I agree that using a bigger card is a good idea. Look at it this way:  
wear levelling isn't going to mess with blocks that contain (relatively)  
static data, so wear levelling will only cause logical blocks to migrate  
round the free space on the card.  

In the light of that, I recently decided that my 4GB card was a bit small  
since 'stuff' on on the ext4 partition occupies 2.7 GB I've recently  
migrated my Raspbian setup onto an 8GB card.

That was simple:  
- connect two SD readers to a bigger Fedora box
- use cfdisk to create partitions on the new card
- use mke2fs and mkfs.dos to format the partitions
- use dd to shift all the data across.  

At this time, although all the data have been moved correctly, mounting  
the new SD card on the Fedora box showed that something thought the new  
ext4 partition was the same size as it had been on the 4GB card. However,  
resize2fs soon fixed that.  

Moved the card onto my RPi. It booted immediately and the free space is  
now up from around 1.3GB to 4.3GB.

One oddity, though some of the lower level views of Raspbian partitions  
seem to show that its using a 4Kb block size. Might this be an SD card  
optimisation? I believe that the SD card's internal block size (the one  
used for wear levelling) is 4Kb - it just seems like a somewhat oversized  
block for a Linux filing system, though equating it to wear-levelling  
units would make a lot of sense.


--  
martin@   | Martin Gregorie
gregorie. | Essex, UK
We've slightly trimmed the long signature. Click to see the full one.
Re: disable journalling?
On 03/03/14 23:14, Martin Gregorie wrote:
Quoted text here. Click to load it

This is true.

Quoted text here. Click to load it

However, apart from blocks that haven't been written to at all, the  
controller has no idea of "free space". Once a block has been written it  
assumes there is data on it, even though the file may have been deleted  
from the file system. This is where TRIM comes into play. It can tell  
the SD card which blocks are currently in use and which really are free.
Quoted text here. Click to load it

Two things identify the filesystem size. One is the partition table  
which identifies how much space has been reserved for the file system on  
the storage medium. The other is the filesystem superblock.

BTW you don't need to use mkfs to create filesystems on the new card if  
you are using dd to copy an existing fs onto it. dd will overwrite all  
the formatting info that mkfs. creates.
Quoted text here. Click to load it

4KB is the normal blocksize for ext[34] filesystems. All my Debian based  
PCs have 4KB block sizes on their hard drives.


Re: disable journalling?
On Tue, 04 Mar 2014 04:55:04 +0000, Dom wrote:

Quoted text here. Click to load it
OK, so resize2fs wrote the new FS size to all superblocks in the  
partition. I was wondering what it had updated. Obvious with hindsight.

Quoted text here. Click to load it
I knew that dd would only overwrite just under 4 GB of the new partition,  
but not what it would make of the MS fs blocks occupying the rest of it,  
I thought that reformatting before using dd would be sensible.
  
Quoted text here. Click to load it
Thanks - I had a scan round but didn't find anything definite. Is 4KB an  
actual block size or is it really a cluster of four 1KB blocks? I picked  
up hints that ext4 tends to work with clusters of blocks as its read/
write unit and that it might favour 4KB clusters that contain 4 blocks.

Presumably and ext[34] fs can't put data from more than one file on a  
block, so, as many Linux files are smaller than 4KB, using 4KB blocks  
would seem to be fairly wasteful of disk space.



--  
martin@   | Martin Gregorie
gregorie. | Essex, UK
We've slightly trimmed the long signature. Click to see the full one.
Re: disable journalling?
Quoted text here. Click to load it

It still hasn't caught up with ReiserFS...

However, it is not as important as it used to be, except maybe for a
system that stores mail in maildir format.  Today, the average file
size is a lot more than 20 years ago.

Re: disable journalling?
On 04/03/14 21:29, Martin Gregorie wrote:
Quoted text here. Click to load it

As the filesystem doesn't know about the additional blocks (until you  
run resize2fs), it doesn't care what is in them. Running resize2fs will  
update and create new superblocks as needed. It won't touch any blocks  
that are then defined as free space as there is no data to write to them  
yet.
Quoted text here. Click to load it

I believe it is actual 4KB blocks. They seem to be arranged in larger  
groups so a new file may take a number of blocks, but when the space  
starts to get limited those extra blocks will be reallocated to other  
files to make most use of the available space. Initially allocating a  
number of blocks to a file helps to prevent any fragmentation - as long  
as there is sufficient free space.
Quoted text here. Click to load it

It's only a small amount of wastage, and the larger hard disks currently  
available use 4KB sectors anyway, so using smaller block sizes would be  
slow - to write a 512Byte logical sector would mean reading 4KB,  
updating the content of 1/8 of it and writing it out again instead of  
just writing 4KB in the first place.

When you do a dumpe2fs to check the file system status, compare the  
Inode Count to the Block Count. I think you'll find that it's at least 2  
blocks per inode. So that is a minimum of 2 blocks per file (more if you  
allow for linked files.


Re: disable journalling?
On Wed, 05 Mar 2014 07:01:13 +0000, Dom wrote:

Quoted text here. Click to load it
Noted.

OK

Just for fun I wrote a script that pipes the output of "ls -lR  $1/*"  
into a small AWK program that reports the number of files it finds along  
with max, min and average file sizes. Rather to my surprise, it shows the  
average file size to be around 5-5.5 KB, so quite a lot bigger than I  
expected to find.


--  
martin@   | Martin Gregorie
gregorie. | Essex, UK
We've slightly trimmed the long signature. Click to see the full one.
Re: disable journalling?
On Wed, 05 Mar 2014 03:11:25 +0000, Johny B Good wrote:

Quoted text here. Click to load it
More than I wish to remember, but IIRC all FAT file systems used 512 byte  
blocks. The problem was always the size of the FAT table, which had a  
fixed size and in which each entry corresponded to a disk block (FAT8).  
Eventually disks outgrew this limit and the quick fix was to say that  
each FAT entry referenced a cluster of 8 contiguous blocks (FAT12 and  
FAT16). After that the cluster size increased still further.

As you said, a bloody nightmare - only two copies of the FAT table and  
AFAIK nothing in the blocks to link them back to the FAT table which, in  
turn had no detectable integrity checking features so, as we all know  
full well, it was a very fragile structure. And don't even think of  
mention fragmentation the the fun & games of finding a reliable defragger.  
Hint: that wasn't the M$ one - I've seen *that* wonderfiul bit of  
software totally destroy a disk's contents more than once.
    

--  
martin@   | Martin Gregorie
gregorie. | Essex, UK
We've slightly trimmed the long signature. Click to see the full one.
Re: disable journalling?
On Wed, 5 Mar 2014 22:11:30 +0000 (UTC), Martin Gregorie

Quoted text here. Click to load it

 I'm afraid that's contrary to my experience. Perhaps you're recalling
early DOS versions compounded by running it on disks with bad sectors?

 I'm quite confident in the versions current from as far back as
win95OSR2. Indeed, it proved to be quite bombproof a few years back
when I was dealing with a system that had a peculiar instability
induced by a compatability issue between the hard disk controller and
the detected UDMA mode which caused random system lockups (but, not so
random when running defrag).

 Having found something (defrag) that would make the issue
'repeatable', I used it to good effect to determine an effective cure.

 In the process of repeated lockups during defrag attempts, I never
once saw any issues of data loss (nor did the ensuing CHKDSK /R scan
have to fix any problems, validating the robustness of the algorithms
used by defrag).
--  
Regards, J B Good

Re: disable journalling?
On Thu, 06 Mar 2014 16:02:18 +0000, Johny B Good wrote:

Quoted text here. Click to load it
The case I'm thinking of was a brand new Olivetti, but I don't now  
remember it is was running WfW 3.11 or W95 - it was certainly in 1994.  
We'd been using the machine for less than a week when this happened.
  
I don't recall how the disk got corrupted - not my PC - and I'm not ever  
sure it was a crash. Anyway, Windows was a bit unhappy but was running,  
so we tried the Windows defragger and 5 minutes later the disk was  
completely unusable. Can't have been a bad disk, because we reformatted  
it and reinstalled Windows and applications successfully. That wasn't the  
only M$ defrag disaster I've seen  - just the one I remember best.

After that it was a matter of 'once bitten, twice shy' and I've never  
used a M$ defragger since - why bother when the one in Norton Utilities  
was so much better.


--  
martin@   | Martin Gregorie
gregorie. | Essex, UK
We've slightly trimmed the long signature. Click to see the full one.
Re: disable journalling?
On Thu, 6 Mar 2014 22:01:16 +0000 (UTC), Martin Gregorie

Quoted text here. Click to load it

 So, MSDOS 6.xx and WFWG then (win95 didn't appear until August '95 -
the year it was _supposed_to_ ship - the proper version with FAT32
support didn't appear until a year later).

 I think the dos version of defrag was prone to trashing the data if
the FS had uncorrected consistency errors. It was always best to run
scandisk beforehand to make sure FS errors weren't lying in wait to
screw up the defrag run.

Quoted text here. Click to load it

 With WFWG, I don't believe there ever was a windows version of
defrag. You needed to exit the windows shell before running defrag
from the command line.

 I can't recall whether the windows shell would prevent defrag being
launched but I suspect that if it didn't and you launched it from
within windows without windows closing automatically to dump you back
to DOS, you'd likely suffer corruption from windows' background
activity interfering with the defrag activity. It's all so long ago
now so ICBW.

Quoted text here. Click to load it

 Agreed! I must admit that I preferred Peter Norton's defrag tool over
the MS one when dealing with FAT volumes. I suspect the defrag tool in
win95OSR2 was vastly improved over the earlier DOS versions (but,
again, this was a long time ago so only have a hazy impression that
this was so).

 Certainly by the time I discovered the joys of win2KSP4 sometime
around 2004/5, I had every confidence in the MS defrag utility and
stopped using the Peter Norton utilities, probably coincident with
Peter selling out to Symantec around that time.

 My experience with defrag's robustness as a test tool on that
problematic system I was refurbishing (winXP most likely to be the
installed OS but possibly win2k) only strengthened my confidence in
the windows version of defrag. I've certainly never been given cause
to doubt its reliabilty since at least the days of win95OSR2.
--  
Regards, J B Good

Re: disable journalling?
On Fri, 07 Mar 2014 01:31:50 -0000, Johny B Good  

Quoted text here. Click to load it

WinME's defrag is better than w98's, which I feel was the same as w95's;
you might want to post to a w98 group!

FU's set.

Quoted text here. Click to load it


--  
It's a money /life balance.

Re: disable journalling?

Quoted text here. Click to load it

I seem to recall this being configurable bo don recall ever
encoutering a format that didn't use 512.

Quoted text here. Click to load it

I've never heard of FAT-8 -- was it something from MS-DOS 1.0 ?

Quoted text here. Click to load it

FAT-12 could handle upto 4096 allocation units (aka clusters),  
for disks upto 2 megytes in size it was one sector per cluster.

Quoted text here. Click to load it

FAT table copies were optional too

--  
Neither the pheasant plucker, nor the pheasant plucker's son.


Re: disable journalling?

Quoted text here. Click to load it

 Most probably PC-DOS 1 and in relation to floppy disk storage.

 Funnily enough, I re-invented a FAT8 system for a solenoid controlled
data cassette deck File System in the early 80s to use with a Transam
Tuscan S100 kit built computer.

 It's possible that my friend who'd given me helpful hints on creating
a file system may have been familiar with the PC-DOS FAT8 FS (I don't
recall him mentioning FAT8 or DOS specifically) so it might not have
been the coincidence it seems to be in retrospect.

 In this case the physical block size was 2KB so an 8 bit pointer in
what I called the LNKTBL (FAT in MS's terminology) was more than ample
to address the 168 blocks each side of a C60 sized data cassette tape
(332KB max usable capacity a couple of the 2k blocks were used to
duplicate the directory and LNKTBL data. The duplication being
required to avoid unrecoverable FS errors in the event of a crash or
power outage mid write.

Quoted text here. Click to load it

 I don't think that was _ever_ true. The FATs _had_ to be duplicated
simply to guard against the very real and ever present risk of crashes
or power outage mid write as per my own tape FS.

 Despite its apparent fragileness, it nevertheless served its function
quite well in practice. As long as you ran a FS consistency check
(scandisk or chkdsk) upon resumption from a crash or unforeseen power
loss, the FAT duplication did an effective enough job.

 True, it may not have had as much redundency in the FS as its CP/M
predecessor (which, imo, seemed to have a wasteful amount of
redundency at a time when CPU grunt was provided by 8 bit 2MHz clocked
8080 or Z80 chips) but it proved sufficient in practice for all
intents and purposes.

 In all my years of experience with FAT, I've never considered it to
be "A bloody nightmare". I've created enough Frankenstein
monstrousities in my time and done my fair share of scandisk/chkdsk FS
repairs and can only recall one event where I lost data due to FS
trouble on a FAT32 volume on a disk that had generated bad clusters
due to being overheated by a mere 5 deg at most.

 Luckily, the one folder's worth of data that proved irretreivable
just happened to be backed up on another drive so no net loss of data.

 Admittedly the NTFS volume, once I'd sorted out a testbed PC to run
the job on, proved to be the more successful chkdsk run with no data
loss involved. FAT only becomes "A Nightmare" when run on unstable
systems in the habit of frequently crashing at the slightest
provocation, a system no self respecting user would want to trust his
precious data to anyway.
--  
Regards, J B Good

Re: disable journalling?
Quoted text here. Click to load it

 It's certaily true now:  see the man page for mkdosfs

 and this page from Microsoft.
  
 http://support.microsoft.com/kb/140418
  
Quoted text here. Click to load it




--  
Neither the pheasant plucker, nor the pheasant plucker's son.


Re: disable journalling?
On Fri, 07 Mar 2014 12:34:37 +0000, Jasen Betts wrote:

Quoted text here. Click to load it
Most likely. I don't think I've ever seen it: only heard of it and at  
256 sectors it would have been a rather small disk. All I know was that  
it was around right at the start of MS-DOS, certainly before my time.  

I don't remember using any DOS box before 1986, and that was almost  
certainly a 12MHz PC-AT and using those new-fangled 1.2 MB floppy disks.



--  
martin@   | Martin Gregorie
gregorie. | Essex, UK
We've slightly trimmed the long signature. Click to see the full one.

Site Timeline