USB drives, caching and sync

Op Mon, 11 Jan 2010 12:04:40 +0100 schreef Noob :

Read caching and write caching are two different uses.

Or use open() with O_SYNC.

Your documentation should state which calls affect data only and which affect also metadata.

POSIX calls should be reentrant/thread-safe unless specified otherwise.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

Assuming kB is a typo and you mean MB - 770 kB is small nowadays - I am still not sure you will be better off if you turn the cache off. Depending on the OS, this may result in more disk overhead than you are happy with - e.g. if the OS updates the directory entry every time you write to the file (last modified); it can get even worse if it decides to do that recursively for the directory entries of the entire path (I don't know how many OSs would do the latter; I know DPS would do it if the directories are explicitly of that type, which I don't think I ever really used after I implemented it.... :-) ).

Well this will depend on the OS but one which works won't have a problem with it :-). The process which first gains access will do the "sync", the second one will find it has nothing to do etc.

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

You're right. I had write-caching in mind.

Unfortunately, vfs_open does not seem to accept an O_SYNC flag.

I would assume that most writes require an update of the FAT, which one might consider metadata?

I don't think OS21/OS+ claims POSIX conformance. I will have to check.

Regards.

Yes sorry, most writes require a metadata update at some point, but there are only a few ways of forcing the system to do it immediately.

In fact, all OS calls should be reentrant/thread-safe unless specified otherwise.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

I like this solution :-)

So, if two threads are concurrently streaming data to the disk, and one thread calls sync, all the data (and metadata) written so far by both threads should be flushed to disk, right?

If the second thread is "in the middle" of a write, the OS would queue the flush, so that everything happens as expected, right?

Regards.

It is, I believe, common practice.

Yes.

Either the OS has received the data to be written, or it hasn't. When it hasn't, a sync cannot affect it.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

On a single processor system, how could two threads call sync simultaneously ?

Unless the system call is explicitly made preemptable, assuming user/kernel mode switching during the call, this could not happen.

At least in most operating system, calls to OS services is implemented with some kind of change mode to kernel traps, which are served in a more or less similar way as interrupts in kernel mode. Some hardware even have instructions with names like "software interrupt" to activate the service routine in kernel mode.

No other user mode thread will get control, unless this "interrupt" returns one way or an other, so there should not be such problems.

Of course the "OS call" seen by the user may do some jobs in the user mode, before switching to kernel mode, so this code may be vulnerable to task switching, if not implemented properly.

It would be a pretty lousy OS if sync() wasn't pre-emptible, given that it has to be one of the slowest system calls there is. Think about it: sync() spends most of its time blocked waiting for the disk to indicate either that it's ready for more data or has completed outstanding writes.

Consider a scenario where thread A needs to write 100 MB to the hard disk drive i.e. thread A calls write(fd, buf, 100*1000*1000);

When the OS has "processed" approximately half the data, another thread calls sync.

Would the sync operation be queued and carried out only after the write completes?

Or would the OS perform the sync operation "immediately", flushing data and metadata cached up to this point? In that case, when the write completes, the first half of the data would be guaranteed to be written to disk, while parts of the second half may still be cached.

(I think the second scenario is more plausible.)

In other words, I think I'm asking: is there some form of atomicity of file system calls?

Regards.

The answer depends on your system. Often the only guarantee is that the flush operation is scheduled when sync() returns.

Often it does, but the granularity of the 'atoms' depends on the FS implementation.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

The only usage for sync() I have seen is to write out any buffers prior to shutting down the entire system or disconnect a removable media.

In such a situation, it is really preferable to have a blocking call. Recent Linux versions use blocking calls, while at least some older HP-UX versions the sync() returned before writing out everything (and hence the instruction of entering twice the sync command manually before powering down, so that the first sync completed, while the operator entered the second sync :-).

Does the OS21 have fsync() to write out only a single file ? Of course, writing to the file and fsyncing should be done from the same thread.

Maybe, maybe not. The OS can switch between threads as it sees fit. If a write() takes a long time, it may suspend the thread and schedule another thread, or it may not. If the other thread calls sync(), it may start it immediately, or it may decide that's a good point to suspend the thread.

I don't know about OS21/OS+, but POSIX provides very few atomicity guarantees (and those which it does provide may not always be honoured; it used to be a standing joke that "NFS" stood for "Not a File System" because of its failure to provide even minimal atomicity).

Nowadays, that's normally handled by umount(). With a sync(), nothing prevents the device from being modified the moment the sync() completes. With umount(), the call will fail if the device is still in use; as soon as the umount() commences, further operations are excluded.

[With the root filesystem, it's more complex, as it's always "in use". Typically, it's remounted read-only once everything except init has been killed.]

In general, a "sync" command (as opposed to the sync() system call) isn't guaranteed to fully synchronise the filesystem, as its execution may result in the last-access time of the executable (or a shared library on which it depends) being updated after the sync() call completes.

In the case of a USB HDD, I must call sync every time I close a file I've written to, because I don't know when the user will unplug the disk.

There is a vfs_fflush, but I think vfs_close already calls it.

My problem is that, if I don't call vfs_sync after closing a file, the disk is still considered "dirty", and the OS forces an fsck when it sees the disk again.

OK, but many threads may be writing to the disk "at the same time" i.e. concurrently, whereas vfs_sync is supposed to flush everything to disk (even data and metadata from other threads).

Regards.

but not to the same file I hope?

Perhaps you're reading over the initial 'f' of fsync.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

thread.

Each thread writes to its own file, in its own directory.

I don't quite understand.

I think you're saying fsync works on a file, whereas sync works on an entire volume, but I'm not getting your point.

On my system, fsync is called vfs_fflush, and I've tried calling it right before closing a file, but the OS still runs fsck when I replug the drive. Calling sync seems to make the OS happy.

Regards.

Paul asked about fsync, but your reply only mentioned other functions. That was my point.

Says who? fsync and fflush are supposed to be different things, with different purposes and differing behaviour.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

In the OS I'm using, there's no fsync, and the documentation for fflush states:

"vfs_fflush() writes any outstanding data (and metadata) for this file to the physical device."

vfs_fflush is closer to POSIX fsync than to POSIX fflush, don't you agree?

Reference for myself:

Regards.

It is not about what is closer, it is about what works for you.

And since your FS implementation is clearly unequal to POSIX, I believe that only people familiar with it can help you further.

--
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)