Low Bandwidth Disk Operations Saturate the Drive

I often see my computer limited by the activity of the disk. It will be at 100% according to Task Manager, yet the actual activity will be well below 1 MB/S. Other times the activity will be in the 10's of MB/S and it won't be throttling the computer.

What's up with that?

Task Manager measures the data transfer rate as well as "Active time" which is measured in %. What is that measuring exactly? I read somewhere it has to do with the utilization of the disk I/O, but why would it be clogged at a low data rate? Do transactions hang the disk interface?

--

Rick C
Reply to
rickman
Loading thread data ...

You don't give enough details to make helpful comments but if you are talking about spinning rust technology then the drive seek times will dominate for random access to small files. If you do a lot of that sort of thing putting your data onto an SSD will make a big difference.

Typical spinning rust is peak 150MB/s, 4kR 1MB/s W 2MB/s mixed 0.3MB/s Whereas an SSD is much faster 500MB/s, 4kR 40MB/s W 80MB/s mixed 50MB/s (needs to be on SATA 3 with the right cable to get full bandwidth) M2 PCIe SSD interface is even faster if your PC supports it peak 3GB/s.

It's waiting for the bit of disk with the data on to come round. You can make some hybrid gains with the Intel SSD cache technology but now that decent sized SSDs have come down in price it isn't really worth it.

The other possibility is that you have a badly behaved AV program that is baulking on some of the files and slowing things down a lot. I have seen that happen sometimes on very large emails in the past.

--
Regards, 
Martin Brown
Reply to
Martin Brown

Not enoght information, but perhaps you have virtual memory issues due to insufficient RAM installed. The virtual memory swap in/swap out is not accounted as normal disk activity by many OSes.

--
Fletto i muscoli e sono nel vuoto.
Reply to
dalai lamah

If it's windows 7 I would say you have a bad driver somewhere and your cache is thrashing. Rus that stupid windows experience benchmark and have a look at (IIRC) suggestion or addition information

--
Using Opera's mail client: http://www.opera.com/mail/
Reply to
David Eather

are ALL users tasks being shown, the clogging task may be hidden

Reply to
makolber

In the former case, your filesystem is doing a whole lot of individual (small) reads and writes, scattered across some or all of the disk surface. Every "random" read is likely to take on the rough order of

10 milliseconds to complete - time for the drive to move the head to the proper track, and time for the drive platter to rotate into the correct position to read the sector(s) involved. A small read (say, 4k bytes) takes almost as long as a large read (e.g. 1 megabyte) because the head-seek and rotational latency is the same.

In the latter case (high performance), the filesystem is doing large reads and writes - "streaming" a substantial amount of data to/from a contiguous section of the drive. In these cases, you pay only one "seek and rotate" time penalty for each operation, no matter how large it is (to a first approximation) and the number of sectors read/written per second goes up very sharply.

That's the percent of time that the host computer has issued a "read" or "write" command to the drive, and the drive hasn't said "OK, it's done."

On simpler drives, each read or write transaction does "hang" the interface until the actual physical operation completes.

On medium-sophistication drives, the drive itself has a memory cache. When you read a sector, the read will complete almost immediately if the sector in question is still in the cache. When you write data, the drive will accept it into the cache, tell the computer that the write is complete, and then actually transfer the new data to the drive "in the background" when time is available. This can significantly increase performance. It has limits, though - the cache is of a limited size, and during writes it'll eventually fill up... the drive still has to take the time to move the head to the right position for each cached write, and then wait for the platter to rotate into position.

More sophisticated drives have "native command queueing", in which the drive can accept two or more read or write requests from the host, and then complete them in whatever order makes sense to the drive for best performance. A sophisticated filesystem can take advantage of this. It eliminate the head seek and rotational delay, but it can allow the drive to minimize it by scheduling reads and writes in the most efficient order.

Solid-state drives don't have head-movement or rotational delay, and tend to handle "lots of small reads and writes" a good deal better than hard drives do.

Back when I was doing filesystem and media-storage work at a former place of employment, I recommended a rule of thumb to the developers: plan out their I/O strategy so that no more than 1/3 of the time would be required to "move the head and rotate the platter" to do I/Os. This usually meant "throw memory at the problem - allocate lots of large buffers, and do your reads and writes in big chunks". The time penalty for reading more data than you needed, and then throwing away the parts you didn't actually need, was a lot less than the time penalty involved in doing lots of small reads that each required a head seek.

Reply to
Dave Platt

Did something happen to your message? It seems to have lost some characters.

--

Rick C
Reply to
rickman

Task Manager is hard-coded to hide some tasks, *always*. If you use the low-level APIs you can see them all.

I once wrote a virtual memory use program which does that, displaying the resident/virtual working set for every DLL in every process (with the exception of one or two sekrit system services). You can snapshot the page table in running processes! Available on request.

Clifford Heath.

Reply to
Clifford Heath

When was the last time you defragment your disks ?

Especially log files,which are written with small fragments added might contain hundreds of fragments all over th disk. If this file is frequently accessed, most of the time goes to moving the heads.

Also look at the fragmentation of the page file. There should be only one fragment on each disk. Moving the page file to a (SSD) disk with low activity may also help.

Reply to
upsidedown

My understanding is Windows defrags continuously. That was one of the tasks hitting my disk hard one of the many times I've checked. The defragger is not often a culprit though.. It's often something like "system" or "core" something. Sometimes when I pack a Eudora mailbox it drags the disk for what seems to be too long. That usually only impacts Eudora though. The other cases impact the entire computer.

--

Rick C
Reply to
rickman

I believe you schedule when to run defrag in win7 etc. and by default it is set to something like Wednesday 1am

Reply to
Lasse Langwadt Christensen

A few years ago I can across a Seagate drive that threw a lot of ECC errors. It was a SATA drive. It still worked, but the number of errors was very high and it was a noticably slow.

Cheers

Reply to
Martin Riddle

That has been the claim for NTFS, but I am not convinced about that.

I just run the Analyze function of the Defragger on a disk that was defragged only a few months ago. It found 1000 fragmented files split into 12000 fragments,

Of my 1000 fragmented files were mail or usenet news that are frequently updated by new messages, some with hundreds of fragments,

Reply to
upsidedown

Apparently the drive is doing a lot of bad block replacements.

Even if the file had originally been in continuous sectors, the heads have to jump to the replacement block and then continue the original read sequence.

Reply to
upsidedown

could be fragmentation.

--
This email has not been checked by half-arsed antivirus software
Reply to
Jasen Betts

Are you sure that there are no stupid programs causing excessive page fault rates ?

In many Windows programming examples, first open a file, determine the file size, malloc() a user mode buffer for the whole file and read it into that buffer with a single read.

This works of course OK for small files.

However, when the file size exceeds the free (physical) memory, things get hairy. About two decades ago when the physical memory size was well below 1 iB a picture display program did just that. Then the video display was added. With small MPEG clips, thinks worked OK.

Now, what happens, if you feed a 1 GB+ MPEG file to it ? Trying to read the 1 GB file will be paused at say 100 MB when no free physical memory is available and the OS starts to save the 100 MB of virtual memory into the page file, freeing that physical memory, so that the next 100 MB can be read into physical memory and so on.

The OS performs this copying of 100 MB chunks from original MPEG file to the page file. The application doesn't note anything special, but the user notices high disk activity.

Only when all these 100 MB chunks were copied to the page file could the actual display start. This then required rereading the video file from the page file in small chunks to be displayed at a time.

The correct way of doing this would be using memory mapped files, just declaring the mapping to a virtual memory array, but not doing any physical I/O to the disk at first.

Making a read memory reference into the virtual memory array, would cause page faulting into the physical memory, using the highly efficient OS virtual memory paging services. This would only cause read requests to the original video file, no need to writes to and subsequent reads from the page file. When a virtual page has been processed, there is no need to save it, since the original is already on the disk for next view.

If such stupid program designs (such as reading all at once) are done, this can severely reduce total system performance.

Reply to
upsidedown

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.