I see a need for a hardware DD equivalent, but FAST compared to what little i found so far (speeds up to 14 GB per minute).
To do this, the device must do all of the work in hardware, and simultaneously read and write; there can be a time difference from read and write.
No vendor tells how they do their copy, and that 14 GB/min tells me that they do not do the copy simultaneously, but do the copy in a read-block then write-block fashion.
The hardware should approach the SATA revision I/II/III (1.5/3.0/6.0 Gbps) drive speeds, which is up to 30 times faster that the clumsy 14 GB/min.
Do you know of any hardware that approaches the SATA speeds?
Some years ago, when my employer's disk-duplicating house found themselves unable to copy disks fast enough on the then-available commercial duplicators (which were PIO-only) I put together a clone-a-tron made out of a standard PC motherboard, some multi-port PCI/SATA interface cards, standard Linux, and a simple disk copying application I threw together (from "dd" I believe).
The tricks required were: the drives were opened in "O_DIRECT" mode, so that data was read directly from the SATA interface into the application's memory buffers (bypassing the normal kernel buffer-and-copy process). The app buffers had to be page-aligned in order to pull this off.
The app was multi-threaded. One thread would read a big chunk (1 MB or so) from the source drive into a direct-memory buffer. It would then "hand off" this buffer to a second thread, which write it to the first target drive... when this was done, the second thread handed it to a third, which would write it to the second target drive, etc. The operating system wouldn't allow two "direct I/O" operations from the same buffer at the same time, so the writes had to be passed along in this fashion,
There were more buffers than drives, so the "pipeline" would fill up nicely.
The limit to copying speed might be that of the drives (in Ultra DMA mode), or might be limited by the motherboard's PCI and memory bus (during copies to multiple fast drives).
One PCI system with two add-on two-port cards could do a 1-in, 4-out copy quite a bit faster than the commercial duplicator, and it was a lot less expensive. I understand that the commercial guys were rather shocked when they found we hadn't waited two years for them to ship a DMA-based duplicator, but made one ourselves in a few days for a quarter of their price :-)
The equivalent today would be to use PCIe interfaces, I think.
There *may* be high-end SATA interfaces which can do direct port-to-port copies, or direct card-to-card over PCIe without having to go to host DRAM, but I don't know any specifics about what's available along those lines.
Sure. NVMe, SATA Express, NGFF/M.2, PCIe Gen 3.0, etc. If you're thinking of buying rotating mechanical memory, you're investing in the stone age. Everything worthwhile today uses SSD and you're not going to get 14GB/min out of a spinning hard disk, even at 10K or 15KRPM, because they're all limited by the SATA 3 interface speed.
This is a bit old now, but covers the problem quite well: "SSDs have a problem: They're getting too fast, too soon"
What's happening is the SSD is much faster than todays commodity hardware. The SSD kinda caught the industry by surprise. When SSD's dropped in price, there no approved bus specification available capable of taking advantage of the speed of an SSD. Anyway, read the article.
--
Jeff Liebermann jeffl@cruzio.com
150 Felker St #D http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann AE6KS 831-336-2558
Interesting. A colleague used a similar technique (but with a single mag-tape drive as output) to speed up backups of our HP systems in
1987. Because the language was not multi-threaded, he created multiple large buffers in shared memory, and forked the process several times, using semaphores to pass buffers. At the time, he increased backup speeds from 10-15kb/s (!) to over 120kb/s. You can understand that he had plenty of time to work on the program while waiting to change the multiple nightly backup tapes.
That program got adopted by HP as their preferred backup solution. If you worked on an HP-UX system, you might recall when "ftio" came out.
But back on the original topic... the ability of flash to write smaller block sizes will change the structure of DBMS software, and will require the interconnect to handle very large numbers of small I/Os with now overheads. It's no longer necessary to require DBMS to move data in 64kb chunks (8x8kB transfers), which is the average of the last decade. Much bigger transfers must be fast too, but efficient small transfers would be worthwhile.
Thank you very much for the info. Yes, the rub is the makers of that "Standalone 2.5 / 3.5? SATA Hard Drive Duplicator and Eraser w/ High Duplication Speed up to 14GBpm" and other similar devices are not talking. And that seems to be the fastest...
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.