OT: DD equivalent..

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View

   I see a need for a hardware DD equivalent, but FAST compared to what  
little i found so far (speeds up to 14 GB per minute).

   To do this, the device must do all of the work in hardware, and  
simultaneously read and write; there can be a time difference from read  
and write.

   No vendor tells how they do their copy, and that 14 GB/min tells me  
that they do not do the copy simultaneously, but do the copy in a  
read-block then write-block fashion.

   The hardware should approach the SATA revision I/II/III (1.5/3.0/6.0  
Gbps) drive speeds, which is up to 30 times faster that the clumsy 14  

   Do you know of any hardware that approaches the SATA speeds?

   Use: for multi-TERRORbyte drive archiving.


Re: OT: DD equivalent..

Quoted text here. Click to load it

Some years ago, when my employer's disk-duplicating house found
themselves unable to copy disks fast enough on the then-available
commercial duplicators (which were PIO-only) I put together a
clone-a-tron made out of a standard PC motherboard, some multi-port
PCI/SATA interface cards, standard Linux, and a simple disk copying
application I threw together (from "dd" I believe).

The tricks required were:  the drives were opened in "O_DIRECT" mode,
so that data was read directly from the SATA interface into the
application's memory buffers (bypassing the normal kernel
buffer-and-copy process).  The app buffers had to be page-aligned in
order to pull this off.

The app was multi-threaded.  One thread would read a big chunk (1 MB
or so) from the source drive into a direct-memory buffer.  It would
then "hand off" this buffer to a second thread, which write it to the
first target drive... when this was done, the second thread handed it
to a third, which would write it to the second target drive, etc.
The operating system wouldn't allow two "direct I/O" operations from
the same buffer at the same time, so the writes had to be passed along
in this fashion,

There were more buffers than drives, so the "pipeline" would fill up

The limit to copying speed might be that of the drives (in Ultra DMA
mode), or might be limited by the motherboard's PCI and memory bus
(during copies to multiple fast drives).

One PCI system with two add-on two-port cards could do a 1-in, 4-out
copy quite a bit faster than the commercial duplicator, and it was a
lot less expensive.  I understand that the commercial guys were rather
shocked when they found we hadn't waited two years for them to ship a
DMA-based duplicator, but made one ourselves in a few days for a
quarter of their price :-)

The equivalent today would be to use PCIe interfaces, I think.

There *may* be high-end SATA interfaces which can do direct
port-to-port copies, or direct card-to-card over PCIe without having
to go to host DRAM, but I don't know any specifics about what's
available along those lines.

Re: OT: DD equivalent..
On 25/04/16 07:44, Dave Platt wrote:
Quoted text here. Click to load it

Interesting. A colleague used a similar technique (but with a single
mag-tape drive as output) to speed up backups of our HP systems in
1987. Because the language was not multi-threaded, he created multiple
large buffers in shared memory, and forked the process several times,
using semaphores to pass buffers. At the time, he increased backup
speeds from 10-15kb/s (!) to over 120kb/s. You can understand that
he had plenty of time to work on the program while waiting to change
the multiple nightly backup tapes.

That program got adopted by HP as their preferred backup solution.
If you worked on an HP-UX system, you might recall when "ftio" came

But back on the original topic... the ability of flash to write
smaller block sizes will change the structure of DBMS software,
and will require the interconnect to handle very large numbers of
small I/Os with now overheads. It's no longer necessary to require
DBMS to move data in 64kb chunks (8x8kB transfers), which is the
average of the last decade. Much bigger transfers must be fast too,
but efficient small transfers would be worthwhile.

Clifford Heath.

Re: OT: DD equivalent..
Dave Platt wrote:
Quoted text here. Click to load it
   Thank you very much for the info.
   Yes, the rub is the makers of that "Standalone 2.5 / 3.5? SATA Hard  
Drive Duplicator and Eraser w/ High Duplication Speed up to 14GBpm" and  
other similar devices are not talking. And that seems to be the fastest...

Re: OT: DD equivalent..
On Sun, 24 Apr 2016 11:46:45 -0800, Robert Baer

Quoted text here. Click to load it

Sure.  NVMe, SATA Express, NGFF/M.2, PCIe Gen 3.0, etc.  If you're
thinking of buying rotating mechanical memory, you're investing in the
stone age.  Everything worthwhile today uses SSD and you're not going
to get 14GB/min out of a spinning hard disk, even at 10K or 15KRPM,
because they're all limited by the SATA 3 interface speed.

This is a bit old now, but covers the problem quite well:
"SSDs have a problem: They're getting too fast, too soon"

What's happening is the SSD is much faster than todays commodity
hardware.  The SSD kinda caught the industry by surprise.  When SSD's
dropped in price, there no approved bus specification available
capable of taking advantage of the speed of an SSD.  Anyway, read the

Jeff Liebermann     snipped-for-privacy@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
We've slightly trimmed the long signature. Click to see the full one.
Re: OT: DD equivalent..
Jeff Liebermann wrote:
Quoted text here. Click to load it
* Which is quoted as 6 Gigabytes per Second - almost 30 times FASTER.

Quoted text here. Click to load it

Re: OT: DD equivalent..
On 25/04/16 18:36, Robert Baer wrote:
Quoted text here. Click to load it

There are 8 bits in a byte. SATA 3 is 6G-bits-per-second, not bytes.
And the protocol overhead slows it down further.

Re: OT: DD equivalent..


Quoted text here. Click to load it

it's not the interface (connect a SSD to a sata3 interface and you get 500M
B/sec). From a HD you get 150MB/s burst and aroun 100 MB/sec sustained.

Bye Jack

Site Timeline