Speeeeed, Powerrrrrrr, or how to transfer 250Gb of tiny files?

Like the title says (think Jeremy Clarkson): Powerrrr, Speeed.
I would like to backup my CF-19's SSD more regularly. It contains loads and loads of small files, slowing any backup through USB 2 or 1394 to an increadible slow speed that takes well over 10 hours to back everything up.

Now I could buy the new mk6 (I'm sure there are enough people that would like to buy my mk5 for $5 ) and use the USB3.0 but unfortunately I don’t have this kind of money laying around. (It would give me finally get the fingerprintscanner I want so badly).
So now I’m looking for the next best thing, and I think I can use a raid1 buffalo linkstation mini. It uses 2 500Gb HDD and Ethernet and has the ability to be connected directly to the gigabit port of the 19.
Another solution would be to use a flush expresscard 54 with USB 3.0 ports and then use that for fast data transfers.
Are there any thoughts/ideas to this??

 

if you have them on a separate partition, or if you can image the entire partition, this way you don't have to worry about how many small files you have; there are lots of programs that can do that.
Without filesystem overhead on the external drive (esp. random writes), you should be able to transfer at sequential speed, about 30-32MB/s on USB2.0 (there still is the USB protocol overhead).
Any external drive should be able to write sequentially at that speed, so you should be good to go in ~2.6 hours.

Another solution would be to archive everything with ZIP, TAR, etc (with zero or low compression), and have 1-2-3-4 big archive files, created locally on you SSD (your 600GB should allow for that), and THEN transfer them over, externally, at the same maximum speed. There is additional time need for creating the archive, but since you have a good SSD, I bet it won't take more than 1hr maybe 2hrs.
It might be worth trying to create the archives directly on the external drive. (random read from your ssd wouldn't be an issue, and the external HDD should write them seq).
You can use Total Commander for creating the archives (it has built in support for ZIP, TAR, additional plugins for all kinds of files).

 

I want them to be secure in case of theft, SSD faillure, virus or alike things. so copying them to a different partition isn't a sollution.
But the idea to zip the main bulk of small files (all in all 15Gb in millions of files and hundereds of maps but all clustered in one map) and back it up like that, makes certanly sense! Very good suggestion!


I'm still wondering which way to go as an USB3.0 addition to my 19 sounds attractive, but I'm not sure if the sustained speeds are going to beat gigabit?! Does anybody here have experience with this?

 

the first idea was to image the partition (save the entire partition directly to a file on the external drive).
Gigabit should give you more than 100MB/s, USB 3.0 a lot more speed than 2.0 (never tested it though), BUT it all depends on what you copy the file(s) to, and in what way (few large files, or many small ones).

What is you external drive setup ? I assume, from you post, it has USB 2.0 and Firewire (400), and some 7200rpm 3.5 hdd.
A RAID1 solution it's a sane way to backup, although it would be at most as fast as a single drive setup. A RAID10 would still give you redundancy, and also, theoretically, twice the speed. But it needs 4 drives, and these storage solutions tend to be rather pricey (even without the drives).

Too bad Toughbooks don't have eSATA, that would be the best solution. Or Thunderbolt, on 19mk6 or 31mk3.
I played with a Cardbus eSATA adapter (cheap VIA 6421 SATA controller in it), on my C1, and ~40MB/s was the max speed I got, even using a X25-M.

 

External I have a high speed USB firewire 400 2.5" enclosure with an OZC 128Gb SSD. Not a particular fast drive but way faster than the 2.6Mbps that I get now with the tiny files.
When transfering large files (1-4Gb files, around 100Gb total, I don't get higher then 20Mbps on either usb or firewire.

I think I go the USB 3 way. Flush mounted expresscard 54, so it can stay inside and the 19 stays IP65, and I'll always have a non/low powered USB3 wth me! And with a tiny USB2 cable I can make it even powered. But most important is that I can use the drive/enclosure it on any USB2 port as well. So I can use it on any computer/laptop!
Once I get it to work, I'll buy a bigger/faster SSD. For now the 128Gb is good enough.

If I'm lucky I can get it this evening. Ill report back on how it goes!

I've got it!
It's a startech 2 port flush mount 54mm superspeed USB 3.0 Expresscard, it has an extra usb2 for power and a sabrent enclosure with USB 3-B port. Cost me around 60 in total. Not cheap, but speed comes at a price (as always )
The expresscard fits in nicely and I can keep the baydoor shut when I don't need it. Nice!!!

First quick test shows well over 60Mb per second for large files and around 6Mb per second for the tiny files. That's a good 3 times as fast. I now have the drive formatted at NTFS, does that make any difference when compared to exFAT?
At this moment my computer is indexing some 60.000 emails, so that might have an impact as well. I'll test it again later on.

 

Check whether that adapter overheats under workload and on standby.
My adapter (simple chinese one) is very fast, but also veeery hot. So I insert it only when I really need it (very rare).

You can try to use your USB 3.0 device connected to USB 2.0 port. Very interesting results in my case

 

thereis no adapter, it's a power-cable fromthe regular usb 2 port to the express card. I think USB 2.0 has only 500mAh maximum? So it's nowhere as powerfull as a regular USB3 port, but i'ts more than enough for a SSD!
Tried connecting it and it worked. Slower, but still worked!
What was the result on yours?

 

@Alecgold....good going, sounds like you are having fun.

As indicated above, the current back-up drive is temporary....just thought I would mention that OCZ are labelled as having the highest failure rates in the SSD industry.

Just sayin'....so that you know the score.

 

Thanks for the heads-up OG!

I know OZC doesn't have the best rate, that's why I have an intel inside my 19.
But I had this OZC for testing around, so I figured if I got everything working I buy another Intel or Samsung, large enough for a good backup and use that.

But the USB3 is working good. It's not blasting fast, but it does make a backup about 3 times as fast. And the difference between 10 hours or just over 3 hours is the difference between an all nighter and just an evening. Makes it much more easy to do for me.

 

If you want secure, why not made a TrueCrypt file, store all your little files in that container, and to back up, simply copy the single encrypted file?