21 hours to copy 1.6 TB

[JohnnyJazz]

I''m copying 1.6 TB from my NAS to an external disk. They are linked directly through the USB 3.0 port, I'm not going through the network. I started 10 hours ago and I'm only half way. Is there any way to speed up the process ?

[Gulfsailor]

USB 3 is not the limiting factor. It could transfer the 1.7TB within the hour. Most likely your external HDD write speed is just low. An average external HDD will write at 20-25MB/s. So that means around 20 hours to write 1.7TB

[Chicog]

I assume the drive is a USB 3.0 drive?

If it is you should be getting 70+MB/S.

[KhunBENQ]

I am afraid @gulfsailor has got the point.

I have just read through some testreports.

There are often highly exaggerated specs for write rates ("peak" rates, "sequential" bla bla in the 100 MB/s+ range).

Practical test with big files show rates of 50 MB/s e.g.

So if the poster does not really have the fastest of the fast, the 20 hours is not too unrealistic.

The thread is an interesting lesson BTW.

(Magnetic) hard drive capacities have grown excessively in the last years.

But the write speeds have not grown proportionally.

So many others will at some time be confronted with similar problems:

how and where to move/backup my terabytes of movies/pictures/multimedia files?

I am regularly synchronizing ("backup") data from my laptop to my desktop via Gbit Ethernet cable.

By theory about 100 MB/s would be possible.

Practical rates around 30 MB/s.

Edited December 8, 2014 by KhunBENQ

[canopus1969]

There is also the problem with the number of files

[JusMe]

Wouldn't help in actually reproducing a complete drive, but using Carbon Copy Cloner regularly means that only the files that have changed are written to the external drive.

Leave it a long time, and there's a lot to copy and therefore takes a long time; do it regularly and it can be just a few minutes.

I use it for backups, and as long as I don't leave it too long, I get it completed in a very short time.

[Chicog]

I've got about 6Tb to copy over USB 2.0 sometime this year. Not looking forward to that.

[mikebike]

Making removable drives native format rather than FAT can help.

[MJCM]

I've got about 6Tb to copy over USB 2.0 sometime this year. Not looking forward to that.

555555555555.

I had same thing a couple of months (however it was just under a Tb).

I started the Copy (Sync, but I forgot to turn off, "ask if files are different") left home, came back a couple hours later expecting the copy to be finished. But was greeted with a message after around 4% of the copy, "Do you want to overwrite this file"

Edited December 8, 2014 by MJCM

[Tywais]

There is also the problem with the number of files

Indeed. If I have a single 1.6TB file and transferring on USB3.0 the time will be extensively less than if I transferred 10,000 files equating to 1.6TB. The latency is the opening, creating directory entry, copying the file and closing the file at both ends of the transfer cycle. I ran a test once where I had an external box with eSata and another one with USB 3.0, same drive types, and the transfer speeds were pretty much identical indicating drive limitation.

[Tywais]

Transfer time calculator - http://techinternets.com/copy_calc?do

For 1.6GB with 10% overhead on USB3 at 25MB/s comes out very close to 21 hours.

[manarak]

OP, are you sure you don't use the network?

Where do you initiate the copy process from? If you use a client, i.e. your workstation connected to the NAS through the network, then the copy process will be managed by the workstation over the network.

The only way to ensure the network is not used is if you initiate the copy process on the NAS directly, either using the NAS' GUI (if such function 'copy to local device' is supported) or command line on the server itself.

[jpinx]

Copying in Linux is a bit better. If it's a fresh backup you can use dd which copies the whole drive as raw data - no overhead for individual files. If it's a refresh of a backup rsync will only copy over new stuff. Both commands need various options to be used according to the specific situation.

Edited December 8, 2014 by jpinx

[Chicog]

I've got about 6Tb to copy over USB 2.0 sometime this year. Not looking forward to that.

555555555555.

I had same thing a couple of months (however it was just under a Tb).

I started the Copy (Sync, but I forgot to turn off, "ask if files are different") left home, came back a couple hours later expecting the copy to be finished. But was greeted with a message after around 4% of the copy, "Do you want to overwrite this file"

Fortunately it's from a nearly full Drobo to a completely empty one.

[Anthony5]

Check if the disk in the server is running in DMA or POI mode.

[JohnnyJazz]

OP, are you sure you don't use the network?

90% sure, unless I missed something

The only way to ensure the network is not used is if you initiate the copy process on the NAS directly, either using the NAS' GUI (if such function 'copy to local device' is supported).

That's what I did.

Thanks for all the answers. The reason I make such a big copy is because I want to take my movie collection with me during the holidays. The process is long but imagine if I had to carry with me the same in DVDs ...

[wgdanson]

OP, are you sure you don't use the network?

90% sure, unless I missed something

The only way to ensure the network is not used is if you initiate the copy process on the NAS directly, either using the NAS' GUI (if such function 'copy to local device' is supported).

That's what I did.

Thanks for all the answers. The reason I make such a big copy is because I want to take my movie collection with me during the holidays. The process is long but imagine if I had to carry with me the same in DVDs ...

That's a LOT of movies, or a very long holiday. If you are just going to stay in and watch movies, why not stay at home? Or download as and when needed.

[JohnnyJazz]

That's a LOT of movies, or a very long holiday. If you are just going to stay in and watch movies, why not stay at home? Or download as and when needed.

Who said I'm going to watch all these movies myself ? How did you infer that ?

It's our traditional family Christmas gathering, 20 something people, including kids of all ages and grand parents. As I can't guess what people will want to see, I take everything. And I will have probably more interesting things to do during my holidays that to fight with lousy internet connection. Does that make sense to you ?

Edited December 9, 2014 by JohnnyJazz