Best way to move a lot of data

[robblok]

What have learned to counter oilinki his sluggishness is to zip my data first to one big file and then transfer it over the network. This makes things much faster. This helps a lot when you have a lot of small files. Later you have to unzip again but it helps a lot and makes things work a lot faster.

[robblok]

Compressing and collecting 1TB misc data files to a single file would take ages and it would require huge amount of resources.

I'm using rsync for transferring files from one location to another. It also offers compression. The other benefit is, that data transfer can be stopped and later continued, if there is a need for it.

While on topic. If OP have a new 1TB disk, it would be highly recommended to do house cleaning, before filling up the new disk. If you get the disk first full, then remove unwanted files, this might cause fragmentation later on, when you insert new files to the disk.

Guess you don''t have a powerful computer.. I let my computer compress and store 20 gb of data everyday. It goes faster as copying it to the other network driver uncompressed (if you take compressing time into account) Speeds far exceeding your speeds.

I tested what was faster.. copying it uncompressed.. or compressing first and then copying.. the later option won easily..

Edited May 17, 2015 by robblok

[NeverSure]

A .zip file, I believe is limited to 4 gig of data. The newer .zip64 version is for practical purposes unlimited. Win XP Explorer can't read .zip64.

I agree that 4 gig is a large amount of data, but it wouldn't contain the contents of a HDD.

[robblok]

A .zip file, I believe is limited to 4 gig of data. The newer .zip64 version is for practical purposes unlimited. Win XP Explorer can't read .zip64.

I agree that 4 gig is a large amount of data, but it wouldn't contain the contents of a HDD.

I rar it every day 20 gb and get it stored on an external drive it goes far faster as copying it uncompressed.. even if you take into account the compressing time.

[NeverSure]

Fragmentation is disk specific rather than file specific. One way to defrag data is to copy it from a fragmented disk to a blank disk. The file system puts the data where there's room for it which results in fragmentation over time.

[NeverSure]

My HDD gets imaged to an external once a week, and Cobian Backup 11 (free) runs every night at 2:00 AM. Cobian shows the actual files on the external which can be individually copied if needed. I don't care how long it takes because it all happens while I'm sleeping. The image and Cobian each take fewer than 30 minutes.

Copying individual files is another matter due to referencing the index and gathering everything together with the read/write heads. An image doesn't care because it's an image.

[NeverSure]

A .zip file, I believe is limited to 4 gig of data. The newer .zip64 version is for practical purposes unlimited. Win XP Explorer can't read .zip64.

I agree that 4 gig is a large amount of data, but it wouldn't contain the contents of a HDD.

I rar it every day 20 gb and get it stored on an external drive it goes far faster as copying it uncompressed.. even if you take into account the compressing time.

If you're doing backups that way you should try Cobian Backup. It uses the Windows Volume Shadow Copy Service (VSS) so it can override the File Protection System which is what might put up an error message saying "file can't be copied/moved/etc. because it's in use". It can also override file size and name restrictions and with the VSS it takes a snapshot of the whole task so that if something changes during the backup it doesn't matter.

Once you tell it what files and folders you want backed up, and where you want the backup to go, it remembers and will run on the schedule you set. It will delete old backups once you tell it how many to save and then your HDD or partition never fills up.

Cheers

[2fishin2]

If you are in BKK go to Pantip or Fortune town and pick up one of these. I have been using these type devices for many years now to make ghost and duplicate hardrives, even before SATA came along. With SATA its just plug the hard drive in, use the appropriate software and got to work. Simple cheap and effective. This is a top model that cost me 2500 baht, cheaper ones with only one or 2 docks are in the neighborhood of 1000-1500.

[robblok]

Guess you don''t have a powerful computer.. I let my computer compress and store 20 gb of data everyday. It goes faster as copying it to the other network driver uncompressed (if you take compressing time into account) Speeds far exceeding your speeds.

I tested what was faster.. copying it uncompressed.. or compressing first and then copying.. the later option won easily..

1TB of data is quite a different ball game than 20GB of data. If one have a enough memory, the whole compressed archive creation could be in theory done within RAM, without need for very slow disk I/O operations when creating the archive.

And no, I don't have very powerful computers. I don't need one.

That explains why it does not work for you and it works for me. I got 20gb of files (mostly 2kb -10kb). It takes ages to send this to an other drive over the network. Now by compressing it first I safe many hours and I do have a real powerful computer and use it.

[robblok]

A .zip file, I believe is limited to 4 gig of data. The newer .zip64 version is for practical purposes unlimited. Win XP Explorer can't read .zip64.

I agree that 4 gig is a large amount of data, but it wouldn't contain the contents of a HDD.

I rar it every day 20 gb and get it stored on an external drive it goes far faster as copying it uncompressed.. even if you take into account the compressing time.

If you're doing backups that way you should try Cobian Backup. It uses the Windows Volume Shadow Copy Service (VSS) so it can override the File Protection System which is what might put up an error message saying "file can't be copied/moved/etc. because it's in use". It can also override file size and name restrictions and with the VSS it takes a snapshot of the whole task so that if something changes during the backup it doesn't matter.

Once you tell it what files and folders you want backed up, and where you want the backup to go, it remembers and will run on the schedule you set. It will delete old backups once you tell it how many to save and then your HDD or partition never fills up.

Cheers

Thanks, but I use a program that works good already. So no need to change that. But I have done it manual a few times. My way saves hours of time IF there are many small files. Just try copying 700mb of small 1-200kb files or 1 700mb file over the network and then report back.

[Don Mega]

I use dropbox, then I can access my data from any machine, anywhere. I guess you don't have an apple machine as icloud offers similar to dropbox. Monthly cost for 1 tb is £7.99 or £11 per month for unlimited space. I've been using Dropbox for years and swear by it for file sharing (big files in particular that are too big for attachment)

I run a Synology Diskstation at home with 16tb of storage, I can access it anywhere in the world.... it costs me nothing per month.

[robblok]

I use dropbox, then I can access my data from any machine, anywhere. I guess you don't have an apple machine as icloud offers similar to dropbox. Monthly cost for 1 tb is £7.99 or £11 per month for unlimited space. I've been using Dropbox for years and swear by it for file sharing (big files in particular that are too big for attachment)

I run a Synology Diskstation at home with 16tb of storage, I can access it anywhere in the world.... it costs me nothing per month.

That is quite ok for people who are not really demanding. If your Synology station is in Thailand the speed with what you can access it and download stuff is far below that of dropbox. Because dropbox has servers anywhere in the world and your NAS station is limited by the upload (not dowload) speeds in Thailand. In general these are 1mb or so and if your in an other part of the world.. you would be lucky to get 30% of that speed.

I got a Qnap Nas station and it has the same limitations.

[NeverSure]

Please understand the Windows VSS service. Attached is a screenshot of my Cobian folder. It contains two folders with two backups so figure half of that file count and half of that data is going to a USB external in less than 25 minutes with no input from me other than initial setup. It's on a scheduler and does that every night while I sleep and then deletes the oldest folder/backup automatically.

Edited May 17, 2015 by NeverSure

[robblok]

Please understand the Windows VSS service. Attached is a screenshot of my Cobian folder. It contains two folders with two backups so figure half of that file count and half of that data is going to a USB external in less than 25 minutes with no input from me other than initial setup. It's on a scheduler and does that every night while I sleep and then deletes the oldest folder/backup automatically.

Isnt that the way it is supposed to work ? I mean if you use any backup software, I would not accept it if it did not work like that. Damm even 20 years ago I did the same thing but with tape streamers.

[robblok]

That explains why it does not work for you and it works for me. I got 20gb of files (mostly 2kb -10kb). It takes ages to send this to an other drive over the network. Now by compressing it first I safe many hours and I do have a real powerful computer and use it.

Your case is irrelevant. You forgot that we are not talking about 20GB of your data, the OP stated that he/she has 1TB of data.

Do try with 1TB of data and you will see.

On addition to that. To be able to compress 1TB of misc files within TB's of RAM with 50% compression rate, would still require 1.5TB of local disk space on both ends of the transfer.

The steps when creating compressing archives

1 Read the data from the disk a (originating computer)

2 Create compressed archive

3 Write the archive to disk a

4 Transfer the archive to disk b (destination computer)

5 Write the archive to the disk b

6 Uncompress the archive to disk b

7 remove the compressed archive on disk b

While steps 1 and 2 as well as 4 and 5 can be done simultaneously, there is still quite a few steps, which slow down.

I give a command rsync -arvP /diskA /diskB which does reading disk B, transferring and writing disk B simultaneously.

My mistake i forgot that not everyone has 10tb space of harddisks in their system. Space was not a problem for me just speed. I will try it with one TB and report back.

The solution of just swapping HD's is the one I would use. It would go fastest.

But as i said.. if you got multiple small files its best to compress them in one and then send over the network. That is a fact it will go faster as not compressing. Even with 1TB.

But your right in this case The OP has no space to spare.. so it wont work.

[NeverSure]

Please understand the Windows VSS service. Attached is a screenshot of my Cobian folder. It contains two folders with two backups so figure half of that file count and half of that data is going to a USB external in less than 25 minutes with no input from me other than initial setup. It's on a scheduler and does that every night while I sleep and then deletes the oldest folder/backup automatically.

Isnt that the way it is supposed to work ? I mean if you use any backup software, I would not accept it if it did not work like that. Damm even 20 years ago I did the same thing but with tape streamers.

You didn't use the VSS 20 years ago because it came out with SP2 for XP. It came out in Server 2003. It defeats the stop screens you might see when copying such as "can't move/copy/etc". because the file is in use." That's the Windows File Protection System. It also defeats the "can't copy/move/etc. because the file name is too long." By defeat I mean you don't get a stop or a message. It just copies it anyway.

VSS also takes a snapshot of what you want so that any changes while the copying is happening don't affect the copy.

These issues may be a problem for the OP as he tries to copy or move so many files to another disk from a running computer using Ethernet or USB. He may be continually stopped and have to tell Windows what to do. THAT's time consuming.

[robblok]

Please understand the Windows VSS service. Attached is a screenshot of my Cobian folder. It contains two folders with two backups so figure half of that file count and half of that data is going to a USB external in less than 25 minutes with no input from me other than initial setup. It's on a scheduler and does that every night while I sleep and then deletes the oldest folder/backup automatically.

Isnt that the way it is supposed to work ? I mean if you use any backup software, I would not accept it if it did not work like that. Damm even 20 years ago I did the same thing but with tape streamers.

You didn't use the VSS 20 years ago because it came out with SP2 for XP. It came out in Server 2003. It defeats the stop screens you might see when copying such as "can't move/copy/etc". because the file is in use." That's the Windows File Protection System. It also defeats the "can't copy/move/etc. because the file name is too long." By defeat I mean you don't get a stop or a message. It just copies it anyway.

VSS also takes a snapshot of what you want so that any changes while the copying is happening don't affect the copy.

These issues may be a problem for the OP as he tries to copy or move so many files to another disk from a running computer using Ethernet or USB. He may be continually stopped and have to tell Windows what to do. THAT's time consuming.

I thought we were talking about data.. usually you get the file in use things if you are copying an operating system. For data in general you don't have that problem.

[Don Mega]

I use dropbox, then I can access my data from any machine, anywhere. I guess you don't have an apple machine as icloud offers similar to dropbox. Monthly cost for 1 tb is £7.99 or £11 per month for unlimited space. I've been using Dropbox for years and swear by it for file sharing (big files in particular that are too big for attachment)

I run a Synology Diskstation at home with 16tb of storage, I can access it anywhere in the world.... it costs me nothing per month.

That is quite ok for people who are not really demanding. If your Synology station is in Thailand the speed with what you can access it and download stuff is far below that of dropbox. Because dropbox has servers anywhere in the world and your NAS station is limited by the upload (not dowload) speeds in Thailand. In general these are 1mb or so and if your in an other part of the world.. you would be lucky to get 30% of that speed.

I got a Qnap Nas station and it has the same limitations.

I use it as my media server.

It performs exactly the same no matter what country Iam in at the time.

[robblok]

It is faster to transfer a single large file instead of lots of little files because of the overhead of negotiating the transfer. The negotiation is done for each file, so transferring a single file it needs to be done once, transferring n files means it needs to be done n times.

You will save yourself a lot of time if you zip first before the transfer.

Lots of little files will also be more expensive to write to the file system than a single large file. It needs to do things like:

• Check the file name is unique
• Write out the file table entry
• 
  

As you get more and more files in a directory this can become quite costly. And each of these steps can add latency to the copy process and slow the whole thing down.

This was what i was talking about when i said zip or rar it first.

Edited May 17, 2015 by robblok

[robblok]

I use dropbox, then I can access my data from any machine, anywhere. I guess you don't have an apple machine as icloud offers similar to dropbox. Monthly cost for 1 tb is £7.99 or £11 per month for unlimited space. I've been using Dropbox for years and swear by it for file sharing (big files in particular that are too big for attachment)

I run a Synology Diskstation at home with 16tb of storage, I can access it anywhere in the world.... it costs me nothing per month.

That is quite ok for people who are not really demanding. If your Synology station is in Thailand the speed with what you can access it and download stuff is far below that of dropbox. Because dropbox has servers anywhere in the world and your NAS station is limited by the upload (not dowload) speeds in Thailand. In general these are 1mb or so and if your in an other part of the world.. you would be lucky to get 30% of that speed.

I got a Qnap Nas station and it has the same limitations.

I use it as my media server.

It performs exactly the same no matter what country Iam in at the time.

Sure but dropbox has the speed advantage for data, for a media server you don't need high speeds. I got NAS drive like you a quite good one 30k baht, and I still prefer dropbox for data especially if my NAS is in Thailand and i need it somewhere else.

Remember with a NAS drive your limited by the UPLOAD speed of your internet package.. that is usually 1mb.. can download with 30mb from dropbox so dropbox would be 30times faster. (or more/less depending on your internet connection)

Anyway for your use it works, for others dropbox might be faster.

[Don Mega]

By the time I upload to dropbox then provide a download link.....

My NAS is not far off the pace as everything is stored there.

Edited May 17, 2015 by Don Mega

[maidee]

one question ... is this a collection of all your downloaded movies?

if yes, just delete what you don't need, don't be sentimental

[robblok]

one question ... is this a collection of all your downloaded movies?

if yes, just delete what you don't need, don't be sentimental

I actually keep my movies that i downloaded sometimes even re watch them. I do the same for TV series.

[2fishin2]

You should purchase a larger USB drive and keep your media on that vs your computer's main OS drive. That would be the simplest and easiest thing to do.