USB3 speed

What sort of speeds should I get from USB3? Currently I have a Crucial X6 SSD plugged into a USB3.1 Gen 1 port and CrystalDiskMark gives me about 480MBps on sequential reads and writes. But real world transfers I am seeing 250MBps write and about the same for read, highly dependent on OS. Fastest I have found is Debian 12 at 250MBps. Of course in Windows there is the Momentum cache from Crucial which gives 5000MBps r/w on CrystalDiskMark which is effectively writing to RAM. But I don't use Windows much.

I would suggest that the USB is probably working fine, the issue is more likely the disk you are writing to or from plus its operating system overheads. You can of course create a file system in RAM using ramdisk on Linux. Tutorial here https://www.linuxbabe.com/command-line/create-ramdis... - although I had to use 'top' rather than 'htop'

That is a good tutorial for a RAM disk but what I need is a large cache like the Crucial Momentum feature in Windows (2Gb-4Gb) but for Linux as I am backing up 2Gb files to an external SSD, the Crucial X6 1Tb.

Sorry, I was only suggesting that as a means of determining whether your USB3 was working OK.

In reply to a post by DFScale:

Sorry, I was only suggesting that as a means of determining whether your USB3 was working OK.

Sorry you lost me on that.

A caching program, like Momentum, is never going to speed up the overall transfer time. If there is a bottleneck between the computer bus and the disk than the data transfer rate is still limited by that bottleneck.

A caching program will give the illusion of faster transfer rates - but you don't see, behind the scenes, that it hasn't finished writing the data to the disk. So you can't swith the computer off until all the data has been flushed.

Look at an analogy. Suppose your bath took five minutes to empty because of a restricted waste pipe width. You could put another tank in between the bath and the outflow with a much bigger pipe joining them. Then the bath would appear to empty in - say - 1 minute. But the water is in the intermediate tank still draining out at the restricted flow rate. It still takes 5 minutes to transfer all the water; all you have done is to introduced another overhead and another delay because of that.

Now with a bath that doesn't matter - it is, to all intents and purposes empty. But in your case the backup isn't complete - the data is still in RAM waiting to be flushed to the disk. Turn the computer off - or have a poiwer cut - and that data is lost and your backup is corrupt. You can't beat the laws of physics.

What does it matter how long the backup takes? Just leave it to do its own thing whilst you get on with whatever else you were doing. We are long past the age of DOS when you had to wait for one program to finish before running another one.

And don't worry about benchmarks; they are designed to sell you something not to measure real-world performance.

I went through a similar exercise recently:

When USB 4 (throughput) isn't quite claimed...in Mac-land

In reply to a post by TinyMongomery:

A caching program, like Momentum, is never going to speed up the overall transfer time. If there is a bottleneck between the computer bus and the disk than the data transfer rate is still limited by that bottleneck.

A caching program will give the illusion of faster transfer rates - but you don't see, behind the scenes, that it hasn't finished writing the data to the disk. So you can't swith the computer off until all the data has been flushed.

Look at an analogy. Suppose your bath took five minutes to empty because of a restricted waste pipe width. You could put another tank in between the bath and the outflow with a much bigger pipe joining them. Then the bath would appear to empty in - say - 1 minute. But the water is in the intermediate tank still draining out at the restricted flow rate. It still takes 5 minutes to transfer all the water; all you have done is to introduced another overhead and another delay because of that.

Now with a bath that doesn't matter - it is, to all intents and purposes empty. But in your case the backup isn't complete - the data is still in RAM waiting to be flushed to the disk. Turn the computer off - or have a poiwer cut - and that data is lost and your backup is corrupt. You can't beat the laws of physics.

What does it matter how long the backup takes? Just leave it to do its own thing whilst you get on with whatever else you were doing. We are long past the age of DOS when you had to wait for one program to finish before running another one.

And don't worry about benchmarks; they are designed to sell you something not to measure real-world performance.

Yes I see now suppose I will have to suffer 2 hour backups for my machine.

In reply to a post by Pheasant:

I went through a similar exercise recently:

When USB 4 (throughput) isn't quite claimed...in Mac-land

Interesting read and yes I am seeing similar low speeds in real world but tests are not too bad.

In reply to a post by Banger:

what I need is a large cache like the Crucial Momentum feature in Windows (2Gb-4Gb) but for Linux as I am backing up 2Gb files to an external SSD, the Crucial X6 1Tb.

Is it specifically 2GB files that you are backing up or a hard drive with 2GB files? May be rsync [there might be a pun in there] will do what you want?

rsync will back up a file system to a clone of that filesystem, only backing up anything which has changed since the last backup. I believe it can run in the background too, which could address your concerns about USB speed if your workflow will accommodate this.

Perhaps you need to review your backup strategy. A regime of full backups once in a while, with intervening incremental or differential backups, should normally only take a few minutes with the occasional longer time. And do you really need to back up all of that data every time?

What backup program do you use? Something like Back in Time, which is quite similar to Apple’s Time Machine, might suit you.

3.1 Gen 1 is the same as 3.0 which is 5 Gbps theoretical. I've seen a 1 Gigabit ethernet run at 900 Mbps over a 3.0 connection, so high speeds are possible.

Any anti-malware tools running on your machine, and the drivers of the USB port, and lots of other things can make different drives quicker and slower. Check reviews.

In reply to a post by DFScale:

In reply to a post by Banger:

what I need is a large cache like the Crucial Momentum feature in Windows (2Gb-4Gb) but for Linux as I am backing up 2Gb files to an external SSD, the Crucial X6 1Tb.

Is it specifically 2GB files that you are backing up or a hard drive with 2GB files? May be rsync [there might be a pun in there] will do what you want?

rsync will back up a file system to a clone of that filesystem, only backing up anything which has changed since the last backup. I believe it can run in the background too, which could address your concerns about USB speed if your workflow will accommodate this.

Why 2gb files. I am using foxclone sometimes rescuezilla as redundancy which splits files backups into 2gb chunks which I believe is more efficient for the SSD cache and can backup individual partitions. Backup strategy is a full backup once a month (patch Tuesday) before MSFT updates and once a week before rolling updates of my linux partitions. Linux paritions only take 20 mins but I have to boot into foxclone because you cant backup a live system. Macrium the windows alternative isnt really Linux aware and diffs or incrementals copy the whole linux partition at a time and havent tested a restore with macrium.

So far I have found the fastest running OS is Debian 12.9 live ISO with Foxclone runs around 9GB/min and 15GB/min for a verify which cuts the time down.

I dont know if a cache will make much difference as when I write files to USB stick they are fast in Solus but when I reboot it takes ages to flush the cache and the USB light stays on for about 5 minutes after hitting the restart button.

I have just tried Acronis and it took overnight to backup 1tb to my external 3tb WD on USB3 so that is a non starter. I know HDD is only 125MB/s but overnight seems a long time.

Image backup tools are fine for occasional use for disaster recovery, but totally inappropriate for regular backups. No wonder your backups take so long.

Use a backup program not a cloning program.

In reply to a post by TinyMongomery:

Image backup tools are fine for occasional use for disaster recovery, but totally inappropriate for regular backups. No wonder your backups take so long.

Use a backup program not a cloning program.

I was using Macrium with diff and incremental but it was taking as long as a clone.

Cloning is the wrong level of file system abstraction. Imagine the consequences of a duff sector on your clone disk. It means that the whole image can only be a failure. Whereas a backup of files will ride on and not use bad sectors.

Does your backup disk have plenty of free space?

And it makes it a nightmare to restore individual files.

Incremental backups using sync (or some wrapper program) should only take minutes unless you are changing huge amounts of data per day. Even when using a mechanical hard disk.

Personally, l wouln’t use an SSD as the destination for backups.

In reply to a post by TinyMongomery:

Does your backup disk have plenty of free space?

I have 4 backup disks, 2 x 1tb portable SSDs and 2 x 3tb HDDs which I rotate. With compression the 1tb disks have about 200gb spare after a backup.

If each backup is using 800GB, with compression, I’d suggest that you are not doing incremental backups.

Disk space down to 20% free probably impinges on backup speed somewhat, but not to the extent you describe. Do you format the disks before the backups?

In reply to a post by TinyMongomery:

If each backup is using 800GB, with compression, I’d suggest that you are not doing incremental backups.

Disk space down to 20% free probably impinges on backup speed somewhat, but not to the extent you describe. Do you format the disks before the backups?

I usually delete and trim the SSDs of old backups. HDDs just delete old snapshots.

Just tried to restore a disk selectively with Acronis True Image and it failed at 75% wiping the hard disk. Fortunately Foxclone came to the rescue and restored successfully. Wont be subscribing to Acronis.

Keep to the 3-2-1 rule of backups and most importantly; regularly test 👍

That's a good demonstration of the dangers that DFScale pointed out of using cloning rather than backing up. (That's apart from the immense time saving of using a proper backup program.)

But it's good to test your backups. Too many people forget that aspect.

I spent many years managing backups for an international corporation and I am firmly of the belief that the best policy is a continuous backup system, where any changed files - and only changed files - are regularly saved and a certain number of generations of each file kept. (Obviously there is an initial phase when every file is saved.) I had the advantage of using very sophisticated backup software (IBM's Tivolli) which stored initial backups on disk and also to a tape library. When the disk storage was filling up files were shifted to the tape library. (And, of course, copies of the tapes were moved offsite every day.) Backups of a huge total amount of data from roundabout 1,000 users were almost instantaneous, despite the fact that they were happening over a - by today's standards - slow network, and any recently deleted files could be retrieved in a matter of minutes. That was very popular with my users.

This is much the way that Apple's Time Manager works; it is by far the best personal backup software (and it's free!). Similar programs exist for Windows and Linux.

Backups serve two purposes: 1. Disaster recovery. Very rarely needed in practice; 2. Recovery of corrupted or accidently deleted data. Far more common. Cloning programs address the first use, but not the second. Backup programs address both, but make disaster recovery slightly more complicated. To me that doesn't matter as it is such a rare scenario.

In reply to a post by Pheasant:

Keep to the 3-2-1 rule of backups and most importantly; regularly test 👍

Ted Rogers and Dusty Bin?

Well just done a complete system backup x2 with Macrium Reflect to HDDs and it took 4 hours each. 2 hours to backup and 2 hours to verify - not taking any chances. Will test later. Good thing about Macrium is you can mount the backup and extract individual files if needed from each drive and it is free or was.

Macrium failed to restore the partition table so non booting machine. Back to FoxClone to restore sanity.

In reply to a post by Pheasant:

Keep to the 3-2-1 rule of backups and most importantly; regularly test 👍

Amen

Back it up
Back it up
Back it up

So I have decided on my backup routine. I will do a full Foxclone image once a month possibly to SSD as it is less likely to have bad blocks which my 3TB WD have despite only being powered on for 2 years. I will do a weekly differential with Macrium now that I have a base backup as it only takes about 30 minutes to do a diff and they can be restored fairly quickly or individual files can be picked out of the backup.

Had to retire one of my 3TB WDs (I have 4) as when I was running a backup with Macrium it disappeared from the system and Macrium said backup aborted nothing to write to. This might have happened with Acronis as it was the same disk but Acronis didn't complain. Why it did this I suspect overheating. Anyway now retired.