Have an Ubuntu machine with a large collection of files (2.7TB, thousands of directories, in just under 2 million files). I need a daily incremental file backup so that a user can easily navigate through the backup as if they were browsing any other file directory (via Windows Explorer and the assigned SMB unit).
Until the backup became so large, a combination of using cp -al to rotate the destination folder from the most recent backup to the previous day using hard links and then doing an rsync to the destination folder of the copy Most recent security work worked just fine. However, I moved the script from the NAS (so I was "pulling" the files to make a backup) to a new server and now I'm running the script on the source server instead of the device. destination of the backup.
I am not sure if the change from a pull to a push is causing the problem or if the set of files is too large, but the script is failing and there is no error in the output of the CPU or rsync with the verbose log. It just stops, and I'm finding that the cp and rsync processes are still running in memory and do not seem to be doing anything. It's as if rsync & # 39; fails & # 39; but it is not completely removed from memory.
The date of origin, approximately 95% or more, does not change because they are file data, but MAY change from time to time. Then, the obvious solution would be to segment the backup to make only the most recent directories and then make a backup of the quite static directories separately in a less frequent interval. Or, switch to a completely different backup solution.
But as I said, the restrictions are that the backup should be easily explorable in Windows Explorer through a mapped drive.
So I'm wondering if there's any rsync option (or other trick) that I can use to speed up the backup? It's almost as if what you need is that rsync can know if a file has been changed in a directory without having to read the file information for each file and THEN immerse yourself in the directory.
I'm using rsync with the options: -rlptgoh (oops, I just noticed that I took the v to see more details at some point, well, I'll keep adding that and see if I get additional information about the problem)
But are you still interested if there are suggestions on a better approach given the requirements, or some other combination of rsync options? The ability to navigate incrementals through Windows Explorer actually throws a curve ball into the typical recommendations of system administrators, as they generally do not meet that requirement.