My daily home dir backup script to hot backup machine

WalterDnes

Here is my daily backup script that backs up my home directory daily from my main desktop PC to my “hot backup” machine. The script has to wake up the target machine, do the rsync, and then poweroff the target machine. I don’t actually have a user named “misc”, but I’ve shoved a bunch of static stuff into /home/misc. One thing that is not static is the drive C: image for the ArcaOS (successor to OS/2) VM under QEMU. I’ve still got Galactic Civilizations for OS/2. It’s a l-o-o-o-n-g turn-based game. Because I usually play daily, the disc image is included in the daily backup.

Set up requirements

===================

Definitions:
“source machine” is the machine that is being backed up.
“target machine” is the machine that is receiving the backup.
“Wake-On-LAN” involves sendng packet(s) to a “powered-down” machine.
“Wake-On-Lan” has to be enabled in the target machine’s BIOS.
The target machine’s BIOS should NOT be set for “deep sleep” when powered down, so that it can respond to “Wake-On-Lan” requests.
Because the “powered-down” target machine does not have an IP address, we have to talk direct to its ethernet card’s MAC ID
Some linux distros have the older “ipconfig” program while others have the newer “ip” program.
webpage https://www.ubuntumint.com/find-mac-address-linux/ shows how to determine the MAC ID of an ethernet card using either “ipconfig” or “ip”. This has to be run on the target machine when it is powered up.
For the backup to work, both machines need to have “rsync” and “ssh” installed and sshd listener at bootup.
Booby trap… make sure that you do NOT copy over any “.ssh” directories. It’s OK when backing up to a USB drive, but not to another machine.
The source machine will need “wakeonlan” or equivalent program installed.
The backup script exists on my source machine as “/root/bin/bkup”. I originally ran it manually, but then decided to go fully automated with a cron job every day at 3:30 AM. Root’s crontab is…

30 03 * * * /root/bin/bkup

Things you’ll have to change

============================

The target machine’s MAC ID will have to replace “xx:xx:xx:xx:xx:xx”. See above for how to determine MAC ID.
My target machine is named “dev3”. Replace it with the name (will IP address work?) of your target machine.
My new target machine has an NVME drive and boots up in under 30 seconds!!! So I set the two sleep statements to 30 and 15 seconds. If your machine boots up from a “spinning rust” drive, allow more time for boot (and retry) by increasing the values with the “sleep” commands.
Change “/home/waltdnes” to your actual home dir

==================================================

The script follows in the reply. Suggestions and questions are welcome. It took me a while to wrap my head around the “–filter=” syntax, but when I did, I realized just how powerful it is.

WalterDnes

WalterDnes Arrrgh. The Flarum board software treats pound signs (comments) as special formatting, and butchers up the post beyond belief. Here is the script attached as a zip file. Extract bkup and mark executable to implement.

bkup.zip

837B

Donoreo

I ran OS/2 back in the day. I skipped Win 9.x and kept using OS/2. I never played GC, since I do not play games, but heard lots of good things about it.

WalterDnes

WalterDnes
I think I’ve figured out gow to “de-fang” weird formatting… use “insert code”. The only bother is that it insists on colourizing keywords. Oh, and how can you tell that Flarum was made for smartphones with desktop PCs being second-class citizens?
Anyhow, here is the script “in living colour”.

#!/bin/bash
#
# Send wakeup signal to remote machine (dev3)
wakeonlan xx:xx:xx:xx:xx:xx
#
# Wait 30 seconds and send one ping packet to check for a connection
sleep 30
ping -c 1 dev3
#
# If ping fails ( ${?} is not 0 ) wait 15 seconds and try again
if [ ${?} -ne 0 ] ; then
   echo "No connection after 30 seconds; waiting 15 more seconds."
   sleep 15
   ping -c 1 dev3
   if [ ${?} -ne 0 ] ; then
#
# If still no connection, give up.
     echo "No connection after 45 seconds; terminating backup attempt."
     echo "Backup failed $( date +%Y%m%d\ %H%M ) " >> /root/statuslog.txt
     exit
   fi
fi
#
# Housekeeping: "cd /" and remove old rsync and error log files.
cd /
rm -f /root/rserrors.txt /root/rsync_log.txt
#
# The actual rsync command
rsync --no-devices --no-specials --keep-dirlinks --links \
      --group --itemize-changes --owner --perms --progress \
      --recursive --stats --times --verbose \
      --filter='- */.cache' \
      --filter='- */.ssh' \
      --filter='- */.moonchild\ productions/pale\ moon/*/storage/' \
      --filter='- /home/waltdnes/pm' \
      --filter='+ /' \
      --filter='+ /home/' \
      --filter='+ /home/waltdnes/***' \
      --filter='+ /home/misc/' \
      --filter='+ /home/misc/qemu/' \
      --filter='+ /home/misc/qemu/arca/' \
      --filter='+ /home/misc/qemu/arca/arcac.img' \
      --filter='- *' \
      --log-file=/root/rsync_log.txt 2>/root/rserrors.txt \
      / waltdnes@dev3:/
#
# After finishing, power off the remote machine
ssh root@dev3 /sbin/poweroff
echo "Backup succeeded $( date +%Y%m%d\ %H%M ) " >> /root/statuslog.txt

Donoreo

I like editors that colourize keywords, it really helps to catch typos.

taraf

Two suggestions…

You could use a while loop to keep sleeping the script for 15s (or longer) until the backup server has booted.
You could alternately use netcat to open a daemon on a specific port & listen for a connection from the backup server, though that would require a corresponding script on the backup server to execute at boot in order to complete the connection. Then you just have to wait until the backup server closes the connection to know it’s ready. (and if the backup server sends a zero-length packet to the port in question it’ll disconnect immediately).

The 2nd suggestion is cleaner, but really only works if there’s only one system that’s backing up to one system. You can kinda get around that by giving the boot script on the backup server multiple hosts to connect to via netcat, but up to you. 🙂

The 3rd semi-suggestion is to use rtcwake on the backup server to have it wake itself up at a specific time, if for some reason wakeonlan isn’t working

EDIT – adding the code for the while loop, just in case this works. change this section:

ping -c 1 dev3
#
# If ping fails ( ${?} is not 0 ) wait 15 seconds and try again
if [ ${?} -ne 0 ] ; then
   echo "No connection after 30 seconds; waiting 15 more seconds."
   sleep 15
   ping -c 1 dev3
   if [ ${?} -ne 0 ] ; then
#
# If still no connection, give up.
     echo "No connection after 45 seconds; terminating backup attempt."
     echo "Backup failed $( date +%Y%m%d\ %H%M ) " >> /root/statuslog.txt
     exit
   fi
fi

to this:

counter=0
maxcounter=5
sleeptimer=15

ping -c 1 dev3
[[ $? -eq 0 ]] && counter=$maxcounter
while [ $counter -lt $maxcounter ]
do
  sleep $sleeptimer
  ping -c 1 dev3
  [[ $? -eq 0 ]] && counter=$maxcounter || counter=$counter+1
done

ping -c 1 dev3
if [ $? -ne 0 ]; then
   echo "No connection after $((${maxcounter}*${sleeptimer})) seconds; terminating backup attempt."
   echo "Backup failed $( date +%Y%m%d\ %H%M ) " >> /root/statuslog.txt
   exit
 fi

WalterDnes

I coded the the script for my specific situation. “It works for me”.

Donoreo

WalterDnes I coded the the script for my specific situation. “It works for me”.

So why did you share it then? I would share so that others could adapt make use of it, or potentially improve it. You are now suggesting not to do that. It makes no sense.

taraf

WalterDnes Ok. Glad it works for you 🙂

If you’re interested, I use Déjà Dup on my laptop. https://apps.gnome.org/DejaDup/

I don’t have to worry about waking my NAS up because it’s hosting a few containers (including the main DNS for my home network), but otherwise it works. 🙂

WalterDnes

Donoreo I would share so that others could adapt make use of it, or potentially improve it. You are now suggesting not to do that. It makes no sense.

I don’t expect others to follow me lockstep or visa versa. I’m sharing so that others can have a starting point to adapt TO THEIR SPECIFIC USE CASES. One size does not fit all.

Donoreo

WalterDnes I may have interpreted it as you being dismissive of suggestions. As we all know, it is hard to tell intent when someone is writing. I also had very little sleep, so I was probably off.