Arch Linux: Services and Timers

Creating a functioning backup.service and backup.timer pair

Context
Arch Linux is an incredibly powerful operating system, but documentation (including answers on StackOverflow) and Red Hat on how to accomplish specific tasks are often a bit convoluted and not beginner-friendly.Case in point, I have been trying to create a timer-based backup service for my arch linux server, which I use to create a coding environment I can access on my Chromebook (thank you to the code-server project). Yes, I could just install the crontab package from the Arch AUR repository, but in the spririt of learning and of keeping my server operating system as minimal and lightweight as possible, I will use the built-in service-timer pair functionality in systemctl.

The ArchWiki pages provide great detail on what systemctl services and timers do, but not a lot of information on how to set them up.
So, I am just going to dive in and see if I can create one without breaking my server and document my progress along the way.

Project Goals
Create weekly and nightly backup timers using the Arch Linux .service-.timer pairs
WEEKLY BACKUP: create a zipped, tarball backup of my entire drive. If my server crashes and I need to re-install Arch, I want to be able to restore all my custom bash files from before.
NIGHTLY BACKUP:: git (or git-annex) commit incremental backup of my projects nightly so I have a nightly version-controlled backup of my important projects.

Current project state
Currently I have my function saved as backup.sh in /root. To execute the backup, I have to run ./backup.sh in bash
NOTE: To avoid errors in the jupyter notebook container, I have changed the folder names to reflect sample folders created in the container

In [2]:
#
# backup.sh
# tar backup script to be run weekly
# backup all system files to /mnt/backup/Linux Backups/Arch-server

# mount External HD
sudo mount /dev/sdXYZ /mnt/path_to_HD

# set timestamp
TIMESTAMP=$(date +%m.%d.%Y);

#set destination filename
FILENAME="code-server-$TIMESTAMP.tar";

#set destination path
DESTINATION="backup_folder_ex"   # replace with "mnt/path_to_HD/backup_folder"


# set source folder
SOURCEFOLDER="source_folder"   # replace with "root"

echo "Backing up $SOURCEFOLDER to $DESTINATION/$FILENAME"
date
echo

#create backup
tar -cpzf $DESTINATION/$FILENAME $SOURCEFOLDER

#print confirmation
echo
echo "Backup Finished"
echo "Source folder: $SOURCEFOLDER"
echo "Saved as: $DESTINATION/$FILENAME"


# unmount External HD for security
sudo umount /mnt/path_to_HD
echo "External HD disconnected"

Backing up source_folder to backup_folder_ex/code-server-12.17.2019.tar.gz
Tue Dec 17 20:07:15 UTC 2019


Backup Finished
Source folder: source_folder
Saved as: backup_folder_ex/code-server-12.17.2019.tar.gz

Weekly Automation

I have decided to use the Realtime timer (description from ArchWiki below) so that I can set it to run sometime when I most likely won't be awake and using the server coding environment. If for some reason I leave the server off for over a week or on the day it is supposed to run, I definitely want to employ the option Persistent=True Screenshot 2019-12-17 at 2.26.00 PM.png
From the ArchWiki systemd, systemd/Timers, and the systemd.service man-page, it looks like I need to create a backup.timer file and a backup.service file and put them in the /etc/systemd/system folder. The backup.service file will also have to call the backup.sh script. In the /etc/systemd/system/mutli-user.targer.wants folder, I also found the docker.service and ssh.service files, which will serve as a template:

In [ ]:
# docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

####################################################################################
# ssh.service


[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service
After=sshdgenkeys.service
After=network.target

[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target


backup.service

In [ ]:
# /etc/systemd/system/multi-user.target.wants/backup.service

[Unit]
Description=tar backup of all system files to be run weekly

[Service]
# will move backup.sh to /usr/bin
ExecStart=/usr/bin/backup.sh
# set type to 'oneshot: this is useful for scripts that do a single job and then exit. 
# You may want to set RemainAfterExit=yes as well so that systemd still considers the 
# service as active after the process has exited.'
Type=oneshot
RemainAfterExit=yes
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target


backup.timer

In [ ]:
# /etc/systemd/system/backup.timer

[Unit]
Description=Run backup.sh weekly

[Timer]
# OnCalendar=weekly ... based on ArchWiki page, it looks like default 'weekly' is Mon at 12am
# but just to be sure it performs the backup while I'm offline, I am going to change it a bit
OnCalendar=Mon 02:00:00
Persistent=true

[Install]
WantedBy=timers.target


Testing
Time to put the files in place and try activating the service by using systemctl enable backup.service

Output of systemctl enable backup.service:
Failed to enable unit: Unit file backup.service does not exist.
Didn't work. Let's try putting both files one folder up just in the etc/systemd/system

Output of systemctl enable backup.service now:
Created symlink /etc/systemd/system/multi-user.target.wants/backup.service → /etc/systemd/system/backup.service.
Looks like it worked! Out of curiosity, let's see if the symlink put anything new in the multi-user.target.wants folder...yep, looks like the symlink created a copy of backup.service in the multi-user.target.wants folder. Now let's start it and figure out how to make sure it is running.


Start backup.service

In [ ]:
systemctl start backup.service


Failed to start backup.service: Unit backup.service has a bad unit file setting.
See system logs and 'systemctl status backup.service' for details.
In [ ]:
systemctl status backup.service

● backup.service - tar backup of all system files to be run weekly
     Loaded: bad-setting (Reason: Unit backup.service has a bad unit file setting.)
     Active: inactive (dead)
Dec 17 18:20:07 code-server systemd[1]: backup.service: Service has Restart= set to either always or on-success, which isn't allowed for Type=oneshot services. Refusing.

Oops, looks like I can't use the restart=always option. Below is the updated backup.service file:

In [ ]:
# /etc/systemd/system/multi-user.target.wants/backup.service

[Unit]
Description=tar backup of all system files to be run weekly

[Service]
# will move backup.sh to /usr/bin
ExecStart=/usr/bin/backup.sh
# set type to 'oneshot: this is useful for scripts that do a single job and then exit. 
# You may want to set RemainAfterExit=yes as well so that systemd still considers the 
# service as active after the process has exited.'
Type=oneshot
RemainAfterExit=yes
KillMode=process

[Install]
WantedBy=multi-user.target

Let's try again and see if it works now:
systemctl status backup.service

● backup.service - tar backup of all system files to be run weekly
     Loaded: loaded (/etc/systemd/system/backup.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2019-12-18 16:45:26 CST; 45s ago
    Process: 3086964 ExecStart=/usr/bin/backup.sh (code=exited, status=203/EXEC)
   Main PID: 3086964 (code=exited, status=203/EXEC)


Dec 18 16:45:26 code-server systemd[1]: Starting tar backup of all system files to be run weekly...
Dec 18 16:45:26 code-server systemd[3086964]: backup.service: Failed to execute command: Exec format error
Dec 18 16:45:26 code-server systemd[3086964]: backup.service: Failed at step EXEC spawning /usr/bin/backup.sh: Exec format error
Dec 18 16:45:26 code-server systemd[1]: backup.service: Main process exited, code=exited, status=203/EXEC
Dec 18 16:45:26 code-server systemd[1]: backup.service: Failed with result 'exit-code'.
Dec 18 16:45:26 code-server systemd[1]: Failed to start tar backup of all system files to be run weekly.</pre>

Based on some research (this answer from stackexchange was helpful), it looks like all this was because I was missing the "shebang" to call the bash interpreter (i.e. #!/bin/bash) in the first line of the .sh file. After adding that, systemctl start backup.service worked!


Check Status
Alright, now that we've got backup.service started, let's see where we're at in terms of it running
Enabled services:
systemctl list-unit-files | grep backup

    backup.service                                                   enabled
    backup.timer                                                     disabled

Looks like we still need to enable backup.timer, so let's go ahead and do that...looks like that worked!
Looks like everything is working, and a run of systemctl restart backup.service has created my first systemd-triggered archive file, and from here on out it should be triggered by backup.timer

In [ ]: