Day 10 - Scheduling tasks
Introduction
Linux has a rich set of features for running scheduled tasks. One of the key attributes of a good sysadmin is getting the computer to do your work for you (sometimes misrepresented as laziness!) - and a well configured set of scheduled tasks is key to keeping your server running well.
The time-based job scheduler cron(8) is the one most commonly used by Linux sysadmins. It’s been around more or less in it’s current form since Unix System V and uses a standardized syntax that’s in widespread use.
Using at to schedule oneshot tasks
If you’re on Ubuntu, you will likely need to install the at package first.
sudo apt update
sudo apt install at
We’ll use the at command to schedule a one time task to be ran at some point
in the future.
Next, let’s print the filename of the terminal connected to standard input (in Linux everything is a file, including your terminal!). We’re going to echo something to our terminal at some point in the future to get an idea of how scheduling future tasks with at works.
vagrant@ubuntu2204:~$ tty
/dev/pts/0
Now we’ll schedule a command to echo a greeting to our terminal 1 minute in the future.
vagrant@ubuntu2204:~$ echo 'echo "Greetings $USER!" > /dev/pts/0' | at now + 1 minutes
warning: commands will be executed using /bin/sh
job 2 at Sun May 26 06:30:00 2024
After several seconds, a greeting should be printed to our terminal.
...
vagrant@ubuntu2204:~$ Greetings vagrant!
It’s not as common for this to be used to schedule one time tasks, but if you ever needed to, now you have an idea of how this might work. In the next section we’ll learn about scheduling time-based tasks using cron and crontab.
For a more in-depth exploration of scheduling things with at review the
relevant articles in the further reading section below.
Using crontab to schedule jobs
In Linux we use the crontab command to interact with tasks scheduled with
the cron daemon. Each user, including the root user, can schedule jobs that run
as their user.
Display your user’s crontab with crontab -l.
vagrant@ubuntu2204:~$ crontab -l
no crontab for vagrant
Unless you’ve already created a crontab for your user, you probably won’t have one yet. Let’s create a simple cronjob to understand how it works.
Using the crontab -e command, let’s create our first cronjob. On Ubuntu, if
this is you’re first time editing a crontab you will be greeted with a menu to
choose your preferred editor.
vagrant@ubuntu2204:~$ crontab -e
no crontab for vagrant - using an empty one
Select an editor. To change later, run 'select-editor'.
1. /bin/nano <---- easiest
2. /usr/bin/vim.basic
3. /usr/bin/vim.tiny
4. /bin/ed
Choose 1-4 [1]: 2
Choose whatever your preferred editor is then press Enter.
At the bottom of the file add the following cronjob and then save and quit the file.
* * * * * echo "Hello world!" > /dev/pts/0
NOTE: Make sure that the
/dev/pts/0file path matches whatever was printed by yourttycommand above.
Next, let’s take a look at the crontab we just installed by running crontab -l
again. You should see the cronjob you created printed to your terminal.
vagrant@ubuntu2204:~$ crontab -l
* * * * * echo "Hello world!" > /dev/pts/0
This cronjob will print the string Hello world! to your terminal every minute until we remove or update the cronjob. Wait a few minutes and see what it does.
vagrant@ubuntu2204:~$ Hello world!
Hello world!
Hello world!
...
When you’re ready uninstall the crontab you created with crontab -r.
Understanding crontab syntax
The basic crontab syntax is as follows:
* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)
- Minute values can be from 0 to 59.
- Hour values can be from 0 to 23.
- Day of month values can be from 1 to 31.
- Month values can be from 1 to 12.
- Day of week values can be from 0 to 6, with 0 denoting Sunday.
There are different operators that can be used as a short-hand to specify multiple values in each field:
| Symbol | Description |
|---|---|
| * | Wildcard, specifies every possible time interval |
| , | List multiple values separated by a comma. |
| - | Specify a range between two numbers, separated by a hyphen |
| / | Specify a periodicity/frequency using a slash |
There’s also a helpful site to check cron schedule expressions at crontab.guru.
Use the crontab.guru site to play around with the different expressions to get an idea of how it works or click the random button to generate an expression at random.
Your Tasks Today
- Schedule daily backups of user’s home directories
- Schedule a task that looks for any backups that are more than 7 days old and deletes them
Automating common system administration tasks
One common use-case that cronjobs are used for is scheduling backups of various
things. As the root user, we’re going to create a cronjob that creates a
compressed archive of all of the user’s home directories using the tar utility.
Tar is short for “tape archive” and harkens back to earlier days of Unix and
Linux when data was commonly archived on tape storage similar to cassette tapes.
As a general rule, it’s good to test your command or script before installing it
as a cronjob. First we’ll create a backup of /home by manually running a
version of our tar command.
vagrant@ubuntu2204:~$ sudo tar -czvf /var/backups/home.tar.gz /home/
tar: Removing leading `/' from member names
/home/
/home/ubuntu/
/home/ubuntu/.profile
/home/ubuntu/.bash_logout
/home/ubuntu/.bashrc
/home/ubuntu/.ssh/
/home/ubuntu/.ssh/authorized_keys
...
NOTE: We’re passing the
-vverbose flag totarso that we can see better what it’s doing.-czfstand for “create”, “gzip compress”, and “file” in that order. Seeman tarfor further details.
Let’s also use the date command to allow us to insert the date of the backup
into the filename. Since we’ll be taking daily backups, after this cronjob
has ran for a few days we will have a few days worth of backups each with it’s
own archive tagged with the date.
vagrant@ubuntu2204:~$ date
Sun May 26 04:12:13 UTC 2024
The default string printed by the date command isn’t that useful. Let’s output
the date in ISO 8601 format, sometimes referred to as the “ISO date”.
vagrant@ubuntu2204:~$ date -I
2024-05-26
This is a more useful string that we can combine with our tar command to
create an archive with today’s date in it.
vagrant@ubuntu2204:~$ sudo tar -czvf /var/backups/home.$(date -I).tar.gz /home/
tar: Removing leading `/' from member names
/home/
/home/ubuntu/
...
Let’s look at the backups we’ve created to understand how this date command is being inserted into our filename.
vagrant@ubuntu2204:~$ ls -l /var/backups
total 16
-rw-r--r-- 1 root root 8205 May 26 04:16 home.2024-05-26.tar.gz
-rw-r--r-- 1 root root 3873 May 26 04:07 home.tar.gz
NOTE: These
.tar.gzfiles are often called tarballs by sysadmins.
Create and edit a crontab for root with sudo crontab -e and add the following
cronjob.
0 5 * * * tar -zcf /var/backups/home.$(date -I).tar.gz /home/
This cronjob will run every day at 05:00. After a few days there will be several
backups of user’s home directories in /var/backups.
If we were to let this cronjob run indefinitely, after a while we would end up
with a lot of backups in /var/backups. Over time this will cause the disk
space being used to grow and could fill our disk. It’s probably best
that we don’t let that happen. To mitigate this risk, we’ll setup another
cronjob that runs everyday and cleans up old backups that we don’t need to store.
The find command is like a swiss army knife for finding files based on all
kinds of criteria and listing them or doing other things to them, such as
deleting them. We’re going to craft a find command that finds all of the
backups we created and deletes any that are older than 7 days.
First let’s get an idea of how the find command works by finding all of our
backups and listing them.
vagrant@ubuntu2204:~$ sudo find /var/backups -name "home.*.tar.gz"
/var/backups/home.2024-05-26.tar.gz
...
What this command is doing is looking for all of the files in /var/backups
that start with home. and end with .tar.gz. The * is a wildcard character
that matches any string.
In our case we need to create a scheduled task that will find all of the files
older than 7 days in /var/backups and delete them. Run sudo crontab -e and
install the following cronjob.
30 5 * * * find /var/backups -name "home.*.tar.gz" -mtime +7 -delete
NOTE: The
-mtimeflag is short for “modified time” and in our casefindis looking for files that were modified more than 7 days ago, that’s what the +7 indicates. Thefindcommand will be covered in greater detail on Day 11 - Finding things….
By now, our crontab should look something like this:
vagrant@ubuntu2204:~$ sudo crontab -l
# Daily user dirs backup
0 5 * * * tar -zcf /var/backups/home.$(date -I).tar.gz /home/
# Retain 7 days of homedir backups
30 5 * * * find /var/backups -name "home.*.tar.gz" -mtime +7 -delete
Setting up cronjobs using the find ... -delete syntax is fairly idiomatic of
scheduled tasks a system administrator might use to manage files and remove
old files that are no longer needed to prevent disks from getting full. It’s not
uncommon to see more sophisticated cron scripts that use a combination of tools
like tar, find, and rsync to manage backups incrementally or on a schedule
and implement a more sophisticated retention policy based on real-world use-cases.
System crontab
There’s also a system-wide crontab defined in /etc/crontab. Let’s take a look
at this file.
vagrant@ubuntu2204:~$ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
# You can also override PATH, but by default, newer versions inherit it from the environment
#PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
By now the basic syntax should be familiar to you, but you’ll notice an extra
field user-name. This specifies the user that runs the task and is unique to
the system crontab at /etc/crontab.
It’s not common for system administrators to use /etc/crontab anymore and
instead user’s are encouraged to install their own crontab for their user, even
for the root user. User crontab’s are all located in /var/spool/cron. The
exact subdirectory tends to vary depending on the distribution.
vagrant@ubuntu2204:~$ sudo ls -l /var/spool/cron/crontabs
total 8
-rw------- 1 root crontab 392 May 26 04:45 root
-rw------- 1 vagrant crontab 1108 May 26 05:45 vagrant
Each user has their own crontab with their user as the filename.
Note that the system crontab shown above also manages cronjobs that run daily,
weekly, and monthly as scripts in the /etc/cron.* directories. Let’s look
at an example.
vagrant@ubuntu2204:~$ ls -l /etc/cron.daily
total 20
-rwxr-xr-x 1 root root 376 Nov 11 2019 apport
-rwxr-xr-x 1 root root 1478 Apr 8 2022 apt-compat
-rwxr-xr-x 1 root root 123 Dec 5 2021 dpkg
-rwxr-xr-x 1 root root 377 Jan 24 2022 logrotate
-rwxr-xr-x 1 root root 1330 Mar 17 2022 man-db
Each of these files is a script or a shortcut to a script to do some regular
task and they’re run in alphabetic order by run-parts. So in this case
apport will run first. Use less or cat to view some of the scripts on your
system - many will look very complex and are best left well alone, but others
may be just a few lines of simple commands.
vagrant@ubuntu2204:~$ cat /etc/cron.daily/dpkg
#!/bin/sh
# Skip if systemd is running.
if [ -d /run/systemd/system ]; then
exit 0
fi
/usr/libexec/dpkg/dpkg-db-backup
As an alternative to scheduling jobs with crontab you may also create a script
and put it into one of the /etc/cron.{daily,weekly,monthly} directories and
it will get ran at the desired interval.
A note about systemd timers
All major Linux distributions now include “systemd”. As well as starting and stopping services, this can also be used to run tasks at specific times via “timers”. See which ones are already configured on your server with:
systemctl list-timers
Use the links in the further reading section to read up about how these timers work.
Further reading
- freeCodeCamp: Job Scheduling in RHEL – cron and at Explained with Examples
- nixCraft: How To Add Jobs To cron Under Linux or UNIX
- Arabesque: Cron best practices
- Linode: Using Cron to Schedule Tasks for Certain Times or Intervals
- Arch Wiki: A good overview of systemd/Timers
- How to Use Systemd Timers as a Cron Replacement
License
Some rights reserved. Check the license terms here