[Solved] Trouble setting systemd timer

I’m trying to move away from cron jobs, not that they don’t work, but I want to get on with the times and also learn some things.

I created two user timers (and the associated services), one for backing up my data and the second to upload to B2. I’m using two scripts I had in my cron jobs for a few years and they worked without problems. But with systemd timers both scripts fail with exit code 15 (process terminated) and I have no idea why.

I run Debian 12 Bookworm.

Here’s the output for the status of the upload service:


<span style="color:#323232;">> systemctl --user status rclone-up.service
</span><span style="color:#323232;">○ rclone-up.service - Run rclone up for b2
</span><span style="color:#323232;">     Loaded: loaded (/home/clmbmb/.config/systemd/user/rclone-up.service; disabled; preset: enabled)
</span><span style="color:#323232;">     Active: inactive (dead)
</span><span style="color:#323232;">TriggeredBy: ● rclone-up.timer
</span><span style="color:#323232;">
</span><span style="color:#323232;">Apr 11 06:10:39 tesla systemd[1698218]: Starting rclone-up.service - Run rclone up for b2...
</span><span style="color:#323232;">Apr 11 06:12:18 tesla systemd[1698218]: rclone-up.service: Main process exited, code=killed, status=15/TERM
</span><span style="color:#323232;">Apr 11 06:12:18 tesla systemd[1698218]: rclone-up.service: Failed with result 'signal'.
</span><span style="color:#323232;">Apr 11 06:12:18 tesla systemd[1698218]: Stopped rclone-up.service - Run rclone up for b2.
</span><span style="color:#323232;">Apr 11 06:12:18 tesla systemd[1698218]: rclone-up.service: Consumed 12.811s CPU time.
</span>

Also, here’s the log created by rclone while running:


<span style="color:#323232;">2024/04/11 06:10:42 INFO  : integrity.2376: Copied (new)
</span><span style="color:#323232;">2024/04/11 06:10:43 INFO  : hints.2376: Copied (new)
</span><span style="color:#323232;">2024/04/11 06:10:43 INFO  : nonce: Copied (replaced existing)
</span><span style="color:#323232;">2024/04/11 06:10:47 INFO  : config: Updated modification time in destination
</span><span style="color:#323232;">2024/04/11 06:10:55 INFO  : index.2376: Copied (new)
</span><span style="color:#323232;">2024/04/11 06:11:40 INFO  :
</span><span style="color:#323232;">Transferred:      443.104 MiB / 2.361 GiB, 18%, 16.475 MiB/s, ETA 1m59s
</span><span style="color:#323232;">Checks:              1503 / 1503, 100%
</span><span style="color:#323232;">Transferred:            4 / 19, 21%
</span><span style="color:#323232;">Elapsed time:       1m0.8s
</span><span style="color:#323232;">Transferring:
</span><span style="color:#323232;"> *                                   data/2/2328: 19% /502.259Mi, 2.904Mi/s, 2m19s
</span><span style="color:#323232;"> *                                   data/2/2329: 52% /500.732Mi, 10.758Mi/s, 22s
</span><span style="color:#323232;"> *                                   data/2/2330: 14% /501.598Mi, 3.150Mi/s, 2m15s
</span><span style="color:#323232;"> *                                   data/2/2331:  0% /500.090Mi, 0/s, -
</span><span style="color:#323232;">
</span><span style="color:#323232;">2024/04/11 06:12:18 INFO  : Signal received: terminated
</span>

Where should I look to get some more information about what’s going on? Why would the service be terminated like that?

LE:

Setting TimeoutSec=infinity inside the [Service] section of the unit file seems to help. Not 100% if it’s a good idea, but I’ll experiment with it.

rollingflower, (edited )

Here is my template


<span style="color:#323232;">sudo cat > /etc/systemd/user/rsync-backup.service <<EOF
</span><span style="color:#323232;">[Unit]
</span><span style="color:#323232;">Description=do rsync backups with some conditions
</span><span style="color:#323232;"># After=network-online.target
</span><span style="color:#323232;">
</span><span style="color:#323232;">[Service]
</span><span style="color:#323232;">Type=oneshot
</span><span style="color:#323232;"># require a power connection (optional)
</span><span style="color:#323232;"># ExecStartPre=sh -c '[ $(cat /sys/class/power_supply/AC/online) = 1 ]'
</span><span style="color:#323232;">
</span><span style="color:#323232;"># require battery over 40%
</span><span style="color:#323232;"># ExecStartPre=sh -c '[ $(cat /sys/class/power_supply/BAT0/capacity) -ge 40 ]'
</span><span style="color:#323232;">
</span><span style="color:#323232;"># require the connected network to NOT be "metered"
</span><span style="color:#323232;"># ExecStartPre=sh -c '! $(nmcli -t -f GENERAL.METERED dev show | grep -q 'yes')'
</span><span style="color:#323232;">
</span><span style="color:#323232;">ExecStart=/home/user/.local/bin/rsync-backup
</span><span style="color:#323232;"># you might add everything you need
</span><span style="color:#323232;"># ExecStart=/path/to/something/else
</span><span style="color:#323232;">
</span><span style="color:#323232;"># delete old logs (disabled for testing)
</span><span style="color:#323232;"># ExecStartPost=rm -f /var/log/rsync-backups.log
</span><span style="color:#323232;"># log the updates
</span><span style="color:#323232;"># ExecStartPost=sh -c 'echo "Last backup: $(date)" > /var/log/rsync-backup.log'
</span><span style="color:#323232;"># write errors to log
</span><span style="color:#323232;">StandardError=file:/var/log/rsync-backups.log
</span><span style="color:#323232;">
</span><span style="color:#323232;"># GUI message
</span><span style="color:#323232;">#ExecStartPost=/usr/bin/notify-send -t 0 -a "Backup" "rsync backup finished." "$(output of some command if you want infos about the backup)"
</span><span style="color:#323232;">
</span><span style="color:#323232;"># run with low priority, when idling
</span><span style="color:#323232;"># Nice=15
</span><span style="color:#323232;">IOSchedulingClass=idle
</span><span style="color:#323232;">
</span><span style="color:#323232;"># when conditions were not met, try again after 15 minutes
</span><span style="color:#323232;"># Restart=on-failure
</span><span style="color:#323232;"># RestartSec=900
</span><span style="color:#323232;">
</span><span style="color:#323232;">[Install]
</span><span style="color:#323232;">WantedBy=multi-user.target
</span><span style="color:#323232;">EOF
</span>

Timer file:


<span style="color:#323232;">sudo cat > /etc/systemd/user/rsync-backup.timer <<EOF
</span><span style="color:#323232;">[Unit]
</span><span style="color:#323232;">Description=do rsync backups with some conditions
</span><span style="color:#323232;">
</span><span style="color:#323232;">[Timer]
</span><span style="color:#323232;">OnCalendar=daily
</span><span style="color:#323232;">Persistent=true
</span><span style="color:#323232;">EOF
</span>

(I think the unit is needed)

That is a slightly modified variant of my automatic rpm-ostree system updates which took an hour or so with the help of ChatGPT and a lot of testing around.

Systemd services are lit.

If you add a “repeat when conditions are not met” you need another timer to start it. Like 2 loops, one big loop to start the process, and one small loop to keep trying until conditions are met. I do that with my system updates to prevent them being done

  • with low battery (or even using an AC requirement)
  • over a metered network
  • when the system is busy
clmbmb,

Thanks for the suggestion. I’ll look into this too.

clmbmb,

I can’t see anywhere that a [Timer] section can be added to a service unit. I tried it and it doesn’t work:

Apr 11 13:16:15 computer systemd[2593025]: /home/clmbmb/.config/systemd/user/test.service:10: Unknown section ‘Timer’. Ignoring.

The service/script ran as it should, but the Timer section is ignored, as you can see.

rollingflower,

Okay good notice. You should put that in a deparate .timer file then

sir_pronoun,

Is it on the horizon for cron to be deprecated?

clmbmb,

I don’t think so, but I wanted to try systemd timers.

sir_pronoun,

Sure, not trying to second guess you. What do you think of them so far? Is it worth the effort to learn them?

clmbmb,

Yes, for sure. Timers are more versatile than what cron jobs can do.

orcrist,

Nothing wrong with learning new tricks, but it’s worth mentioning on the side that sometimes a cron job is the right tool.

Magickmaster,

I think I read something about there being timeouts for slow jobs, which backups definitely are. Really unsure though, it has been a while. Try spawning independent worker programs!

clmbmb,

I just read a bit some internet posts and the documentation. I set TimeoutSec=infinity inside the service and I set the timer to run it now. I’ll see if that helps.

somethingsomethingidk,

Can you share your service file?

clmbmb, 

<span style="color:#323232;">[Unit]
</span><span style="color:#323232;">Description=Run rclone up for b2
</span><span style="color:#323232;">
</span><span style="color:#323232;">[Service]
</span><span style="color:#323232;">Type=oneshot
</span><span style="color:#323232;">ExecStart=/zet/Users/radu/backup/rclone_up.sh
</span><span style="color:#323232;">
</span><span style="color:#323232;">[Install]
</span><span style="color:#323232;">WantedBy=rclone-up.timer
</span>
Oinks, (edited )
@Oinks@lemmy.blahaj.zone avatar

Does your script fork at some point (and might exit before the rsync job is completed)? Because then you need to use Type=forking instead of simple or oneshot, otherwise systemd will start trying to clean up child processes when the script exits.

Edit: Actually considering the time span involved Type=forking will not solve your issue because it will timeout, if this is the problem you need to change your script to not do that.

clmbmb,

No, my script doesn’t fork and I don’t think rclone does that either.

Here’s the script (pretty simple):


<span style="color:#323232;">#!/bin/bash
</span><span style="color:#323232;">
</span><span style="color:#323232;">repos=(fotorepo multirepo persorepo appconfigs)
</span><span style="color:#323232;">
</span><span style="color:#323232;">if pidof -x rclone >/dev/null; then
</span><span style="color:#323232;">  echo "Process already running"
</span><span style="color:#323232;">  exit 1
</span><span style="color:#323232;">fi
</span><span style="color:#323232;">
</span><span style="color:#323232;">for repo in "${repos[@]}"; do
</span><span style="color:#323232;">    inidate=$(date +'%Y%m%d_%H%M%S')
</span><span style="color:#323232;">    /usr/bin/rclone -v --log-file=/backup/borg/logs/${repo}_b2sync_${inidate}.log sync /backup/borg/${repo} b2:${repo}
</span><span style="color:#323232;">    if [[ $? -eq 0 ]]; then
</span><span style="color:#323232;">      MSGDATE=$(date +'%d/%m/%y %T')
</span><span style="color:#323232;">      mesaj="[${MSGDATE}] Upload for ${repo} was successful."
</span><span style="color:#323232;">      curl -H "Title: B2 Upload" -H "Tags: arrow_double_up" -d "${mesaj}" "${URL}"
</span><span style="color:#323232;">      #sendmsg "[${MSGDATE}] Upload for <b>${repo}</b> was <b><u>successful</u></b>."
</span><span style="color:#323232;">    else
</span><span style="color:#323232;">      MSGDATE=$(date +'%d/%m/%y %T')
</span><span style="color:#323232;">      mesaj="[${MSGDATE}] Upload for ${repo} has failed. Check the logs."
</span><span style="color:#323232;">      curl -H "Title: B2 Upload" -H "Tags: warning" -H "prio:high" -d "${mesaj}" "${URL}"
</span><span style="color:#323232;">      #sendmsg "[${MSGDATE}] Upload for <b>${repo}</b> has <b><u>failed</u></b>. Check the logs."
</span><span style="color:#323232;">    fi
</span><span style="color:#323232;">    enddate=$(date +'%Y%m%d_%H%M%S')
</span><span style="color:#323232;">    mv /backup/borg/logs/${repo}_b2sync_${inidate}.log /backup/borg/logs/${repo}_b2sync_${enddate}.log
</span><span style="color:#323232;">done
</span>
Oinks, (edited )
@Oinks@lemmy.blahaj.zone avatar

Indeed, that all looks fairly innocuous. Just in case, you are sure that you didn’t just accidentially kill or killall rclone or bash?

Perhaps wrapping the script in might help debug where the offending signal is coming from.

clmbmb,

Just in case, you are sure that you didn’t just accidentially kill or killall rclone or bash?

No. The process runs at night. Only if my dog started learning Linux and tested something! That makes me wonder…

somethingsomethingidk,

You could try setting TimeoutStopSec=“infinity” for the service. There may be a default timeout for services and its killing rclone before it can finish because the oneshot type is considered “starting” until the program exits.

Oinks,
@Oinks@lemmy.blahaj.zone avatar

TimeoutStopSec applies to the ExecStop command, TimeoutStartSec would be the culprit here. I’m not sure why there would be a default timeout of specifically 1:39 minutes though.

somethingsomethingidk,

Yeah you’re right, it’s time for bed lol

The default start timeout is disabled by default for oneshot.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • linux@lemmy.ml
  • ethstaker
  • DreamBathrooms
  • cubers
  • mdbf
  • everett
  • magazineikmin
  • Durango
  • Youngstown
  • rosin
  • slotface
  • modclub
  • kavyap
  • GTA5RPClips
  • ngwrru68w68
  • JUstTest
  • thenastyranch
  • cisconetworking
  • khanakhh
  • osvaldo12
  • InstantRegret
  • normalnudes
  • Leos
  • tacticalgear
  • tester
  • megavids
  • anitta
  • provamag3
  • lostlight
  • All magazines
    •