Programming

This bit me so many times I have

# USE ABSOLUTE PATHS, ASSUME ALL COMMANDS ARE UNDEFINED, SPECIFY ALL VERSIONS, REDIRECT OUTPUT

at the top of my local crontab so I remember from when I take something from it to prod, lol.

Edit: formatting whoopsie

You know I'd totally forgot about this but you're reminding me of my own insane debugging experiences over a decade who

hey, at least you caught it within a day.

EDIT : Next time make the cron job run when you're finished deploying it.

That's one reason why I like using systemd timers instead of cronjobs. You can easily test it by running the unit instead of setting the job to "in a minute".

Downside is that you don't easily get notification mails for any output.

cron isn't using your login shell. that's the trap.

but put PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin at the top of the crontab, then test the exact command once with env -i HOME="$HOME" PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /bin/sh -c '...', and you catch most of the "worked in my terminal" stuff before it turns into tomorrow's missing backup. MAILTO or a heartbeat catches the next failure.

Looks like you need to collect your crontab logs somewhere you can read.

I'd recommend self-hosting healthchecks and using runitor for your cronjobs. Runitor pings healthchecks when the cronjob starts, then pings it again on completion with the status (success or fail) along with the stdout and stderr.

Healthchecks can be configured to expect a ping periodically (once per day, once per hour, whatever) and alert you if it doesn't receive one.

For backups, Borgmatic has a healthchecks integration.