How to Monitor Cron Jobs That Fail Silently

Define what completion means

Write one sentence before adding monitoring: “Yesterday's export exists, contains the expected reporting date, and can be read by the downstream system before its deadline.” For a backup, distinguish a new archive from a verified restore. A daily success ping cannot establish restoreability by itself.

Create one check per independently actionable job. If ten jobs share one heartbeat URL, one successful job can conceal nine missing jobs.

Wire success after verification

Healthchecks' cron documentation (opens in a new tab) describes sending a request when a job completes. The following Bash wrapper adds an application-specific verification step. Replace both commands and load HC_URL from your scheduler's restricted environment; it is the complete secret ping URL. Do not paste it into a public issue.

#!/usr/bin/env bash
set -u
: "${HC_URL:?Set the complete success heartbeat URL}"

if /opt/jobs/export-report; then
  if /opt/jobs/verify-report; then
    if curl --fail --silent --show-error --max-time 10 \
      --retry 2 "$HC_URL" >/dev/null; then
      exit 0
    fi
    printf '%s\n' 'Job output verified, but heartbeat delivery failed' >&2
    exit 3
  fi
  printf '%s\n' 'Job finished, but output verification failed' >&2
  exit 2
fi
printf '%s\n' 'Job command failed' >&2
exit 1

The two /opt/jobs/ programs are placeholders you must implement. The verifier should check freshness and the expected reporting period, not merely whether a file left over from yesterday exists. Log a job identifier and run identifier with failures, without logging credentials or report contents.

This wrapper sends only success. The receiver's missed-deadline alert is therefore essential. A provider-specific failure endpoint can add earlier failure notification, but never replace the deadline. Preserve the original command's exit code too if another system depends on its exact value.

Set the expected schedule

Configure the receiver with the intended schedule, its time zone, and a grace period based on observed normal runtime plus delivery jitter. For a job with a business deadline, monitor that deadline rather than just “once per day.” Confirm how your scheduler handles daylight-saving changes and overlapping runs.

For example, a report due at 06:00 needs a late-report alert early enough for a person to recover it. A receiver that waits until tomorrow to call it missing is correctly configured for the wrong operational requirement.

Run the failure matrix

Use a test job and a separate test heartbeat URL.

Handle the confusing case

“The job succeeded but monitoring says it failed” can mean that heartbeat delivery failed. Check the job's verified output and the receiver's last accepted request separately. Retrying the business job may duplicate an invoice or export; retry notification independently when that is appropriate.

Keep concurrency control, timeout behavior and restore tests in the job's runbook. Monitoring a job does not make rerunning it safe.