Make MongoDB and MySQL Backup Failures Visible

Build a minimal backup inventory

Start with one row per database: service owner, database engine and version, backup method, schedule and time zone, destination, maximum acceptable data age, and last restore result. Store a stable job identifier. A shared “all backups OK” ping lets one healthy database hide another job that was removed months ago.

Track the run's start, finish, process exit code, artifact identifier, and upload result. Start time explains a hung job; it does not advance the last successful backup time. Never treat yesterday's existing file as evidence that today's run produced output.

Choose consistency for the actual database

For MySQL 8.4, mysqldump --single-transaction (opens in a new tab) is useful for transactional tables such as InnoDB, but it does not make changing nontransactional tables consistent. Concurrent schema changes also need coordination. Inventory table engines and required routines, events, users, and replication context before selecting the dump options. A successful export of the wrong scope is still an incomplete recovery plan.

For MongoDB, the mongodump documentation (opens in a new tab) distinguishes topology and consistency requirements. A full replica-set dump with the applicable oplog option and matching replay procedure is different from dumping one database while writes continue. Do not apply a replica-set recipe to a sharded cluster or assume a successful command captured one consistent point across changing collections. Select a supported snapshot or coordinated backup procedure for the deployment.

Keep completion logic explicit

Use separate stages whose failures stop the run:

acquire job lock
record run_id and start time
create database-consistent backup in a unique temporary location
check command result and expected artifact structure
upload to the configured backup destination
verify the stored artifact can be read with recovery credentials
record completion and stored artifact identity
send this job's success heartbeat
release lock

These are implementation steps, not executable commands. If using a shell pipeline, ensure a failed dump cannot be concealed by a successful compression or upload command. Keep credentials out of command output and restrict backup files, which may contain the full application data set.

An upload check should verify the object from the destination rather than only trusting the upload client's exit message. A checksum detects altered bytes; it does not prove database consistency or that the restore operator still possesses the decryption key.

Detect absence outside the backup script

Configure an independent deadline (opens in a new tab) for each job. A disabled cron entry never reaches its own error handler. Use observed runtime and recovery needs to set the deadline, then distinguish “backup failed,” “backup missing,” and “backup succeeded but reporting failed.” Retry reporting separately from the backup when rerunning the database operation would be expensive.

Exercise the failure cases

In a test environment, use bad database credentials, a full temporary destination, a rejected upload, and a disabled scheduler. None should advance the last successful completion. Then restore a selected artifact into an isolated database with outbound integrations disabled and verify known records, indexes, and application reads.

Store restore date, backup identifier, validation scope, and elapsed recovery time in a separate field. Keep the completion check frequent and the restore exercise proportionate to recovery risk; never rename the former “verified restore” merely because the job exited zero.