Skip to main content

How to Find Hidden Manual Dependencies in Operations

GuideWritten by oncall.fyi editorialPublication approved by Burak YApproval recorded 12 September 2026
Sources and verification
Source dates
Oldest source check: 12 September 2026.
Technical verification
Separate technical verification has not been recorded.

Publication approval and technical verification are recorded separately. Automated link checks establish reachability, not accuracy. A checked example verifies only its stated test cases, not the whole article or your production setup.

In short

Inventory recurring operational outcomes and ask what actually triggers each one. Verify the scheduler, credentials, execution owner and completion evidence instead of assuming a script means automation. Move a selected task into a shared, repeat-safe workflow with a measurable result and an owned failure route. Test a missed run and an unsuccessful run before depending on it.

Key takeaways

  • A script is not an automated workflow unless its trigger and outcome are reliable.
  • Measure the business result, not only the process exit code.
  • Shared ownership includes credentials, overlap control and recovery instructions.

Audit outcomes rather than repositories

Start with recurring results the business expects: yesterday's import arrived, certificates renewed, a partner file was delivered, or an expired account lost access. Ask when each result last completed and who can show the evidence.

Then trace backwards. Was a scheduler responsible, did someone remember a calendar event, or does a laptop script run when one engineer is online? A script stored in a shared repository can still depend on a single person's login session and knowledge.

Google's discussion of toil (opens in a new tab) helps frame recurring operational work. Your inventory should identify the specific hidden dependencies before deciding what to automate.

Write an execution contract

FieldQuestion to answer
TriggerWhat starts the work, and in which timezone?
IdentityWhich service account and permissions does it use?
InputWhat must exist before execution is valid?
OutcomeWhich external result proves success?
OverlapWhat happens when a previous run is still active?
RecoveryWho can retry or repair a partial result?

Store this with the task's owner and next review date. Confirm the credentials can be rotated without contacting the original author. Record how a replacement responder gains access through the normal access process.

Distinguish execution from completion

A successful process exit does not establish that a partner received the right file. Add a result check that matches the actual contract: expected row count, output age, accepted receipt or a checksum verified by the receiving system.

Keep start time, completion time and outcome separate. An implementation can publish a timestamp of the last verified success using a monitoring endpoint or supported metrics collector. The Node exporter textfile collector (opens in a new tab) is one option for host-level batch metrics; choose it only when it fits how the job runs and is monitored.

Do not include order identifiers or customer names as metric labels. Keep detailed evidence in the appropriate protected system and put a stable job name in the metric.

Automate one recoverable task first

Choose a task with a clear input, bounded duration and a safe retry story. Move its trigger to the team's scheduler, set a maximum runtime and prevent concurrent runs where overlap could duplicate work. A lock must also have a recovery procedure; a permanently stale lock can hide the same missed-run problem.

Make partial completion explicit. If the task can write half an output before failing, use a staging result and a deliberate finalization step where possible. If its side effects cannot safely be repeated, require the operator to reconcile state before retrying.

Run the absence and failure exercises

Test a skipped schedule, missing input, expired credential, timeout and a run that reports success but produces an invalid result. Confirm the responsible person sees enough context to distinguish these cases. The cron monitoring guide covers the missing-completion path in more detail.

Have a second team member follow the recovery instructions using their own access. They should be able to identify the last complete outcome and explain whether retrying is safe. If the original author must interpret an undocumented file or portal, capture that dependency before declaring the task handed over.

Keep the manual procedure available during the pilot, with a rule preventing it from running concurrently with automation. Retire it only after the shared workflow has passed a complete schedule and recovery exercise.

Did this help?

Your answer helps us improve this guide. We save only the page and your choice for 30 days.

No name, email, or incident details are requested.

Sources

Vendor facts change. Each source below shows the date this page last checked it.

  1. Eliminating toil Google SRE. Checked 12 September 2026.
  2. Node exporter Prometheus. Checked 12 September 2026.

Related

One practical idea, occasionally

The On-Call Brief: short field notes, templates, and operational lessons.

Follow the field guide

Email subscriptions are not open yet. Read new guides in your feed reader — no email address needed.

Subscribe with RSS