Guides
Each guide answers a decision you have to make, then hands you something you can use: a checklist, a table, a test, or a template.
Fundamentals
- Your First On-Call Shift: A Calm, Practical ChecklistWhat to set up before your first on-call shift, what to do in the first five minutes of a page, when to escalate, and how to recover afterwards.Updated 12 September 2026
- How to Choose Lightweight Log Monitoring for a Small SystemSeparate logs from metrics and uptime checks, estimate retention needs, and test a small log collection setup before adding a larger stack.Updated 11 September 2026
- How to Design an On-Call Schedule That Actually WorksDesign an on-call schedule with clear ownership, a sustainable rotation length, predictable handoffs and a way to find coverage gaps — including what to do when the team is too small.Updated 3 September 2026
- What Is On-Call? A Practical Guide for Engineering TeamsWhat on-call means, the seven parts of a working on-call system, and how a small team can set up a humane first rotation.Updated 3 September 2026
Alert quality
- Alert Fatigue: How to Reduce Noise Without Hiding RiskHow to spot alert fatigue, apply an actionability test to every alert rule, and cut paging volume without losing coverage of real risk.Updated 3 September 2026
- How to Add Alert and Recording Rules to a PromQL BackendCheck query compatibility, scheduled rule evaluation, recording output and alert delivery before adding a ruler to a PromQL-compatible backend.Updated 12 September 2026
- Alert on DNS Failure Without Depending on That DNSDetect a local DNS failure while keeping notification resolution independent, with explicit target queries, container DNS checks, and a controlled failure test.Updated 12 September 2026
- How to Alert on Missing Sensor Data Separately from ThresholdsDistinguish an abnormal sensor value from an absent, stale or failed measurement path using freshness rules, inventory and controlled sensor tests.Updated 12 September 2026
- How to Audit an On-Call Rotation with Eighty Pages a WeekClassify a busy on-call rotation by real actions, repeated incidents and lost time, then prioritize measured fixes without suppressing important coverage.Updated 12 September 2026
- How to Build a Service Health Summary That Handles Missing DataCount expected services once, distinguish healthy, degraded, unknown and maintenance states, and expose source freshness instead of counting dashboard panels.Updated 12 September 2026
- How to Build a Simple Threshold Alert Interface for OperatorsDesign an operator alert form around approved devices, units, thresholds and durations while keeping routing and rule lifecycle under control.Updated 12 September 2026
- Turn Webhook Logs into a State View That Can Go StaleBuild a current status view from webhook events with per-key deadlines, receiver timestamps, replay protection, and explicit stale or unknown states.Updated 12 September 2026
- How to Choose Direct SQL Alerts or a Prometheus ExporterCompare direct SQL Server alert queries with exporter-based metrics using query cost, freshness, reuse and a controlled database pilot.Updated 12 September 2026
- How to Consolidate MSP Alerts Across Monitoring ToolsNormalize customer monitoring events into an owned incident flow with tenant isolation, lifecycle tracking and connector health checks.Updated 12 September 2026
- Bridge Email-Only Alerts into Webhooks ReliablyBridge email-only alerts into reliable webhooks with authenticated ingestion, stable incident keys, durable processing, duplicate handling, and freshness monitoring.Updated 12 September 2026
- Correlate Uptime Checks Across Two LocationsCorrelate uptime observations across locations with fresh time windows, independent network paths, regional incident states, and explicit handling of missing probes.Updated 12 September 2026
- Investigate 401 Responses That Only Affect Your Uptime ProbeTrace HTTP 401 failures seen only by uptime probes using request equivalence, authentication challenges, redirects, and edge or origin logs before changing policy.Updated 12 September 2026
- Debug HTTP Checks Behind Nginx Without Falling Back to TCPDebug Uptime Kuma HTTP failures behind Nginx by comparing the actual URL, DNS, SNI, redirects, authentication, and application response from the same network.Updated 12 September 2026
- How to Decide Between a Page and a TicketClassify alerts by response deadline, impact and available action, then move nonurgent signals into an owned ticket queue without losing coverage.Updated 12 September 2026
- How to Deduplicate Grafana Alerts Before Posting to SlackDistinguish Grafana grouping from Slack message deduplication, persist incident-to-message mappings and test retries, recovery and new alert occurrences.Updated 12 September 2026
- Detect Backups Skipped by Daylight Saving TimeCatch backup runs skipped at daylight-saving transitions using explicit scheduler time zones, last-success age, catch-up policy, and controlled clock tests.Updated 12 September 2026
- Catch Blank HTTP 200 Pages with Content and Browser ChecksDetect empty HTTP 200 pages with response assertions and browser tests that check useful content, JavaScript failures, and successful heartbeat delivery.Updated 12 September 2026
- Diagnose Disappearing Monitors Before Rebuilding the DatabaseInvestigate disappearing Uptime Kuma monitors by separating UI and socket failures from persisted data loss, preserving backups, and testing changes on a clone.Updated 12 September 2026
- Diagnose Timeouts Across Every Uptime Monitor at OnceInvestigate simultaneous Uptime Kuma timeouts by comparing independent probes, database latency, scheduler delay, container limits, and safe version changes.Updated 12 September 2026
- Find Devices That Have Silently Lost Backup CoverageFind silent backup coverage gaps by reconciling managed devices, assigned policies, recent recovery points, and restore evidence instead of failed jobs alone.Updated 12 September 2026
- How to Find Unused Prometheus Metrics Without Breaking AlertsCombine query history, dashboards, recording rules and alert dependencies before dropping Prometheus metrics, with a canary change and explicit recovery limits.Updated 12 September 2026
- How to Interpret Linux Load Average Before Creating an AlertRead Linux load average alongside CPU capacity, utilization, pressure and user impact so a fixed number does not become a misleading incident threshold.Updated 12 September 2026
- How to Keep Kubernetes Alerts Working During a Central Monitoring OutageSeparate central metric storage from local alert evaluation, label clusters consistently, and test remote-write and notification failure paths.Updated 11 September 2026
- How to Limit Prometheus Metric Cardinality at the SourcePrevent runaway Prometheus series by bounding labels before instrumentation, measuring series growth and validating that cardinality controls preserve useful totals.Updated 12 September 2026
- Load-Test an Uptime Platform Before Adding 12,000 TargetsPlan an uptime monitoring load test for 12,000 services using check rate, timeout concurrency, retention, discovery, and measurable detection delay.Updated 12 September 2026
- Measure Remote-Site Latency with Push ChecksMeasure latency and packet loss from remote sites, transmit bounded observations, and distinguish degraded connectivity from missing or stale reports.Updated 12 September 2026
- How to Monitor AI Workflow Quality Beyond UptimeMeasure AI task completion and grounded output alongside availability, latency and cost using representative evaluation cases and explicit unknown results.Updated 12 September 2026
- How to Monitor Cron Jobs That Fail SilentlyCatch jobs that never start, fail, or finish without valid output. Add success heartbeats, explicit deadlines, and a practical failure-test matrix.Updated 11 September 2026
- How to Monitor Intermittently Connected Sites Without False AlarmsDistinguish expected offline windows from missing contact and local service failure, with per-site deadlines and tests for delayed event replay.Updated 12 September 2026
- How to Get Alerts When Your Internet or Monitoring Server Goes DownDetect a failed internet connection or monitoring host with an independent heartbeat and an external website check, then test each failure separately.Updated 11 September 2026
- Detect IPv6 Outages Hidden by Healthy IPv4 ChecksCatch IPv6 outages hidden by IPv4 fallback with separate forced-family HTTP checks, IPv6-capable probe locations, and explicit partial-availability status.Updated 12 September 2026
- Make MongoDB and MySQL Backup Failures VisibleMake MongoDB and MySQL cron backups observable with preserved exit codes, per-database freshness, consistency requirements, artifact checks, and isolated restores.Updated 12 September 2026
- How to Monitor Multiple AKS Clusters with Centralized PrometheusDesign a central AKS metrics view with unique cluster identities, outbound collection, overlapping-network checks and explicit loss-of-telemetry alerts.Updated 12 September 2026
- How to Monitor Private Networks Without Opening Inbound PortsMonitor cameras, NVRs and internal services with outbound health heartbeats. Separate device health from sender health and detect missing reports.Updated 11 September 2026
- Monitor Mail Acceptance Beyond an Open SMTP PortMonitor SMTP from protocol response through test-mailbox delivery, separating port availability, mail acceptance, delayed delivery, and independent alerting.Updated 12 September 2026
- How to Monitor SSL Certificate Expiration Before It Breaks AccessInventory service certificates, check expiry and trust separately, and verify the certificate actually served after renewal, including private PKI boundaries.Updated 11 September 2026
- How to Monitor Traditional Infrastructure Alongside KubernetesCombine server, network and Kubernetes monitoring around service ownership while preserving the discovery and protocol needs of each environment.Updated 12 September 2026
- How to Monitor Website Uptime Independently of Your Hosting ProviderMeasure public website availability with external HTTP and content checks. Record sampling limits and test errors, timeouts and incorrect content.Updated 11 September 2026
- How to Plan Prometheus Architecture for Multiple ClustersPlan collection, local alerting, central queries and retention from measured series volume and failure requirements before adding Kubernetes clusters.Updated 12 September 2026
- Publish Upcoming Maintenance Before It StartsPublish upcoming maintenance with a shared event record for customer notices and alert suppression, including time zones, cancellations, and delayed starts.Updated 12 September 2026
- Reconciling Alert State After the Monitoring Network RecoversAvoid replaying stale down and up notifications after the monitoring network returns by reconciling fresh observations, current incidents, and queued history.Updated 12 September 2026
- How to Reduce False-Positive Alerts Without Hiding OutagesClassify noisy alerts, tune persistence and grouping, and test sustained faults and missing data before changing production paging rules.Updated 11 September 2026
- Stop Uptime Checks from Flooding Your DNS LogsMeasure excessive DNS traffic from uptime monitors, distinguish client queries from recursive work, and reduce load without hiding DNS or address changes.Updated 12 September 2026
- How to Retire Stale Alerts Without Losing CoverageAudit unused alert rules against service inventory, dependencies and historical incidents, then remove them with a reversible coverage check.Updated 12 September 2026
- Route Certificate Expiry and Availability Alerts to Different TeamsSeparate certificate expiry from HTTP availability so infrastructure and network teams receive the right alert without muting certificate or probe failures.Updated 12 September 2026
- Keep Monitoring Email Working with Modern AuthenticationKeep monitoring email compatible with modern authentication by checking SMTP client capabilities, provider policy, OAuth relay design, and credential expiry tests.Updated 12 September 2026
- How to Separate Production and Non-Production AlertsKeep development and QA signals out of production paging with trusted environment labels, explicit routes and negative delivery tests.Updated 12 September 2026
- How to Set Different Retention Periods for SMART and Operational MetricsKeep long-term disk-health history without retaining every high-frequency metric forever, using explicit storage boundaries, aggregation choices and freshness checks.Updated 12 September 2026
- How to Suppress Downstream Alerts During a WAN OutageUse explicit dependency labels and scoped Alertmanager inhibition to reduce unreachable-device noise while keeping independent failures visible.Updated 12 September 2026
- How to Design Two-Way Alert State Sync with SlackSeparate Grafana signal state from human acknowledgement, then design Slack actions with authorization, durable processing, reconciliation and loop prevention.Updated 12 September 2026
- Test DNS Answers, Not Just DNS Server AvailabilityVerify that DNS returns the expected blocked and allowed answers, accounting for Pi-hole blocking modes, address families, resolver identity, and stale checks.Updated 12 September 2026
- A Successful Backup Job Can Still Produce an Unrestorable DatabaseValidate PostgreSQL backups with supported consistency methods and isolated restores that check data, roles, extensions, and recovery time beyond a zero exit code.Updated 12 September 2026
- How to Test Prometheus Alert Rules Before Removing the Old RuleValidate Prometheus rule behavior, confirm the loaded rule through the API, and test notification routing before retiring an existing alert.Updated 12 September 2026
- Test Whether Failed Uptime Emails Are RetriedTest whether Uptime Kuma email failures are retried, queued, or lost using a controlled SMTP relay and evidence across reconnects and process restarts.Updated 12 September 2026
- How to Turn Vendor Status API Responses into Consistent AlertsConvert polled status snapshots into durable incident transitions with stable keys, stale-data handling and recovery tests.Updated 12 September 2026
- How Grafana, Prometheus and Alertmanager Share Alerting WorkTrace collection, query, rule evaluation, routing and delivery across a Grafana stack, including what continues when a browser or server is unavailable.Updated 12 September 2026
- How to Use PrometheusRule Resources with Another Rule EngineBridge Kubernetes PrometheusRule resources to an alternative evaluator with explicit selectors, conversion ownership and create-update-delete tests.Updated 12 September 2026
- Should Uptime Alerts Depend on Home Assistant?Decide whether to route uptime alerts through Home Assistant by testing shared dependencies, notification targets, maintenance behavior, and fallback delivery.Updated 12 September 2026
- Uptime Kuma Shows Down but the Container Is Running: How to Diagnose ItCheck the URL from Uptime Kuma’s network, then isolate Docker DNS, ports, TLS and HTTP response problems without hiding a real outage.Updated 11 September 2026
Response practice
- Escalation Policies: How to Build a Reliable Fallback PathHow to design an escalation policy with sensible timeouts, independent fallbacks, and a worst-case pass time that fits your response objective.Updated 3 September 2026
- How to Add Logs, Deployment Context and Runbooks to AlertsGive responders the service, time window, deployment and evidence links they need, then test whether another engineer can investigate from the alert.Updated 12 September 2026
- How to Assign an Incident Commander in a Small TeamDecide when a small incident needs a coordinator, divide technical work and communication, and rehearse explicit command transfer without adding unnecessary roles.Updated 12 September 2026
- How to Automate Deployment Health Checks and Reduce Night WatchesReplace one manual release-watch step with a bounded health gate, clear stop conditions and a tested rollback, then measure night-time intervention.Updated 12 September 2026
- How to Build an Incident Timeline While You RespondCapture observations, decisions and rejected hypotheses during an incident so the handoff and postmortem preserve evidence instead of relying on memory.Updated 12 September 2026
- How to Find Hidden Manual Dependencies in OperationsFind recurring work hidden in personal scripts and manual portal checks, then define ownership, repeat safety, completion checks and failure notification.Updated 12 September 2026
- How to Fix Missed On-Call Notifications on iPhone and AndroidTrace a missed page from incident creation to routing, provider delivery and phone settings, then test fallback channels and escalation.Updated 12 September 2026
- How to Give an AI Assistant Useful Production Incident ContextPrepare a bounded, redacted evidence packet with service version, timeline and read-only links so an assistant can investigate without inventing production facts.Updated 12 September 2026
- How to Limit AI-Assisted Incident Investigation CostsSet per-task token, tool, time and spend limits for AI investigation, with reserved budget, cancellation and a useful human handoff when limits are reached.Updated 12 September 2026
- How to Route Alerts to the Right Service OwnerBuild a service-to-owner routing map, handle unknown labels explicitly, and test escalation without adding more overlapping rules.Updated 12 September 2026
- How to Separate Incident Mitigation from Root Cause AnalysisRestore service with controlled mitigation, preserve evidence, and investigate deployment triggers and failed defenses in a separate follow-up workflow.Updated 12 September 2026
- How to Use the First Five Minutes After an Incident AcknowledgementGive responders a practical starting sequence for impact, ownership, evidence and safe mitigation, with an explicit handoff when another team must act.Updated 12 September 2026
- How to Summarize Incident Telemetry Before Sending It to an AI AgentFilter incident telemetry by service and time, retain representative failures and counterexamples, and attach evidence references and sampling limits.Updated 12 September 2026
- How to Write and Track Postmortem Actions That Can ShipTurn postmortem findings into scoped work with owners, capacity and completion evidence, then make unresolved risk visible in the existing team backlog.Updated 12 September 2026
- How to Write an Incident Handoff That Survives the PostmortemKeep one evidence-based incident timeline with ownership, impact, decisions and recovery so the technical team does not have to reconstruct it later.Updated 12 September 2026
- On-Call Handoffs: Transfer Risk, Not Just the PagerHow to run an on-call handoff that moves current risk, temporary mitigations and upcoming events to the incoming responder, not just the rota entry.Updated 3 September 2026
- How to Write an Incident Runbook a Tired Responder Can UseA structure for incident runbooks that works at 3am: trigger, first five minutes, safety boundaries, mitigation with rollback, and how to keep them true.Updated 12 September 2026
- How to Test an On-Call Paging Path End to EndA controlled procedure for verifying that a page reaches the right person on every channel, escalates on time, and leaves evidence behind.Updated 3 September 2026
People and policy
- How to Handle Leave and Fairness in an On-Call RotationPlan on-call leave without coverage gaps, compare night and weekend burden, and verify overrides without treating equal shift counts as equal workload.Updated 12 September 2026
- How to Reduce DevOps Interruptions Without Losing Support RequestsCreate one request intake, separate emergencies, rotate triage and measure repeated work so a small platform team can protect project time.Updated 12 September 2026
- How to Replace Dashboard Vigilance with Actionable AlertsTurn the conditions engineers watch on dashboards into tested signals, explicit response expectations and independent delivery checks before ending manual watch.Updated 12 September 2026
- How to Set On-Call Response Times and Backup ExpectationsDefine acknowledgement, investigation and recovery separately, including what happens when the on-call engineer is away from a laptop.Updated 12 September 2026
- How to Test On-Call Schedule Changes, Overrides and DeparturesCompare effective coverage before and after an on-call schedule edit, including time zones, daylight saving, leave and team departures.Updated 11 September 2026
- On-Call Compensation Models for Engineering TeamsThe common shapes of on-call pay — allowance, standby, active response, minimum callout, premium periods and recovery time — and how to choose between them.Updated 3 September 2026
Migration and evaluation
- Grafana OnCall OSS Alternatives After ArchivalWhat archival changed, what still works in a running self-hosted install, and how to choose between replacing the notification channels and moving on.Updated 3 September 2026
- How to Choose On-Call Software for a Small TeamCompare on-call tools using a requirements worksheet, total-cost formula and real delivery tests, including what Slack integration must actually do.Updated 12 September 2026
- Hosted vs Self-Hosted On-Call: How to Compare the Real WorkCompare hosted and self-hosted on-call tools using a responsibility map, maintenance budget and a recovery exercise that includes the paging system itself.Updated 12 September 2026
- How to Evaluate an In-House On-Call Tool Before Replacing ItAssess a homegrown paging system with failure tests, maintenance ownership and a build-versus-buy worksheet instead of assuming a replacement is better.Updated 11 September 2026
- How to Evaluate SSO for a Lightweight On-Call ToolTest OIDC sign-in, authorization, account removal and emergency access in an on-call tool without mistaking a login button for a complete identity lifecycle.Updated 12 September 2026
- How to Evaluate Slack Incident Management for a Small TeamTest Slack incident workflows for a 15–20-person team, including responder permissions, acknowledgement, coordination and a working fallback when chat fails.Updated 12 September 2026
- How to Migrate Datadog Dashboards to Grafana Without Losing Query MeaningCompare critical Datadog and Grafana panels by metric identity, aggregation, units and fixed time windows before switching responders to the new dashboards.Updated 12 September 2026
- How to Move from Elasticsearch to Loki Without Losing Incident QueriesRedesign log labels and query patterns for Loki, preserve useful structured fields and validate common incident investigations before retiring Elasticsearch.Updated 12 September 2026
- How to Migrate from PagerDuty Without Losing Alert CoveragePlan a PagerDuty exit with a service inventory, shadow receiver, pilot acceptance tests and rollback before your renewal deadline.Updated 12 September 2026
- How to Migrate Opsgenie Terraform Without Losing Alert CoverageMap Opsgenie Terraform resources to destination behavior, inspect isolated plans and reconcile users, integrations, maintenance and schedule coverage before cutover.Updated 12 September 2026
- How to Prepare for a PagerDuty Renewal and Compare Contract OptionsPrepare an on-call renewal with usage evidence, a notice-date calendar, monthly-versus-annual scenarios and a tested migration fallback.Updated 12 September 2026
- How to Test an Opsgenie Replacement with Your DevelopersRun an Opsgenie replacement pilot covering direct alerts, Slack actions, shift changes, mobile delivery and developer permissions before committing to migration.Updated 12 September 2026
- How to Test Composite Alert Parity During a MigrationVerify combined alert conditions across engines with truth tables, grouping checks, missing-data cases and observed notification timing.Updated 12 September 2026