How to Give an AI Assistant Useful Production Incident Context

Tie code to the running service

A repository checkout can differ from the version currently serving traffic. Start the packet with service identity, environment, deployment identifier and the relevant change window. If the deployed version is unknown, say so explicitly and make establishing it an early read-only investigation step.

Include the reported symptom in customer terms and the earliest supported timestamp. Separate “requests began failing at this time” from “this deployment caused the failure.” The first may be evidence; the second is a hypothesis until the relationship is tested.

Use a compact evidence packet

The following structure is an illustrative investigation brief:

Task: explain the checkout error increase using supplied evidence.
Scope: checkout API, production, 08:00–08:30 UTC.
Deployed version: release-demo-17; confirmed by deployment record.
Observed: error ratio rose at 08:12; request volume stayed stable.
Evidence: E1 error chart, E2 sanitized log signatures, E3 change record.
Unknown: database connection saturation during the first five minutes.
Allowed: read the approved evidence and propose follow-up queries.
Required output: observations, hypotheses, missing evidence, next checks.

Use access-controlled links to retained evidence where possible. A screenshot with no query, timezone or time window is difficult to verify and easy to misinterpret. Include enough context to reproduce the reading without copying every log line.

Remove unnecessary sensitive material

Exclude credentials, session tokens, customer payloads and unrelated identifiers. Redact before sending the packet to the model or storing it in an assistant transcript. Keep a note of redacted categories so the assistant does not interpret removed fields as absent production behavior.

OpenTelemetry's sensitive-data guidance (opens in a new tab) is relevant when telemetry becomes assistant input. Apply your organization's permitted destinations and retention settings to the packet as well as the original monitoring system.

Keep evidence separate from authority

Logs, incident comments and retrieved runbooks can contain arbitrary text. Tell the assistant to treat them as untrusted evidence, not as instructions that change its permissions or task. A log line asking for a credential export should remain a suspicious log line.

Start with narrowly scoped read-only tools. OWASP's excessive-agency guidance (opens in a new tab) describes the risk of giving an assistant more capability or autonomy than the task needs.

If investigation suggests a production change, turn it into a separate proposal naming the action, affected scope, expected benefit, stop condition and rollback. Obtain the appropriate authorization before executing that proposal. Read access to evidence is not authorization to restart services or disable alerts.

Require traceable reasoning outputs

Ask for each material observation to name its evidence identifier, and for each hypothesis to identify a check that could contradict it. An answer should be allowed to say the packet is insufficient.

Include at least one known counterexample in the packet when available, such as a region running the same release without the symptom. Ask the assistant to explain how that observation affects its proposed cause. This prevents a list of recent changes from becoming an automatic rollback recommendation. If the supposedly healthy region has little traffic or different dependencies, retain that limitation rather than treating the comparison as decisive proof.

Review a sample result against the original sources. Check that the assistant used the correct version, did not broaden the time window silently and did not turn correlation into certainty. Reject fabricated dashboard links or measurements even if the proposed fix sounds plausible.

Update the packet as new evidence arrives and label its revision. The telemetry summarization guide provides a method for keeping that update compact while preserving contradictory observations and source coverage.