How to Design Two-Way Alert State Sync with Slack

Define what two-way synchronization means

Before selecting an integration, write down which object each system owns. A Grafana alert instance, a notification group, a human incident and a Slack message are related objects, not interchangeable IDs. A button labelled “resolve” becomes dangerous when nobody agrees which of those objects it changes.

Grafana documents rule state and health (opens in a new tab) as results of evaluation. Design human acknowledgement as a separate field in the incident system. If a responder closes an incident while the source still fires, show both facts and apply an explicit reopening policy. Do not claim that changing a Slack message resets a failing query.

Give each field one authority

Keep the source tenant, alert identity, incident occurrence, channel and message timestamp in persistent storage. Store observation and receipt times separately. If the source provides a trustworthy version or sequence, use it to reject stale transitions. If it does not, fetch current state or retain uncertainty; arrival order alone does not establish event order.

Make Slack commands safe to retry

Verify Slack requests (opens in a new tab) against the raw body, signature and timestamp before processing them. Map the Slack user to an authorized incident-system identity and verify access to that incident. Channel membership or knowledge of an incident ID is not sufficient authorization.

Persist an accepted command in a durable queue, then return the required response promptly. Slack's interaction documentation (opens in a new tab) requires an HTTP acknowledgement within three seconds. That response means the integration received the interaction; it must not be presented as a completed incident acknowledgement when the downstream API is still unavailable.

Use a stable command identity to make retry processing idempotent. Show “request pending” while necessary, and replace it with confirmed success or an actionable failure. Log the actor, intended transition, incident identity and outcome without logging tokens or private incident content unnecessarily.

Prevent echoes and lost updates

A Slack command can change the incident service, whose webhook then returns to the integration. Treat that callback as confirmation of the state, not a new command to replay. Tag origin and causation IDs in your own event envelope, and render only when the canonical view changes.

Use atomic updates or a per-incident work queue when commands race. If two responders acknowledge simultaneously, the API's actual result decides ownership. If a delayed recovery callback arrives after a new occurrence began, match it to the old occurrence rather than closing the new incident.

A Slack bot can update messages it posted (opens in a new tab). Store the channel and timestamp returned by the original message creation. A plain incoming webhook plus no persistent identity is not enough to implement this design. Treat API rate limits, revoked permissions and an unavailable channel as visible delivery failures.

Exercise the failure matrix

Use a test workspace and a nonpaging integration. Check duplicate callbacks, simultaneous acknowledgement, revoked user access, a worker crash after enqueue, an incident API timeout, recovery followed by refiring and a stale recovery delivered last. Restart the integration during the exercise to prove the mapping survives process memory loss.

After an outage, compare current incident and signal state with the stored mapping, then repair the Slack view. Preserve a timestamp and a stale indicator until reconciliation succeeds. Keep a direct incident link and an independent urgent notification path; Slack is one interface, not proof that the paging channel works.

This is an integration architecture and acceptance checklist. It does not assert that MonoDuty or any other provider implements all these operations. For the simpler repeated-message problem, use the Grafana-to-Slack deduplication procedure.