How to Deduplicate Grafana Alerts Before Posting to Slack

Diagnose which duplicate you have

Save a small set of sanitized webhook envelopes and Slack message timestamps. Determine whether they represent one event retried, one rule firing for multiple instances, a configured reminder, or two routes sending the same signal. Fix overlapping production routes before introducing another integration layer to hide the problem.

Grafana grouping settings (opens in a new tab) control which alerts travel together and the timing of initial, changed and repeated notifications. A repeated notification may be deliberate. Agree whether reminders should update the original message, add a thread reply or notify again; automatically hiding them can defeat the response policy.

Choose a message model

Choose one message per incident occurrence or one message per notification group. In the first model, a batch containing five alert instances may need five existing mappings. In the second, the message must display which members are firing and which recovered. A group is not healthy merely because one member resolved.

The webhook payload (opens in a new tab) includes alert fingerprints and a group key. A fingerprint describes labels; the group key describes grouping. Neither alone expresses every lifecycle boundary your incident model needs. Namespace identities by tenant and source, and retain occurrence information so a later recurrence does not inherit an old acknowledgement.

Avoid adding changing metric values, rendered summaries or webhook receipt timestamps to the identity. Doing so guarantees a new key on each delivery. If you intentionally change an identity label, record that as a new alert identity and account for the old one during reconciliation.

Persist before posting

A small bridge needs a durable inbox for accepted events, a canonical current-state record and a message mapping. Validate the sender using the configured authentication or signature mechanism, enforce input bounds and commit the event before confirming acceptance. Serialize processing by the chosen identity.

Post a new message only for a new occurrence. For an existing occurrence, compare the rendered state and update the mapped message if needed. Use the appropriate Slack message API and its supported permissions; chat.update (opens in a new tab) updates messages belonging to the authenticated sender, including its bot's own messages.

Handle uncertain delivery explicitly

The difficult case is a timeout after Slack accepted a new post but before the bridge stored the timestamp. Blindly posting again can create a duplicate. Keep the operation in an uncertain state and reconcile using available message metadata or an operator review before another creation attempt. Do not promise exactly-once delivery unless the complete protocol and persistence design provide it.

For rate limits, respect the server's retry instructions and keep pending work durable. Do not hold an incoming request open indefinitely while trying Slack. If the bridge cannot accept an event safely, return a failure according to the sender's documented retry behavior rather than acknowledging and discarding it.

Test complete alert occurrences

In a nonproduction channel, send one firing event twice, a changed value with the same identity, a recovery and a later firing episode. Restart the worker between two deliveries. Replay an old recovery after the later firing. Also test a batch with one recovered member and one still firing.

Expected outcomes: duplicates do not create fresh incident messages; meaningful changes appear; a new occurrence becomes visible; stale data cannot silently close it; and failures remain inspectable. Record behavior when messages are deleted, the bot loses access or the mapping store is unavailable.

Use a separate two-way state model if responders need acknowledgement buttons. Deduplicating chat messages does not implement incident permissions, escalation or recovery semantics by itself.