How to Monitor Multiple AKS Clusters with Centralized Prometheus

Choose where collection and evaluation happen

Start with three separate decisions: where metrics are scraped, where they are stored and queried, and where alert rules run. A central dashboard does not require a single central scraper. Collectors inside each cluster can reach local targets and send authorized telemetry outward.

Azure Monitor's managed Prometheus overview (opens in a new tab) describes its in-cluster agent, data collection rules and Azure Monitor workspace. It also documents remote write from self-managed Prometheus. Choose the supported onboarding path for your environment and region rather than pasting a generic remote-write URL into an unconfigured cluster.

Document the destination workspace, collection identity, permitted endpoints and alert evaluator for each cluster. Keep customer or environment access boundaries explicit. A shared workspace and a cluster label do not, by themselves, guarantee that one team cannot query another team's metrics.

Assign a unique identity before sharing storage

Two clusters can have the same display name in different subscriptions or resource groups. Two private networks can also reuse the same pod or node addresses. Use an inventory that maps a stable telemetry identity to the full cluster resource identity and responsible team.

For example, use deliberately unique aliases such as teama_prod_westeurope_01 and teamb_prod_westeurope_01. Microsoft documents the managed collector's cluster alias setting (opens in a new tab), including normalization of unsupported characters and the need to keep rule onboarding consistent with the alias. Check the emitted value after normalization; different input names can otherwise converge on the same label.

For self-managed Prometheus, external labels (opens in a new tab) can identify a collector's outgoing telemetry. They do not overwrite a label already present on a series. Reserve your identity field and inspect actual remote samples instead of assuming the configuration won a collision.

Overlapping private networks need two separate tests

First test transport. From each cluster, verify DNS, the chosen TLS endpoint, authorization and the network policy or private connectivity required by the supported collection path. Outbound telemetry can remove the need for a central scraper to route directly to every private pod address. It does not bypass a firewall or create an authorized network path.

Then test data identity. Send a known demo metric from the same private address and job name in two isolated test clusters. Query centrally and confirm that the results remain distinguishable by your cluster identity. If they merge, adding a route or peering connection will not fix the labeling problem.

Do not expose unauthenticated metrics endpoints publicly merely to make central scraping convenient. Choose collection and access patterns that meet your environment's network requirements.

Prove coverage with an inventory comparison

Start with one metric that your selected collection profile actually exports. In a Prometheus-compatible query interface, a diagnostic expression such as the following can help inspect the series that exist:

count by (cluster) (up)

This is an inspection example, not a complete cluster-health alert. Missing up data may indicate a collection profile difference, a stopped collector or broken transport. Compare observed clusters with a separate expected-cluster inventory; a query cannot enumerate an absent cluster from samples it never received.

Keep each failure scoped to the demo workload. Record when data stopped, when the central view recognized the gap and when a person was notified.

Keep the outage response independent enough

Decide which critical failures still need local evaluation or another delivery path if central storage is unavailable. Test that decision using the central-outage monitoring procedure. Conversely, a local evaluator cannot notify through a failed network unless another route exists.

Finish the pilot with an inventory of expected clusters, current sample freshness and unresolved collection gaps. Expand in batches so an incorrect alias or authorization rule affects a small scope. Centralization is successful when responders can locate trustworthy evidence for each cluster and recognize when that evidence is missing.