Scenario 5

A 404 dashboard panel was empty even though the query looked reasonable.

This one shows why “no data” is not automatically a Grafana or Prometheus failure.

SymptomEmpty panel

CauseNo traffic + strict filter

FixGenerate traffic

Best labeljob regex

What looked wrong

The panel was blank

A query targeting 404 activity returned no data, which initially looked like missing metrics or a broken dashboard.

Request volume                  No data
Successful vs error requests   No data
HikariCP connections           No data
JDBC active connections        No data

What was actually happening

The query window had nothing to show

No recent 404 traffic had been generated yet, and the original filter was too narrow for the metric labels. Once traffic was generated and the query was widened to the right application label, the panel started returning data.

Signal The dashboard rendered but returned empty series

The panel itself loaded correctly; only the chart data stayed blank.

Reading The filter was too strict for recent traffic

No 404 traffic had been generated yet and the query scope was too narrow.

Decision Generate requests and widen labels

Use an application-level regex label and create real traffic before judging the panel.

Generate real traffic

kubectl -n payment-exception-review-local port-forward svc/payment-exception-review-service 8080:80
curl http://localhost:8080/actuator/health
curl http://localhost:8080/api/payment-exceptions/payexc-100045/bob

Working query

sum(
  rate(http_server_requests_seconds_count{
    job=~".*payment-exception-review.*",
    status="404"
  }[5m])
)