Scenario 2

The Prometheus API was healthy. The request was wrong.

This is a good example of an “observability failure” that was actually just a bad API contract on the client side.

Symptombad_data

CauseMissing query

HealthyPrometheus API

LessonValidate the call

Symptom

A validation error looked like a platform issue

Calling /api/v1/query without a PromQL expression returned a parsing error. That can look alarming until you realize the endpoint was reachable and simply rejecting an incomplete request.

{
  "status": "error",
  "errorType": "bad_data",
  "error": "invalid parameter \"query\""
}

Diagnosis

The API contract requires a real query

Prometheus was fine. The failure was only that api/v1/query expects a query=... parameter. Once the request shape was corrected, the platform validated normally.

Signal The API answered with structured JSON

This was an application-level validation response, not a timeout.

Reading Prometheus was healthy

A live error payload proves the endpoint was up and parsing the request.

Decision Fix the request shape

Supply a real PromQL expression instead of debugging the cluster stack.

Broken request

curl http://127.0.0.1:9090/api/v1/query

Working validation

curl 'http://127.0.0.1:9090/api/v1/query?query=up'
curl http://127.0.0.1:9090/api/v1/targets