Scenario 3
Grafana could not validate a datasource pointed at localhost.
The real issue was context: localhost from the laptop shell is not localhost from inside the Grafana container.
Bad URL127.0.0.1:9090
Correct URLService DNS
RiskDuplicate datasource
FixReuse provisioned default
Symptom
Datasource validation kept failing
Testing Prometheus in Grafana with http://127.0.0.1:9090 did not work, even though the local port-forward was valid from the workstation shell.
Grafana datasource test failed
URL: http://127.0.0.1:9090
Error: connection refused
Diagnosis
Grafana lives inside the cluster
Inside Grafana, 127.0.0.1 points back to the Grafana container itself. The datasource had to use the in-cluster Kubernetes service name instead. The best fix was to keep the provisioned default datasource and avoid creating a duplicate one manually.
Signal
The datasource test failed inside Grafana
The URL worked from the laptop, but not from the dashboard container.
Reading
Loopback resolved to the wrong process
127.0.0.1 inside Grafana points back to Grafana itself.
Decision
Use Kubernetes service DNS
Keep the provisioned datasource and target Prometheus through in-cluster networking.
Good validation flow
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090
curl 'http://127.0.0.1:9090/api/v1/query?query=up'Correct datasource URL
http://kube-prometheus-stack-prometheus.monitoring:9090