Support path prefix

[oncipriani]

We could not get prom-label-proxy to work with our external Mimir installation. All queries returned 404.

We believe the error is caused because Mimir exposes its Prometheus API under /prometheus and prom-label-proxy does not seem to support paths in its upstream.

We tried setting --upstream to https://mimir.our.org.domain/prometheus and https://mimir.our.org.domain. Accessing http://prom-label-proxy:8080/prometheus/api/v1/query?query=up and http://prom-label-proxy:8080/api/v1/query?query=up returns 404 in all cases.

Accessing Mimir directly via https://mimir.our.org.domain/prometheus/api/v1/query?query=up returns the metrics, as expected.

Also, adding some kind of debug logs would really help. #336

[yuvaldekel]

I would like to work on this

[simonpasquier]

We tried setting --upstream to https://mimir.our.org.domain/prometheus and https://mimir.our.org.domain. Accessing http://prom-label-proxy:8080/prometheus/api/v1/query?query=up and http://prom-label-proxy:8080/api/v1/query?query=up returns 404 in all cases.

Have you tried setting --upstream=https://mimir.our.org.domain/prometheus while querying http://prom-label-proxy:8080/prometheus/api/v1/query?query=up?

I've not attempted with Mimir but with a vanilla Prometheus, it worked:

$ ./prometheus  --web.route-prefix=/prometheus
time=2026-05-05T15:15:59.826+02:00 level=INFO source=main.go:1564 msg="updated GOGC" old=100 new=75
time=2026-05-05T15:15:59.827+02:00 level=INFO source=main.go:689 msg="Leaving GOMAXPROCS=22: CPU quota undefined" component=automaxprocs
time=2026-05-05T15:15:59.827+02:00 level=INFO source=memlimit.go:198 msg="GOMEMLIMIT is updated" component=automemlimit package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=60134130892 previous=9223372036854775807
...

$ ./prom-label-proxy -label job --upstream http://localhost:9090/prometheus --insecure-listen-address :8080
2026/05/05 15:16:26 Listening insecurely on [::]:8080

$ curl 'localhost:8080/api/v1/query?query=up&job=prometheus'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","cluster-id":"foo","instance":"localhost:9090","job":"prometheus"},"value":[1777986999.837,"0"]}]}}

[oncipriani]

Have you tried setting --upstream=https://mimir.our.org.domain/prometheus while querying http://prom-label-proxy:8080/prometheus/api/v1/query?query=up?

Yes. It returns 404:

$ podman run -it --rm -p 8080:8080 quay.io/prometheuscommunity/prom-label-proxy:v0.13.0 --upstream='https://mimir.our.org.domain/prometheus' --insecure-listen-address :8080 --label=cluster --label-value=eks-segsa-01
2026/05/05 16:52:16 Listening insecurely on [::]:8080

$ curl -Ssg 'http://localhost:8080/prometheus/api/v1/series?limit=1' --data-urlencode 'match[]=up{cluster="eks-segsa-01"}'
404 page not found

Querying Mimir directly returns the metrics:

$ curl -Ssg 'https://mimir.our.org.domain/prometheus/api/v1/series?limit=1' --data-urlencode 'match[]=up{cluster="eks-segsa-01"}' | jq
{
  "status": "success",
  "data": [
    {
      "__name__": "up",
      "app": "alloy",
      "availability_zone": "sa-east-1a",
      "cluster": "eks-segsa-01",
      "container": "alloy",
      "instance": "10.13.33.124:12345",
      "job": "integrations/alloy",
      "k8s_cluster_name": "eks-segsa-01",
      "namespace": "grafana-stack",
      "pod": "alloy-55b7c7c597-jw6r9",
      "region": "sa-east-1",
      "source": "kubernetes",
      "workload": "alloy",
      "workload_type": "replicaset"
    }
  ],
  "warnings": [
    "results truncated due to limit"
  ]
}

[simonpasquier]

ack there's something different between Prometheus and Mimir then. But before moving forward with #361 I'd rather add debug logs to understand exactly what happens.