Summary
I was testing the prom-label-proxy v0.6.0 + Grafana 9.1.8 and I noticed that when using HTTP POST method with the /api/v1/series endpoint, prom-label-proxy is passing two different value for the match[] parameter: one in the POST body and the other one in the URL query parameters.
This leads to incorrect results. For instance the thanos querier seems to read only the match[] value in the query parameter in this situation which is pretty much garbage.
For instance if we are doing:
curl 127.0.0.1:39090/api/v1/series -X POST -H 'X-Tenant-Id: foo' --data-urlencode "match[]=up"
Then the resulting HTTP query sent by prom-label-proxy is (in JSON format as recorded by gohrec and URL-decoded) will be:
{
"Body": "match[]={__name__=\"up\",tenant_id=\"foo\"}",
"URI": "/api/v1/series?match[]={tenant_id=\"foo\"}"
}Depending on its implementation, the upstream may only read the match[] value set in the URI (which is the case for the Thanos querier). Even if it is true that the upstream should only read the match[] value in the request body with Content-Type: application/x-www-form-urlencoded, this is still an issue in the way prom-label-proxy is writing the resulting query.
Full response recorded by gohrec
{
"Protocol": "HTTP/1.1",
"Headers": [
"Accept-Encoding: gzip",
"Accept: */*",
"Content-Length: 61",
"Content-Type: application/x-www-form-urlencoded",
"User-Agent: curl/7.58.0",
"X-Forwarded-For: 172.29.0.1",
"X-Tenant-Id: foo"
],
"ContentLength": 61,
"Body": "match[]={__name__=\"up\",tenant_id=\"foo\"}",
"Trailers": [],
"TransferEncodings": null,
"RemoteAddr": "172.29.0.2:39070",
"Host": "127.0.0.1:39090",
"Method": "POST",
"Path": "/api/v1/series",
"Query": [
"match[]: {tenant_id=\"foo\"}"
],
"URI": "/api/v1/series?match[]={tenant_id=\"foo\"}"
}
Steps to reproduce
version: '3'
services:
http-recorder:
image: frxyt/gohrec:latest
command: ["record", "--listen=:8080"]
volumes:
- ./logs:/gohrec/log
prom-label-proxy:
image: quay.io/prometheuscommunity/prom-label-proxy:v0.6.0
ports:
- 127.0.0.1:39090:39090
command:
- "-header-name=X-Tenant-Id"
- "-label=tenant_id"
- "-upstream=http://http-recorder:8080"
- "-insecure-listen-address=0.0.0.0:39090"
- "-enable-label-apis"curl 'http://127.0.0.1:39090/api/v1/series' -X POST -H 'X-Tenant-Id: foo' --data-urlencode "match[]=up" -o /dev/null -sS
- See recorded HTTP query in
./logs/[date]/
Expected behavior
The HTTP request sent to the upstream should be:
POST /api/v1/series HTTP/1.1
X-Tenant-Id: foo
Content-Type: application/x-www-form-urlencoded
match[]={__name__="up",tenant_id="foo"}
instead of
POST /api/v1/series?match%5B%5D=%7Btenant_id%3D%22foo%22%7D HTTP/1.1
X-Tenant-Id: foo
Content-Type: application/x-www-form-urlencoded
match[]={__name__="up",tenant_id="foo"}
Summary
I was testing the prom-label-proxy v0.6.0 + Grafana 9.1.8 and I noticed that when using HTTP POST method with the
/api/v1/seriesendpoint, prom-label-proxy is passing two different value for thematch[]parameter: one in the POST body and the other one in the URL query parameters.This leads to incorrect results. For instance the thanos querier seems to read only the
match[]value in the query parameter in this situation which is pretty much garbage.For instance if we are doing:
Then the resulting HTTP query sent by prom-label-proxy is (in JSON format as recorded by gohrec and URL-decoded) will be:
{ "Body": "match[]={__name__=\"up\",tenant_id=\"foo\"}", "URI": "/api/v1/series?match[]={tenant_id=\"foo\"}" }Depending on its implementation, the upstream may only read the
match[]value set in the URI (which is the case for the Thanos querier). Even if it is true that the upstream should only read thematch[]value in the request body withContent-Type: application/x-www-form-urlencoded, this is still an issue in the way prom-label-proxy is writing the resulting query.Full response recorded by gohrec
{ "Protocol": "HTTP/1.1", "Headers": [ "Accept-Encoding: gzip", "Accept: */*", "Content-Length: 61", "Content-Type: application/x-www-form-urlencoded", "User-Agent: curl/7.58.0", "X-Forwarded-For: 172.29.0.1", "X-Tenant-Id: foo" ], "ContentLength": 61, "Body": "match[]={__name__=\"up\",tenant_id=\"foo\"}", "Trailers": [], "TransferEncodings": null, "RemoteAddr": "172.29.0.2:39070", "Host": "127.0.0.1:39090", "Method": "POST", "Path": "/api/v1/series", "Query": [ "match[]: {tenant_id=\"foo\"}" ], "URI": "/api/v1/series?match[]={tenant_id=\"foo\"}" }Steps to reproduce
./logs/[date]/Expected behavior
The HTTP request sent to the upstream should be:
instead of