Clarify key vs token generation for use with github actions

[eureka-cpu]

I was able to create a token easily enough and add it to my project, but I get the following error:

error: invalid value '***' for '--token <TOKEN>': Failed to validate JWT (JSON Web Token): ***                                                                              
For more information, try '--help'.  

I then went to look at the docs again, only to see that it says to use --key and says to check out the key docs for more info. The key docs do not have information on generating a key for use as a repository secret like the token docs do so it's a bit confusing.

[epompeii]

Hey @eureka-cpu thank you for using Bencher, and I'm sorry you're running into trouble.

Yes, the switch from --token to --key is relatively new and a bit mid-stream. I'm very sorry for the poor and confusing documentation. That is something that I'm actively working towards improving.

Going forward, I would recommend using --key. You can get a Project scoped key by going to your Project's Console page and selecting Keys then clicking the ➕ Add button: https://bencher.dev/console/projects/<your-project-slug>/keys/add

User scoped keys are also in the works: #874
And user tokens (JWTs) will soon be deprecated.

[eureka-cpu]

Thanks for the reply, I ended up getting it working shortly after. I think also it would be a nice touch to allow keys to not expire, it's just one less thing I have to worry about updating. For now I just set the ttl to 1B.

[kaimast]

Couldn't you print a warning or disallow --token? That would have helped a lot.

[epompeii]

@kaimast thanks for the report, though I want to flag that the version pin is most likely a coincidence rather than the actual fix.

The action doesn't couple the CLI version to the ref you pin. With the default version: latest it installs the version baked into the bundled openapi.json and downloads that exact release binary. Right now both bencherdev/bencher@main and bencherdev/bencher@v0.6.8 resolve to the same v0.6.8 binary, and --token has strictly required a JWT in v0.6.6, v0.6.7, and v0.6.8 alike. So swapping the ref can't make --token accept a non-JWT value.

The "Failed to validate JWT" error means something other than a user JWT was passed to --token / BENCHER_API_TOKEN: typically a bencher_* API key, or an empty/garbled secret (the *** is just GitHub masking it). The durable fix is to use --key / BENCHER_API_KEY with a bencher_user_* key instead of --token.

[epompeii]

Couldn't you print a warning or disallow --token? That would have helped a lot.

@kaimast I have a draft PR up to add more helpful CLI errors when the wrong credential type is used: #916
Please, let me know if this lines up with what you were thinking.