Skip to content

[tinker] Return prompt and top-k prompt logprobs from sample requests - #1945

Open
pcmoritz wants to merge 1 commit into
mainfrom
tinker-prompt-logprobs
Open

[tinker] Return prompt and top-k prompt logprobs from sample requests#1945
pcmoritz wants to merge 1 commit into
mainfrom
tinker-prompt-logprobs

Conversation

@pcmoritz

Copy link
Copy Markdown
Collaborator

sample requests carrying prompt_logprobs / topk_prompt_logprobs came back with both fields unset, which the Tinker SDK surfaces as "Tinker response did not include prompt_logprobs".

Three gaps, all in the glue between the API layer and the backends:

  • api.py accepted topk_prompt_logprobs and dropped it when building types.SampleInput, so it never reached a backend. A positive top-k now also implies prompt_logprobs (both come off the same prompt forward pass).
  • The SkyRL-Train backend never put include_prompt_logprobs / topk_prompt_logprobs in the sample body, so vLLM was never asked for them, and then read the result as a list-of-lists when RemoteInferenceClient.sample returns flat per-prompt-token lists -- position 0 (always None) would have been returned as the whole field. Only the first sample of each request asks for prompt logprobs now, since all num_samples samples share one prompt.
  • types.SampleOutput.prompt_logprobs was typed list[float], which rejects the None at position 0.

The two forwarding clients in extra/ also gain prompt-logprob support via vLLM's /v1/completions, and the vLLM -> Tinker conversion is now shared (convert_vllm_prompt_logprobs) so the three paths cannot drift.

The JAX backend returns an explicit error for topk_prompt_logprobs: tx's generator only produces the prompt tokens' own logprobs, so there is no per-position distribution to build top-k from.

…uests

`sample` requests carrying `prompt_logprobs` / `topk_prompt_logprobs` came
back with both fields unset, which the Tinker SDK surfaces as "Tinker
response did not include prompt_logprobs".

Three gaps, all in the glue between the API layer and the backends:

- `api.py` accepted `topk_prompt_logprobs` and dropped it when building
  `types.SampleInput`, so it never reached a backend. A positive top-k now
  also implies `prompt_logprobs` (both come off the same prompt forward pass).
- The SkyRL-Train backend never put `include_prompt_logprobs` /
  `topk_prompt_logprobs` in the sample body, so vLLM was never asked for
  them, and then read the result as a list-of-lists when
  `RemoteInferenceClient.sample` returns flat per-prompt-token lists --
  position 0 (always None) would have been returned as the whole field.
  Only the first sample of each request asks for prompt logprobs now, since
  all `num_samples` samples share one prompt.
- `types.SampleOutput.prompt_logprobs` was typed `list[float]`, which
  rejects the None at position 0.

The two forwarding clients in `extra/` also gain prompt-logprob support via
vLLM's `/v1/completions`, and the vLLM -> Tinker conversion is now shared
(`convert_vllm_prompt_logprobs`) so the three paths cannot drift.

The JAX backend returns an explicit error for `topk_prompt_logprobs`: tx's
generator only produces the prompt tokens' own logprobs, so there is no
per-position distribution to build top-k from.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for requesting top-k prompt logprobs (topk_prompt_logprobs) alongside standard prompt logprobs. It adds a helper function convert_vllm_prompt_logprobs to standardize the conversion of vLLM's prompt logprobs into Tinker's response format. The SkyRL-Train backend is updated to forward and parse these top-k logprobs, while the JAX backend is updated to explicitly reject requests with topk_prompt_logprobs > 0 as it is unsupported. Unit tests have also been added to verify this new functionality. I have no additional feedback to provide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant