Skip to content

feat: implement truncate_prompt endpoint#501

Open
AlexFoxalt wants to merge 6 commits into
developmentfrom
feat/28-support-truncate_prompt-endpoint
Open

feat: implement truncate_prompt endpoint#501
AlexFoxalt wants to merge 6 commits into
developmentfrom
feat/28-support-truncate_prompt-endpoint

Conversation

@AlexFoxalt

Copy link
Copy Markdown
Collaborator

fixes #28

Summary

Adds the DIAL truncate_prompt deployment sub-endpoint (POST /openai/deployments/{deployment_id}/truncate_prompt), the dry-run/planning counterpart to inline chat_completion truncation. Given a batch of chat-completion requests each with a max_prompt_tokens budget, it returns the indices of messages that would be discarded to fit — without invoking model generation (only token counting).

Implementation

  • Modeled on the existing tokenize endpoint (hand-rolled FastAPI route reusing the SDK's TruncatePromptRequest/TruncatePromptResponse models).
  • Reuses the app's existing truncation algorithms so dry-run results match live truncation:
    • GPT (GPT4O/GPT4O_MINI/GPT_GENERIC) → multi_modal_truncate_prompt with tiktoken.
    • vLLM (VLLM/QWEN3_ASR_VLLM) → coarse truncate_prompt with VllmTokenizer.
    • Other chat models (RESPONSES/MISTRAL/DATABRICKS/COMPLETIONS) → coarse truncate_prompt via the reused tokenize tokenizer builder.
    • Image/video/audio/anthropic deployments → 404.
  • Per-input error isolation: a bad input yields a TruncatePromptError in its output slot without failing the batch; framework-level errors propagate through the existing exception handler.

Test plan

  • New tests/unit_tests/test_truncate_prompt_endpoint.py: fits ([]), over-budget (system + last retained), missing max_prompt_tokens isolation, 404 for unsupported deployment, vLLM with mocked upstream tokenizer, 422 on invalid input shape.

@AlexFoxalt
AlexFoxalt requested a review from adubovik as a code owner July 16, 2026 07:47
@AlexFoxalt AlexFoxalt linked an issue Jul 16, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support truncate_prompt endpoint

1 participant