Updated message trim to handle message better and support tool calling.

[cableman]

See https://github.com/AarhusAI/aarhusai-docker/blob/main/guardrails/README.md for more information about the message trim guardrail.

New configuration knobs

• default_max_context_tokens (default 8192) and max_context_tokens_by_model map -- context-window size is now resolved per-model, falling back through: per-model map --> litellm.get_max_tokens --> global default. Removes the silent 8192 fallback footgun on fleets with mixed 8k/32k/128k models.
• pop_trailing_tool_messages (global) and pop_trailing_tool_messages_by_model -- opt-in for strict chat templates that reject tool-terminal conversations. Defaults to preserving trailing tool messages, since User --> Asst{tool_calls} --> Tool is the normal agent-loop shape.

New tool-calling repair logic

• _repair_tool_call_pairings -- after trim_messages truncates history, drops orphan role: tool messages and strips orphan tool_calls from assistant messages so vLLM doesn't reject the conversation.
• _ensure_last_is_user / _sanitize_messages -- guarantees the conversation doesn't end on an assistant message; appends {"role": "user", "content": "Please continue"} when needed. Sanitize is now run on every request, not just trimmed ones.

Behavior changes

• Removed dependency on litellm.litellm_core_utils.prompt_templates.common_utils.get_completion_messages and the ensure_alternating_roles path replaced by the in-house sanitize.
• Token recount now happens after sanitize (whether or not trimming occurred), and max_tokens is recalculated accordingly.
• Debug log line updated from max(512, … * 0.90) to max(256, … * 0.75) (note: this is only the log string -- the actual calculation lives in _calculate_safe_completion_tokens).

[lilosti]

Dette er et potentielt fiks til os2ai/Feedback#1.
Der er ændringer i Guardrail for at imødekomme brugen af tool calls.
Disse ændringer ændrer også på den generelle brug af guardrail, og forhåbentlig også det oprindelige issue.
Vi kan ikke genskabe det oprindelige issue på andet end Holstebros installation, og kan derfor ikke teste om det virker.

[SigneA-hm]

@lasseborly vil du lave code review? Vi kører den udenom den igangværende RC som et hot fix og ser om det kan være med til at løse Holstebros udfordring

[lasseborly]

I have a hard time wrapping my head around the actual logic and how it solves the apparent problems. Not that it does not, I am just very removed from the problem space.

[hypesystem]

I think I'm in doubt as to what the purpose of the message trimming guardrail is. Is there a reason we are trimming messages at all? It seems like trying to trim them is what causes the behavior users are reporting back as bad, but I am not sure I understand the motivation for having trimming in the first place?

[hypesystem]

@cableman We talked about adding a short comment with a link from here to some documentation. In the documentation we would like to document:

• Reasons for adding this guardrail: models that run out of context window.
• Principles in message trimming: how do we approach the problem, how do we weigh options.

[lilosti]

@hypesystem create a new issue regarding how to handle documentation in the project, it should not be decided on a case to case basis. For now read the documentation here: os2ai/documentation#6

[hypesystem]

@lilosti I'm not sure exactly what this means or what outcome you would like from it, but we're happy to improve general processes where it makes sense! Let's try and talk about it in person to land it at some point 😄

[hypesystem]

The code makes sense when considering the documentation 😄

I still think a one-liner on why this code exists would make sense, if not a link to external documentation.

I added a few small notes on validation and a single one on less significant logic. Nothing blocking, fine to merge with or without fixes.

[hypesystem]

A beaut, I declare! Good work 😄

[hypesystem]

Nice! ⭐