TRL is a convenient library to train large language model (LLM) using reinforcement learning (RL). However, it is still too new, the interface is not well-developed yet. rollout_func is a low-level interface to write your own rollout for RL and environment_factory is a high-level interface to train your model with external environemnt, however, how it parse the model output for tool use is hard coded and not available for every model (check add_response_schema in trl/trainer/grpo_trainer.py and trl/chat_template_utils.py)
TRL-ENV addresses the middle-level with a very simple environment interface
type Action = str
type Delta = str
type Seed = str
class Env(Protocol):
reward: float
alive: bool
def reset(self, seed: Seed) -> tuple[Env, Delta]: ...
def step(self, action: Action) -> tuple[Env, Delta]: ...It is similar to tool call if not the same. Note that, rollout_func is an experimental feature of TRL, this library is subject to break at anytime
It is important to note that, tokenizer is additive if each chunk starts and ends with special tokens, in the case of qwen3 each chunk starts with <|im_start|> and <|im_end|>
tok(a ++ b) = tok(a) ++ tok(b)
where a and b are texts and ++ is concatenation. We call it essentially additive (not actually additive)
transformers despite after 8 years of development (as of 2026) is still not stable. For example, not all models has Tokenizer.parse_response which should be a basic function that must be implemented from the beginning. TRL-ENV requires Tokenizer.parse_response to be existed by Processor interface
Language = str
class Processor(Protocol):
def init_system_input(self, prompt: Language) -> str: ...
def append_user_input(self, prompt: Language) -> str: ...
def parse_agent_output(self, completion: Language) -> tuple[str, str]: ...TRL-ENV provides a very simple example for training agentic LLM. See experiment/examples
uv pip install vllm==0.12.0 flash-attn==2.8.3 --torch-backend=cu126 --no-build-isolationgit clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install -r requirements/cpu.txt --index-strategy unsafe-best-match
uv pip install -e .