max_tokens <= 100 always returns finish_reason="length" regardless of actual completion

[ShayanShamsi]

Used the following test script to check the context window but fails at even ~1000 tokens.

import argparse
from openai import OpenAI

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--api_key", required=True)
    parser.add_argument("--base_url", required=True)
    args = parser.parse_args()

    client = OpenAI(base_url=args.base_url, api_key=args.api_key)
    
    # Test usually safe lengths
    token_counts = [1000, 4000, 8000, 16000, 32000]
    
    print(f"Testing endpoint: {args.base_url}")
    
    for tokens in token_counts:
        print(f"\nTesting prompt size: ~{tokens} tokens...", end=" ", flush=True)
        
        # Create a dummy prompt of roughly 'tokens' length (1 token ~= 4 chars)
        prompt = "test " * tokens
        
        try:
            response = client.chat.completions.create(
                model="zai-org/GLM-5-FP8",
                messages=[{"role": "user", "content": prompt}],
                max_tokens=10,
                temperature=0
            )
            
            content = response.choices[0].message.content
            reason = response.choices[0].finish_reason
            
            if content is None or reason == "length":
                print("❌ FAILED (Context Limit Hit)")
                print(f"   -> The endpoint cannot handle {tokens} tokens.")
                break
            else:
                print("✅ SUCCESS")
                
        except Exception as e:
            print(f"❌ ERROR: {e}")
            break

if __name__ == "__main__":
    main()

uv run python test_limit.py \
  --api_key "<token>" \
  --base_url "https://api.us-west-2.modal.direct/v1"
Testing endpoint: https://api.us-west-2.modal.direct/v1

Testing prompt size: ~1000 tokens... ❌ FAILED (Context Limit Hit)
   -> The endpoint cannot handle 1000 tokens.

[ShayanShamsi]

Did some more testing (with MiniMax M2.5 in OpenCode):

Description

When calling the Modal endpoint https://api.us-west-2.modal.direct/v1 with max_tokens set to 100 or lower, the API always returns finish_reason="length" even when the model generates far fewer tokens than requested.

Steps to Reproduce

curl -X POST "https://api.us-west-2.modal.direct/v1/chat/completions"
-H "Content-Type: application/json"
-H "Authorization: Bearer "
-d '{"model": "zai-org/GLM-5-FP8", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 10}'
Expected Behavior

• max_tokens=10 should return up to 10 tokens or until natural stop
• finish_reason="length" should only appear when max_tokens is actually reached

Actual Behavior

• max_tokens <= 100: Always returns finish_reason="length" regardless of completion length
• max_tokens >= 200: Works correctly

---

Evidence

Test 1: Varying max_tokens values

for mt in 1 2 5 10 20 50 100 200; do
curl -s -X POST "https://api.us-west-2.modal.direct/v1/chat/completions"
-H "Content-Type: application/json"
-H "Authorization: Bearer "
-d "{"model": "zai-org/GLM-5-FP8", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": $mt}"
done | python3 -c "import sys,json; [print(f'max_tokens={i+1}:', json.loads(line)['choices'][0]['finish_reason']) for i,line in enumerate(sys.stdin)]"

Output:

max_tokens=1: length
max_tokens=2: length
max_tokens=5: length
max_tokens=10: length
max_tokens=20: length
max_tokens=50: length
max_tokens=100: length
max_tokens=200: stop

---

Test 2: Full response with max_tokens=1

curl -s -X POST "https://api.us-west-2.modal.direct/v1/chat/completions"
-H "Content-Type: application/json"
-H "Authorization: Bearer "
-d '{"model": "zai-org/GLM-5-FP8", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 1}'

Output:
{
choices: [{
finish_reason: length,
message: {
content: null,
reasoning_content: Let
}
}],
usage: {
completion_tokens: 1,
prompt_tokens: 6,
total_tokens: 7
}
}

Note: completion_tokens=1 equals max_tokens=1, but finish_reason="length" is incorrect since the model naturally stopped after generating just 1 token (the word "Let").

---

Test 3: Without max_tokens parameter (works correctly)

curl -s -X POST "https://api.us-west-2.modal.direct/v1/chat/completions"
-H "Content-Type: application/json"
-H "Authorization: Bearer "
-d '{"model": "zai-org/GLM-5-FP8", "messages": [{"role": "user", "content": "Hello"}]}'

Output:

{
choices: [{
finish_reason: stop,
message: {
content: Hello! I'm the GLM language model trained by Z.ai. How can I assist you today?
}
}],
usage: {
completion_tokens: 189,
prompt_tokens: 6,
total_tokens: 195
}
}

Works correctly with finish_reason="stop".