[Bug] extract_edges hard-codes max_tokens=16384, overriding the model's real budget; truncated JSON then surfaces as a messageless Exception

[drmm]

Summary

extract_edges hard-codes a 16,384 output-token cap and passes it as an explicit max_tokens=, which overrides the model's real output budget. On a model graphiti itself records as supporting 65,536 output tokens, edge extraction is therefore capped at a quarter of capacity. When a document produces more edge JSON than that, the reply is truncated mid-string, json.loads fails, and — on the Gemini path — the error surfaces as a bare Exception with no message.

Related but not the same: #811 asks for the constant to be hoisted. This issue is about the cap causing silent, hard-to-diagnose ingest failure, and about the messageless exception that hides it. #763 is the mirror-image problem (config max_tokens ignored) on the OpenAI clients.

Version: graphiti-core 0.30.2 (eaa4128), Python 3.12, Windows 10, Kuzu store, GeminiClient(model='gemini-2.5-flash', thinking_config=ThinkingConfig(thinking_budget=0)).

1. The cap overrides the model

graphiti_core/utils/maintenance/edge_operations.py:141

extract_edges_max_tokens = 16384

passed at edge_operations.py:205:

llm_response = await llm_client.generate_response(
    prompt_library.extract_edges.edge(context),
    response_model=ExtractedEdges,
    max_tokens=extract_edges_max_tokens,   # <-- explicit, so it wins
    ...
)

GeminiClient._resolve_max_tokens (graphiti_core/llm_client/gemini_client.py:175-201) documents its own precedence, and an explicit parameter is rule 1:

1. Explicit max_tokens parameter passed to generate_response()
2. Instance max_tokens set during client initialization
3. ...

So the per-model table three dozen lines above it is never consulted for this call — even though it says (gemini_client.py:51-57):

GEMINI_MODEL_MAX_TOKENS = {
    'gemini-2.5-pro': 65536,
    'gemini-2.5-flash': 65536,
    'gemini-2.5-flash-lite': 64000,
    ...

There is no configuration knob: setting LLMConfig(max_tokens=...) does not help, because the call site passes its own value.

2. What that looks like at runtime

Ingesting a corpus of 81 markdown documents (~1 MB total), a single larger document blows the cap:

ERROR 🦀 LLM generation failed parsing as JSON, will try to salvage.
ERROR Raw output (truncated): {"edges": [{"source_entity_name": "ORCHESTRATOR", "target_entity_name": ...
ERROR Error in generating LLM response: Failed to parse structured response:
      Unterminated string starting at: line 1 column 43665 (char 43664)
ERROR 🦀 LLM generation failed and retries are exhausted.
ERROR Max retries (2) exceeded. Last error:

43,665 characters of dense JSON is almost exactly 16,384 tokens — the reply is being cut at the cap, not by the model. salvage_json is attempted and returns None for these. Retrying twice cannot help: the same prompt regenerates the same over-long answer.

Note thinking_budget=0 is already set, so this is not #1868 (thinking tokens consuming the output budget) — this is the raw cap.

3. The error arrives with no message

graphiti_core/llm_client/gemini_client.py:361:

        except Exception as e:
            ...
            logger.error(f'Error in generating LLM response: {e}')
            raise Exception from e

raise Exception from e constructs an argument-less Exception. Anything that catches and formats it — a bulk runner, an MCP server, a user's except Exception as exc: log(exc) — gets an empty string:

[13-24/81] FAIL  209.9s  Exception:

The cause survives only because logger.error on the line above happens to print it. raise (bare, preserving the original) or raise Exception(str(e)) from e would cost nothing. The same pattern appears at :344 where the message is preserved (raise Exception(f'Failed to parse structured response: {e}') from e), so this looks like an oversight rather than intent.

4. Blast radius in add_episode_bulk

One oversized document fails the entire batch. In the run above, chunk [13-24/81] — twelve documents — was lost to a single file that overflowed, because add_episode_bulk (graphiti.py:1290) has no per-episode error isolation. With the recommended chunking from its own docstring (graphiti.py:1345-1348), larger chunks mean a bigger blast radius from one bad document.

Suggested fixes

1. Make the edge-extraction cap respect the model. Either drop the explicit max_tokens= at edge_operations.py:205 and let _resolve_max_tokens consult GEMINI_MODEL_MAX_TOKENS, or make the constant a floor rather than a ceiling: max_tokens=max(extract_edges_max_tokens, model_max). Hoisting it (refactor: hoist EXTRACT_EDGES_MAX_TOKENS constant #811) would also make it overridable.
2. gemini_client.py:361 — preserve the message: bare raise, or raise Exception(str(e)) from e.
3. Optional, higher value: isolate per-episode failures in add_episode_bulk so one unparseable document does not discard its whole chunk.

Happy to open a PR for (1) and (2) if that's useful.

[drmm]

Concrete blast-radius data for point 4, from finishing the run.

Twelve documents were marked failed by a single add_episode_bulk chunk. Only one of them was actually over the cap — orchestrator/references/cli-programs.md, 30,627 bytes. The other eleven were simply its chunk-mates, and they include:

bytes	file
30,627	orchestrator/references/cli-programs.md ← the only real failure
19,459	orchestrator/SKILL.md
13,528	orchestrator/plan.md
…	…
289	orchestrator/state/README.md

A 289-byte document failed to ingest because a 30 KB document shared its batch. Re-running those eleven with chunk=1 ingests them without incident.

So the cost of one over-cap document is not one document — it is the whole chunk, and the docstring at graphiti.py:1345-1348 actively encourages larger chunks for throughput. The two recommendations interact badly: chunk larger for speed, lose more per failure.

This also makes the failure hard to attribute. The runner sees one messageless Exception for twelve files (point 3) and no indication which document caused it; the only way to find out is a bisect at chunk=1.

Isolating per-episode failures inside add_episode_bulk — collect the exception against its episode and carry on, returning it in AddBulkEpisodeResults — would contain the damage to the document that actually overflowed, independently of whether the cap in point 1 is changed.

[Zsanz3]

I'd like to take a minimal fix for the hard-coded max_tokens=16384 in extract_edges.

Approach: Remove (or replace with a floor relative to the model's resolved max via _resolve_max_tokens) the hard cap at the call sites in edge_operations.py, so Gemini clients can use their real model budget. Also keep the original exception message when raising in gemini_client.py instead of dropping it. Leaving add_episode_bulk as a follow-up to keep the PR small.

Keeping this separate from #1868. Feedback welcome before I open the PR.

[drmm]

Two further data points from working around this on an 81-document corpus.

custom_extraction_instructions is a partial workaround. Since the cap can't be raised from configuration, the only lever left is asking the model to emit less. Passing consolidating instructions through add_episode_bulk(..., custom_extraction_instructions=...) — "consolidate related facts into a single broad edge rather than one per mention, keep facts under 25 words, completeness of the JSON matters more than exhaustiveness" — recovered 7 of 12 documents that had failed on every prior attempt.

That is worth knowing, but it is a workaround with a sharp edge: it trades extraction quality for the ability to finish at all, and the user has no way to know which documents silently got a thinner extraction.

The cut-off is entity density, not document size. This surprised me and it makes the failure hard to predict:

bytes	file	result
19,459	orchestrator/SKILL.md	succeeds with consolidation — 104 entities, 89 facts
16,441	tasker/resources/supervision.md	still fails
30,627	orchestrator/references/cli-programs.md	still fails
131,438	historical-manual.md	still fails
330,808	tasker/resources/architecture.md	still fails

A smaller file fails while a larger one succeeds, because a dense reference document yields far more edge JSON per byte than a longer prose document. So there is no file-size threshold a caller could check in advance to know whether ingest will work — which is what makes the messageless Exception (point 3) especially costly: you cannot predict the failure and you cannot diagnose it after the fact.

Final coverage on this corpus was 76/81. The five that remain are all above 15 KB and all failed on this cap.