gemini-heal

Resilience toolkit for the Google Gemini API. Rate limiting, MALFORMED_FUNCTION_CALL recovery, and circuit breaking — extracted from production.

---

We built this at Emotix while running Gemini inside a multi-agent research pipeline. Three problems kept surfacing:

MALFORMED_FUNCTION_CALL — Gemini's function calling silently breaks when tool arguments contain large strings. Google confirmed it as a P2 bug (still open). We spent days debugging it before finding the workaround.

429s with no backoff — The free tier is 10 RPM. Without a proper rate limiter the process would hang, queue up hundreds of requests, and eventually crash.

JSON wrapped in markdown — Even with responseMimeType: 'application/json', Gemini returns ```json ``` blocks. Small thing, but it breaks JSON.parse in production.

This library is our fixes, packaged up.

---

What's inside

Module	What it does
GeminiRateLimiter	Adaptive token-bucket — halves RPM on 429, recovers slowly, circuit-breaks when queue is too deep
GeminiClient	Text completion with rate limiter integration, cost tracking, and markdown stripping
ToolCaller	Forced tool calling with MALFORMED_FUNCTION_CALL retry + structured output fallback
stripMarkdownCodeBlock	Strips ```json ``` wrappers from Gemini responses
isRateLimitError / extractRetryAfterMs	429 detection helpers

---

Install

npm install gemini-heal @google/generative-ai

---

Usage

Rate limiter

import { GeminiRateLimiter } from 'gemini-heal';

const limiter = new GeminiRateLimiter({ rpm: 60 });

// Call before every Gemini request
await limiter.acquire();

// Call when you get a 429
limiter.reportRateLimit();

// Check before calling Gemini — returns true when queue wait exceeds threshold
if (limiter.shouldSkip()) {
  // route to a fallback model
}

// Observability
console.log(limiter.getStats());
// { currentRpm: 60, queueDepth: 0, total429s: 0, estimatedWaitMs: 0, ... }

// Always call destroy() on shutdown to clean up timers
limiter.destroy();

The limiter automatically halves RPM on each 429 (floor: 2 RPM) and recovers by +2 RPM every 60 seconds of clean traffic. No configuration needed for the happy path — it adapts.

---

Text completion

import { GeminiClient, GeminiRateLimiter } from 'gemini-heal';

const limiter = new GeminiRateLimiter({ rpm: 60 });

const client = new GeminiClient({
  apiKey: process.env.GOOGLE_AI_API_KEY!,
  model: 'gemini-2.0-flash',
  rateLimiter: limiter,
});

const result = await client.complete(
  'You are a helpful assistant.',
  'Summarize the current state of LLM tool calling.',
);

console.log(result.content);
console.log(`Tokens: ${result.totalTokens} | Cost: $${result.cost.toFixed(6)} | ${result.latencyMs}ms`);

---

Forced tool calling

This is the main reason we made this public.

FunctionCallingMode.ANY is supposed to guarantee a tool call. It doesn't. When arguments contain long strings (typically 1000+ characters), Gemini returns finishReason: MALFORMED_FUNCTION_CALL with no output. Two things fix it:

1. Prompt instruction — telling Gemini to properly JSON-escape string values cuts MALFORMED errors by ~90%. We verified this over 500+ production calls.
2. Structured output fallback — if function calling still fails after N retries, we switch to responseMimeType: 'application/json' + responseSchema. This bypasses the function-calling code path entirely. No MALFORMED possible.

import { ToolCaller, GeminiRateLimiter, ToolDefinition } from 'gemini-heal';

const tools: ToolDefinition[] = [
  {
    type: 'function',
    function: {
      name: 'analyze_market',
      description: 'Analyze a market segment and return structured insights.',
      parameters: {
        type: 'object',
        properties: {
          summary: { type: 'string' },
          opportunities: { type: 'array', items: { type: 'string' } },
          risks: { type: 'array', items: { type: 'string' } },
        },
        required: ['summary', 'opportunities', 'risks'],
      },
    },
  },
];

const limiter = new GeminiRateLimiter({ rpm: 60 });
const caller = new ToolCaller(process.env.GOOGLE_AI_API_KEY!, { rateLimiter: limiter });

const result = await caller.run(
  'You are a market research analyst.',
  'Analyze the AI coding assistant market in 2025.',
  tools,
  'analyze_market',
  { maxAttempts: 3 },
);

console.log(result.strategy); // 'function_calling' or 'structured_output'
console.log(result.attempt);  // which attempt succeeded (1-indexed)
console.log(result.args);     // the tool arguments, ready to use

The strategy field is useful for monitoring — if you're seeing structured_output too often, something in your tool schema is triggering the Gemini bug.

---

Tool definitions format

ToolCaller accepts tools in OpenAI's JSON Schema format. If you're already using OpenAI-format tools you can pass them directly — no conversion needed.

---

Bring your own logger

Both GeminiClient and ToolCaller accept a logger option compatible with pino, winston, or any { info, warn, error, debug } interface:

import pino from 'pino';

const caller = new ToolCaller(apiKey, { logger: pino() });

By default, logs go to console.log / console.warn / console.error.

---

API reference

GeminiRateLimiter

new GeminiRateLimiter(options?: RateLimiterOptions)

Option	Type	Default	Description
rpm	number	GEMINI_RPM_LIMIT env, or 10	Max requests per minute
maxAcceptableWaitMs	number	15000	shouldSkip() returns true above this

Method	Description
acquire()	Await a rate-limiter slot before calling Gemini
reportRateLimit(retryAfterMs?)	Tell the limiter a 429 was received
shouldSkip()	Circuit breaker — true when queue wait exceeds threshold
getEstimatedWaitMs()	Current queue depth in milliseconds
getStats()	Returns RateLimiterStats for observability
destroy()	Clean up internal timers

---

GeminiClient

new GeminiClient(options: GeminiClientOptions)

Option	Type	Default
apiKey	string	required
model	string	gemini-2.0-flash
temperature	number	0.7
maxTokens	number	8192
jsonMode	boolean	false
rateLimiter	GeminiRateLimiter	—
logger	Logger	console

Returns CompletionResult: { content, model, inputTokens, outputTokens, totalTokens, latencyMs, cost }.

---

ToolCaller

new ToolCaller(apiKey: string, options?: { logger?, rateLimiter? })

caller.run(systemPrompt, userMessage, tools, toolName, options?)

Run option	Type	Default
model	string	gemini-2.0-flash
temperature	number	0
maxAttempts	number	3
maxTokens	number	4096

Returns ToolCallResult: { toolName, args, attempt, strategy }.

---

Known Gemini issues this addresses

MALFORMED_FUNCTION_CALL Tracked in googleapis/python-genai#1120 and googleapis/google-cloud-java#11782. Gemini returns an error finish reason with no output when function call arguments contain large strings. The prompt-level fix in ToolCaller handles the common case; the structured output fallback handles everything else.

JSON in markdown code blocks responseMimeType: 'application/json' does not always prevent Gemini from wrapping output in ```json ```. stripMarkdownCodeBlock() is a one-liner fix you can use anywhere.

No built-in rate limit handling The @google/generative-ai SDK throws on 429 but provides no backoff. GeminiRateLimiter gives you a proactive token bucket so you never hit the limit in the first place, plus automatic recovery when you do.

---

Running tests

npm install
npm test

24 unit tests covering rate limiter behavior, utility functions, and edge cases. Live API integration tests are not included — they'd consume your quota.

---

Contributing

Issues and PRs are welcome. If you've hit a Gemini quirk that isn't handled here, open an issue with a minimal reproduction. We're particularly interested in:

• New MALFORMED_FUNCTION_CALL patterns
• Model-specific behavior differences (2.0 Flash vs 2.5 Pro vs 1.5 Pro)
• Retry strategies for streaming responses

---

License

MIT — see LICENSE.

---

Built by the Emotix team.