LLMCarbon-API

Project Overview

Introduction

Large Language Models (LLMs) like GPT-4, Claude, and Mistral are transforming industries, but their environmental impact—especially during inference—remains opaque. Most tools underestimate the true cost of LLM inference, ignoring how context size and token handling dramatically increase computational requirements and, consequently, carbon emissions.

Our project builds on the LLMCarbon methodology but introduces a crucial innovation: accurate modeling of inference cost growth as a function of both prompt and context tokens. Unlike other tools, we highlight and quantify how context management can push the cost from linear to quadratic growth, and how efficiency techniques can bring it closer to linear again.

Why Token Counting Matters

The Hidden Cost of Context

Inference cost isn’t just about the number of output tokens. Each new token generated by an LLM requires reprocessing the entire context window, so the cost grows with both the length of the prompt and the number of tokens already generated. This leads to a quadratic cost curve in naïve implementations:

For each token generated, the model re-reads all previous tokens in the context. If you generate N tokens, the total number of tokens processed is roughly proportional to N². Most existing tools ignore this, underestimating the true energy and carbon impact of LLM inference.

Our Approach: Two Cost Models

Naïve/Quadratic Model:

Every new token requires the model to process all previous tokens (context + prompt + generated). Total compute: O(N²), where N is the number of tokens in the context window.

Efficient/Linear Model:

With advanced techniques (e.g., caching, attention optimizations), the model only processes new tokens, reducing compute to O(N). Our tool can estimate both scenarios, showing the real-world impact of engineering choices.

Key Features

Accurate Token Accounting:

We count all tokens—including prompt, context, and generated output—to avoid underestimating inference cost.

Dual Cost Estimation:

See both the worst-case (quadratic) and best-case (linear) inference costs, depending on model and infrastructure.

Environmental Impact Calculation:

Integrates with the Impact Framework to estimate carbon emissions based on true computational load.

Transparent Methodology:

All formulas and assumptions are documented for reproducibility and comparison.

Example Calculation

Suppose a prompt of 300 tokens and an answer of 100 tokens.

Naïve (Quadratic) Cost:

• Turn 1: LLM process 300 tokens (prompt)
• Turn 2: LLM process 400 tokens (300 previous prompt + 100 corresponding answer) + 300 current prompt = 700 tokens
• Turn 3: LLM process 800 tokens (600 previous prompts + 200 corresponding answers) + 300 current prompt = 1100 tokens

We can check that the cost in each turn rises linearly. The total cost of having N turns in a conversation instead grows quadratically.

Efficient (Linear) Cost:

• Each turn processes only the new token Total tokens processed: 100

Why This Matters

Ignoring context cost leads to severe underestimation of LLM environmental impact. Our tool provides a more honest, actionable assessment, empowering people to consider best practices when using this kind of tools and organizations to make greener choices and optimize their LLM deployments.

References

• LLMCarbon Project (GitHub)
• Impact Framework
• Key research on LLM compute and carbon impact (Narayanan et al., Ahmad et al.)