This project sends Codex OpenTelemetry logs to a local Grafana LGTM stack. The stack includes Grafana, Loki, Tempo, and Prometheus and is intended for local development and experimentation.
The recommended way to start the stack is Docker Compose. It loads the version-controlled dashboard and uses a named volume for persistent data:
docker compose up -dIf codex-observability is already running from the earlier docker run --rm command, stop it before switching to Compose:
docker stop codex-observability
docker compose up -dStopping that container does not delete the codex-lgtm-data named volume.
Grafana is available at http://localhost:3000. The initial username and
password are both admin.
The checked-in .codex/config.toml sends Codex logs to
the OTLP/HTTP endpoint exposed by this container at
http://localhost:4318/v1/logs.
Codex reads personal defaults from ~/.codex/config.toml. If that file does
not exist, run the following commands from the root of this repository:
mkdir -p "$HOME/.codex"
cp .codex/config.toml "$HOME/.codex/config.toml"The cp command replaces an existing global configuration. If the file
already exists, back it up before copying or editing it:
cp "$HOME/.codex/config.toml" "$HOME/.codex/config.toml.backup"Then either overwrite it:
cp .codex/config.toml "$HOME/.codex/config.toml"Or preserve your other personal settings by copying only the [otel] and
[otel.exporter.otlp-http] sections into the existing file. Do not create
duplicate [otel] tables; merge their keys into the existing table instead.
Keep log_user_prompt = false unless exporting prompt text is an intentional
privacy decision. Codex still exports prompt length and other operational
metadata when prompt logging is disabled.
Start a new Codex session after installing the configuration, submit a prompt, and then exit the session so the asynchronous exporter can flush its batch.
The codex-lgtm-data named volume is mounted at /data, where the LGTM image
stores Grafana's database and the Loki, Prometheus, Tempo, and Pyroscope data.
This means UI-created dashboards and collected telemetry survive container
replacement. Running docker compose down removes the container and network
but leaves this volume intact. Running docker compose down -v also deletes
the volume and its stored dashboards and telemetry.
For a technical overview of the volume and the different roles of Grafana,
Loki, Prometheus, Tempo, Pyroscope, and the OpenTelemetry Collector, see
docs/codex-lgtm-data.md.
The named volume is local Docker state and should not be committed to a repository. For a portable dashboard, this repository instead uses Grafana's file provisioning:
grafana/dashboards/codex-observability.jsonis the version-controlled dashboard definition.grafana/provisioning/dashboards.yamltells Grafana where to load that definition.compose.yamlmounts both paths read-only into the LGTM container.
The provisioned Codex Observability dashboard appears in the Codex folder. Grafana checks the JSON file for changes every 30 seconds. It is marked read-only in the UI because the repository file is the source of truth.
To develop the dashboard visually, make a copy in Grafana, edit the copy, then export its JSON and replace the repository JSON file. Review the diff and commit it like any other source file. Grafana never writes UI changes back to the mounted provisioning file.
Useful lifecycle commands are:
docker compose logs -f lgtm
docker compose stop
docker compose start
docker compose downIn Grafana, open Explore, select the Loki data source, and begin with:
{service_name=~"codex.*"}
Codex's service name depends on the client that started the session and is
commonly codex_cli_rs or codex_exec. If the query returns nothing, discover
the stream without assuming a service name:
{service_name=~".+"} | event_name=~"codex\\..+"
Open one result's log details and confirm its indexed labels and structured
metadata. Loki normalizes OTel attribute dots to underscores, so Codex's
event.name, conversation.id, and error.message normally appear as
event_name, conversation_id, and error_message in LogQL.
The examples below use this stream selector:
{service_name=~"codex.*"}
Useful queries include:
{service_name=~"codex.*"}
| event_name=~"codex\\..+"
| line_format `{{.event_name}} model={{.model}} conversation={{.conversation_id}} duration={{.duration_ms}}ms status={{.http_response_status_code}} success={{.success}} error={{.error_message}}`
The line_format stage builds a readable message from Loki structured
metadata because Codex OTel records may have an empty log body. Fields that do
not apply to a particular event are displayed as empty values.
{service_name=~"codex.*"} | event_name="codex.conversation_starts"
This event is useful for inspecting the model, Codex version, reasoning settings, sandbox policy, approval policy, and configured MCP servers.
{service_name=~"codex.*"}
| event_name="codex.api_request"
| error_message!=""
{service_name=~"codex.*"}
| event_name="codex.tool_result"
| success="false"
{service_name=~"codex.*"}
| event_name="codex.sse_event"
| event_kind="response.completed"
Expand a result to see input_token_count, output_token_count, and, when
present, cached_token_count. Codex versions using WebSockets may emit the
corresponding completion information on codex.websocket_event, so include
that event type if it appears in your logs.
{service_name=~"codex.*"} | event_name="codex.tool_decision"
Use this to review approved and denied tool requests and whether the decision came from configuration or a user.
Create a dashboard, add the following panels using the Loki data source, and
set the dashboard refresh interval to something modest such as 30 seconds.
Grafana's $__interval and $__range variables automatically follow the
selected dashboard time range.
Visualization: Time series
sum(count_over_time(
{service_name=~"codex.*"} | event_name="codex.conversation_starts" [$__interval]
))
This shows newly started Codex conversations per interval.
Visualization: Time series. This uses a rolling five-minute window and
creates one series per event_name.
sum by (event_name) (
count_over_time(
{service_name=~"codex.*"}
| event_name!=""
[5m]
)
)
The service selector uses the regex operator =~; using = would search for
a service whose literal name is codex.*.
Visualization: Time series
sum(count_over_time(
{service_name=~"codex.*"} | event_name="codex.api_request" [$__interval]
))
Visualization: Gauge or Stat, unit Percent (0-100)
100 *
sum(count_over_time(
{service_name=~"codex.*"}
| event_name="codex.api_request"
| error_message!="" [$__range]
))
/
sum(count_over_time(
{service_name=~"codex.*"} | event_name="codex.api_request" [$__range]
))
Visualization: Time series, unit milliseconds
quantile_over_time(
0.95,
{service_name=~"codex.*"}
| event_name="codex.api_request"
| unwrap duration_ms [$__interval]
)
Visualization: Time series. Add these as two queries in the same panel and
name their legends input and output.
sum(sum_over_time(
{service_name=~"codex.*"}
| event_name="codex.sse_event"
| event_kind="response.completed"
| unwrap input_token_count [$__interval]
))
sum(sum_over_time(
{service_name=~"codex.*"}
| event_name="codex.sse_event"
| event_kind="response.completed"
| unwrap output_token_count [$__interval]
))
The provisioned dashboard also shows the largest model context window and automatic compaction threshold reported by conversations in the selected time range. Both fields are optional in Codex telemetry.
max(max_over_time(
{service_name=~"codex.*"}
| event_name="codex.conversation_starts"
| unwrap context_window [$__range]
))
max(max_over_time(
{service_name=~"codex.*"}
| event_name="codex.conversation_starts"
| unwrap auto_compact_token_limit [$__range]
))
The per-request context panel graphs the largest input context sent during each dashboard interval:
max(max_over_time(
{service_name=~"codex.*"}
| event_name="codex.sse_event"
| event_kind="response.completed"
| unwrap input_token_count [$__interval]
))
This is a per-request measurement rather than a continuous live utilization gauge. History compaction can cause the value to fall between turns.
Visualization: Time series
sum(count_over_time(
{service_name=~"codex.*"}
| event_name="codex.tool_result"
| success="false" [$__interval]
))
Visualization: Logs
{service_name=~"codex.*"}
| event_name=~"codex.api_request|codex.tool_result|codex.tool_decision"
| error_message!="" or success="false" or decision="denied"
If a panel reports a parse error or no data, inspect a matching entry in Explore and adjust the field name or value to the metadata emitted by the installed Codex version. In particular, approval values and WebSocket event fields may vary between client versions.
| Ponytail | Caveman | |
|---|---|---|
| Main goal | Write less and simpler code | Produce shorter explanations and responses |
| Changes reasoning? | Yes—pushes Codex toward YAGNI, standard-library features, existing dependencies, native platform features, and minimal diffs | Mostly changes communication style: terse, compressed, “caveman-like” wording |
| Typical effect | Fewer abstractions, dependencies, files, and lines of code | Fewer output tokens and less conversational text |
| Best for | Preventing overengineering | Reducing verbose agent chatter |
| Risk | May underbuild features in aggressive modes | Responses may become harder to read or omit helpful explanation |
| Modes | lite, full, ultra, off |
lite, full, ultra, plus normal/off commands |
Without either:
I’ll create a reusable abstraction, configuration type, service layer, and test utilities to support this feature.
With Ponytail:
The platform already supports this. Use the native API and change three lines.
With Caveman:
Native API exists. Change three lines. No abstraction needed.
Ref: https://github.com/DietrichGebert/ponytail
- Install the Codex CLI first
- Follow the official installation instruction, the plugin is then located in
~/.codex/plugins/cache/ponytail/ponytail/<VERSION> - Reload VSCode
Ref: https://github.com/JuliusBrussee/caveman
- Follow the official installation instructions
- Reload VSCode