Skip to content

Prevent Goggles memory amplification and contain worker pressure #230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
erskingardner merged 2 commits into from
Jul 10, 2026
Merged

Prevent Goggles memory amplification and contain worker pressure #230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9758791
Prevent Goggles memory amplification
erskingardner Jul 10, 2026
5fd6e04
Address memory hardening review feedback
erskingardner Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,23 @@ DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS=0
DJANGO_SECURE_HSTS_PRELOAD=0
DATABASE_URL=postgres://goggles:replace-with-long-random-database-password@db:5432/goggles
GOGGLES_MAX_DUMP_BYTES=52428800
GOGGLES_MAX_DUMP_RECORDS=50000
GOGGLES_MAX_JSONL_LINE_BYTES=2097152
GOGGLES_MAX_ACTION_EVENTS_PER_REQUEST=50000
GOGGLES_AGENT_EXPORT_MAX_EVENTS=50000
GOGGLES_FILE_UPLOAD_MEMORY_BYTES=1048576
GOGGLES_UPLOADS_ENABLED=1

@erskingardner erskingardner Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: this PR documents the pre-existing GOGGLES_UPLOADS_ENABLED kill switch for the first time but leaves its sibling GOGGLES_EXPORTS_ENABLED (streaming group export) undocumented in .env.example. Since this section is already being touched, worth adding it too.

GOGGLES_EXPORTS_ENABLED=1
GOGGLES_WEB_MEMORY_LIMIT=16g
GOGGLES_WEB_CPUS=2.0
GOGGLES_WEB_PIDS_LIMIT=256
GOGGLES_WEB_LOG_MAX_SIZE=20m
GOGGLES_WEB_LOG_MAX_FILES=5
GOGGLES_WEB_WORKERS=3
GOGGLES_WEB_THREADS=4
GOGGLES_WEB_TIMEOUT_SECONDS=300
GOGGLES_WEB_MAX_REQUESTS=500
GOGGLES_WEB_MAX_REQUESTS_JITTER=50
GLITCHTIP_DSN=https://d550950965a64eb689f5e289416faa42@glitch.ipf.dev/1
GLITCHTIP_SECURITY_ENDPOINT=https://glitch.ipf.dev/api/1/security/?glitchtip_key=d550950965a64eb689f5e289416faa42
GLITCHTIP_ENVIRONMENT=production
Expand Down
41 changes: 40 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,23 @@ DJANGO_SECURE_HSTS_INCLUDE_SUBDOMAINS=0
DJANGO_SECURE_HSTS_PRELOAD=0
DATABASE_URL=postgres://goggles:replace-with-long-random-database-password@db:5432/goggles
GOGGLES_MAX_DUMP_BYTES=52428800
GOGGLES_MAX_DUMP_RECORDS=50000
GOGGLES_MAX_JSONL_LINE_BYTES=2097152
GOGGLES_MAX_ACTION_EVENTS_PER_REQUEST=50000
GOGGLES_AGENT_EXPORT_MAX_EVENTS=50000
GOGGLES_FILE_UPLOAD_MEMORY_BYTES=1048576
GOGGLES_UPLOADS_ENABLED=1
GOGGLES_EXPORTS_ENABLED=1
GOGGLES_WEB_MEMORY_LIMIT=16g
GOGGLES_WEB_CPUS=2.0
GOGGLES_WEB_PIDS_LIMIT=256
GOGGLES_WEB_LOG_MAX_SIZE=20m
GOGGLES_WEB_LOG_MAX_FILES=5
GOGGLES_WEB_WORKERS=3
GOGGLES_WEB_THREADS=4
GOGGLES_WEB_TIMEOUT_SECONDS=300
GOGGLES_WEB_MAX_REQUESTS=500
GOGGLES_WEB_MAX_REQUESTS_JITTER=50
GLITCHTIP_DSN=https://d550950965a64eb689f5e289416faa42@glitch.ipf.dev/1
GLITCHTIP_SECURITY_ENDPOINT=https://glitch.ipf.dev/api/1/security/?glitchtip_key=d550950965a64eb689f5e289416faa42
GLITCHTIP_ENVIRONMENT=production
Expand All @@ -135,6 +152,18 @@ POSTGRES_USER=goggles
POSTGRES_PASSWORD=replace-with-long-random-database-password
```

Compose reads container resource and logging limits while it parses the Compose
file, before a service's `env_file` is applied. The default `.env` works for
both purposes automatically. When using a custom file, supply it through both
mechanisms so values such as `GOGGLES_WEB_MEMORY_LIMIT` are not silently left at
their defaults:

```sh
export GOGGLES_ENV_FILE=/absolute/path/to/goggles.env
docker compose --env-file "$GOGGLES_ENV_FILE" config
docker compose --env-file "$GOGGLES_ENV_FILE" up -d --build
```

`GLITCHTIP_DSN` enables server-side exception reporting and 5% performance
tracing by default. `GLITCHTIP_SECURITY_ENDPOINT` enables report-only CSP
violation reporting in browsers; that endpoint contains a public project key and
Expand Down Expand Up @@ -285,7 +314,17 @@ docker compose exec web python manage.py shell -c "from forensics.models import

- Audit logs preserve raw engine ids, group refs, message ids, digests, payload metadata, raw lines, raw uploaded text, user agents, and source IPs; protect the database and backups accordingly.
- Brain disk encryption is the expected at-rest protection for v1.
- Upload size defaults to 50 MiB via `GOGGLES_MAX_DUMP_BYTES`.
- Upload size defaults to 50 MiB via `GOGGLES_MAX_DUMP_BYTES`. Processing is
additionally bounded to 50,000 non-empty records and 2 MiB per JSONL line;
multipart bodies spool to disk after 1 MiB. Over-complex uploads are retained
as one quarantined raw artifact instead of being expanded into per-line ORM
objects.
- Projection APIs default to 100 rows and cap requests at 500. Action-history
scans and synchronous agent exports have separate 50,000-event safety caps.
- The Compose web service defaults to a configurable 16 GiB no-swap cgroup
limit (`GOGGLES_WEB_MEMORY_LIMIT`) and recycles Gunicorn workers after a
jittered request budget. Keep those host-protection limits in place even if
application limits are raised.
- Purge stored audit data without removing users or upload tokens with
`manage.py purge_audit_data`. Run it with `--dry-run` first, then
`--confirm-delete-audit-data` to perform a deployment cutover. Rebuild the
Expand Down
31 changes: 25 additions & 6 deletions config/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,19 +276,38 @@ def scrub_glitchtip_event(event, _hint):
LOGOUT_REDIRECT_URL = "login"

GOGGLES_MAX_DUMP_BYTES = int(os.environ.get("GOGGLES_MAX_DUMP_BYTES", 50 * 1024 * 1024))
GOGGLES_MAX_DUMP_RECORDS = int(os.environ.get("GOGGLES_MAX_DUMP_RECORDS", 50_000))
GOGGLES_MAX_JSONL_LINE_BYTES = int(os.environ.get("GOGGLES_MAX_JSONL_LINE_BYTES", 2 * 1024 * 1024))
GOGGLES_MAX_ACTION_EVENTS_PER_REQUEST = int(
os.environ.get("GOGGLES_MAX_ACTION_EVENTS_PER_REQUEST", 50_000)
)
GOGGLES_AGENT_EXPORT_MAX_EVENTS = int(os.environ.get("GOGGLES_AGENT_EXPORT_MAX_EVENTS", 50_000))
GOGGLES_UPLOADS_ENABLED = env_bool("GOGGLES_UPLOADS_ENABLED", True)
# Operational kill-switch for the streaming group-export endpoint, mirroring the
# upload toggle. Lets an operator shed a resource-intensive read surface without a
# redeploy.
GOGGLES_EXPORTS_ENABLED = env_bool("GOGGLES_EXPORTS_ENABLED", True)
DATA_UPLOAD_MAX_MEMORY_SIZE = GOGGLES_MAX_DUMP_BYTES
FILE_UPLOAD_MAX_MEMORY_SIZE = GOGGLES_MAX_DUMP_BYTES
FILE_UPLOAD_MAX_MEMORY_SIZE = min(
GOGGLES_MAX_DUMP_BYTES,
int(os.environ.get("GOGGLES_FILE_UPLOAD_MEMORY_BYTES", 1024 * 1024)),
)
for setting_name in (
"GOGGLES_MAX_DUMP_BYTES",
"GOGGLES_MAX_DUMP_RECORDS",
"GOGGLES_MAX_JSONL_LINE_BYTES",
"GOGGLES_MAX_ACTION_EVENTS_PER_REQUEST",
"GOGGLES_AGENT_EXPORT_MAX_EVENTS",
"FILE_UPLOAD_MAX_MEMORY_SIZE",
):
if globals()[setting_name] <= 0:
raise ImproperlyConfigured(f"{setting_name} must be a positive integer.")
# The upload endpoint only ever ingests a single file part, and every part at
# or under FILE_UPLOAD_MAX_MEMORY_SIZE is buffered in RAM. Capping the number
# of files at 1 stops a multipart request from accumulating many sub-threshold
# parts (each passing the per-file size check) into multiple gigabytes of
# resident memory. MaxDumpSizeUploadHandler additionally bounds the cumulative
# bytes across parts as defense in depth.
# or under FILE_UPLOAD_MAX_MEMORY_SIZE is buffered in RAM. Keep that threshold
# much smaller than the accepted dump size so normal multipart uploads spool to
# a temporary file before ingestion. Capping the number of files at 1 stops a
# multipart request from accumulating many sub-threshold parts, while
# MaxDumpSizeUploadHandler additionally bounds cumulative bytes across parts.
DATA_UPLOAD_MAX_NUMBER_FILES = 1

SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
Expand Down
23 changes: 20 additions & 3 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@ services:
build: .
init: true
restart: unless-stopped
# Keep a pathological request inside the Goggles cgroup instead of letting
# it exhaust host RAM and swap. Equal memory/swap limits disable additional
# swap allocation for this container.
mem_limit: ${GOGGLES_WEB_MEMORY_LIMIT:-16g}
memswap_limit: ${GOGGLES_WEB_MEMORY_LIMIT:-16g}
Comment on lines +40 to +41

@coderabbitai coderabbitai Bot Jul 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and inspect the compose file around the cited lines.
file="docker-compose.yml"
wc -l "$file"
cat -n "$file" | sed -n '1,140p'

# Show only the service block if present.
rg -n "mem_limit|memswap_limit|env_file|GOGGLES_WEB_MEMORY_LIMIT|GOGGLES_ENV_FILE" "$file"

Repository: marmot-protocol/goggles

Length of output: 3742

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect repository references for the environment-file mechanism and memory limit variable.
rg -n "GOGGLES_ENV_FILE|GOGGLES_WEB_MEMORY_LIMIT|docker compose|docker-compose" .

# Look for docs explaining how the env file is meant to be supplied to Compose.
fd -a -t f 'README*|*.md' .

Repository: marmot-protocol/goggles

Length of output: 2126

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read the documentation around the memory limit configuration and env-file usage.
sed -n '120,150p' README.md
printf '\n---\n'
sed -n '60,90p' docs/deployment.md
printf '\n---\n'
sed -n '1,80p' .env.example 2>/dev/null || true

Repository: marmot-protocol/goggles

Length of output: 4029

Use a Compose-time source for the memory limit. env_file only sets container environment, so changing GOGGLES_WEB_MEMORY_LIMIT inside GOGGLES_ENV_FILE will not update mem_limit or memswap_limit; Compose will keep the default unless the value is present in the shell or Compose .env. Either document that requirement or read the limit from the same config path as the rest of the deployment settings.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docker-compose.yml` around lines 40 - 41, Update the memory-limit
configuration used by the Compose service: the ${GOGGLES_WEB_MEMORY_LIMIT:-16g}
interpolation in mem_limit and memswap_limit only reads Compose-time environment
sources, not GOGGLES_ENV_FILE. Either document that this variable must be
defined in the shell or Compose .env, or change the deployment configuration so
both fields read the limit from the same source as the other settings.

mem_swappiness: 0
cpus: ${GOGGLES_WEB_CPUS:-2.0}
pids_limit: ${GOGGLES_WEB_PIDS_LIMIT:-256}
env_file: ${GOGGLES_ENV_FILE:-.env}
Comment on lines +40 to 45

@erskingardner erskingardner Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: GOGGLES_WEB_MEMORY_LIMIT is silently ignored when set via a custom GOGGLES_ENV_FILE.

mem_limit/memswap_limit use single-$ interpolation, which Compose resolves at compose-file-parse time from the invoking shell's environment or the literal ./.env file Compose auto-loads next to the compose file. The gunicorn flags below use $$-escaped interpolation, resolved inside the container at runtime from whatever file env_file: ${GOGGLES_ENV_FILE:-.env} points to. These are two different resolution mechanisms, and GOGGLES_ENV_FILE only participates in the second one.

Verified with docker compose config (with GOGGLES_WEB_MEMORY_LIMIT=24g set only inside a custom GOGGLES_ENV_FILE target):

mem_limit: "17179869184"      # 16 GiB fallback — the 24g override was NOT applied
memswap_limit: "17179869184"
...
--workers $${GOGGLES_WEB_WORKERS:-3}   # correctly threads through to the container

Moving the same value into a top-level .env (or exporting it in the shell) resolves correctly to 24 GiB. So the memory ceiling — the headline container-hardening feature of this PR — is only configurable through the file Compose auto-loads, not through the pre-existing GOGGLES_ENV_FILE override (this repo's own CI sets GOGGLES_ENV_FILE: .env.example). An operator who follows .env.example/README and puts GOGGLES_WEB_MEMORY_LIMIT=24g in a custom env file will see the gunicorn knobs take effect while the actual cgroup ceiling silently stays at 16 GiB.

Worth either (a) documenting the constraint explicitly ("GOGGLES_WEB_MEMORY_LIMIT must be exported in the shell or the top-level .env, not merely placed in the GOGGLES_ENV_FILE target"), or (b) restructuring so all six new tunables resolve the same way (e.g. document docker compose --env-file "$GOGGLES_ENV_FILE" as the required invocation).

depends_on:
db:
Expand All @@ -45,9 +53,18 @@ services:
command: >
sh -c "python manage.py migrate --noinput &&
python manage.py collectstatic --noinput &&
gunicorn config.wsgi:application --bind 0.0.0.0:8000
--workers 3 --threads 4 --timeout 300
--max-requests 500 --max-requests-jitter 50"
exec gunicorn config.wsgi:application
--bind 0.0.0.0:8000
--workers $${GOGGLES_WEB_WORKERS:-3}
--threads $${GOGGLES_WEB_THREADS:-4}
--timeout $${GOGGLES_WEB_TIMEOUT_SECONDS:-300}
--max-requests $${GOGGLES_WEB_MAX_REQUESTS:-500}
--max-requests-jitter $${GOGGLES_WEB_MAX_REQUESTS_JITTER:-50}"
logging:
driver: json-file
options:
max-size: ${GOGGLES_WEB_LOG_MAX_SIZE:-20m}
max-file: ${GOGGLES_WEB_LOG_MAX_FILES:-5}
healthcheck:
test:
[
Expand Down
12 changes: 10 additions & 2 deletions docs/api-v1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,19 @@ Projection endpoints support these filters where the field applies:
- `epoch`
- `from_ms`
- `to_ms`
- `limit`: defaults to `500`, capped at `5000`
- `limit`: defaults to `100`, capped at `500`
- `offset`: defaults to `0`

Compatibility note: releases before the memory-pressure hardening defaulted to
500 rows and allowed up to 5,000. API consumers that relied on those larger
pages must now follow `has_more` and advance `offset` in batches of at most 500.

Paginated responses include:

```json
{
"pagination": {
"limit": 500,
"limit": 100,
"offset": 0,
"returned": 10,
"has_more": false,
Expand All @@ -68,6 +72,10 @@ Action attribution endpoints also support:
- `origin`: for example `local_user`, `system`, or `observed_group_event`
- `action`: the normalized human-action name, such as `send_message`

Action pagination includes `scan_truncated` and `scan_limit`. If
`scan_truncated` is true, Goggles bounded the request to the newest safe action
window; use narrower filters rather than requesting an unbounded group history.

## Sensitivity Metadata

Derived objects that may contain decrypted content, full author identifiers, raw
Expand Down
40 changes: 40 additions & 0 deletions docs/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,43 @@ Capacity model — size the database for it:
The edge proxy (Caddy) streams `reverse_proxy` responses by default, so no proxy
change is required; `nginx` serves only static assets and is not in the export
request path.

## Memory-pressure deployment

The performance hardening release does **not** require purging audit data.
Deploy it with uploads paused so no old worker continues a high-memory ingest:

Set the Compose environment source once before running the commands below. This
explicit `--env-file` is required for Compose-time resource and logging limits;
the service-level `env_file` alone only populates the container environment.

```sh
export GOGGLES_ENV_FILE="${GOGGLES_ENV_FILE:-.env}"
```

1. Set `GOGGLES_UPLOADS_ENABLED=0` in the production environment.
2. Recreate the web service so the changed environment and Compose resource
limits take effect:
`docker compose --env-file "$GOGGLES_ENV_FILE" up -d --build --force-recreate web`.
3. Confirm the container has the expected 16 GiB memory/no-swap boundary (or
the value set in `GOGGLES_WEB_MEMORY_LIMIT`) by resolving the actual Compose
container rather than assuming a project-specific name:
`web_container_id="$(docker compose --env-file "$GOGGLES_ENV_FILE" ps -q web)"; test -n "$web_container_id"; docker inspect "$web_container_id"`.
Wait for the health check to pass.
4. While uploads remain paused, exercise an authenticated group overview,
delivery tab, and evidence tab while watching `docker stats`.
5. Set `GOGGLES_UPLOADS_ENABLED=1` and recreate the web service again with the
same `--env-file` command from step 2.
6. Perform a representative upload while watching `docker stats`, then recheck
the group overview, delivery tab, and evidence tab.

Do not use `purge_audit_data` for this deployment. The query changes avoid
hydrating stored raw bodies without changing their schema or deleting evidence.

The Compose service keeps three threaded workers by default, recycles each
after a jittered 500-request budget, and constrains the whole web container to
a configurable 16 GiB default (`GOGGLES_WEB_MEMORY_LIMIT`) with no additional
swap. CPU, PID, and Docker log rotation limits are configurable through the
adjacent `GOGGLES_WEB_*` settings. During an incident, set
`GOGGLES_WEB_WORKERS=1` before the recreate to prevent concurrent amplification;
restore the measured production worker count only after memory remains stable.
82 changes: 82 additions & 0 deletions forensics/admin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from django.contrib import admin
from django.db.models.functions import Length, Substr
from django.urls import reverse
from django.utils.html import format_html
from django.utils.http import urlencode
Expand Down Expand Up @@ -97,6 +98,12 @@ class AuditFileAdmin(admin.ModelAdmin):
"events_link",
)

def get_queryset(self, request):
# Admin changelists/autocomplete render metadata only. Keep the complete
# upload body out of those page-sized query results; the change-page
# preview intentionally resolves the one deferred body it displays.
return super().get_queryset(request).defer("raw_text", "user_agent")

@admin.display(description="Raw text (preview)")
def raw_text_preview(self, obj):
"""Bounded, read-only preview of the uploaded JSONL.
Expand Down Expand Up @@ -167,6 +174,78 @@ class AuditEventAdmin(admin.ModelAdmin):
)
autocomplete_fields = ("audit_file", "group")
show_full_result_count = False
# Raw evidence is immutable forensic input and can be as large as the
# complete quarantined upload. Keep it out of Django's generated form and
# expose one bounded line preview plus an intentional link to the complete
# evidence response.
exclude = (
"raw_line",
"raw_event",
"raw_kind",
"raw_context",
"context_human_action",
"context_transport",
"context_engine",
"context_group",
"context_convergence",
"context_source",
)
readonly_fields = ("raw_line_preview", "evidence_link")

def get_queryset(self, request):
return (
super()
.get_queryset(request)
.defer(
"raw_line",
"raw_event",
"raw_kind",
"raw_context",
"context_human_action",
"context_transport",
"context_engine",
"context_group",
"context_convergence",
"context_source",
)
)
Comment on lines +195 to +211

@erskingardner erskingardner Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: AuditEventAdmin change page still renders an unbounded raw blob, unlike its AuditFileAdmin sibling in this same PR.

This defer() correctly keeps the heavy columns out of the paginated changelist (verified by the new test_admin_changelists_do_not_select_verbatim_evidence_columns). But there's no equivalent of AuditFileAdmin.raw_text_preview (which caps rendering at RAW_TEXT_PREVIEW_CHARS = 2000 specifically because "a single upload can be tens of megabytes"). AuditEventAdmin has no exclude/custom readonly_fields for raw_line, so the auto-generated ModelForm resolves the deferred field and renders it verbatim into an editable <textarea>.

This matters more after this PR than before it: save_invalid_upload() stores the entire raw upload body (up to GOGGLES_MAX_DUMP_BYTES, 50 MiB default) into a single AuditEvent.raw_line for a quarantined file, and this PR adds two new triggers for that path (GOGGLES_MAX_DUMP_RECORDS / GOGGLES_MAX_JSONL_LINE_BYTES violations) on top of the pre-existing ones (bad UTF-8, generic ingest exceptions).

Confirmed directly: opening the admin change page for a quarantined event with a 2 MiB raw_line returns a 2.17 MB HTML response containing the blob verbatim; with a full 50 MiB quarantined upload this would be a ~50 MB single-request response, one .../audit_event/<pk>/change/ click away for any admin operator.

Given AuditFileAdmin was given exactly this treatment in this same PR for exactly this reason, AuditEventAdmin should probably get a bounded raw_line_preview/raw_event_preview readonly display too — or at minimum exclude those fields from the form and link out to audit_file_raw_text/the JSON evidence API the way AuditFileAdmin.events_link links out instead of inlining.


@admin.display(description="Raw line (preview)")
def raw_line_preview(self, obj):
if obj is None or obj.pk is None:
return "\u2014"
# Slice in SQL: resolving the deferred TextField first would still pull
# a quarantined 50 MiB upload into the worker merely to render 2,000
# characters of it.
row = (
AuditEvent.objects.filter(pk=obj.pk)
.annotate(
preview=Substr("raw_line", 1, RAW_TEXT_PREVIEW_CHARS),
total_chars=Length("raw_line"),
)
.values("preview", "total_chars")
.get()
)
preview = row["preview"] or ""
total_chars = row["total_chars"] or 0
notice = (
f"Showing first {RAW_TEXT_PREVIEW_CHARS} of {total_chars} characters."
if total_chars > RAW_TEXT_PREVIEW_CHARS
else f"{total_chars} characters."
)
return format_html(
'<p><em>{}</em></p><pre style="max-height: 24em; overflow: auto; '
'white-space: pre-wrap;">{}</pre>',
notice,
preview,
)

@admin.display(description="Complete evidence")
def evidence_link(self, obj):
if obj is None or obj.pk is None:
return "\u2014"
url = reverse("api-event-evidence", kwargs={"event_id": obj.pk})
return format_html('<a href="{}">Open JSON evidence</a>', url)

def lookup_allowed(self, lookup, value, request):
if lookup == "audit_file__id__exact":
Expand All @@ -181,6 +260,9 @@ class AnalysisRunAdmin(admin.ModelAdmin):
autocomplete_fields = ("group", "created_by")
readonly_fields = ("created_at",)

def get_queryset(self, request):
return super().get_queryset(request).defer("report_json")


@admin.register(DeliveryArtifact)
class DeliveryArtifactAdmin(admin.ModelAdmin):
Expand Down
5 changes: 5 additions & 0 deletions forensics/analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,11 @@ def audit_files_for_group(group):
# both the groups (M2M) and events relations.
return (
AuditFile.objects.filter(groups=group)
# ``raw_text`` can be tens of MiB and none of the group metadata views
# consume it. Loading it here made every overview/evidence/export
# metadata query retain the full body of every related upload. The raw
# body remains available through the dedicated detail/download paths.
.defer("raw_text", "user_agent")
.annotate(
group_event_count=Count("events", filter=Q(events__group=group), distinct=True),
)
Expand Down
Loading