Increased disk and memory usage after in-place upgrade from Dgraph v24.1.2 to v25.3.8

[gooohgb]

Hi Dgraph team,
We observed a significant increase in disk and memory usage after performing an in-place upgrade of a Dgraph Alpha cluster from v24.1.2 to v25.3.8.
Environment
Previous version: v24.1.2
Upgraded version: v25.3.8
Upgrade type: in-place upgrade using the existing data directories
Cache configuration remained unchanged across versions:size-mb=96384
percentage=0,80,20
remove-on-update=false

Observed behavior

After the upgrade, the upgraded environment shows noticeably higher disk usage and memory usage compared with the same workload/configuration before upgrade.
From /debug/prometheus_metrics, we observed that on upgraded Alpha nodes:
dgraph_disk_used_bytes is significantly higher than expected.
badger_size_bytes_lsm and badger_size_bytes_vlog do not fully explain the total filesystem usage reported by dgraph_disk_used_bytes.
process_resident_memory_bytes / dgraph_memory_proc_bytes are also higher after upgrade.
badger_compaction_current_num_lsm and badger_write_pending_num_memtable were 0 in sampled metrics.
Example pattern from upgraded nodes:
badger_size_bytes_lsm ~99-112 GiB
badger_size_bytes_vlog ~2 GiB
dgraph_disk_used_bytes ~131-150 GiB
process_resident_memory ~188-191 GiB
For comparison, nodes still running v24.1.2 show disk usage much closer to badger_size_bytes_lsm + badger_size_bytes_vlog.
Question
Could there be any known issue or expected behavior in v25.3.8 when opening or continuing to operate on data directories created by v24.1.2, especially around Badger storage, compaction, temporary files, memory mapping, or filesystem usage accounting?
We would appreciate guidance on what additional metrics, logs, or directory-level checks would help determine whether this is expected after upgrade or indicates a leak/stale file accumulation issue.

[matthewmcneely]

Hi @gooohgb — thanks for the detailed metrics, that's a good starting point. Two things
up front, then what I need to take this further.

dgraph_disk_used_bytes isn't measuring Dgraph's directories

This one trips people up regularly. That metric comes from a statfs() call on the
postings mount, not from a walk of p/:

https://github.com/dgraph-io/dgraph/blob/main/x/disk_metrics_linux.go#L41-L49

So it reports used bytes for the entire filesystem holding --postings — which
includes the Raft WAL in w/, whatever is under --tmp, any exports or backups written
to that volume, and anything else sharing the mount. badger_size_bytes_lsm + badger_size_bytes_vlog was never expected to add up to it.

That behavior is identical in v24.1.2 and v25.3.8 (the only change to that file between
the two tags is a ticker-leak fix and an int64 cast), so the gap you're seeing on the
v25 nodes isn't a v25 accounting change. It most likely reflects what else lives on
those volumes.

For the same reason, comparing dgraph_disk_used_bytes across a v25 node and a v24 node
only tells us something if both mounts hold the same things.

One real v25 change that could explain the memory side

v25 added a uid-array materialization on the read path that v24.1.2 doesn't have
(#9430). When a uid posting list is read from disk, Dgraph now builds the full uid set as
an uncompressed []uint64 — 8 bytes per uid — rather than iterating the packed encoding.

The catch, given your config: that work is not gated on the posting-list cache being
enabled. You're running percentage=0,80,20, so the posting-list cache is off, which
means every uid posting-list read pays for the materialization and then throws it away.
Pure overhead, no reuse. I think that's a bug and I'm filing it separately.

Worth trying as an experiment on one node: give the posting-list cache a non-zero share,
e.g.

--cache "size-mb=96384; percentage=20,60,20; remove-on-update=false"

That won't remove the allocation, but it lets the result get reused instead of discarded.
I can't predict the net effect on your workload, so please treat it as a measurement
rather than a fix, and watch dgraph_posting_list_cache_hit_ratio alongside RSS.

Also note that the badger block and index caches are allocated off the Go heap (via
jemalloc), so an RSS meaningfully above your 96 GB size-mb is expected rather than a
leak on its own.

What would help me localize this

Directory-level numbers, which is the piece the metrics can't give us:

du -sh /path/to/p /path/to/w /path/to/tmp
du -sh --max-depth=1 <mount root>
df -h <mount>

on one upgraded alpha and one still on v24.1.2.

Then:

1. Are the v24 and v25 nodes actually comparable? Same Raft group, same tablets? If
   they hold different predicates the disk comparison doesn't mean much. Ideally, a
   before/after on the same node across the upgrade.
2. A heap profile from an upgraded alpha: curl <alpha>:8080/debug/pprof/heap > heap.out,
   plus curl <alpha>:8080/debug/jemalloc for the off-heap side. Both are safe to
   collect on a live node.
3. Host or container memory limit, and whether you're seeing OOM kills or just higher
   steady-state RSS.
4. Full alpha command line, particularly whether --badger was overridden.
5. Cluster shape and workload: number of alphas and groups, replication factor,
   approximate dataset size, and whether the workload is read-heavy or write-heavy.
6. How long since the upgrade? Rollups and badger compaction can take a while to
   settle after a version change, and LSM size during catch-up isn't steady state.
7. Upgrade path: straight v24.1.2 → v25.3.8, and were the p/ and w/ directories
   reused untouched, or was there any export/import or backup/restore in between?

For what it's worth, I checked and ruled out several plausible culprits: the badger
option defaults, the cache-flag plumbing, NumVersionsToKeep, and the rollup trigger
thresholds are all unchanged between the two versions, and v25 does no data migration
when opening a v24 p/ directory.