[dgraph v25] failed when calling List.rollup: while encoding: cannot iterate through the list

[hankhsutc]

Describe the bug

seeing error logs failed when calling List.rollup: while encoding: cannot iterate through the list in alpha pods.
It seems to happen on nodes with large-sized data. we added @lang directive for multi-languages in the predicate.

logs

W0724 09:43:14.902606       1 mvcc.go:181] Error rolling up key [failed when calling List.rollup: while encoding: cannot iterate through the list: cannot initialize iterator when calling List.iterate minTs: 58256931, plist: splits:1  splits:8523883114575535643, mutationMap: Committed List: map[58550095:postings:{uid:17043778602460988961  value: {large size html data}

To Reproduce

Expected behavior

No error logs

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

Version : v25.3.4
deployed in k8s

Additional context

Add any other context about the problem here.

[matthewmcneely]

Hi @hankhsutc, thanks for the report. I traced the error chain and it narrows
down to a single call site, but the log line is truncated right before the part
we need.

The chain is mvcc.go:181 → List.Rollup → rollup → encode → iterate →
pIterator.seek. That last call can only fail two ways:

1. deleteBelowTs (N) must be greater than the minTs in the list (M) — a
   delete-all marker in the mutable layer below the rollup timestamp.
2. cannot read initial list part for list with base key <hex> — the
   multi-part list references a split key that can't be read back from Badger.

Your plist shows splits:1 splits:8523883114575535643, so this is a genuine
multi-part list and (2) is on the table. The two have very different causes and
very different fixes, so I need to know which one you're hitting.

What I need

The real cause is at the end of that log line, after the full posting list
dump. There's a bug in our logging (mvcc.go:181 has the format args reversed)
that also puts the key at the end, so the tail of the line gives us both. Could
you grab the last few hundred characters?

kubectl logs <alpha-pod> \
  | grep -F 'failed when calling List.rollup' \
  | head -1 | tail -c 800

[hankhsutc]

Hi @matthewmcneely,
I can't see the key at the end.
seems the logs are truncated due to large size content.
I will upgrade once this fix gets released. #9808.
let me know if anything I can help, thanks.