On master @ 58cee73, doProcessors rebinds the batch to whatever each processor returns (batch/batch.go:416-436). A processor that follows the standard Go idiom return nil, err on failure silently wipes the entire batch from accounting: downstream processors receive nil, the final per-item Error scan sees nothing, and only the single stage-wide error is reported — per-item tracking (a documented core feature) is lost. Intentional truncation (Filter) is indistinguishable from accidental loss, so the engine cannot warn.
PR #66 fixes the Filter-specific instance (errored items passed through), but any custom processor can still do this.
Ask: make the contract explicit and loud on the Processor interface godoc — "return every item you did not deliberately drop; on error, return the input slice, not nil" — and consider having the engine treat a nil return with a non-nil error as "keep previous items" (defensive, zero cost on the happy path).
Relations: #79 (error identity/shape), #77 (processor requirements), PR #66 (Filter instance). From a deep engine review.
Scope addition (from the PR #66 review discussion): the same engine-level guard should also consider returned slices containing nil elements — today a nil item from a buggy processor panics in Transform/Channel/Error and the engine's final error scan. A per-processor nil check was declined on #66 as inconsistent piecemeal defense; if nil tolerance is wanted, it belongs here, in the engine/contract.
On master @ 58cee73,
doProcessorsrebinds the batch to whatever each processor returns (batch/batch.go:416-436). A processor that follows the standard Go idiomreturn nil, erron failure silently wipes the entire batch from accounting: downstream processors receive nil, the final per-itemErrorscan sees nothing, and only the single stage-wide error is reported — per-item tracking (a documented core feature) is lost. Intentional truncation (Filter) is indistinguishable from accidental loss, so the engine cannot warn.PR #66 fixes the Filter-specific instance (errored items passed through), but any custom processor can still do this.
Ask: make the contract explicit and loud on the
Processorinterface godoc — "return every item you did not deliberately drop; on error, return the input slice, not nil" — and consider having the engine treat a nil return with a non-nil error as "keep previous items" (defensive, zero cost on the happy path).Relations: #79 (error identity/shape), #77 (processor requirements), PR #66 (Filter instance). From a deep engine review.
Scope addition (from the PR #66 review discussion): the same engine-level guard should also consider returned slices containing nil elements — today a nil item from a buggy processor panics in Transform/Channel/Error and the engine's final error scan. A per-processor nil check was declined on #66 as inconsistent piecemeal defense; if nil tolerance is wanted, it belongs here, in the engine/contract.