[RFC] Reduce per-call overhead of CPU inplace prediction (config parse, model-view rebuild)

[agene0001]

Following up on #12306 and #12307 (single-row CPU inplace-predict overhead): after both fixes, I re-profiled the same workload (single row, 127 features, 50 trees, nthread=1, XGBoosterPredictFromDense in a loop via a proxy DMatrix, Apple M3 Pro, ~4.5us/call) to see what remains. The call is still dominated by per-call setup rather than tree traversal:

Cost	~share of call	Notes
Json::Load of the config string	~17%	config bytes are typically identical call-to-call
Json::Load of the array-interface string (inside ArrayAdapter)	~16%	only the data pointer/shape change between calls
HostModel / GBTreeModelView construction	~19%	rebuilt per call: per-tree view variants, allocations, memcpys — model unchanged between calls
Actual prediction work (FVecFill + traversal)	~15%
SetArray remainder, output handling, misc	rest

So roughly half of a single-row inplace predict is recomputing state that is invariant across calls in a serving loop. Three directions, in decreasing value/effort ratio:

1. Cache the constructed model view (HostModel) in PredictionCacheEntry (or on GBTree), invalidated when trees are added or tree_begin/tree_end differ. This also benefits the DMatrix predict path.
2. Memoize the parsed prediction config: if the config bytes equal the previous call's (per booster, thread-local), reuse the parsed Json. Alternatively a "prepared predict" handle in the C API, though that's a bigger surface change.
3. Lighter array-interface parsing for the adapter path (the document schema is small and fixed; a specialized parser or a Json::Load fast path could cut it substantially).

Before writing any code: is there maintainer appetite for work in this direction, and if so, which approach would you prefer for (1) and (2)? Happy to implement and benchmark — same methodology as the two merged PRs.

🤖 Generated with Claude Code

[trivialfis]

Thank you for the optimizations and the investigation. I think the existing prediction method is a dead end for low-latency inference. The entire implementation has been optimized for batch prediction and to support high-level languages like Python/R.

We hope that dedicated inference libraries such as https://github.com/rapidsai/nvforest (among others) can support low-latency inference. If you are aiming for efficient single-sample inference, I would say XGBoost's predictor is far from optimal, and the existing API is unlikely to achieve that goal.

For most of the inference libraries out there (not just trees), one needs to first construct an immutable model representation (like the host model in XGBoost), persist it in memory, even in the CPU cache if possible, then reuse it for all incoming samples, hence, they are usually modeled as a dedicated long running inference service.

[RAMitchell]

I don't mind simple optimisations here, but cacheing can get a bit annoying to maintain. Maybe the json is worth optimising if its not too hard, it looks like you could get up to another 30%.