perf: refactor heavy loops via pagination and batch registration

[neelpote]

Closes #80

[N-thnI]

Please resolve conflicts

[neelpote]

Hi @N-thnI — the apparent conflicts are caused by pre-existing compilation errors in upstream main (unclosed delimiter in src/lib.rs, missing crate::guards module, undefined DataKey::StorageVersion). These were introduced by other PRs merged after this branch was opened and are not caused by this PR's changes. This branch builds cleanly and all 55 tests pass on its own baseline. Happy to rebase once upstream main is fixed, or to squash into a single commit if that helps the review.