feat: Order iceberg_scan and prune matching

[tomlarkworthy]

Addresses #746: Top-N queries over time-sorted data (e.g. ORDER BY ts DESC LIMIT N) read every file before the limit filters them out.

Uses DuckDB core's set_scan_order table-function callback (used by native table_scan). When a LIMIT is present over an ORDER BY on a numeric/temporal column, reorder the data files by the order column's per-file bounds and drop files that cannot contain a top-N row, so the scan touches only the relevant files.

When a WHERE is present the ordering also helps choose a more efficient order for expansion (done by existing core optimizers).

A fair amount of this was generated by AI. Let me know if this is helpful or not (this is quite a good win though for timeseries usecases)

[Tmonster]

There are only happy path tests.
Can we also check what happens if we run a large insert and the values we eventually order on are not ordered?

i.e

create table t1 (a int);
insert into t1 select (range%60) a from range(100); -- file 1, lower bound 0
insert into t1 select (range%60) + 5 a from range(100); -- file 2, lower bound 5
insert into t1 select (range%60) + 2 a from range(100); -- file 3, lower bound 2
insert into t1 select (range%60) + 3 a from range(100); -- file 4, lower bound 3
insert into t1 select (range%60) + 1 a from range(100); -- file 4, lower bound 1

select a from t1 order by a asc limit 10;

Here the lower bounds for each file are scattered a bit more, and I wonder how the logic reacts to this. I would expect every file to still be scanned because there is no guarantee there are multiple instances of the lower bound per file.

[Tmonster]

Also, can you retarget main for this?

[tomlarkworthy]

ok added a new test for equally overlapping files, it scanned everything but still got the right answer (pruning is not breaking correctness).
Also added a test for mostly overlapping files, it scanned less than all of them, indicating the pruning has value in realistic scenarios when things are not perfectly mutually exclusive ranges.

rebased on main. Thanks for checking it!

[Tishj]

I understand the changes, I'm just worried that this introduces full metadata materialization
So I'm leaning on the side of not merging this

EDIT:
I took a better look, it runs after pushdown, we would end up materializing anyways, only now we're doing it eagerly as opposed to lazily.

That's fine actually, if there's a limit anyways and an order by, that sounds like the right call

[Tishj]

I have one request regarding the null counts, but other than that I think it's ready to merge

[tomlarkworthy]

I understand the changes, I'm just worried that this introduces full metadata materialization So I'm leaning on the side of not merging this

EDIT: I took a better look, it runs after pushdown, we would end up materializing anyways, only now we're doing it eagerly as opposed to lazily.

That's fine actually, if there's a limit anyways and an order by, that sounds like the right call

Yes exactly, we only cut down on IO work or perform the IO work in a better order.

[Tishj]

Thanks!