cat: preserve lookup values by demoting order to Unordered on misalignment

[ghyatzo]

Hello,

Currently, cat (or Base.cat) of two AbstractDimArrays (or AbstractDimStacks) along a dimension whose lookups are ForwardOrdered but not strictly contiguous silently drops the lookup values, replacing them with NoLookup(). For example:

dd_1 = DimArray(1:3, Ti([Date(2011), Date(2013), Date(2015)]))
dd_2 = DimArray(1:3, Ti([Date(2010), Date(2012), Date(2014)]))
cat(dd_1, dd_2; dims=Ti)     # ─╯ warns "misaligned" and returns Ti(NoLookup(...))

Once the coordinates are gone, there is no way to sort/recover them sortby (or Base.sort) also drops lookups (differently, but just as fatally). This makes it impossible to concatenate disjoint time-series chunks and then sort the result.

Closes #1207

Approach

Replace the check_cat_lookups(...)::Bool return with a Symbol tri-state:

Symbol	Meaning	Action in _cat
:ok	contiguous ordered, or any NoLookup	proceed normally (_vcat_dims)
:demote	misaligned/mixed-order but no exact value overlap	values preserved via new _vcat_dims_demote, order → Unordered(), span → Irregular
:error	exact value overlap (intersect(lookups...) non-empty)	throw(DimensionMismatch(...))
:drop	span/step mismatch	rebuild(catdim, NoLookup()) (unchanged)

The new _vcat_dims_demote helper reuses the existing _vcat_index machinery (the same maybeshiftlocus + reduce(vcat, …) from the success path) to concatenate the raw lookup values, then rebuilds the dimension with order=Unordered() and an Irregular span. This keeps the coordinates intact so downstream operations like sortperm + indexing (or a future sortby) can recover a ForwardOrdered result.

For the fallbacks hcat/vcat/vcat(::Dimension,…) the Symbol return is handled but behaviour is unchanged: they still warn and fall back to Base.hcat(map(parent,…)) / Base.vcat(map(parent,…)).

Example after the fix

julia> cat(dd_1, dd_2; dims=Ti)
┌ Warning: `Ordered` lookups are misaligned at 2014-01-01 and 2010-01-01
│ on dimension Ti.
└ @ DimensionalData .../methods.jl:557
┌ 6-element DimArray{Int64, 1} ┐
├──────────────────────────────┴───────────────────────────────────── dims ┐
  ↓ Ti Sampled{Date} [Date("2011-01-01"), …, Date("2014-01-01")]
      Unordered Irregular Points
└──────────────────────────────────────────────────────────────────────────┘
 2011-01-01  1
 2013-01-01  2
 2015-01-01  3
 2010-01-01  1
 2012-01-01  2
 2014-01-01  3

The values are preserved. A subsequent sortby (or ds[Ti=sortperm(lookup(ds,Ti))])
recovers ForwardOrdered.

What changed

src/array/methods.jl

• check_cat_lookups → returns ::Symbol instead of ::Bool
• _check_cat_lookup_order → returns :ok/:demote/:error; tracks an
  accumulated Set to detect exact value overlap
• _check_cat_lookups → two-level chain: order check passes through to span
  check only on :ok; :demote short-circuits (span validation is meaningless
  when demoted)
• _check_cat_lookups(D, ::Regular, …) / ::Explicit / ::Irregular → return
  :ok/:drop (converted from true/false)
• New _vcat_dims_demote(d1, ds…) — concatenates values and rebuilds with
  order=Unordered(), span=Irregular(combined bounds) for Intervals /
  Irregular(nothing,nothing) for Points
• _cat call site: branches on :ok / :demote / :error / else (:drop)
• hcat, vcat, vcat(::Dimension) call sites: only :ok succeeds; anything
  else falls back (unchanged behaviour)
• Fixed Unorderd typo → Unordered in _cat_lookup_intersect_warn

test/methods.jl

• Flipped @test_warn "misaligned" cat(da, db; dims=2) → @test_throws
  (identical Y lookups = exact overlap, now correctly errors)
• New "misaligned ordered lookups are demoted to Unordered" testset
  (reproduces cat drops lookups, no avenues for sorting? #1207, asserts Unordered + values preserved)
• New "overlap throws DimensionMismatch via cat" testset

Open questions

1. hcat/vcat inconsistency: should they also demote on misalignment
   rather than falling back to a plain Array? Currently out of scope — only
   _cat was changed. Maybe a separate PR?
2. sortby companion: a helper like
   sortby(x, dim) = set(x[rebuild(dims(x,dim), sortperm(lookup(x,dim)))], dim => ForwardOrdered())
   is a natural partner to this fix. Should it be added in a follow-up PR
   or included here?
3. Overlap behaviour: currently overlap (exact shared values) throws
   DimensionMismatch in _cat. For hcat/vcat it still warns and
   falls back. Should this be made consistent in a follow-up?

Related:

• fix cat for rasterstacks #1014 touches the same code but it's orthogonal to this one, both can be merged together mechanically.

Disclosure

PR was an initial hand implementation of #1207 (comment), then checked and refined with an agent.

[codecov]

Codecov Report

❌ Patch coverage is 76.31579% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.16%. Comparing base (f5af202) to head (4ece4f1).

Files with missing lines	Patch %	Lines
src/array/methods.jl	76.00%	18 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1208      +/-   ##
==========================================
+ Coverage   80.15%   80.16%   +0.01%     
==========================================
  Files          61       61              
  Lines        5851     5890      +39     
==========================================
+ Hits         4690     4722      +32     
- Misses       1161     1168       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ghyatzo]

test failures are unrelated (Makie) and I have all greens on 1.12.6 locally

[ghyatzo]

pushed a fix to recover a pretty bad regression in perfomance on the happy path. I was computing the intersection set too eagerly. Now it's comparable to master.

julia> dd_1_c = DimArray(1:1000, Ti(DateTime.(1000:1999)));
julia> dd_2_c = DimArray(1:1000, Ti(DateTime.(2000:2999)));
julia> @btime cat($dd_1_c, $dd_2_c, dims=Ti)

this PR (before): 35.000 μs (80 allocations: 87.31 KiB)
this PR (after): 4.100 μs (66 allocations: 32.98 KiB)
v0.30.1: 3.557 μs (75 allocations: 33.12 KiB)

[rafaqz]

This is a really nice PR. We never put the time in to properly resolve these Unordered lookups properly before but its correct.

My main comment is that the status symbols could more clearly communicate the status or action it triggers.

[felixcremer]

I am wondering whether we should rather return the order types from check_cat_lookups instead of symbols?
This would enable to specify the order that results from the combination of the lookups of a cat without having to reinvent now nomenclature.

[rafaqz]

I think the symbols specify more than just the order, like Nolookup is :ok too

[ghyatzo]

what about this for the status

• :ok -> :simple (but maybe :ok is fine as well?)
• :demote -> :unordered
• :error -> :intersecting
• :drop -> :conflicting

it separates more the "what" from the "what we should do about it" which now are indeed a bit intertwined.

EDIT: went with :mismatch instead of :conflicting