fix: skip column-less empty views so an annotation-less dataset materializes and queries#6
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the Query tab crashing on any dataset without annotations. Found while running the indexation/exploration workflow against repo.layers.pub on 0.4.0:
lairs materializeof a dataset with no annotation layers wrote anannotations.parquetwith zero columns, andQueryEngine.open(the Query tab) then failed because DuckDB cannotread_parqueta column-less file:This blocked
lairs tui --data(and the Query tab generally) for every UD treebank on repo.layers.pub, since their annotations are not forward-reachable from the corpus.Changes
materialize. A view whose schema cannot be derived from its rows (an empty view) has no columns;materializenow skips it instead of writing a Parquet that readers reject.QueryEngine.open. It now skips any Parquet it cannot register as a view (a column-less or otherwise unreadable file) and raises only when none are readable, so one empty view no longer fails the whole engine open. This also makes it robust to such files left by older materializations.materializeskips a column-less view (writes only the populated one);QueryEngine.openskips a 0-column Parquet and still queries the good view, and raises when none are readable. Live integration: an annotation-less corpus on the dockerized PDS materializes to anexpressionsview only and the Query engine opens and queries it.0.4.0->0.4.1.Verified live against repo.layers.pub:
materialize <UD_Nayini>now writes onlyexpressions.parquet, andQueryEngine.run_sql("SELECT id, text FROM expressions")returns its 10 sentences.tui --dataopens.Type of change
Checklist
uv run ruff format --check lairs testspassesuv run ruff check lairs testspasses (includingANN401)uv run ty check --error-on-warningpassesuv run pytestpassesuv run pytest --run-integration -m integrationpasses against the Docker test PDSuv run --group docs mkdocs build --strictpassestests/CHANGELOG.mdupdated (a new## [0.4.1]Fixed entry)@dataclass,pydantic,Any, orobjectannotation introduced; no em-dashesRelated
No tracked issue. Found during live exploration of repo.layers.pub after the 0.4.0 release.