feat(tui): browse, index, and mute dataset sources from a Discover tab#7
Merged
Conversation
12897cb to
8b2e965
Compare
3688f23 to
6a34f16
Compare
6a34f16 to
ffb98c5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a way to view available dataset sources from the TUI and select datasets to
index, with the public Layers PDS (
repo.layers.pub) as a built-in default. ThatPDS is deliberately kept off the firehose (so a small operator running thousands
of accounts is not blacklisted), which makes it otherwise undiscoverable; a
configured source fixes that, and the TUI grows a Discover tab, launch
auto-indexing, and permanent muting with review.
Changes
Sourcemodel and asources.tomlconfig(under the XDG config directory, overridable with
LAIRS_SOURCES_FILE) name thePDS and relay endpoints lairs crawls. A built-in default ships for
repo.layers.pub; a user can add sources or override or disable a built-in.lairs index build --source <name>andlairs sources listuse them.4). Browses the configured sources, crawls a chosen sourcein the background, and lists each dataset with its state (
indexed,new, ormuted).enter/spaceindexes anewdataset (it then shows on Explore) orpermanently mutes an
indexedone;rre-crawls. The crawl reuses the samelistRepospath aslairs index buildthrough a new streamingdiscoverfunction in
lairs.discovery.--index, the TUI opens a default index(under the XDG state directory, overridable with
LAIRS_INDEX_DIR) and crawlsthe enabled sources on launch, indexing every newly discovered, non-muted
dataset so Explore fills in on its own.
lairs tui --no-auto-indexskips it.MutedDatasetand excluded from auto-indexing until unmuted. A settings modal(
ctrl+s) lists the sources and every muted dataset with an unmute action.happens on the UI thread against one shared
DiscoveryIndex, so there is nevera concurrent writer. Explore reloads from disk on tab activation and after the
auto-index commit.
PdsClient(discover happy/skip paths,mute lifecycle, auto-index respects mutes, unreachable-source and crawl-failure
handling, the Discover/Settings/Explore-reload flows, and the
space/rbindings), plus live
@pytest.mark.integrationcoverage against the dockerizedtest PDS: real crawl provenance, mute-respecting re-crawl, the full app
end-to-end over a real
PdsClient(listRepos->describeRepo->listRecords-> index -> Explore), and a multi-page drain at the default pagesize (205 corpora across 3 pages, no drops or duplicates).
sources, settings, and auto-indexing.
Type of change
Checklist
uv run ruff format --check lairs testspassesuv run ruff check lairs testspasses (includingANN401; noAnyin annotations)uv run ty check --error-on-warningpassesuv run pytestpasses (the unit and functional suite)uv run pytest --run-integration -m integrationpasses against the Docker test PDSuv run --group docs mkdocs build --strictpasses (when docs or a public API changed)tests/CHANGELOG.mdupdated for user-visible changes@dataclass,pydantic,Any, orobjectannotation introduced; no em-dashesRelated issues
Follow-on to the discovery index and cross-account materialization work; does not
close a specific tracked issue.