Skip to content

feat(tui): browse, index, and mute dataset sources from a Discover tab#7

Merged
aaronstevenwhite merged 2 commits into
mainfrom
feat/discovery-sources
Jun 30, 2026
Merged

feat(tui): browse, index, and mute dataset sources from a Discover tab#7
aaronstevenwhite merged 2 commits into
mainfrom
feat/discovery-sources

Conversation

@aaronstevenwhite

@aaronstevenwhite aaronstevenwhite commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a way to view available dataset sources from the TUI and select datasets to
index, with the public Layers PDS (repo.layers.pub) as a built-in default. That
PDS is deliberately kept off the firehose (so a small operator running thousands
of accounts is not blacklisted), which makes it otherwise undiscoverable; a
configured source fixes that, and the TUI grows a Discover tab, launch
auto-indexing, and permanent muting with review.

Changes

  • Configured dataset sources. A Source model and a sources.toml config
    (under the XDG config directory, overridable with LAIRS_SOURCES_FILE) name the
    PDS and relay endpoints lairs crawls. A built-in default ships for
    repo.layers.pub; a user can add sources or override or disable a built-in.
    lairs index build --source <name> and lairs sources list use them.
  • Discover tab (4). Browses the configured sources, crawls a chosen source
    in the background, and lists each dataset with its state (indexed, new, or
    muted). enter/space indexes a new dataset (it then shows on Explore) or
    permanently mutes an indexed one; r re-crawls. The crawl reuses the same
    listRepos path as lairs index build through a new streaming discover
    function in lairs.discovery.
  • Launch auto-indexing. With no --index, the TUI opens a default index
    (under the XDG state directory, overridable with LAIRS_INDEX_DIR) and crawls
    the enabled sources on launch, indexing every newly discovered, non-muted
    dataset so Explore fills in on its own. lairs tui --no-auto-index skips it.
  • Permanent muting with review. A muted dataset is recorded as a
    MutedDataset and excluded from auto-indexing until unmuted. A settings modal
    (ctrl+s) lists the sources and every muted dataset with an unmute action.
  • Concurrency. Background workers only do network reads; every index write
    happens on the UI thread against one shared DiscoveryIndex, so there is never
    a concurrent writer. Explore reloads from disk on tab activation and after the
    auto-index commit.
  • Tests. Unit coverage with a faked PdsClient (discover happy/skip paths,
    mute lifecycle, auto-index respects mutes, unreachable-source and crawl-failure
    handling, the Discover/Settings/Explore-reload flows, and the space/r
    bindings), plus live @pytest.mark.integration coverage against the dockerized
    test PDS: real crawl provenance, mute-respecting re-crawl, the full app
    end-to-end over a real PdsClient (listRepos -> describeRepo ->
    listRecords -> index -> Explore), and a multi-page drain at the default page
    size (205 corpora across 3 pages, no drops or duplicates).
  • Docs. The explorer guide and API reference document the Discover tab,
    sources, settings, and auto-indexing.

Type of change

  • Bug fix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a fix or feature that changes existing behaviour)
  • Documentation
  • Refactor, test, or chore (no user-visible change)

Checklist

  • uv run ruff format --check lairs tests passes
  • uv run ruff check lairs tests passes (including ANN401; no Any in annotations)
  • uv run ty check --error-on-warning passes
  • uv run pytest passes (the unit and functional suite)
  • uv run pytest --run-integration -m integration passes against the Docker test PDS
  • uv run --group docs mkdocs build --strict passes (when docs or a public API changed)
  • Tests added or updated for new behaviour, mirrored 1:1 under tests/
  • CHANGELOG.md updated for user-visible changes
  • No @dataclass, pydantic, Any, or object annotation introduced; no em-dashes
  • Commits follow Conventional Commits, imperative mood, with no AI-assistant mentions

Related issues

Follow-on to the discovery index and cross-account materialization work; does not
close a specific tracked issue.

@aaronstevenwhite aaronstevenwhite force-pushed the feat/discovery-sources branch from 12897cb to 8b2e965 Compare June 29, 2026 21:07
@aaronstevenwhite aaronstevenwhite changed the title feat(discovery): configured dataset sources with a built-in repo.layers.pub default feat(tui): browse, index, and mute dataset sources from a Discover tab Jun 29, 2026
@aaronstevenwhite aaronstevenwhite force-pushed the feat/discovery-sources branch 3 times, most recently from 3688f23 to 6a34f16 Compare June 30, 2026 00:36
@aaronstevenwhite aaronstevenwhite force-pushed the feat/discovery-sources branch from 6a34f16 to ffb98c5 Compare June 30, 2026 00:45
@aaronstevenwhite aaronstevenwhite merged commit 9770d36 into main Jun 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant