ADR-0011: Pivot to tdl-raw transformer architecture - #9
Merged
Conversation
tg-export's v0.1.0 live Telethon exporter was built under a misread of the paired msgbrowse decision (ADR-0022), which delegates Telegram to `tdl` — a Go binary that logs in by importing an installed Telegram Desktop session (no phone/code/2FA) and dumps messages including the raw MTProto struct via --raw. The two tools are complementary tiers, not rivals: tdl is the one-click session+dump tier; tg-export is the fidelity tier that transforms tdl's --raw output into the curated schema_version:1 contract. The fidelity core (mapping.py) was already a pure, offline transform over Telethon-shaped objects — it never needed a live client. So this deletes the live half and reuses the jewel. Source: - Delete auth.py, export.py (live walk), reliability.py, media.py (Telethon download); drop the telethon + platformdirs deps (only jsonschema remains). - Add archive.py (contract writer/manifest, salvaged from export.py), adapter.py (the single tdl-raw -> Telethon-shape seam), transform.py (adapter -> mapping -> schema -> archive pipeline). - cli.py collapses to one non-interactive command: `transform --input <tdl-export> --output <dir>`. - errors.py: retire the auth/network sentinels; exit codes are OK / bad-arg / bad-input / runtime. adapter.py is a verified skeleton: the pipeline runs end-to-end and every emitted message + manifest validate against the shipped schema, but the exact tdl --raw field names (and whether the dump carries the users[]/chats[] entity map for offline sender-name resolution) are TODO(tdl-shape), pending a real `tdl chat export --raw` dump. Senders degrade to stable id-only contacts when unresolved (msgbrowse ADR-0003). Records reconciled: add ADR-0011; mark ADR-0002/0006/0009 superseded; amend ADR-0001; banner SPEC-0001 spec.md + design.md (contract/fidelity stay authoritative, acquisition half retired). Tests: drop the live-path suites; refactor synthetic.py/conftest.py off the fake client; add test_transform.py. 51 pass, ruff clean, fully offline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0157yR7hQK8qugYMhUK8j918
…p dead live-path code Review fixes for the ADR-0011 pivot: - export_from_doc now requires account.id and validates chat type against the shipped manifest schema enum, so a structurally malformed tdl doc exits with the dedicated malformed-input code 5 up front instead of dying mid-transform (or post-write) as an unclassified runtime error (exit 1). - Drop the unused 'out' parameter from adapter._resolve_sender. - Remove code stranded by the pivot: jsonio.read_manifest / write_ndjson (only callers were the deleted live modules), errors.MalformedArgumentError and errors.exit_code_for (unreferenced), and the ndjson_writer append mode with its stale --since/M5 docs. - Cover the transform CLI (happy path + exit-code mapping) and the two new malformed-input rejections in tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WLMXDUhjouS4DqszQ3VcyF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements ADR-0011, pivoting tg-export from a live Telethon MTProto client to a stateless transformer that consumes
tdl --rawexports. The tool no longer performs authentication, session management, or network I/O — it is now a pure offline JSON-to-JSON converter that reshapes tdl's gotd-flavoured output into the msgbrowse contract.Key Changes
Removed (live client surface)
src/tg_export/export.py— the dialog walk, message iteration, and Telethon-coupled export orchestrationsrc/tg_export/auth.py— credential resolution, login, and session managementsrc/tg_export/media.py— media download and flood-wait handlingsrc/tg_export/reliability.py— flood-wait survival helperstest_auth.py,test_media.py,test_reliability.py,test_incremental.py,test_cli_export.py,test_export.py,test_fake_client.py,test_golden.py)Added (transform pipeline)
src/tg_export/adapter.py— tdl-raw JSON → Telethon-shaped object reshaper; the single seam coupling tg-export to tdl's outputsrc/tg_export/transform.py— the new workhorse: loads tdl export, adapts each message, maps through the unchangedmappingmodule, validates, and writes NDJSON + manifestsrc/tg_export/archive.py— salvaged from retiredexport.py; protocol-agnostic manifest assembly and per-chat index buildingtests/test_transform.py— transform pipeline tests over in-memory tdl-raw documents (VERIFICATION GATE pending real tdl dump)Modified (contract & CLI)
src/tg_export/cli.py— singletransformcommand (nologin,doctor,export,chats); reads tdl dump path, writes archivesrc/tg_export/errors.py— removed auth/network-specific codes; addedMalformedInputErrorfor malformed tdl dumpssrc/tg_export/__init__.py— updated docstring to reflect transformer roleREADME.md— repositioned as tdl consumer, not live exporterCHANGELOG.md— v0.2.0 entry documenting the pivotpyproject.toml— updated descriptiontests/conftest.py— removedFakeTelegramClientfixture; logging reset onlytests/synthetic.py— updated docstring; fixtures now feed the adapter/mapper directlytests/test_schema.py— removed golden-file validation (golden fixtures deleted)tests/test_packaging.py— version bumped to 0.2.0Implementation Details
Adapter verification gates (ADR-0011): The adapter is a skeleton pending confirmation against a real
tdl chat export --rawdump. Three gates are documented:Message/MessageServicewithreply_to,fwd_from,reactions,action,entities,mediafrom_idpeer resolution through theusers[]/chats[]indexsize,mime_type, and file metadataByte-stability (ADR-0004): Inherited from
jsoniomodule; no clock beyond manifestgenerated_atseamIncrementality moved upstream: tdl's time-window
https://claude.ai/code/session_0157yR7hQK8qugYMhUK8j918