feat: DLRET reconstruction table as primary output#1
Merged
Conversation
Add the verified research report on how CRSP generates DLRET across all delisting situations (Shumway -30% / Shumway-Warther -55% / worthless -1 corrections; the 501/502 up-migration bug), and the approved design spec making the per-delisting DLRET reconstruction table the library's primary output via a central EnrichedDelistRecord type. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8 TDD tasks: dlret.py hub (resolve_dlret/DlretResult + mixed cash+stock), bmp_correction re-point, 501/502 fix, EnrichedDelistRecord + enrich, build_dlret_table + CSV, firm-month stock-leg threading, classify_universe wiring to output/dlret.csv, and docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h mixed cash+stock
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion back-compat) The no-consideration ABSTAIN branch fired before the price guard, so a merger with neither consideration terms nor a valid last_trade_close returned 0.0 instead of NaN, silently turning a firm-month drop into a ~-100% return and breaking compute_dlret's documented value-preservation invariant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… source) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ock terms, clamp confidence Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…maps; av_listing import Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nk comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(dedup _read_map) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…al verify') Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reads EDGAR merger filings with an LLM and returns structured consideration (cash leg, stock exchange ratio, acquirer), lifting cash+stock/stock-only DLRET coverage from 10 to 98 deals (407/461 total computed). The acquirer price and target last-trade close are joined from the raw Tiingo panel; a sanity gate (|terminal/last_close-1| <= tol) rejects mis-resolved acquirers. New modules: - raw_tiingo.py: nominal-close price-panel join - llm_client.py: injectable OpenAI JSON client (json_schema -> json_object -> bare fallback; last-resort drops temperature) - filing_selection.py: filing-tier selection shared with PayoutExtractor (its 53 tests stay byte-identical) - llm_merger_extractor.py: the extractor (prompt constant + version, excerpt windowing, disk cache, completeness gate) - scripts/eval_merger_extractor.py: live calibration harness over 10 labeled deals (proven 10/10 before fan-out) Wired into classify_universe.py behind --extract-merger-terms-llm. When the LLM reads a deal as all-stock, the payout extractor's (mis-read) cash is suppressed so build_dlret_table cannot inflate a stock-only deal into a bogus cash_plus_stock (the MRD 65x bug). 190 offline tests; e2e validated (median |dlret| 0.9%, max 13.8%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the LLM merger-terms auto-extraction command (network: SEC + OpenAI) and architecture pointers for llm_merger_extractor.py / llm_client.py / raw_tiingo.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two sanity-gated coverage wins on the LLM merger path: - Pure-cash deals where the regex payout extractor found nothing now fall back to the LLM's cash (sanity-gated vs last_close): +5 (AMED, MCW, MSP, NRE, TWKS). - ACQUIRER_RENAMES maps a deal-era acquirer ticker to the symbol the raw Tiingo panel files its history under (UTX->RTX, ECA->OVV, COG->CTRA, HRS->LHX, Q->IQV, SXCI->CTRX, SPF->CAA, ESV->VAL), so the acquirer price resolves though the issuer was later renamed: +6 (COL, NFX, XEC, XLS, IMS, CHSI). ATW/RDC/RYL stay blank — their renamed price didn't reconcile and the gate correctly rejected them. Computed DLRET 407 -> 418/461; no regressions (10 labeled deals unchanged, all 104 stock-leg deals within +/-15%). 190 tests green. Diagnosed but not fixed: 7 of 8 "found-nothing" mergers (MON, TIN, THOR, STR, ...) carry wrong observed_delist_dates in the input (MON 2022-12-23 vs the 2018 Bayer close), so filing selection misses the real 8-K -- an input-data issue, separate from the extractor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every delisting in output/dlret.csv now carries a numeric DLRET (was 418/461, now 461/461). A completed merger or a fund/non-equity closure with a known last price but no computable consideration has terminal value ≈ that last price — merger arbitrage closes the gap to the deal value before the last trade, and a fund/ETF redeems at NAV ≈ its last trade. So DLRET ≈ 0 is the maximum-likelihood estimate, not a missing value. These rows are emitted as a new ASSUMED_PAR method at "low" confidence (terminal_value = last_close), so the value is filled but never mistaken for a realized/computed return — readers can filter on dlret_confidence. Requires a positive last price (no denominator otherwise; a no-price abstain still renders blank). Table-only: enrich() applies the fill; the firm-month facade (compute_dlret / bmp_correction) is byte-for-byte unchanged. The 43 filled rows = 26 merger abstains (acquirer not in the price panel, sanity-rejected, or wrong-dated) + 17 expiration/non-equity closures (mostly recycled tickers whose original company was acquired, e.g. HOT→Marriott now a Defiance ETF). 192 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Using DLRET to build training labels and backtests" section: the two formulas (label = (1+RET)(1+DLRET)-1; exit = last_close×(1+DLRET) = terminal_value), a pandas recipe straight off output/dlret.csv, a dlret_confidence trust guide, and per-bucket intuition. Also reconcile the DLRET-policy table and the AET worked example with the shipped behavior: document the assumed_par fill (no empty / no silent zero), and correct AET to its real last trade close ($212.70 → dlret -0.21%) — the old example divided by the pre-announcement $190 and showed a +11.6% that double-counts the premium already earned in RET. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
output/dlret.csv: a per-delisting DLRET reconstruction table built on a self-explaining DLRET hub (dlret.py) and a centralEnrichedDelistRecord+build_dlret_table/write_dlret_csv(reconstruction.py).terminal = cash + stock_ratio×acquirer_price.classify_universe.pygains--last-trade-closes/--merger-terms/--recoveries/--dlret-outputflags.EXCHANGE_TRANSFER, notCOMPLIANCE_FAILURE.bmp_correction.pyre-points at the DLRET hub (back-compat re-exports).Update — LLM-based cash+stock merger extractor (
d2ba93c…de3f77e)Replaces hand-written
--merger-termsCSVs with an LLM that reads deal terms straight from EDGAR, behind--extract-merger-terms-llm.llm_merger_extractor.py(extractor + versioned prompt +cache/llm/cache),llm_client.py(injectable OpenAI JSON client, json_schema→json_object→bare fallback),raw_tiingo.py(nominal-close price join),filing_selection.py(filing-tier picker shared withpayout_extractor.py).|terminal/last_close−1| ≤ tol, default 0.15) drops any term that doesn't reconcile, rejecting mis-resolved acquirers. Explicit--merger-termsrows always override the LLM.scripts/eval_merger_extractor.py, 10/10) before fan-out.ACQUIRER_RENAMESmap (UTX→RTX, ECA→OVV, COG→CTRA, HRS→LHX, Q→IQV, SXCI→CTRX, SPF→CAA, ESV→VAL) resolves later-renamed acquirers' panel prices (+6).de3f77e): a completed merger / fund closure with a known last price but no computable consideration has terminal ≈ last price (arbitrage closes the gap; funds redeem at NAV), so it's filled asASSUMED_PAR(DLRET 0, low confidence) rather than left blank — filterable viadlret_confidence, never mistaken for a computed return. Table-only; the firm-month facade is unchanged.Recall on the full universe (461 delistings):
dlret_methodmix: cash_only 216 · stock_only 56 · cash_plus_stock 48 · exchange_transfer_zero 45 · assumed_par 43 (low-conf ≈0) · shumway_nyse_amex 32 · shumway_nasdaq 21. The 43assumed_parrows are completed mergers whose acquirer isn't in the US price panel (foreign/private/delisted), sanity-rejected, or wrong-dated, plus fund/recycled-ticker closures (e.g. HOT→Marriott, now a Defiance ETF).Test plan
conda run -n rdagent4qlib pytest)PayoutExtractor's 53 tests byte-identical after thefiling_selection.pyrefactorclassify_universe.py --extract-merger-terms-llm→output/dlret.csv(461/461 valued, 0 blank); AET −0.21% / AGN −0.01% reconcile; sanity gate caught the MRD 65× regression; no stock-leg outliers >15%🤖 Generated with Claude Code