Scores real earnings announcements with two methods -- a real Loughran & McDonald (2011) finance dictionary and a from-scratch transformer trained on the Financial PhraseBank corpus -- then runs a market-model event study testing whether either score predicts post-earnings returns. Built from the "Project 2" spec in @chrispathway's ML+quant finance list.
Headline finding, stated honestly up front: with 5 real events (the maximum this project could responsibly gather in one sitting -- see "Data provenance" below), no correlation here is statistically distinguishable from noise, and the README says so explicitly rather than dressing up small-sample numbers as a result. What's genuinely useful is the methodology -- a correct market-model event study, a real dictionary baseline, a real from-scratch transformer, and the infrastructure to run this at the scale (dozens to hundreds of events) where it would actually produce a trustworthy answer.
This sandbox's bash tool can only reach package registries and GitHub
(verified directly -- huggingface.co, sec.gov, and market-data sites
all return 403 from the egress proxy). That has two consequences, both
handled explicitly rather than worked around silently:
- FinBERT's actual pretrained weights are unreachable. They live on Hugging Face Hub. Rather than fake a "transformer-based" scorer, this project trains a real (small) transformer from scratch on the real Financial PhraseBank corpus -- see From-scratch transformer.
- Bulk historical data can't be fetched from the sandbox at all.
Every price series and press release in
data/raw/was fetched viaweb_search/web_fetchat the conversation level (broader access than the bash sandbox), then hand-transcribed into the repo -- not scraped automatically by code that runs here.data/raw/prices/_build_prices.pydocuments exactly what was fetched and from where.
Prices: stockanalysis.com (S&P Global Market Intelligence for
equities, Tiingo for SPY), fetched 2026-07-31, daily OHLCV for
2026-05-19 through 2026-07-30 (50 trading days), for AAPL, MSFT, META,
AMZN, GOOGL, and the SPY benchmark.
Press releases: official company sources -- apple.com/newsroom,
microsoft.com/investor, prnewswire.com, businesswire.com,
abc.xyz/investor -- each cross-verified against 3-5 independent
republications (Yahoo Finance, SEC 8-K filings, CNBC, etc.) before being
transcribed, specifically to avoid misattributing numbers or quotes to a
real company. Full source citations are in each file's footer
(data/raw/press_releases/*.txt). These are not redistributed as raw text
in this repository -- see Limitations and .gitignore.
Financial PhraseBank: Malo, P., Sinha, A., Korhonen, P., Wallenius, J.,
Takala, P. (2014), "Good debt or bad debt: Detecting semantic
orientations in economic texts," Journal of the Association for
Information Science and Technology, 65(4), 782-796. Licensed CC BY-NC-SA
3.0; obtained via a GitHub mirror since the canonical Kaggle/HF hosts
aren't reachable from this sandbox (data/raw/phrasebank/License.txt).
4,846 sentences, 50%+ annotator agreement, label split 59.4% neutral /
28.1% positive / 12.5% negative (confirmed against the published
distribution).
Why this particular week: AAPL, MSFT, META, AMZN, and GOOGL all reported earnings within days of each other (2026-07-22 to 2026-07-30), with genuinely large and divergent reactions -- MSFT's single-day gain was reportedly the largest one-day market-cap gain for any U.S. company on record, while META and GOOGL fell despite topline beats. That's an unusually information-rich, real natural experiment for a 5-event demonstration sample, not a cherry-picked convenient one.
scoring/lm_scorer.py wraps pysentiment2's LM class, which bundles
the actual Loughran-McDonald finance word lists (140 positive / 893
negative terms in this version -- the well-documented negative skew of
finance vocabulary, not a bug). Score: net_tone = (n_positive - n_negative) / n_total_words, the standard LM-style normalization (dividing
by document length rather than by n_pos + n_neg, which is unstable when
few sentiment words appear -- a real risk given how short and
numbers-heavy press-release prose is).
scoring/transformer/ is a small transformer encoder implemented directly
in JAX (no flax/haiku/torch) -- embedding + learned positional encoding,
2 pre-LN encoder blocks (multi-head self-attention with a padding mask,
position-wise feed-forward), mean-pooling over non-pad positions, linear
3-way classifier head. 64-dim model, 4 heads, ~330K parameters.
Trained on Financial PhraseBank (85/15 train/val split) with
class-weighted cross-entropy (inverse frequency) since the corpus is
59% neutral -- unweighted training collapses to predicting "neutral"
almost everywhere. Adam via optax.
The model overfits fast, which the training script handles by keeping the best-validation-accuracy checkpoint rather than the final epoch: train accuracy reaches 99.6% by epoch 5, while validation accuracy peaks at epoch 10 (76.3%) and degrades by epoch 15 (73.0%) as the model memorizes training examples. This is an expected, well-documented outcome for a ~330K-parameter model on ~4,100 training examples, not a tuning failure -- and 76.3% is a reasonable result for a from-scratch model at this scale (FinBERT itself, pretrained on a large finance corpus before fine-tuning, reports figures in the 86-97% range across various published evaluations -- not a fair comparison to a model trained from random init on 4,100 sentences, but useful context for what the gap buys you).
| class | precision | recall | f1 | support |
|---|---|---|---|---|
| negative | 0.628 | 0.720 | 0.670 | 82 |
| neutral | 0.807 | 0.861 | 0.834 | 433 |
| positive | 0.718 | 0.578 | 0.640 | 211 |
Standard market-model event study (MacKinlay, 1997), implemented in
pricing/returns.py: for each event, fit R_stock,t = alpha + beta * R_SPY,t by OLS over the trading days strictly before the event (so beta
isn't contaminated by the event's own return), then define abnormal return
AR_t = R_stock,t - (alpha_hat + beta_hat * R_SPY,t) and CAR(0,k) = sum(AR_0..AR_k). This is deliberately not simple/mean-adjusted returns,
which would conflate an earnings-specific reaction with whatever the
broad market did that day -- exactly the failure mode that matters here,
since several of these events landed in the same volatile week.
The index arithmetic (which trading day is "day 0", how the estimation
window is sliced, off-by-one risk between price-index-space and
return-index-space) is verified in tests/test_returns.py against a
hand-engineered synthetic case with a known exact abnormal return, checked
to 1e-9 -- this kind of bug is easy to introduce and easy to not notice
(everything still "runs"), so it's tested precisely rather than eyeballed.
Two of the five events (AAPL, AMZN) reported after market close on the last day price data was fetched -- their first full trading session (2026-07-31) hadn't happened yet at data-build time. For those two, the indicated after-hours price move is used as a clearly-flagged proxy for the day-0 return; no abnormal return or CAR can be computed for them (there's no matching benchmark session to regress against, and after-hours prices are thin-volume and provisional). MSFT and META have a realized day-0 return but no further days yet. Only GOOGL (the earliest reporter that week) has a full 5-day post-event window.
This means the "1 to 5 day" analysis the project brief asks for is only fully populated for one ticker out of five. That's disclosed rather than hidden, and it's a genuine, realistic property of running analysis on current-week data rather than a stale backtest -- in live use you'd either wait for the window to close or explicitly model with what's available, which is what this pipeline does.
5 events, all real, paired sentiment scores and returns:
| Ticker | LM net tone | Transformer net score | P(positive) | Day-0 return | CAR (max horizon) |
|---|---|---|---|---|---|
| AAPL | +0.0323 | +0.0039 | 0.00 | -0.0633 * | n/a (proxy only) |
| MSFT | +0.0265 | +0.0678 | 0.20 | +0.1551 | +0.1526 (day 0) |
| META | +0.0127 | -0.0000 | 0.00 | -0.0795 | -0.1080 (day 0) |
| AMZN | -0.0052 | +0.0017 | 0.00 | +0.0953 * | n/a (proxy only) |
| GOOGL | +0.0213 | +0.4740 | 0.47 | -0.0713 | +0.0038 (day +5) |
* after-hours indicated move (proxy; see above).
Rank correlations (n too small for hypothesis testing -- descriptive only):
- LM net tone vs. day-0 return: rho = +0.200 (n=5)
- Transformer net score vs. day-0 return: rho = +0.300 (n=5)
- LM vs. Transformer agreement: rho = +0.500 (n=5)
- LM net tone vs. CAR (n=3: GOOGL/MSFT/META): rho = +1.000
- Transformer net score vs. CAR (n=3): rho = +0.500
On that rho = +1.000: scipy reports p=0.000 alongside it, and that
p-value should be ignored, not celebrated. With n=3 there are only 6
possible rank orderings; perfect agreement happens 1-in-6 by chance alone,
so an honestly-computed exact p-value is about 0.33, not 0.000 -- scipy's
default is an asymptotic (large-n) approximation that breaks down exactly
at small n with a perfect correlation, producing a spuriously tiny number.
This is flagged explicitly because it's a real, easy-to-miss statistical
trap, and reporting p=0.000 without this context would be misleading
regardless of how the correlation itself looks.
What's actually interesting here, read as case studies rather than statistics:
- MSFT is the one case where everything lines up: positive language by both scorers, a real +15.3% abnormal return. Unsurprising -- it's also the most unambiguously positive print of the five (cloud/Azure acceleration, beat-and-raise).
- AAPL and META both read positive-to-neutral in the text (Tim Cook: "proud... strongest June quarter ever"; Zuckerberg: "optimistic about the potential ahead") while the stock fell hard in both cases. Guidance and cost/margin concerns that dominated the actual market reaction (soft iPhone guidance, Meta's legal charges and EPS miss) aren't necessarily visible in the self-congratulatory framing of a headline press release -- a real illustration of why sentiment scored from the announcement text alone is an incomplete predictor, not a pipeline bug.
- GOOGL fell -7.1% on day 0 but the CAR nearly fully round-trips to +0.4% by day +5 -- a small hint of the kind of post-earnings-announcement drift/reversal that's a real, separately-studied phenomenon in the finance literature (Bernard & Thomas, 1989, and a large follow-on literature), and a reminder that day-0 reaction and multi-day drift can tell different stories -- exactly why the brief asks for a 1-5 day window rather than just the announcement-day move.
Full tables: results/metrics/event_study_summary.md,
results/metrics/event_study_results.csv.
pip install -r requirements.txt
pip install -e .
python scripts/train_transformer.py # ~3 min on 1 CPU core; trains + saves the transformer
python scripts/run_event_study.py # scores all 5 events, computes CARs, writes results
pytest tests/ -q # 18 testsTo extend to a real-scale study, add events to data/events.yaml (ticker,
timing, press-release path) and price CSVs to data/raw/prices/ in the
same format -- everything downstream (scoring, CAR computation, rank
correlations) already generalizes to any n; only the "n too small for
hypothesis testing" framing in the writeup would need to change once it's
actually true.
- n=5 is a demonstration sample, not a study. The honest constraint section above explains why (real-time data collection in a sandboxed environment); a real answer to "does sentiment predict post-earnings returns" needs the dozens-to-hundreds of events the original literature uses (Loughran-McDonald 2011 itself uses tens of thousands of 10-Ks).
- Estimation windows are short (up to ~48 trading days available, vs. the ~120-250 days common in academic event studies) because all price history here was hand-fetched for a specific recent window rather than pulled from a bulk historical database.
- The transformer scores full press releases, which are longer than the single sentences it was trained on (Financial PhraseBank sentences are a phrase or two; press releases are hundreds of words). The model mean-pools over the whole document, which dilutes short, strongly-worded passages (like a CEO quote) against long, neutral financial-table-adjacent text. A real system would score sentence-by-sentence and aggregate (exactly what FinBERT-based pipelines typically do) -- noted here as the most valuable next architectural change, not implemented in order to keep scope achievable.
- No cross-validation on the transformer (single 85/15 split) given the small corpus and compute budget; a k-fold estimate of validation accuracy would have wider, more honest error bars than the single 76.3% figure reported.
- Raw press-release text stays local, out of any public GitHub push.
It's present in this delivered copy (the pipeline reads it directly --
scripts/run_event_study.pyneeds the actual files to run), but.gitignoreexcludesdata/raw/press_releases/*.txtso it's never swept into agit push. Company press releases are meant for public reading, but systematically republishing five companies' full official text inside a public code repository is a different, more borderline use than a single researcher scoring it locally -- so only the derived sentiment scores (results/) are meant to end up on GitHub. Full citations for re-fetching are in each file's footer and in "Data provenance" above.
pytest tests/ -q18 tests: LM scorer sanity checks; tokenizer/vocabulary edge cases (padding, truncation, unknown words); the transformer's padding mask (verified to actually change the output, not just silently no-op), gradient flow, and shape contracts; the event-study return/CAR math against a hand-engineered synthetic case with a known-exact answer to 1e-9; and data-loading consistency against the real events file (e.g. that AAPL/AMZN are correctly flagged as day-0-unavailable while MSFT/META/GOOGL are not).
- Malo, Sinha, Korhonen, Wallenius, Takala (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. JASIST.
- Loughran, McDonald (2011). When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks. Journal of Finance.
- Yang, Uy, Huang (2020). FinBERT: A Pretrained Language Model for Financial Communications.
- MacKinlay (1997). Event Studies in Economics and Finance. Journal of Economic Literature.
- Bernard, Thomas (1989). Post-Earnings-Announcement Drift: Delayed Price Response or Risk Premium? Journal of Accounting Research.
- Vaswani et al. (2017). Attention Is All You Need.
- Xiong et al. (2020). On Layer Normalization in the Transformer Architecture.