Repository navigation
The within-organisation regime, measured on three corpora - #3
Merged
Merged
Conversation
… inverting A strongly negative q means the base quantiles were wider than the calibration residuals justify, so CQR tightens them — and a tightening larger than half the base width pushes the lower bound past the upper one. That is not an interval: evaluation.empirical_coverage rejects it, so downstream the choice was between a crash and a defined answer. Surfaced by the metis-core verdict simulation at n=60, where it raised outright. The defined answer is the midpoint. The calibration is saying the truthful band is narrower than nothing, and a zero-width interval reports exactly that — it will miss almost every actual, which is what an over-tightened band deserves — rather than swapping the bounds and fabricating a band the calibration never endorsed. Well-behaved intervals come through bit-identical, so nothing produced before the guard existed changes.
… not test REPORT.md section 7 names one honest gap: every G0 split was a cross-organisation cold start, and the regime a product is deployed in — one organisation's own history — was never measured. Part of it was available for free the whole time. SiP is ONE company: 12,299 estimate events over ten years, 22 developers, 20 project codes, an expert estimate on every row. The gate used it as a text dataset only. The question is deliberately not G0's. G0 asked whether a model can estimate effort from scratch; the answer was no. This asks whether the organisation's own estimation error is conditionally learnable from context available at estimation time, with a leakage split declared column by column before anything is fitted — DeveloperPerformance, TaskPerformance, DeveloperID and StatusCode are the SiP equivalents of the SEERA attributes the G0 audit threw out. Four arms on the same out-of-fold rows. The expert reproduces G0's 41.6% exactly, which is the cross-check that the split is the gate's own. Three findings. Correcting the estimate does not beat the estimate: a measured tie (+0.0% [-1.4, +1.4], McNemar p=1.00) at n=4,533 with 1,121 discordant pairs once the exact ties are dropped. A single global recalibration factor is significantly WORSE (-0.8%, p=2.6e-05) on an estimator already unbiased in the median, with the factor fitted on thousands of rows rather than three. And the expert's headline 41.6% is substantially a logging convention: on 26.3% of test rows the actual equals the estimate to the digit, and dropping those takes the expert to 20.7%. The G0 verdict is untouched: this consumes recorded expert estimates, which protocol.md section 5 keeps permanently outside the gate, and never reads SEERA.
The fair objection to the SiP result is that SiP is ONE organisation with one logging culture. Apache's public issue tracker answers it cheaply: no credentials, and every issue carrying both timeoriginalestimate and timespent is a recorded human estimate paired with a recorded actual, dated, grouped by project. Time tracking is optional there, so the whole tracker yields ~2,100 such issues across ~180 projects — small, but many independent teams rather than one. The corpus is nothing like SiP, and that is the point. SiP's estimator is unbiased in the median; Apache's contributors over-estimate heavily and consistently (pooled mean log-ratio -1.05). So the CENTRE of the correction decides everything: the mean-based factor costs -7.2% PRED(25) against leaving the estimate alone (McNemar p=3e-07), while the median-based one is a clean tie (+0.4%, p=0.66). A fifth arm was added to both experiments to make that visible. Set against SiP the pair says what neither says alone: on an unbiased estimator a global factor subtracts accuracy, on a heavily biased one only a robust centre survives, and neither corpus rewards conditioning on task context beyond that. metis-api's recalibration was rebuilt on exactly this. Declared in the registry with in_gate=False. Never gate-carrying: mined after the gate closed, consumes recorded expert estimates, never touches SEERA.
…nestly TAWOS is the only public corpus with enough within-project history to ask the question at scale, and it ships as a 4.3 GB mysqldump whose README says to load it into MySQL 8. extract_tawos.py reads the Issue table straight out of the dump instead: a scanner that respects SQL quoting and backslash escapes, because issue descriptions contain both plus literal ),( sequences and a regex would silently corrupt exactly the richest rows. Column order comes from the dump's own CREATE TABLE and is verified at run time. The label is the story here, so the report leads with it. TAWOS's Total_Effort_Minutes is derived from in-progress status transitions — a median of 160 hours per issue, which is a LEAD TIME, not hours of work. The real effort label is the JIRA worklog field, and only 900 issues carry both it and a story point. Neither is comparable to a G0 PRED(25), and the report says so before any number. Story points are not hours, so the baseline converts them through the train block's own median outcome-per-point — the conversion a client would actually make. On lead time that beats the estimate-blind floor by +2.2% (p=3e-15), so the points carry signal; a GBM given those same points plus the issue's context lands -0.6% [-1.2, -0.0] against the plain conversion. On the correctly-labelled effort subset the floor is ahead of the conversion. Same shape as SiP and Apache: the estimator that already exists is the one to beat, and none of the model classes tried beats it.
…ne answer And the pilot-partner section stops calling the regime untested: it is tested at task and issue granularity now, negatively, which is why the offer leads with the calibrated interval rather than an expected accuracy gain.
|
This pull request lets the code fetch a URL, touch the filesystem in 3 places, and write a file in 3 places.
What I could not follow43 calls resolve to a name whose type is written nowhere, so what they reach is not established here, and 8 more are in tests and benchmarks, which this report does not judge. Across everything it read that is 51 of 626, 8%. That is too many to list, and where they are is more useful than which they are:
If one of those reaches the network or starts a process, this report does not show it. 16 files read · 13089 ms · against |
It is a Track B dataset with an expert estimate and a date on every row, and the canonical-schema test parametrizes over LOADERS — so adding the loader without adding its expectations failed the case rather than skipping it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
REPORT.md§7 names one honest gap: every G0 split was a cross-organisation cold start,and the regime a product is actually deployed in — one organisation's own history — was
never measured. Part of it was available for free the whole time.
Three corpora, one question, asked with the same shape of experiment: does correcting
an estimator's own estimate, from its own history, beat leaving it alone?
SiP — one company, 12,299 estimate events over ten years, an expert estimate on
every row. The gate used it as a text dataset only. The expert reproduces G0's 41.6%
exactly, which is the cross-check that the split is the gate's own. Correcting it is a
measured tie (+0.0% [−1.4, +1.4], McNemar p = 1.00) at n = 4,533 with 1,121
discordant pairs once the exact ties are dropped. A single global recalibration factor
is significantly worse (−0.8%, p = 2.6e-05) on an estimator already unbiased in the
median. And the expert's headline 41.6% is substantially a logging convention: on 26.3%
of test rows the actual equals the estimate to the digit, and dropping those takes it to
20.7%.
Apache JIRA, mined fresh — ~2,100 issues across ~180 independent teams, answering the
"SiP is only one organisation" objection. The corpus is nothing like SiP, which is the
point: Apache's contributors over-estimate heavily and consistently (mean log-ratio
−1.05), so the centre of the correction decides everything. The mean-based factor
costs −7.2% (p = 3e-07); the median-based one is a clean tie (+0.4%, p = 0.66).
NaCode-Studios/metis-api#3 rebuilds the product's local recalibration on exactly this.
TAWOS — 43,108 story-pointed issues across 28 projects, read out of the 4.3 GB
mysqldump by a streaming extractor rather than a MySQL server. The report leads with the
label, because the label is the story:
Total_Effort_Minutesis derived from in-progresstransitions and is a lead time, not hours of work; only 900 issues carry the real
worklog field alongside a story point. On lead time a team's own point-to-outcome rate
beats the estimate-blind floor by +2.2% (p = 3e-15) and a model given those points plus
context does not beat the plain conversion; on the effort subset the floor is ahead.
One answer, three times: the estimator that already exists is the one to beat, and none
of the model classes tried beats it. What differs is only what a correction is worth
— nothing on an unbiased estimator, a great deal on a biased one, and only with a robust
centre.
Also here: an over-tightened CQR band now collapses to its midpoint instead of returning
lower > upper, whichempirical_coveragerejects outright. Found by the metis-coreverdict simulation at n = 60, where it raised. Well-behaved intervals come through
bit-identical.
None of this is gate-carrying: all three consume recorded expert estimates, which
protocol.md§5 keeps permanently outside the gate, and none reads the sealed SEERAholdout. The G0 verdict is untouched.
Suite green: 117 passed, 9 skipped.