Ets initialisation#403
Merged
Merged
Conversation
Bringing the branch up to speeeeeeed
Bringing the mainstream changes to the branch
Updating the branch from the master
Fixes in ARIMA inits
R-CMD-check failed on all three platforms after the refineHead removal: sma.Rmd, simulate.Rmd and adam.Rmd could not be rebuilt because sma() still passed the removed refineHead argument to the C++ adamCore$fit() method. The extra trailing TRUE made Rcpp overload resolution fail with "could not find valid method". sma() drives adamCore$fit() directly rather than through adam(), so it was the one estimator whose C++ call signature the unit tests never exercised — the break only surfaced when R CMD check rebuilt the vignettes. Add tests/testthat/test_sma.R covering fixed-order fit, automatic order selection, and forecasting so this path is guarded going forward. R suite: 447 pass / 0 fail. All three vignettes rebuild cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
New initialisation option that solves for the initial state by least squares instead of the backcasting filter heuristic. In a Single Source of Error model the in-sample residuals are, at fixed persistence, an affine function of the initial state (for additive models), so the SSE-optimal initials are a single linear solve — no backward pass, no time reversal, no fixed-point iteration to diverge. Motivation: initial="backcasting" diverges for seasonal additive ETS with a trend and a large seasonal smoothing parameter (e.g. ETS(A,A,A), lags=12, persistence (0.6,0.15,0.35)), returning wildly wrong initials (level ~557, trend ~-27 for data ~110) and a large in-sample penalty. The backward pass is a fixed-point iteration with spectral radius > 1 in that regime; more iterations make it worse, and best-iterate / damping / better-seed all fail. gradient sidesteps it entirely with a linear solve. Implementation is a wrapper around the existing provided-initial path (R/adam-gradient.R), so the core estimation code is untouched: fit with backcasting to get the persistence/model, probe the forward pass at unit perturbations of each free initial to build the design, solve, then refit with the solved initials. Phase 1 supports additive ETS (no ARIMA/xreg, single seasonality); everything else falls back to backcasting with a message, so behaviour is unchanged for unsupported specifications. es() inherits it (routes through adam()). Validated on the reported MC grid (seeds 9001+, T=120): the broken cell median deficit goes from +136 to -12 (target -rank = -13), the max deficit is negative across all seeds (no divergent tail), the T-growth is gone, and the healthy low-persistence / m=4 controls are unchanged. Full suite: 456 pass / 0 fail (447 baseline + 9 new gradient tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
Replaces the earlier wrapper (which fitted with backcasting, then solved, then refitted) with gradient as a proper initialType threaded through the estimation machinery. It never runs the backcasting backward pass and never returns a backcasting result. How it works: - gradient joins backcasting/complete in the "initials not in the optimiser's B vector" scaffolding (filler, initialiser, param counting), so the persistence is estimated the usual way while the initials are computed separately. - At each fit evaluation the initials are solved by least squares (adam_fitOrGradient / adam_gradientSolve): a single exact linear solve for additive ETS, Gauss-Newton for multiplicative/mixed. The forward-pass oracle is the C++ reapply method (one call, k+1 probe slices) and the linear solve is the C++ olsCpp (pivoted QR) — both in the shared core, so the Python port will match numerically. - Provided persistence => a single solve. Estimated persistence => the optimiser estimates it while the initials are solved by gradient at every evaluation (nested; no backward pass anywhere). Scope: ETS via adam() and es(); auto.adam() inherits it. ARIMA / xreg / multi-seasonality fall back to backcasting. CES / GUM / (M)SARIMA / occurrence models accept the value (shared match.arg) and fall back to a backcasting-equivalent fit. Proper match.arg: "gradient" is a real choice in the shared commonParametersChecker match.arg, and every function's initial default is the matching 5-value vector (no special-case shims). Validated: on the reported MC grid the previously divergent seasonal +trend cell (ETS(A,A,A), lags=12, high gamma) goes from a large positive in-sample deficit to ~ -rank; controls (m=4, low gamma) are unchanged; provided- and estimated-persistence both healthy. Full suite 461 pass / 0 fail. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
Port the least-squares initial-state solve (initial="gradient") from R/adam-gradient.R to Python. New module core/utils/gradient.py mirrors adam_gradient_solve / adam_gradient_layout / adam_fit_or_gradient and calls the same shared C++ core the R engine uses: reapply(backcast=FALSE) for the forward pass and _ols.ols (pivoted QR) for the solve. Wired into the cost function, estimator and preparator fit sites, plus the checker to accept the "gradient" label. Achieves R<->Python parity to machine precision across ETS(A/M, N/A/Ad/M, N/A/M) at both fixed and optimised persistence (max fitted diff ~5e-13, logLik diff ~1e-9). Three Python-side discrepancies were fixed to get there: - Exclude gradient's ETS initials from the optimiser vector B (they are solved, not optimised) - added "gradient" to the B-construction/filler gates so the multiplicative Gauss-Newton path no longer inherits an optimiser-varied initial. - Re-solve gradient from the pristine msdecompose seed in preparator instead of the evolved post-fit state; skip gradient in the estimator refit. - Use the backcasting-style persistence seed for gradient (matches R), not the optimal seed, so the nested persistence optimiser starts in the same basin. Degrees of freedom: initial="gradient" counts its solved initials as df, the same as initial="optimal", so information criteria are comparable. Backcasting / complete keep their historical zero-initial-df accounting, so model selection is unchanged. When gradient falls back to backcasting (ARIMA / xreg out of scope) it counts df like backcasting too. R suite 461/0, Python suite 723/0 (incl. new tests/test_gradient.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
… from internal helpers CES and MSARIMA are not ETS, so initial="gradient" falls back to backcasting. Fixed the Python fall-back so it is bit-identical to initial="backcasting" (matching R's within-language behaviour): - n_iterations: group "gradient" with backcasting/complete (default 2). The in-scope ETS gradient path still forces 1 internally in the dispatcher; the ARIMA/CES fall-back now uses 2, like backcasting. - ARIMA seed backup/restore in cost_functions.CF now also runs for gradient, so the ARIMA-fall-back CF surface matches backcasting exactly (closes a ~0.07 logLik gap that made MSARIMA gradient converge to a different optimum). - estimator state-refit now runs for out-of-scope gradient (ARIMA/xreg) too, while still skipping in-scope ETS gradient (which re-solves from the pristine seed in preparator). - CES engine: add "gradient" to the backcast / n_iterations / initial gates in ces_model.py, ces/cost_function.py, ces/initialiser.py. Result: MSARIMA and CES initial="gradient" now equal initial="backcasting" to machine precision in Python, mirroring R's fall-back. Also strip the roxygen tags from the internal gradient helpers (adam_gradientSolve / adam_gradientLayout / adam_fitOrGradient) in R/adam-gradient.R - they are internal and must not surface to users. Replaced with plain comments explaining what they do and why. Python suite 725/0 (incl. new CES/MSARIMA gradient fall-back tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
…ARIMA CES and SSARIMA are separate (non-adam) engines, so initial="gradient" must fall back to backcasting. Their fit calls already backcast for gradient, but several B-construction / initial-recording gates still treated gradient as an optimal-like type, so gradient estimated initials in B instead of a clean fall-back: - adam-ces.R: gradient added level/potential/seasonal initials to B (gate at the B builder) and recorded profilesRecentInitial optimal-style. Result: CES gradient differed from backcasting by ~0.04 logLik. - adam-ssarima.R: gradient counted and appended the ARIMA initials to B, so ssarima gradient reported nparam 3 vs backcasting's 2 (logLik coincidentally matched here, but the df was wrong). Added "gradient" to those gates so CES/SSARIMA gradient is now bit-identical to backcasting in both logLik and nparam, matching MSARIMA (which routes through adam()) and the Python side. GUM already excluded gradient from B via its positive optimal/two-stage gate and needed no change; sparma has no such gate. R engine tests pass (test_ces/gum/ssarima 0 failures). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
…nversions Clean up the gradient initial-state solve in both languages so it respects the objects the caller already built instead of recreating or re-typing them: - obsInSample / obs_in_sample is now passed in from the parent (CF, estimator, preparator, forecaster) instead of being recomputed as length(yInSample). - R: yInSample and ot are already (obsInSample x 1) matrices and vecG an (nComponents x 1) matrix, so they go to reapply() untouched — dropping the matrix -> as.numeric -> matrix round-trip and the redundant as.matrix/as.numeric coercions. Only a single yVector is derived for the residual arithmetic (R errors on non-conformable matrix subtraction, so a vector is required there). - The loop-invariant actuals/occurrence matrices are prepared once, not rebuilt inside residualsFor() on every forward-pass evaluation. - Python: y_in_sample / ot arrive as length-obs vectors, so the (obs, 1) reshape is genuine, but it is done once and the resulting y matrix doubles as the broadcasting operand in the residuals; vec_g is kept as its (n_components, 1) column and repeated directly. - Clearer local names (nComponents, yVector, yFitted, arrayProfile, ...). Pure refactor: no behaviour change. R suite 461/0, Python suite 725/0, and R<->Python gradient parity stays exact (max fitted diff ~5e-13, logLik ~1e-11). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
…nd LM Speed up initial="gradient" by moving the whole solve into the shared C++ core as adamCore::gradientSolve (exposed to R via the Rcpp module and to Python via pybind11), replacing the R/Python-level probe loops that duplicated the model matrices into cubes and crossed the language boundary on every cost-function evaluation. - Additive ETS (B+C): the residuals are affine in the initial state, so the design matrix is propagated analytically alongside a single forward pass (sensitivities of every profile cell follow the same linear maps: w'v, F*v, g*e, including the head walk) and the problem is solved by one pivoted-QR least squares (olsCore). One C++ call per evaluation instead of a (nFree+1)-slice reapply plus a separate OLS call. - Multiplicative/mixed (D): finite-difference Gauss-Newton with a backtracking line search, plus a Levenberg-Marquardt fallback that solves the damped normal equations [J; sqrt(lambda) I] when the raw step is ill-conditioned (previously the line search failed outright and the solve stalled at the seed), and a diminishing-returns stop (relative SSE improvement < 1e-4). The solve is deliberately stateless - a cross-evaluation warm start was tried and rejected because it makes the objective history-dependent, which both misleads the optimiser and breaks exact R/Python parity. - The R and Python wrappers become thin: they build the 0/1 probe basis (level/trend spanning the head columns, one column per seasonal cell) and call the shared C++ method; out-of-scope models still fall back to backcasting. Steady-state timings (72 obs, m=12): AAA 0.077s -> 0.025s (~3x, now faster than backcasting's 0.035s); MAM 0.37s -> 0.27s with logLik improved from -181.53 to -178.64; MMM slower (0.50s -> 1.11s) but logLik improves from -181.27 to -178.05 (better than backcasting) - the old Gauss-Newton stalled at the seed, the LM fallback actually converges. The bug-report case (AAA m=12, persistence 0.6/0.15/0.35) keeps its healthy deficit (-7.9 sigma^2 vs backcasting's +3547). Additive results are unchanged (analytic design equals the probe design exactly for affine models); multiplicative results improve. R/Python parity verified exact across ETS(A/M, N/A/Ad/M, N/A/M) at fixed and optimised persistence (max fitted diff ~5e-13, logLik diff ~1e-10). R suite 461/0, Python suite 725/0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
Replace the finite-difference Jacobian in adamCore::gradientSolve's nonlinear
branch (multiplicative/mixed ETS) with exact initial-state sensitivities
propagated by the chain rule in a single forward pass.
New header src/headers/adamGradient.h provides stand-alone derivative
companions to the pure-ETS maps (the originals are untouched):
- adamWvalueJac: d(yhat)/d(states) for every measurement branch (linear,
power-product, and the mixed (w'v)*prod(s^w) forms);
- adamFvalueJac: d(F(v))/d(states) (the matrix itself for T in {N,A};
geometric J_ij = F_ij v'_i / v_j for T='M');
- adamGvalueJac: d(update)/d(states), d/d(error) and d/d(yhat), mirroring the
full branch structure of adamGvalue for BOTH the ADAM maths (ets="adam")
and the conventional maths (ets="conventional", the default) - the two have
different update functions for multiplicative components.
New private adamCore::gradientPassJacobian runs one forward pass carrying the
(nCells x nFree) sensitivity matrix through the head walk, the mixed-model
clamp (derivative zeroed on the flat spot), the occurrence scaling and both
error types, producing residuals and the full Jacobian together.
gradientSolve() gains an `analytic` flag (default TRUE from both wrappers);
the finite-difference path is retained as fallback and validation oracle.
Validation: across all 16 non-additive ETS combos x both ets variants the
analytic and FD solves converge to the same SSE (the single flagged case,
MNA conventional, was checked against a central-difference reference: the
analytic one-iteration solve matches it to 9e-6 while forward FD is off by
3e-2 - the analytic Jacobian is more accurate than the oracle). R/Python
parity exact (13 checks, fitted diff ~5e-13, logLik diff ~1e-10).
Speed: MAM on AirPassengers (the reported case) drops from 8.67s to 0.53s
(~16x) and its Gamma loss improves from 469.32 to 466.68, now beating
backcasting's 467.12; MMM on the 72-obs test series drops 1.11s -> 0.07s.
Additive models are untouched (they already used the analytic design).
Also bump versions: R package 4.5.0 -> 4.5.1 (new NEWS section), Python
package 1.0.6 -> 1.0.7.
R suite 461/0, Python suite 725/0, gradient tests 14/14, ruff+mypy clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
plan-gradient-arbitrary-loss.md: implementation briefing for extending adamCore::gradientSolve to profile the actual estimation loss over the initial states instead of the one-step SSE surrogate. Built on the analytic Jacobian (9b7da03): additive models get an IRLS loop on the exact affine residual surrogate (zero extra model passes), multiplicative models get a weighted Gauss-Newton step (same pass count), and additive multistep losses (MSEh/TMSE/MSCE/GTMSE/GPL) get exact affine designs. Includes the rho/weight table per distribution, interface (lossType enum shared by both wrappers), validation protocol and order of work. Custom user losses fall back to backcasting by design. df-fixes.md: plan for removing the fractional backcasting df heuristic and implementing the efficient df across adam/CES/GUM/SSARIMA/SPARMA and the occurrence models, based on the theory from the backcasting paper. Both plan files are in .Rbuildignore (repo root stays CRAN-clean). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
Implements plan-gradient-arbitrary-loss.md: initial="gradient" now profiles the actual estimation loss over the initial states instead of the one-step SSE surrogate, in shared C++ for exact R/Python parity. - adamGradient.h: rho/psi/psi'' and concentrated-scale helpers per loss (MAE, HAM, dgnorm; exact dlnorm/dgamma/dinvgauss rho for E="M"), IRLS sweeps on the additive affine surrogate (zero-residual rows dropped, no epsilon floors), and the additive multistep solvers: MSEh/TMSE/MSCE one-shot QR, GTMSE/GPL monotone majorise-minimise sweeps on the affine multistep design replicated from the ferrors() recursion. - adamCore::gradientSolve gains lossType/lossParams; the Gauss-Newton branch gets loss-aware steps (IRLS weights for power losses, Newton weights for the likelihood ones) with the rho-based line search and Levenberg-Marquardt acceptance. 'S' (SSE) stays bit-identical. - Mirror-identical loss mapping tables in R/adam-gradient.R and core/utils/gradient.py; custom loss functions fall back to backcasting with a one-time message. All dispatcher call sites (CF, logLik, estimator, preparator) pass the loss context in both languages. - Python bugfixes exposed by the parity grid: scipy invgauss was parameterised inverted in calculate_likelihood() and the QQ plot (two orders of magnitude off R for every dinvgauss fit); the gradient solve leaked solved initials into the shared creator matrices via np.asfortranarray aliasing, so CF/logLik sites now copy and preparator() preserves the pristine seed head across invocations. Validation: per-loss brute-force oracle at fixed persistence (multistep losses match exactly, others within 2e-4); R/Python parity across the 15-case loss grid to 12+ digits with bit-identical direct C++ outputs; R suite 485 passing, Python 730 passing, ruff/mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
Implements plan-gradient-om.md. Previously om() accepted initial="gradient"
but it was a silent no-op: the cost function and refit gated the backcast
flag on c("complete","backcasting") only, so a gradient om fit neither
backcast nor solved and left the initials at the creator seed — measurably
worse than backcasting. Now the om initials are solved by profiling the
occurrence loss over them, via the shared C++ adamCore::gradientSolve.
- adamGradient.h: occurrenceLinkJac (p and dp/dyhat per occurrence type)
and occurrenceErrorJac (d(occurrenceError)/dyhat for the state-update
channel); Bernoulli rho 'B' = -log(1-|r|) in the probability residual
r = o - p, with its IRLS/Newton step rows (always defined, psi' > 0).
- adamCore.h: gradientPassOccurrence and gradientPassJacobianOccurrence run
the occurrence recursion forward, producing the probability residuals and
their analytic Jacobian; gradientSolve gains an O channel (occurrence
always takes the Gauss-Newton branch — the link is nonlinear even for
E='A'; a saturated seed returns failure so the caller backcasts).
- R/adam-gradient.R + python .../gradient.py: mirror-identical om loss
mapper (likelihood->'B', MSE->'S', MAE->'A', HAM->'H', custom->NULL),
o_type threaded through the dispatcher; gradient joins the backcast
fallback group, which alone fixes the no-op.
- R/om.R + python om_cost.py / om.py: both fit sites route through the
dispatcher with oType=occurrenceChar; changeOrigin / model-reuse gates
add "gradient".
Occurrence "fixed" (no estimated initials), custom losses, and om with
ARIMA/xreg fall back to backcasting. omg() is out of scope (coupled solve).
Validation: om gradient reaches optimal-quality logLik at backcasting cost
(MNN odds-ratio, -75.00 gradient vs -75.00 optimal vs -80.26 backcasting,
all three occurrence types); analytic occurrence Jacobian matches finite
differences; R/Python fitted parity to 1e-12 across occurrence type x loss;
gradient MSE solve verified against a manual level grid. R suite 485,
Python 732, ruff/mypy clean.
Note: OM.loglik returns -inf at pathological fixed persistence in Python
(backcasting hits it too) — a pre-existing OM likelihood-eval issue,
unrelated to this change, not triggered on the estimated-persistence path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
OM returned logLik=-inf whenever the parameter vector was empty — fixed
persistence combined with backcasting or gradient initials, which are
solved inside the fit rather than carried in B. nlopt was built with zero
dimensions, never evaluated the objective, and last_optimum_value()
returned +inf, so CF_value was never computed even though the fitted
values were finite and correct. The v0 was fine; the cost simply was
never evaluated.
- Add the empty-B guard: when len(B)==0 evaluate the cost once directly
instead of calling nlopt (mirrors R/om.R's length(B)==0 branch).
- Copy mat_vt in om_cf so the gradient dispatcher's in-place head write
does not leak the solved initials into the shared creator matrices that
the om preparator later re-seeds from (same leak class fixed for ADAM).
OM likelihood logLik now matches R to machine precision across occurrence
types {direct, odds-ratio, inverse-odds-ratio} at both fixed and estimated
persistence. Regression test added.
Note: for the non-likelihood om losses (MSE/MAE/HAM) R reports logLik as
the Bernoulli likelihood of the fitted probabilities floored with
pmax(f,1e-15), whereas Python reports -CF (the raw fitting loss). When the
loss optimum is probabilistically infeasible (p<0, e.g. odds-ratio MAE)
Python surfaces it as -inf rather than flooring it — consistent with the
project's "surface, don't hide the signal" rule. Not changed here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
The occurrence model's reported log-likelihood must be the Bernoulli
likelihood of the fitted probabilities, not the fitting cost. Previously,
for the fit-only losses (MSE/MAE/HAM) om's logLik was -CFValue = minus the
MSE/MAE itself, which is not a likelihood, so logLik and the information
criteria for those fits were meaningless. And the empty-B / fixed-occurrence
paths computed the Bernoulli with a pmax(., 1e-15) floor.
- R (R/om.R, omFinalFit): compute the Bernoulli logLik from the fitted
probabilities whenever loss != "likelihood" (and, as before, when the
estimator deferred it), with lossValue still the minimised loss. Remove
the pmax(., 1e-15) floor — an infeasible fitted probability now surfaces
as NaN/-Inf instead of being clipped.
- Python (core/om.py): same in om_preparator (recompute the Bernoulli and
refresh the IC); drop the 1e-15 floors in om_preparator and _fit_fixed.
loss="likelihood" is unchanged (its cost already is the Bernoulli).
om logLik now matches R to 1e-13 across occurrence types {direct,
odds-ratio, inverse-odds-ratio} x loss {likelihood, MSE} at both fixed and
estimated persistence. An infeasible MAE optimum (p outside [0,1]) yields a
NaN logLik in both languages — the honest, unclipped signal. R suite 485,
Python 733, ruff/mypy clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
The reported lossValue is now the fitting loss evaluated on the final
fitted probabilities — e.g. mean(abs(ot - p)) for MAE — computed without
the infeasibility guard. The guard (return 1e300 when a probability
leaves [0,1]) exists only to steer the optimiser; it was leaking into the
reported loss whenever the loss optimum itself was infeasible, so a fit
whose true MAE was 0.44 reported lossValue = 1e300 (or 0 at fixed
persistence). initial="gradient" with a fit-only loss (MSE/MAE/HAM) minimises
that loss in C++, so its lossValue must be the loss it actually achieved.
logLik stays the Bernoulli likelihood of the fitted probabilities and is
allowed to diverge from lossValue: when the loss optimum is infeasible
(p<0), the Bernoulli is NaN/-Inf — surfaced, not clipped — while lossValue
is the finite loss. loss="likelihood" is unchanged (lossValue = -logLik);
LASSO/RIDGE/custom keep the optimiser's value.
R (omFinalFit) and Python (om_preparator) both recompute lossValue this
way. Parity verified on lossValue and logLik across occurrence types
{direct, odds-ratio, inverse-odds-ratio} x loss {likelihood, MSE, MAE} x
{fixed, estimated} persistence (NaN-aware, all 18 cases match). R suite
485, Python 734, ruff/mypy clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
Two defects made ADAM's reported log-likelihood wrong for non-likelihood losses, diverging from R: 1. logLik = -loss_value. log_Lik_ADAM remapped the loss to "likelihood" and set the loss-implied distribution (MSE<->dnorm, MAE<->dlaplace, HAM<->ds), but wrote them to general["loss_new"]/["distribution_new"] while CF reads general["loss"]/["distribution_new"]. So CF ignored the remap and returned the raw loss, making logLik = -MSE (~-3.6 instead of the Normal likelihood ~-148). Now the remap lands on the keys CF reads (via a copied general dict), mirroring R/adam.R:1004-1012. An explicit distribution is ignored for MSE/MAE/HAM, exactly as R does. 2. ds / dls used scipy t(df=2) as a stand-in for the S distribution, which it is not. distribution="ds" and loss="HAM" therefore reported a wrong logLik. Replaced with the exact S log-density -log(4 s^2) - sqrt(|x-mu|)/s (log-scale variant for dls), verified against greybox::ds. adam logLik and lossValue now match R to ~1e-10 across loss x distribution (likelihood/MSE/MAE/HAM x dnorm/dlaplace/ds/dgnorm, additive; dlnorm/ dgamma/dinvgauss, multiplicative), including every misaligned loss/ distribution pair. Regression tests added. Python suite 735, ruff/mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
adam reported the log-likelihood (and IC) under the loss-implied distribution even when the user explicitly selected a different one: adam(y, loss="MAE", distribution="dnorm") reported the Laplace likelihood, silently ignoring the requested Normal. The distribution is already resolved upstream (default -> loss-implied, explicit -> the user's choice), but logLikADAM / log_Lik_ADAM then re-overrode it back to the loss-implied one for MSE/MAE/HAM, clobbering the explicit selection. Drop that re-override in both languages (R/adam.R logLikADAM, Python log_Lik_ADAM); use the already-resolved distribution, keeping only the loss -> "likelihood" remap so the concentrated likelihood is computed under the selected distribution. distribution="default" is unchanged (still loss-implied); the loss still drives estimation — only the reporting distribution changed. logLik and lossValue match R to ~1e-10 across loss x distribution incl. every misaligned pair (MSE+dlaplace, MAE+dnorm, HAM+dnorm, ...). Test updated to assert the explicit distribution is honoured. R suite 485, Python 738, ruff/mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRdQhv3c5fgxm7GaequJtv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What issue does this PR fix?
Language
Package version
v4.5.1
Development environment
Linux
AI assistance
If yes — how was AI used?
Claude Code