Skip to content

Commit b8885e4

Browse files
docs(adr): record accepted mirt fit-search decisions
Add docs/adr/ for decisions already true on develop: mirt delegation, Zh / S-X2 / Rasch infit gates, AICc and DIC boundaries, and local default as an independent R package. Point ARCHITECTURE.md, README, and docs/papers/README.md at the new index. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>
1 parent 5128d48 commit b8885e4

11 files changed

Lines changed: 265 additions & 1 deletion

ARCHITECTURE.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# kaefa Architecture
22

3-
Last updated: 2026-02-14
3+
Last updated: 2026-08-25
44

55
## Purpose
66

77
`kaefa` is an R package for automated exploratory factor analysis (AEFA).
8+
Accepted decisions already true on this branch are recorded in `docs/adr/`.
89
It provides:
910

1011
- core AEFA execution (`aefa`, `engineAEFA`),

README.Rmd

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ knitr::opts_chunk$set(
1717

1818
The goal of kaefa is to improve researchers' ability to identify unexplained factor structures in complex, cross-classified multilevel data in R. It uses an automated exploratory factor analysis (aefa) framework.
1919

20+
Accepted fit-search decisions are recorded in [`docs/adr/`](docs/adr/).
21+
2022
## Algorithm
2123

2224
The automated exploratory factor analysis (aefa) framework implements a **greedy search algorithm** to efficiently explore the model space and find improved model configurations. The algorithm iteratively:

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ unexplained factor structures in complex, cross-classified multilevel
1010
data in R. It uses an automated exploratory factor analysis (aefa)
1111
framework.
1212

13+
Accepted fit-search decisions are recorded in [`docs/adr/`](docs/adr/).
14+
1315
## Algorithm
1416

1517
The automated exploratory factor analysis (aefa) framework implements a
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# ADR 0001: Delegate IRT/EFA estimation and item-fit to mirt
2+
3+
- Status: Accepted
4+
- Date: 2026-08-25
5+
6+
## Context
7+
8+
kaefa searches unexplained factor structures by fitting candidate IRT/EFA
9+
models and pruning poorly fitting items. The heavy estimation (MML-EM,
10+
rotation, item-fit statistics) is already implemented and validated in
11+
`mirt`. Re-implementing those internals in kaefa would duplicate a
12+
maintained package and drift from its published definitions.
13+
14+
## Decision
15+
16+
IRT/EFA estimation and item-fit statistics are delegated to `mirt`
17+
(Chalmers, 2012). kaefa owns the search loop and the decision rules on
18+
top: `engineAEFA()` estimates candidates through `.mirt` / `.mixedmirt`
19+
wrappers, `evaluateItemFit()` calls `mirt::itemfit()`, and `aefa()`
20+
selects and prunes from that output.
21+
22+
kaefa does not re-implement `P(theta)`, the MML-EM E-/M-step, `S-X2`,
23+
`infit`, or `outfit`. Those remain `mirt`'s responsibility.
24+
25+
## Consequences
26+
27+
- Fit numbers consumed by the search (`Zh`, `S-X2`, `infit`/`outfit`)
28+
come from `mirt` and stay subject to `mirt`'s validation.
29+
- Package-local rules (cutoffs, AICc reconstruction, the DIC boundary)
30+
are recorded in later ADRs and pinned in `docs/papers/README.md`.
31+
- A `mirt` version change can change numeric output without a kaefa
32+
formula change.
33+
34+
## References
35+
36+
Chalmers, R. P. (2012). mirt: A multidimensional item response theory
37+
package for the R environment. *Journal of Statistical Software, 48*(6),
38+
1–29. https://doi.org/10.18637/jss.v048.i06
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# ADR 0002: Standardised log-likelihood misfit Zh decision rule
2+
3+
- Status: Accepted
4+
- Date: 2026-08-25
5+
6+
## Context
7+
8+
The search must decide when an item is misfitting so it can be removed
9+
and the candidate re-estimated. `mirt::itemfit(fit_stats = "Zh")`
10+
supplies the standardised log-likelihood statistic of Drasgow, Levine,
11+
and Williams (1985). kaefa applies a local cutoff; it does not
12+
re-implement `mirt`'s `Zh` internals.
13+
14+
## Decision
15+
16+
The published statistic, pinned in `docs/papers/README.md`, is
17+
18+
Zh = (l0 - E[l0]) / sqrt(Var[l0])
19+
20+
An item is flagged as misfitting when
21+
22+
Zh + qnorm(0.975) / sqrt(n) < qnorm(fitIndicesCutOff / 2)
23+
24+
That is a one-sided lower-tail test at level `fitIndicesCutOff / 2` with
25+
the small-sample correction `qnorm(0.975) / sqrt(n)`. With the default
26+
`fitIndicesCutOff = 0.005` the threshold is `qnorm(0.0025) = -2.807`.
27+
28+
The arithmetic lives in `.zhMisfitCount()` in `R/kaefa.R` and is applied
29+
at the three search sites (rotation scan, best-candidate check, and the
30+
final `ZhCond` gate). Those three sites must stay identical.
31+
32+
## Consequences
33+
34+
- Drift between the three sites changes which rotations and items the
35+
search treats as misfitting. A 2019 debug commit dropped `/sqrt(n)`
36+
from one site; the shared helper exists to prevent that class of
37+
inconsistency.
38+
- `Zh` itself remains a `mirt` computation. Audits compare the local
39+
cutoff to the pinned equation, not a re-derived `Zh`.
40+
41+
## References
42+
43+
Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness
44+
measurement with polychotomous item response models and standardized
45+
indices. *British Journal of Mathematical and Statistical Psychology,
46+
38*(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# ADR 0003: S-X2 and RMSEA.S_X2 misfit gates
2+
3+
- Status: Accepted
4+
- Date: 2026-08-25
5+
6+
## Context
7+
8+
`Zh` is not the only item-fit gate in the search. When `mirt::itemfit()`
9+
returns Orlando and Thissen (2000) `S-X2` columns, kaefa also uses the
10+
limited-information p-value and the RMSEA computed from that statistic.
11+
12+
## Decision
13+
14+
kaefa flags misfit from `S-X2` when either of these holds
15+
(`R/kaefa.R`):
16+
17+
- `p.S_X2 < fitIndicesCutOff`
18+
- `round(RMSEA.S_X2, 2) >= .05`
19+
20+
`S-X2` is computed by `mirt` (`fit_stats = "S_X2"`). The 0.05 RMSEA
21+
close-fit threshold for limited-information item fit follows
22+
Maydeu-Olivares and Joe (2014). kaefa does not re-implement `S-X2`.
23+
24+
## Consequences
25+
26+
- Items can be pruned for a significant `S-X2` p-value or for rounded
27+
RMSEA at or above 0.05 even when `Zh` is acceptable.
28+
- Rounding RMSEA to two decimals is part of the accepted gate, not an
29+
informal display choice.
30+
- The statistic and its RMSEA remain `mirt` output; only the gates are
31+
package-local.
32+
33+
## References
34+
35+
Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for
36+
dichotomous item response theory models. *Applied Psychological
37+
Measurement, 24*(1), 50–64. https://doi.org/10.1177/01466216000241003
38+
39+
Maydeu-Olivares, A., & Joe, H. (2014). Assessing approximate fit in
40+
categorical data analysis. *Multivariate Behavioral Research, 49*(4),
41+
305–328. https://doi.org/10.1080/00273171.2014.911075
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# ADR 0004: infit and outfit only for Rasch unidimensional models
2+
3+
- Status: Accepted
4+
- Date: 2026-08-25
5+
6+
## Context
7+
8+
Wright and Masters (1982) mean-square `infit` and `outfit` are Rasch
9+
fit statistics. `mirt::itemfit(fit_stats = "infit")` can return them,
10+
but they are not defined for the general multidimensional, non-Rasch
11+
candidates `engineAEFA()` explores.
12+
13+
## Decision
14+
15+
`evaluateItemFit()` requests `infit` (which also returns `outfit`) only
16+
when the fitted model has at least one Rasch item type and
17+
`nfact == 1`. Other models skip that `itemfit` call.
18+
19+
kaefa does not re-implement the mean-square residuals. Computation stays
20+
in `mirt`.
21+
22+
## Consequences
23+
24+
- Multidimensional or non-Rasch candidates are judged by `Zh` and, when
25+
available, `S-X2` / `RMSEA.S_X2`, not by `infit`/`outfit`.
26+
- A later request for `infit` on a 2PL or multifactor model would be a
27+
new decision, not an extension of this one.
28+
29+
## References
30+
31+
Wright, B. D., & Masters, G. N. (1982). *Rating scale analysis*. MESA
32+
Press.

docs/adr/0005-aicc-dic-criteria.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# ADR 0005: AICc reconstruction and the DIC boundary
2+
3+
- Status: Accepted
4+
- Date: 2026-08-25
5+
6+
## Context
7+
8+
`aefa()` selects among candidates by an information criterion. Current
9+
`mirt` fits supply `AIC` and `logLik` but do not consistently expose
10+
`AICc`. The maximum-likelihood / MAP models from current `mirt`
11+
versions also do not expose the posterior deviance quantities needed to
12+
reconstruct DIC.
13+
14+
## Decision
15+
16+
AICc is reconstructed only from the Hurvich and Tsai (1989)
17+
small-sample correction. When the fit does not already supply a finite
18+
`AICc`, kaefa recovers `k = (AIC + 2 * logLik) / 2` and applies
19+
20+
AICc = AIC + 2 * k * (k + 1) / (n - k - 1)
21+
22+
The statistic is undefined when `n <= k + 1`; kaefa reports that reason
23+
rather than returning a fabricated finite score.
24+
25+
DIC is accepted only when the fitted model supplies a finite posterior
26+
DIC (Spiegelhalter et al., 2002). DIC is never reconstructed from AIC
27+
and is never relabelled from AIC. `CAIC` is not treated as an alias for
28+
`AICc`.
29+
30+
## Consequences
31+
32+
- Default search can use AIC, AICc, BIC, or saBIC without a posterior
33+
sample.
34+
- Requesting DIC on an ML/MAP `mirt` fit that lacks a finite DIC is an
35+
error, not a silent fall-back to AIC.
36+
- Sequential DIF selection also refuses to substitute AIC for DIC.
37+
38+
## References
39+
40+
Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model
41+
selection in small samples. *Biometrika, 76*(2), 297–307.
42+
https://doi.org/10.1093/biomet/76.2.297
43+
44+
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A.
45+
(2002). Bayesian measures of model complexity and fit. *Journal of the
46+
Royal Statistical Society: Series B, 64*(4), 583–639.
47+
https://doi.org/10.1111/1467-9868.00353
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# ADR 0006: Local default and independent R package
2+
3+
- Status: Accepted
4+
- Date: 2026-08-25
5+
6+
## Context
7+
8+
`ARCHITECTURE.md` describes kaefa as an R package whose runtime path is
9+
`aefa()` / `engineAEFA()`, with `aefaInit()` available for optional
10+
worker setup. In the ContextualWisdomLab ecosystem kaefa is a leaf:
11+
other components may call it, but it must run without those components.
12+
13+
## Decision
14+
15+
Local execution is the default. `aefaInit()` remote workers are
16+
optional. Hosts may be preconfigured with `options(kaefaServers = ...)`,
17+
but nothing in the search requires a remote node.
18+
19+
kaefa remains an independent R package (MSA leaf): it is runnable
20+
without naruon and callable as a dependency. This repository does not
21+
add sibling-repo checkouts or git submodules for ecosystem components.
22+
23+
## Consequences
24+
25+
- Users can run `aefa()` on a local workstation with the package
26+
installed; remote SSH workers are an opt-in.
27+
- Security-sensitive values (keys, tokens) stay out of git history, as
28+
already stated in `ARCHITECTURE.md`.
29+
- Ecosystem wiring to naruon or sibling repositories is out of scope
30+
for this package's default path.
31+
32+
## References
33+
34+
No additional paper. This decision is already stated in
35+
`ARCHITECTURE.md` (local default; optional `aefaInit()`) and in the
36+
package `DESCRIPTION` (standalone R package). The allowed citation list
37+
for this ADR set does not include an execution-topology paper.

docs/adr/README.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Architecture decision records
2+
3+
This directory records accepted decisions that are already true on `develop`.
4+
It does not invent product behaviour. Formula provenance and the pinned
5+
equations remain in [`docs/papers/README.md`](../papers/README.md).
6+
7+
Status dates are 2026-08-25. Citations were live-checked on that date; see
8+
each ADR's References for the required locator.
9+
10+
| ADR | Decision | Status |
11+
| --- | --- | --- |
12+
| [0001](0001-mirt-estimation-delegation.md) | IRT/EFA estimation and item-fit statistics are delegated to `mirt`; kaefa owns search and decision rules | Accepted |
13+
| [0002](0002-zh-misfit-decision-rule.md) | Standardised log-likelihood misfit `Zh` decision rule | Accepted |
14+
| [0003](0003-sx2-rmsea-misfit-gates.md) | `S-X2` and `RMSEA.S_X2` misfit gates | Accepted |
15+
| [0004](0004-rasch-infit-outfit.md) | `infit`/`outfit` requested only for Rasch unidimensional models | Accepted |
16+
| [0005](0005-aicc-dic-criteria.md) | AICc reconstructed via Hurvich and Tsai; DIC accepted only when `mirt` supplies a finite posterior DIC | Accepted |
17+
| [0006](0006-local-default-independent-package.md) | Local execution is the default; kaefa remains an independent R package | Accepted |

0 commit comments

Comments
 (0)