Skip to content

Adjust ftest #628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andreasnoack wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Adjust ftest #628

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6e50def
Adjust ftest
andreasnoack Dec 28, 2025
9289f0d
Rename ftest column dof→df and restore row numbering
andreasnoack Apr 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/src/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,12 +244,12 @@ julia> ols_sq = lm(@formula(y ~ x + x^2), data);

julia> ftest(ols_lin, ols_sq)
F-test: 2 models fitted on 50 observations
────────────────────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────────────────────
[1] 3 1731979.2266 0.9399
[2] 4 1 40.7581 1731938.4685 1.0000 0.0601 1997177.0357 <1e-99
────────────────────────────────────────────────────────────────────────────────
──────────────────────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────────────────────
[1] 1731979.2266 48
[2] 40.7581 47 1731938.4685 1 1997177.0357 <1e-99
──────────────────────────────────────────────────────────────
```

## Methods applied to fitted models
Expand Down
98 changes: 49 additions & 49 deletions src/ftest.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,10 @@ struct SingleFTestResult
pval::Float64
end

mutable struct FTestResult{N}
struct FTestResult{N}
nobs::Int
ssr::NTuple{N,Float64}
dof::NTuple{N,Int}
r2::NTuple{N,Float64}
dof::NTuple{N,Float64}

@nalimilan nalimilan Apr 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better rename the field, if users access it it will be clearer to get an error than silently get the wrong value.

Suggested change
dof::NTuple{N,Float64}
resdof::NTuple{N,Float64}

fstat::NTuple{N,Float64}
pval::NTuple{N,Float64}
end
Expand Down Expand Up @@ -79,10 +78,9 @@ For each sequential pair of linear models in `mod...`, perform an F-test to dete
the one model fits significantly better than the other. Models must have been fitted
on the same data, and be nested either in forward or backward direction.

A table is returned containing consumed degrees of freedom (DOF),
absolute difference in DOF from the preceding model, sum of squared residuals (SSR),
absolute difference in SSR from the preceding model, R², absolute difference in R²
from the preceding model, and F-statistic and p-value for the comparison
A table is returned containing residual degrees of freedom (df),
absolute difference in df from the preceding model, sum of squared residuals (SSR),
absolute difference in SSR from the preceding model, and F-statistic and p-value for the comparison
between the two models.

!!! note
Expand Down Expand Up @@ -114,22 +112,22 @@ julia> bigmodel = lm(@formula(Result ~ 1 + Treatment + Other), dat);

julia> ftest(nullmodel, model)
F-test: 2 models fitted on 12 observations
────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────
[1] 2 3.2292 0.0000
[2] 3 1 0.1283 3.1008 0.9603 0.9603 241.6234 <1e-07
────────────────────────────────────────────────────────────────
──────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────
[1] 3.2292 11
[2] 0.1283 10 3.1008 1 241.6234 <1e-07
──────────────────────────────────────────────

julia> ftest(nullmodel, model, bigmodel)
F-test: 3 models fitted on 12 observations
────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────
[1] 2 3.2292 0.0000
[2] 3 1 0.1283 3.1008 0.9603 0.9603 241.6234 <1e-07
[3] 5 2 0.1017 0.0266 0.9685 0.0082 1.0456 0.3950
────────────────────────────────────────────────────────────────
──────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────
[1] 3.2292 11
[2] 0.1283 10 3.1008 1 241.6234 <1e-07
[3] 0.1017 8 0.0266 2 1.0456 0.3950
──────────────────────────────────────────────
```
"""
function ftest(mods::LinearModel...; atol::Real=0.0)
Expand All @@ -156,23 +154,22 @@ function ftest(mods::LinearModel...; atol::Real=0.0)

SSR = deviance.(mods)

df = dof.(mods)
df = dof_residual.(mods)
Δdf = abs.(_diff(df))
dfr = Int.(dof_residual.(mods))
MSR1 = _diffn(SSR) ./ Δdf
MSR2 = (SSR ./ dfr)
MSR2 = (SSR ./ df)
if forward
MSR2 = MSR2[2:end]
dfr_big = dfr[2:end]
dfr_big = df[2:end]
else
MSR2 = MSR2[1:(end - 1)]
dfr_big = dfr[1:(end - 1)]
dfr_big = df[1:(end - 1)]
end

fstat = abs.((NaN, (MSR1 ./ MSR2)...))
pval = (NaN, ccdf.(FDist.(Δdf, dfr_big), fstat[2:end])...)

return FTestResult(Int(nobs(mods[1])), SSR, df, r2.(mods), fstat, pval)
return FTestResult(Int(nobs(mods[1])), SSR, float.(df), fstat, pval)
end

function show(io::IO, ftr::SingleFTestResult)
Expand All @@ -182,37 +179,36 @@ function show(io::IO, ftr::SingleFTestResult)
return print(io, "p-value: ", PValue(ftr.pval))
end

function show(io::IO, ftr::FTestResult{N}) where {N}
function show(io::IO, ::MIME"text/plain", ftr::FTestResult{N}) where {N}
Δdof = abs.(_diff(ftr.dof))
Δssr = abs.(_diff(ftr.ssr))
ΔR² = abs.(_diff(ftr.r2))

nc = 9
nc = 7
nr = N
outrows = Matrix{String}(undef, nr + 1, nc)

outrows[1, :] = ["", "DOF", "ΔDOF", "SSR", "ΔSSR",
"R²", "ΔR²", "F*", "p(>F)"]
outrows[1, :] = ["", "SSR", "df", "ΔSSR", "Δdf", "F*", "p(>F)"]

# get rid of negative zero -- doesn't matter mathematically,
# but messes up doctests and various other things
# cf. Issue #461
r2vals = [replace(@sprintf("%.4f", val), "-0.0000" => "0.0000") for val in ftr.r2]

outrows[2, :] = ["[1]", @sprintf("%.0d", ftr.dof[1]), " ",
@sprintf("%.4f", ftr.ssr[1]), " ",
r2vals[1], " ", " ", " "]
outrows[2, :] = ["[1]",
@sprintf("%.4f", ftr.ssr[1]),
@sprintf("%.0d", ftr.dof[1]),
" ",
" ",
" ",
" "]

for i in 2:nr
outrows[i + 1, :] = ["[$i]",
@sprintf("%.0d", ftr.dof[i]), @sprintf("%.0d", Δdof[i - 1]),
@sprintf("%.4f", ftr.ssr[i]), @sprintf("%.4f", Δssr[i - 1]),
r2vals[i], @sprintf("%.4f", ΔR²[i - 1]),
@sprintf("%.4f", ftr.fstat[i]), string(PValue(ftr.pval[i]))]
@sprintf("%.4f", ftr.ssr[i]),
@sprintf("%.0d", ftr.dof[i]),
@sprintf("%.4f", Δssr[i - 1]),
@sprintf("%.0d", Δdof[i - 1]),
@sprintf("%.4f", ftr.fstat[i]),
string(PValue(ftr.pval[i]))]
end
colwidths = length.(outrows)
max_colwidths = [maximum(view(colwidths, :, i)) for i in 1:nc]
totwidth = sum(max_colwidths) + 2 * 8
totwidth = sum(max_colwidths) + 2 * (nc - 1)

println(io, "F-test: $N models fitted on $(ftr.nobs) observations")
println(io, '─'^totwidth)
Expand All @@ -221,14 +217,18 @@ function show(io::IO, ftr::FTestResult{N}) where {N}
for c in 1:nc
cur_cell = outrows[r, c]
cur_cell_len = length(cur_cell)
total_pad = max_colwidths[c] - cur_cell_len

padding = " "^(max_colwidths[c] - cur_cell_len)
if c > 1
padding = " " * padding
print(io, " ")
end
if r == 1
left_pad = total_pad ÷ 2
right_pad = total_pad - left_pad
print(io, " "^left_pad, cur_cell, " "^right_pad)
else
print(io, " "^total_pad, cur_cell)
end

print(io, padding)
print(io, cur_cell)
end
print(io, "\n")
r == 1 && println(io, '─'^totwidth)
Expand Down
60 changes: 30 additions & 30 deletions test/runtests.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1607,55 +1607,55 @@ end
ft1a = ftest(mod, nullmod)
@test isnan(ft1a.pval[1])
@test ft1a.pval[2] ≈ 2.481215056713184e-8
@test sprint(show, ft1a) == """
@test sprint(show, "text/plain", ft1a) == """
F-test: 2 models fitted on 12 observations
────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────
[1] 3 0.1283 0.9603
[2] 2 1 3.2292 3.1008 0.0000 0.9603 241.6234 <1e-07
────────────────────────────────────────────────────────────────"""
──────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────
[1] 0.1283 10
[2] 3.2292 11 3.1008 1 241.6234 <1e-07
──────────────────────────────────────────────"""

ft1b = ftest(nullmod, mod)
@test isnan(ft1b.pval[1])
@test ft1b.pval[2] ≈ 2.481215056713184e-8
@test sprint(show, ft1b) == """
@test sprint(show, "text/plain", ft1b) == """
F-test: 2 models fitted on 12 observations
────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────
[1] 2 3.2292 0.0000
[2] 3 1 0.1283 3.1008 0.9603 0.9603 241.6234 <1e-07
────────────────────────────────────────────────────────────────"""
──────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────
[1] 3.2292 11
[2] 0.1283 10 3.1008 1 241.6234 <1e-07
──────────────────────────────────────────────"""

bigmod = lm(@formula(Result ~ Treatment + Other), d)
ft2a = ftest(nullmod, mod, bigmod)
@test isnan(ft2a.pval[1])
@test ft2a.pval[2] ≈ 2.481215056713184e-8
@test ft2a.pval[3] ≈ 0.3949973540194818
@test sprint(show, ft2a) == """
@test sprint(show, "text/plain", ft2a) == """
F-test: 3 models fitted on 12 observations
────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────
[1] 2 3.2292 0.0000
[2] 3 1 0.1283 3.1008 0.9603 0.9603 241.6234 <1e-07
[3] 5 2 0.1017 0.0266 0.9685 0.0082 1.0456 0.3950
────────────────────────────────────────────────────────────────"""
──────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────
[1] 3.2292 11
[2] 0.1283 10 3.1008 1 241.6234 <1e-07
[3] 0.1017 8 0.0266 2 1.0456 0.3950
──────────────────────────────────────────────"""

ft2b = ftest(bigmod, mod, nullmod)
@test isnan(ft2b.pval[1])
@test ft2b.pval[2] ≈ 0.3949973540194818
@test ft2b.pval[3] ≈ 2.481215056713184e-8
@test sprint(show, ft2b) == """
@test sprint(show, "text/plain", ft2b) == """
F-test: 3 models fitted on 12 observations
────────────────────────────────────────────────────────────────
DOF ΔDOF SSR ΔSSR R² ΔR² F* p(>F)
────────────────────────────────────────────────────────────────
[1] 5 0.1017 0.9685
[2] 3 2 0.1283 0.0266 0.9603 0.0082 1.0456 0.3950
[3] 2 1 3.2292 3.1008 0.0000 0.9603 241.6234 <1e-07
────────────────────────────────────────────────────────────────"""
──────────────────────────────────────────────
SSR df ΔSSR Δdf F* p(>F)
──────────────────────────────────────────────
[1] 0.1017 8
[2] 0.1283 10 0.0266 2 1.0456 0.3950
[3] 3.2292 11 3.1008 1 241.6234 <1e-07
──────────────────────────────────────────────"""

@test_throws ArgumentError ftest(mod, bigmod, nullmod)
@test_throws ArgumentError ftest(nullmod, bigmod, mod)
Expand Down
Loading