Different BF with BIC

[solomonchak]

Hi Richard,

I'm confused about why I get different Bayes factors when I calculated it from BIC extracted from the lm models and with lmBF. Example codes are:
`
WaterQual = data.frame (Hard = c(20,4,5,3,1,17,15,20,17,14,15,19,34,36),
Ca = c(3.4,1.7,1.3,1.5,0.4,3.5,3.3,3.5,6.3,5.1,5.2,5.1,12.2,11.9),
Mg = c(1.3,0.4,0.4,0.3,0.3,1.3,1.2,1.3,0.5,0.5,0.8,1.1,0.8,0.8))

Full = lm(Hard ~ Ca + Mg + Ca:Mg, data=WaterQual)
Main = lm(Hard ~ Ca + Mg, data=WaterQual)
deltaBIC = BIC(Full) - BIC(Main)
exp(deltaBIC/2)
BF = 3.308636

bfFull = lmBF(Hard ~ Ca + Mg + Ca:Mg, data=WaterQual)
bfMain = lmBF(Hard ~ Ca + Mg, data = WaterQual)
bfMain / bfFull
BF = 12.44325
`

Thank you very much!

[richarddmorey]

Well, the BIC is an approximation to the Bayes factor under particular priors, which are not the same priors as the one used by BayesFactor (which are the Liang et al, 2008 priors described here: http://www2.stat.duke.edu/courses/Spring09/sta244/Handouts/hyper-g.pdf). The main differences is that the Liang et al prior has a term that takes into account the correlation between the covariates, and while the BIC prior does not (IIRC). In the case of an linear interaction, as you have here, the interaction term is highly correlated with the two covariates. It would take me a bit of delving to work out exactly why they differ, but I'm pretty sure that would be the where the difference would lie.

Do you need a deeper answer than that?

[richarddmorey]

I'll actually take a look at this deeper. I think it might have to do with the complexity correction, as well. The Liang prior may have a steeper penalty for extra parameters than BIC due to the unknown hyperparameter.

[solomonchak]

Hi Richard,

Thanks for the quick reply. It make sense about the correlation between the covariates. But there's still a 100-time difference in BF if I just compared these two models, where the variables are not highly correlated:

Main=lm(Hard ~ Ca + Mg, data=WaterQual)
Red=lm(Hard ~ Ca, data=WaterQual)
deltaBIC = BIC(Main) - BIC(Red)
exp(deltaBIC/2)
BF = 1.200939e-06

bfMain = lmBF(Hard ~ Ca + Mg, data = WaterQual)
bfRed = lmBF(Hard ~ Ca, data=WaterQual)
bfRed / bfMain
BF = 0.0001696674

Hopefully there's a better resolution to this discrepancy. The deeper question is that if I could trust the BF calculated from BICs at all.