ABSTRACT

We present a new approach to model selection and Bayes factor determination, based on Laplace expansions (as in BIC), which we call Prior-based Bayes Information Criterion (PBIC). In this approach, the Laplace expansion is only done with the likelihood function, and then a suitable prior distribution is chosen to allow exact computation of the (approximate) marginal likelihood arising from the Laplace approximation and the prior. The result is a closed-form expression similar to BIC, but now involves a term arising from the prior distribution (which BIC ignores) and also incorporates the idea that different parameters can have different effective sample sizes (whereas BIC only allows one overall sample size n). We also consider a modification of PBIC which is more favourable to complex models.

References

1. Bayarri, M. J., Berger, J. O., Forte, A., & García-Donato, G. (2012). Criteria for Bayesian model choice with application to variable selection. The Annals of Statistics, 40(3), 1550–1577. doi: 10.1214/12-AOS1013 [Crossref], [Web of Science ®], [Google Scholar]
2. Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). New York: Springer-Verlag. [Crossref], [Google Scholar]
3. Berger, J. O., Bayarri, M. J., & Pericchi, L. R. (2014). The effective sample size. Econometric Reviews, 33, 197–217. doi: 10.1080/07474938.2013.807157 [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
4. Berger, J. O., Ghosh, J. K., & Mukhopadhyay, N. (2003). Approximations and consistency of bayes factors as model dimension grows. Journal of Statistical Planning and Inference, 112, 241–258. doi: 10.1016/S0378-3758(02)00336-1 [Crossref], [Web of Science ®], [Google Scholar]
5. Berger, J. O., Pericchi, L. R., & Varshavsky, J. A. (1998). Bayes factors and marginal distributions in invariant situations. Sankhya: The Indian Journal of Statistics. Series A, 60, 307–321. [Google Scholar]
6. Berger, J. O., & Sellke, T. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American statistical Association, 82, 112–122. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
7. Bollen, K. A., Ray, S., Zavisca, J., & Harden, J. J. (2012). A comparison of Bayes factor approximation methods including two new methods. Sociological Methods and Research, 41, 294–324. doi: 10.1177/0049124112452393 [Crossref], [Web of Science ®], [Google Scholar]
8. Chakrabarti, A., & Ghosh, J. K. (2006). A generalization of BIC for the general exponential family. Journal of Statistical Planning and Inference, 136(9), 2847–2872. doi: 10.1016/j.jspi.2005.01.005 [Crossref], [Web of Science ®], [Google Scholar]
9. Drton, M., & Plummer, M. (2017). A Bayesian information criterion for singular models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(2), 323–380. doi: 10.1111/rssb.12187 [Crossref], [Web of Science ®], [Google Scholar]
10. Dudley, R. M., & Haughton, D. (1997). Information criteria for multiple data sets and restricted parameters. Statistica Sinica, 7, 265–284. [Web of Science ®], [Google Scholar]
11. Findley, D. F. (1991). Counterexamples to parsimony and BIC. Annals of the Institute of Statistical Mathematics, 43, 505–514. doi: 10.1007/BF00053369 [Crossref], [Web of Science ®], [Google Scholar]
12. Foygel, R., & Drton, M. (2010). Extended Bayesian information criteria for Gaussian graphical models. In Advances in neural information processing systems (pp. 604–612). [Google Scholar]
13. Haughton, D. (1988). On the choice of a model to fit data from an exponential family. The Annals of Statistics, 16(1), 342–355. doi: 10.1214/aos/1176350709 [Crossref], [Web of Science ®], [Google Scholar]
14. Haughton, D. (1991). Consistency of a class of information criteria for model selection in non-linear regression. Communications in Statistics: Theory and Methods, 20, 1619–1629. doi: 10.1080/03610929108830587 [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
15. Haughton, D. (1993). Consistency of a class of information criteria for model selection in nonlinear regression. Theory of Probability and its Applications, 37, 47–53. doi: 10.1137/1137009 [Crossref], [Web of Science ®], [Google Scholar]
16. Haughton, D., Oud, J., & Jansen, R. (1997). Information and other criteria in structural equation model selection. Communications in Statistics, Part B – Simulation and Computation, 26(4), 1477–1516. doi: 10.1080/03610919708813451 [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
17. Jeffreys, H. (1961). Theory of probability. London: Oxford University Press. [Google Scholar]
18. Kass, R. E., & Raftery, A. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795. doi: 10.1080/01621459.1995.10476572 [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
19. Kass, R. E., & Vaidyanathan, S. K. (1992). Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. Journal of the Royal Statistical Society, 54, 129–144. [Google Scholar]
20. Kass, R. E., & Wasserman, L. (1995). A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. Journal of the American Statistical Association, 90, 928–934. doi: 10.1080/01621459.1995.10476592 [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
21. Pauler, D. (1998). The Schwarz criterion and related methods for normal linear models. Biometrika, 85, 13–27. doi: 10.1093/biomet/85.1.13 [Crossref], [Web of Science ®], [Google Scholar]
22. Raftery, A. E. (1999). Bayes factors and BIC – comment on ‘A critique of the Bayesian information criterion for model selection’. Sociological Methods and Research, 27, 411–427. doi: 10.1177/0049124199027003005 [Crossref], [Web of Science ®], [Google Scholar]
23. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. doi: 10.1214/aos/1176344136 [Crossref], [Web of Science ®], [Google Scholar]
24. Shen, G., & Ghosh, J. K. (2011). Developing a new BIC for detecting change-points. Journal of Statistical Planning and Inference, 141(4), 1436–1447. doi: 10.1016/j.jspi.2010.10.017 [Crossref], [Web of Science ®], [Google Scholar]
25. Stone, M. (1979). Comments on model selection criteria of Akaike and Schwarz. Journal of the Royal Statistical Society, Series B, 41, 276–278. [Google Scholar]
26. Strawderman, W. E. (1971). Proper Bayes minimax estimators of the multivariate normal mean. The Annals of Mathematical Statistics, 42(1), 385–388. doi: 10.1214/aoms/1177693528 [Crossref], [Google Scholar]
27. Tierney, L., Kass, R. E., & Kadane, J. B. (1989). Fully exponential Laplace approximations to expectations and variances of nonpositive functions. Journal of the American Statistical Association, 84(407), 710–716. doi: 10.1080/01621459.1989.10478824 [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
28. Zak-Szatkowska, M., & Bogdan, M. (2011). Modified versions of Bayesian information criterion for sparse generalized linear models. Computational Statistics and Data Analysis, 55, 2908–2924. doi: 10.1016/j.csda.2011.04.016 [Crossref], [Web of Science ®], [Google Scholar]