A fundamental concept in information theory is information entropy, which is used to quantify the degree of uncertainty associated with random variables. Building upon this, relative entropy serves as a measure of the discrepancy between two probability distributions and has been widely studied in statistics. The function of relative entropy in the field of parameter estimation has been extensively investigated. This paper extends existing research by deriving minimum mean relative entropy estimators for the parameters of the Gamma, Laplace, and Rayleigh distributions. Furthermore, we introduce the residual mean relative entropy, a novel measure based on mean relative entropy, and apply it to model comparison. To estimate this measure, we employ kernel density estimation and bootstrap methods. Simulation experiments and empirical analysis demonstrate that the proposed residual mean relative entropy provides an effective new criterion for model comparison.

References

• Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705
• Arizono, I., & Ohta, H. (1989). A test for normality based on Kullback-Leibler information. The American Statistician, 43(1), 20–22.
• Billingsley, P. (1999). Convergence of probability measures (2nd ed.). John Wiley & Sons.
• Chen, G., Gott, J. R., & Ratra, B. (2003). Non-Gaussian error distribution of Hubble constant measurements. Publications of the Astronomical Society of the Pacific, 115(813), 1269–1279. https://doi.org/10.1086/pasp.2003.115.issue-813
• Cover, T. M., & Thomas, J. A. (1991). Elements of information theory (2nd ed.). John Wiley & Sons.
• DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189–228. https://doi.org/10.1214/ss/1032280214
• Efron, B. (1992). Bootstrap methods: Another look at the Jackknife. In S. Kotz & N. L. Johnson (Eds.), Breakthroughs in statistics: Methodology and distribution (pp. 569–593). Springer.
• Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407–499. https://doi.org/10.1214/009053604000000067
• Ghosh, M., & Yang, M. C. (1988). Simultaneous estimation of Poisson means under entropy loss. The Annals of Statistics, 16(1), 278–291. https://doi.org/10.1214/aos/1176350705
• Gupta, M., & Srivastava, S. (2010). Parametric Bayesian estimation of differential entropy and relative entropy. Entropy, 12(4), 818–843. https://doi.org/10.3390/e12040818
• Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals. The Annals of Statistics, 16(3), 927–953.
• Hall, P., & Wilson, S. R. (1991). Two guidelines for bootstrap hypothesis testing. Biometrics, 47(2), 757–762. https://doi.org/10.2307/2532163
• Hu, Q., Guo, M., Yu, D., & Liu, J. (2010). Information entropy for ordinal classification. Science China Information Sciences, 53(6), 1188–1200. https://doi.org/10.1007/s11432-010-3117-7
• Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22(1), 79–86. https://doi.org/10.1214/aoms/1177729694
• Ling, R. F. (1984). Residuals and influence in regression. Technometrics, 26(4), 413–415. https://doi.org/10.1080/00401706.1984.10487996
• Liu, S., Ma, T., & Liu, Y. (2016). Sensitivity analysis in linear models. Special Matrices, 4(1), 225–232. https://doi.org/10.1515/spma-2016-0021
• Mallows, C. L. (2000). Some comments on Cp. Technometrics, 42(1), 87–94.
• Mühlenbein, H., & Höns, R. (2005). The estimation of distributions and the minimum relative entropy principle. Evolutionary Computation, 13(1), 1–27. https://doi.org/10.1162/1063656053583469
• Namdari, A., & Li, Z. S. (2019). A review of entropy measures for uncertainty quantification of stochastic processes. Advances in Mechanical Engineering, 11(6), 1687814019857350. https://doi.org/10.1177/1687814019857350
• Parsian, A., & Nematollahi, N. (1996). Estimation of scale parameter under entropy loss function. Journal of Statistical Planning and Inference, 52(1), 77–91. https://doi.org/10.1016/0378-3758(95)00026-7
• Parzen, E. (1962). On estimation of a probability density function and mode. The Annals of Mathematical Statistics, 33(3), 1065–1076. https://doi.org/10.1214/aoms/1177704472
• Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. https://doi.org/10.1080/01621459.1994.10476870
• Pollard, D. (1984). Convergence of stochastic processes. Springer.
• Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. The Annals of Mathematical Statistics, 27(3), 832–837. https://doi.org/10.1214/aoms/1177728190
• Shannon, C. E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/bltj.1948.27.issue-3
• Silverman, B. W. (1978). Weak and strong uniform consistency of the kernel estimate of a density and its derivatives. The Annals of Statistics, 6(1), 177–184. https://doi.org/10.1214/aos/1176344076
• Vasicek, O. (1976). A test for normality based on sample entropy. Journal of the Royal Statistical Society: Series B (Methodological), 38(1), 54–59. https://doi.org/10.1111/j.2517-6161.1976.tb01566.x
• Zhang, J. (2021). The mean relative entropy: An invariant measure of estimation error. The American Statistician, 75(2), 117–123. https://doi.org/10.1080/00031305.2018.1543139
• Zhang, J., Meng, C., Yu, J., Zhang, M., Zhong, W., & Ma, P. (2023). An optimal transport approach for selecting a representative subsample with application in efficient kernel density estimation. Journal of Computational and Graphical Statistics, 32(1), 329–339. https://doi.org/10.1080/10618600.2022.2084404
• Zheng, Y., Jestes, J., Phillips, J. M., & Li, F. (2013). Quality and efficiency for kernel density estimates in large data. In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data (pp. 433–444). Association for Computing Machinery.

To cite this article: Lei Huang, Danting Wang & Lanpeng Ji (29 Jun 2026): Statistical properties of mean relative entropy and its applications, Statistical Theory and Related Fields, DOI: 10.1080/24754269.2026.2689052

To link to this article: https://doi.org/10.1080/24754269.2026.2689052