Review Articles

Covariate balancing based on kernel density estimates for controlled experiments

Yiou Li ,

a Department of Mathematical Sciences, DePaul University, Chicago, IL, USA

Lulu Kang ,

b Department of Applied Mathematics, Illinois Institute of Technology, Chicago, IL, USA

lkang2@iit.edu

Xiao Huang

b Department of Applied Mathematics, Illinois Institute of Technology, Chicago, IL, USA

Pages 102-113 | Received 12 Aug. 2020, Accepted 18 Jan. 2021, Published online: 03 Feb. 2021,
  • Abstract
  • Full Article
  • References
  • Citations

ABSTRACT

Controlled experiments are widely used in many applications to investigate the causal relationship between input factors and experimental outcomes. A completely randomised design is usually used to randomly assign treatment levels to experimental units. When covariates of the experimental units are available, the experimental design should achieve covariate balancing among the treatment groups, such that the statistical inference of the treatment effects is not confounded with any possible effects of covariates. However, covariate imbalance often exists, because the experiment is carried out based on a single realisation of the complete randomisation. It is more likely to occur and worsen when the size of the experimental units is small or moderate. In this paper, we introduce a new covariate balancing criterion, which measures the differences between kernel density estimates of the covariates of treatment groups. To achieve covariate balance before the treatments are randomly assigned, we partition the experimental units by minimising the criterion, then randomly assign the treatment levels to the partitioned groups. Through numerical examples, we show that the proposed partition approach can improve the accuracy of the difference-in-mean estimator and outperforms the complete randomisation and rerandomisation approaches.

References

  1. Anderson, N. H., Hall, P., & Titterington, D. M. (1994). Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates. Journal of Multivariate Analysis50(1), 41–54. https://doi.org/10.1006/jmva.1994.1033 [Crossref][Web of Science ®], [Google Scholar]
  2. Bernstein, S. (1927). Sur l'extension du théorème limite du calcul des probabilités aux sommes de quantités dépendantes. Mathematische Annalen97(1), 1–59. https://doi.org/10.1007/BF01447859 [Crossref], [Google Scholar]
  3. Bertsimas, D., Johnson, M., & Kallus, N. (2015). The power of optimization over randomization in designing experiments involving small samples. Operations Research63, 868–876. https://doi.org/10.1287/opre.2015.1361 [Crossref][Web of Science ®], [Google Scholar]
  4. Blackwell, M., Iacus, S., King, G., & Porro, G. (2009). cem: Coarsened exact matching in Stata. The Stata Journal9(4), 524–546. https://doi.org/10.1177/1536867X0900900402 [Crossref][Web of Science ®], [Google Scholar]
  5. de Lima, M. S., & G. S. Atuncar (2011). A Bayesian method to estimate the optimal bandwidth for multivariate kernel estimator. Journal of Nonparametric Statistics23(1), 137–148. https://doi.org/10.1080/10485252.2010.485200 [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  6. Duong, T., & Hazelton, M. (2003). Plug-in bandwidth matrices for bivariate kernel density estimation. Journal of Nonparametric Statistics15(1), 17–30. https://doi.org/10.1080/10485250306039 [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  7. Duong, T., & Hazelton, M. L. (2005). Cross-validation bandwidth matrices for multivariate kernel density estimation. Scandinavian Journal of Statistics32(3), 485–506. https://doi.org/10.1111/sjos.2005.32.issue-3 [Crossref][Web of Science ®], [Google Scholar]
  8. Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. The Annals of Statistics32(2), 407–499. https://doi.org/10.1214/009053604000000067 [Crossref][Web of Science ®], [Google Scholar]
  9. Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., & Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology173(7), 761–767. https://doi.org/10.1093/aje/kwq439 [Crossref][Web of Science ®], [Google Scholar]
  10. Gurobi Optimization, LLC (2020). Gurobi optimizer reference manual. [Google Scholar]
  11. Härdle, W. K., Müller, M., Sperlich, S., & Werwatz, A. (2012). Nonparametric and semiparametric models. Springer Science & Business Media. [Google Scholar]
  12. Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press. [Crossref], [Google Scholar]
  13. Jones, M. C., Marron, J. S., & Sheather, S. J. (1996). Progress in data-based bandwidth selection for kernel density estimation. Computational Statistics11, 337–381. [Web of Science ®], [Google Scholar]
  14. Kallus, N. (2018). Optimal a priori balance in the design of controlled experiments. Journal of the Royal Statistical Society: Series B (Statistical Methodology)80(1), 85–112. https://doi.org/10.1111/rssb.12240 [Crossref][Web of Science ®], [Google Scholar]
  15. McHugh, R., & Matts, J. (1983). Post-stratification in the randomized clinical trial. Biometrics39(1), 217–225. https://doi.org/10.2307/2530821 [Crossref][Web of Science ®], [Google Scholar]
  16. Miller, B. L., & Goldberg, D. E. (1995). Genetic algorithms, tournament selection, and the effects of noise. Complex Systems9, 193–212. [Google Scholar]
  17. Morgan, K. L., & Rubin, D. B. (2012). Rerandomization to improve covariate balance in experiments. The Annals of Statistics40(2), 1263–1282. https://doi.org/10.1214/12-AOS1008 [Crossref][Web of Science ®], [Google Scholar]
  18. Morgan, K. L., & Rubin, D. B. (2015). Rerandomization to balance tiers of covariates. Journal of the American Statistical Association110(512), 1412–1421. https://doi.org/10.1080/01621459.2015.1079528 [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  19. Pearl, J. (2000). Causality: Models, reasoning, and inference. Cambridge University Press. [Google Scholar]
  20. Rosenbaum, P. (2017). Observation and experiment: An introduction to causal inference. Harvard University Press. [Crossref], [Google Scholar]
  21. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika70, 41–55. https://doi.org/10.1093/biomet/70.1.41 [Crossref][Web of Science ®], [Google Scholar]
  22. Rubin, D. B. (1980). Randomization analysis of experimental data: The Fisher randomization test comment. Journal of the American Statistical Association75, 591–593. [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  23. Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association100(469), 322–331. https://doi.org/10.1198/016214504000001880 [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  24. Sain, S. R., Baggerly, K. A., & Scott, D. W. (1994). Cross-validation of multivariate densities. Journal of the American Statistical Association89(427), 807–817. https://doi.org/10.1080/01621459.1994.10476814 [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  25. Scott, D. W. (2015). Multivariate density estimation: Theory, practice, and visualization. Wiley. [Crossref], [Google Scholar]
  26. Sheather, S. J., & Jones, M. C. (1991). A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society. Series B (Methodological)53(3), 683–690. https://doi.org/10.1111/rssb.1991.53.issue-3 [Crossref][Web of Science ®], [Google Scholar]
  27. Silverman, B. W. (1986). Density estimation for statistics and data analysis, vol. CRC Press. [Crossref], [Google Scholar]
  28. Simonoff, J. S. (2012). Smoothing methods in statistics. Springer Science & Business Media. [Google Scholar]
  29. Van Laarhoven, P. J., & Aarts, E. H. (1987). Simulated annealing. In Simulated annealing: Theory and applications (pp. 7–15). Springer. [Crossref], [Google Scholar]
  30. Wand, M. P., & Jones, M. C. (1993). Comparison of smoothing parameterizations in bivariate kernel density estimation. Journal of the American Statistical Association88(422), 520–528. https://doi.org/10.1080/01621459.1993.10476303 [Taylor & Francis Online][Web of Science ®], [Google Scholar]
  31. Wand, M. P., & Jones, M. C. (1994). Multivariate plug-in bandwidth selection. Computational Statistics9, 97–116. [Web of Science ®], [Google Scholar]
  32. Wu, C. J., & Hamada, M. S. (2011). Experiments: planning, analysis, and optimization, vol. Wiley. [Google Scholar]
  33. Xie, H., & Aurisset, J. (2016). Improving the sensitivity of online controlled experiments: Case studies at netflix. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 645–654). ACM. [Crossref], [Google Scholar]
  34. Zhang, X., King, M. L., & Hyndman, R. J. (2006). A Bayesian approach to bandwidth selection for multivariate kernel density estimation. Computational Statistics & Data Analysis50(11), 3009–3031. https://doi.org/10.1016/j.csda.2005.06.019 [Crossref][Web of Science ®], [Google Scholar]