ABSTRACT

The goals of any major business transformation programme in an official statistical agency often include improving data collection efficiency, data processing methodologies and data quality. However, the achievement of such improvements may have transitional statistical impacts that could be misinterpreted as real-world changes if they are not measured and handled appropriately.

This paper describes a development work that sought to explore the design and analysis of a times-series experiment that measured the statistical impacts that sometimes occur during survey redesigns. The Labour Force Survey (LFS) of the Australian Bureau of Statistics (ABS) was used as a case study. In the present study:

1. A large-scale field experiment was designed and conducted that allowed the outgoing and the incoming surveys to run in parallel for some periods to measure the impacts of any changes to the survey process; and

2. The precision of the impact measurement was continuously improved while the new survey design was being implemented.

The state space modelling (SSM) technique was adopted as the main approach, as it provides an efficient impact measurement. This approach enabled sampling error structure to be incorporated in the time-series intervention analysis. The approach was also able to be extended to take advantage of the availability of other related data sources (e.g., the data obtained from the parallel data collection process) to improve the efficiency and accuracy of the impact measurement. As stated above, the LFS was used as a case study; however, the models and methods developed in this study could be extended to other surveys.

References

1. Aoki, M. (1987). State space modelling of time series. Heidelberg, Berlin: Springer-Verlag. [Crossref], [Google Scholar]
2. Australian Bureau of Statistics (ABS). (2016). Labour force, Australia, April 2016. (ABS Category Number 6202.0). Canberra, ACT: ABS. Retrieved from http://www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/6202.0Explanatory%20Notes1Apr%202016?OpenDocument. [Google Scholar]
3. Bell, P. (2001). Comparison of alternative labour force survey estimators. Survey Methodology, 27(2), 53–63. [Google Scholar]
4. Durbin, J., & Koopman, S. J. (2012). Time series analysis by state space methods. Oxford: Oxford University. [Crossref], [Google Scholar]
5. Glass, G. V., Willson, V. L., & Gottman, J. M. (2008). Design and analysis of time-series experiments. Charlotte, NC: Information Age. [Google Scholar]
6. Harvey, A. (1989). Forecasting, structural time series models and the Kalman filter. Cambridge: Cambridge University Press. [Google Scholar]
7. Koopman, S. J., Shephard, N., & Doornik, J. A. (2008). Ssfpack 3.0 statistical algorithms for models in state space form. London: Timberlake Consultants Ltd. [Google Scholar]
8. Pfeffermann, D. (1991). Estimation and seasonal adjustment of population means using data from repeated surveys. Journal of Business and Economic Statistics, 9(2), 163–175. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
9. Pfeffermann, D., Feder, M., & Singnorelli, D. (1998). Estimation of autocorrecations of survey errors with application to trend estimation in small areas. Journal of Business and Economic Statistics, 16, 339–348. [Taylor & Francis Online], [Web of Science ®], [Google Scholar]
10. Van den Brakel, J. A. (2008). Design-based analysis of embedded experiments with applications in the Dutch labour force survey. Journal of the Royal Statistical Society, Series A, 171(3), 581–613. doi: 10.1111/j.1467-985X.2008.00532.x [Crossref], [Google Scholar]
11. Van den Brakel, J. A., Griffiths, G., Surzhina, T., Wise, P., Blanchard, J., Zhang, X., & Honchar, O. (2017). A framework for measuring the impact of transitions in official statistics. Australian Bureau of Statistics, Research paper 1351.0.55.158, Canberra, Australia. [Google Scholar]
12. Van den Brakel, J. A., & Krieg, S. (2015). Dealing with small sample sizes, rotation group bias and discontinuities in a rotating panel design. Survey Methodology, 41(2), 267–296. [Web of Science ®], [Google Scholar]
13. Van den Brakel, J. A., & Roels, J. (2010). Intervention analysis with state-space models to estimate discontinuities due to a survey redesign. Annals of Applied Statistics, 4(2), 1105–1138. doi: 10.1214/09-AOAS305 [Crossref], [Web of Science ®], [Google Scholar]
14. Van den Brakel, J. A., Smith, P. A., & Compton, S. (2008). Quality procedures for survey transitions, experiments, time series and discontinuities. Journal for Survey Research Methods, 2(3), 123–141. [Google Scholar]
15. Van den Brakel, J. A., Zhang, X. (Mark), & Tam, S.-M. (2019). Measuring discontinuities in time series obtained with repeated sample surveys. International Statistical Review, forthcoming. [Google Scholar]
16. Zhang, X., & Honchar, O. (2016). Predicting survey estimates by state space models using multiple data sources. (ABS Category Number 1351.0.55.159). Retrieved from http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1351.0.55.159August%202017. [Google Scholar]
17. Zhang, X., Van den Brakel, J. A., Honchar, O., Wong, C., & Griffiths, G. (2017). Using state space models for statistical impact measurement of survey redesigns. (ABS Category Number 1351.0.55.160). Retrieved from http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1351.0.55.160October%20201. [Google Scholar]