A major challenge in spatial transcriptomics (ST) is resolving cellular composition, especially in technologies lacking single-cell resolution. The mixture of transcriptional signals within spatial spots complicates deconvolution and downstream analyses. To uncover the spatial heterogeneity of tissues, we introduce SvdRFCTD, a reference-free spatial transcriptomics deconvolution method, which estimates the cell type proportions at each spot on the tissue. To fully capture the heterogeneity in the ST data, we combine SvdRFCTD with a Bayesian hierarchical negative binomial model with spatial effects incorporated in both the mean and dispersion of the gene expression, which is used to explicitly model the generative mechanism of cell type proportions. By integrating spatial information and leveraging marker gene information, SvdRFCTD accurately estimates cell type proportions and uncovers complex spatial patterns. We demonstrate the ability of SvdRFCTD to identify cell types on simulated datasets. By applying SvdRFCTD to mouse brain and human pancreatic ductal adenocarcinomas datasets, we observe significant cellular heterogeneity within the tissue sections and successfully identify regions with high proportions of aggregated cell types, along with the spatial relationships between different cell types.

References

• Abbott, N. J., Rönnbäck, L., & Hansson, E. (2006). Astrocyte–endothelial interactions at the blood–brain barrier. Nature Reviews Neuroscience, 7(1), 41–53. https://doi.org/10.1038/nrn1824
• Allen, C., Chang, Y., Neelon, B., Chang, W., Kim, H. J., Li, Z., Ma, Q., & Chung, D. (2021). A Bayesian multivariate mixture model for spatial transcriptomics data. bioRxiv.
• The Tabula Muris Consortium, Almanzar, N., Antony, J., Baghel, A. S., Bakerman, I., Bansal, I., Barres, B. A., Beachy, P. A., Berdnik, D., Bilen, B., Brownfield, D., Cain, C., Chan, C. K. F., Chen, M. B., M. F. Clarke, Conley, S. D., Darmanis, S., Demers, A., Demir, K., De Morree, A., … Zou, J. (2020). A single-cell transcriptomic atlas characterizes ageing tissues in the mouse. Nature, 583(7817), 590–595. https://doi.org/10.1038/s41586-020-2496-1
• Banerjee, S., Carlin, B. P., & Gelfand, A. E. (2014). Hierarchical Modeling and Analysis for Spatial Data. Chapman and Hall/CRC.
• Bankhead, P., Loughrey, M. B., Fernández, J. A., Dombrowski, Y., McArt, D. G., Dunne, P. D., McQuaid, S., Gray, R. T., Murray, L. J., Coleman, H. G., James, J. A., Salto-Tellez, M., & Hamilton, P. W. (2017). QuPath: Open source software for digital pathology image analysis. Scientific Reports, 7(1), Article 16878. https://doi.org/10.1038/s41598-017-17204-5
• Biancalani, T., Scalia, G., Buffoni, L., Avasthi, R., Lu, Z., Sanger, A., Tokcan, N., C. R. Vanderburg, Segerstolpe, Å., Zhang, M., Avraham-Davidi, I., Vickovic, S., Nitzan, M., Ma, S., Subramanian, A., Lipinski, M., Buenrostro, J., Brown, N. B., Fanelli, D., …Regev, A. (2021). Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nature Methods, 18(11), 1352–1362. https://doi.org/10.1038/s41592-021-01264-7
• Brooks, S., Gelman, A., Jones, G., & Meng, X.-L. (2011). Handbook of Markov Chain Monte Carlo. Chapman and Hall/CRC.
• Cable, D. M., Murray, E., Zou, L. S., Goeva, A., Macosko, E. Z., Chen, F., & Irizarry, R. A. (2022). Robust decomposition of cell type mixtures in spatial transcriptomics. Nature Biotechnology, 40(4), 517–526. https://doi.org/10.1038/s41587-021-00830-w
• Caligiore, D., Arbib, M. A., Miall, R. C., & Baldassarre, G. (2019). The super-learning hypothesis: Integrating learning processes across cortex, cerebellum and basal ganglia. Neuroscience & Biobehavioral Reviews, 100, 19–34. https://doi.org/10.1016/j.neubiorev.2019.02.008
• Carpenter, A. E., Jones, T. R., Lamprecht, M. R., Clarke, C., Kang, I. H., Friman, O., Guertin, D. A., Chang, J. H., Lindquist, R. A., Moffat, J., Golland, P., & Sabatini, D. M. (2006). CellProfiler: Image analysis software for identifying and quantifying cell phenotypes. Genome Biology, 7(10), R100. https://doi.org/10.1186/gb-2006-7-10-r100
• Chen, K. H., Boettiger, A. N., Moffitt, J. R., Wang, S., & Zhuang, X. (2015). Spatially resolved, highly multiplexed RNA profiling in single cells. Science (New York, N.Y.), 348(6233), aaa6090. https://doi.org/10.1126/science.aaa6090
• Chinta, S. J., & Andersen, J. K. (2008). Redox imbalance in Parkinson's disease. Biochimica et Biophysica Acta (BBA) -- General Subjects, 1780(11), 1362–1367. https://doi.org/10.1016/j.bbagen.2008.02.005
• Clifton, K., Anant, M., Aihara, G., Atta, L., Aimiuwu, O. K., Kebschull, J. M., Miller, M. I., Tward, D., & Fan, J. (2023). STalign: Alignment of spatial transcriptomics data using diffeomorphic metric mapping. Nature Communications, 14(1), 8123. https://doi.org/10.1038/s41467-023-43915-7
• Dong, R., & Yuan, G.-C. (2021). SpatialDWLS: Accurate deconvolution of spatial transcriptomic data. Genome Biology, 22(1), 145. https://doi.org/10.1186/s13059-021-02362-7
• Du, W., Xia, X., Hu, F., & Yu, J. (2024). Extracellular matrix remodeling in the tumor immunity. Frontiers in Immunology, 14, Article 1340634. https://doi.org/10.3389/fimmu.2023.1340634
• Elosua-Bayes, M., Nieto, P., Mereu, E., & Gut, I. (2021). SPOTlight: Seeded NMF regression to deconvolute spatial transcriptomics spots with single-cell transcriptomes. Nucleic Acids Research, 49(9), e50–e50. https://doi.org/10.1093/nar/gkab043
• Erö, C., Gewaltig, M.-O., Keller, D., & Markram, H. (2018). A cell atlas for the mouse brain. Frontiers in Neuroinformatics, 12, 84. https://doi.org/10.3389/fninf.2018.00084
• Geras, A., Shafighi, S. Darvish, Domżał, K., Filipiuk, I., Rączkowska, A., Szymczak, P., Toosi, H., Kaczmarek, L., Koperski, Ł., Lagergren, J., Nowis, D., & Szczurek, E. (2023). Celloscope: A probabilistic model for marker-gene-driven cell type deconvolution in spatial transcriptomics data. Genome Biology, 24(1), 120. https://doi.org/10.1186/s13059-023-02951-8
• Graves, T. L. (2011). Automatic step size selection in random walk metropolis algorithms. arXiv:1103.5986. 
• Haario, H., Saksman, E., & Tamminen, J. (2001). An adaptive metropolis algorithm. Bernoulli, 7(2), 223.
• Hu, C., Li, T., Xu, Y., Zhang, X., Li, F., Bai, J., Chen, J., Jiang, W., Yang, K., Ou, Q., Li, X., Wang, P., & Zhang, Y. (2023). CellMarker 2.0: An updated database of manually curated cell markers in human/mouse and web tools based on scRNA-seq data. Nucleic Acids Research, 51(D1), D870–D876. https://doi.org/10.1093/nar/gkac947
• Hwang, E. S., Hyslop, T., Lynch, T., Ryser, M. D., Weiss, A., Wolf, A., Norris, K., Witten, M., Grimm, L., Schnitt, S., Badve, S., Factor, R., Frank, E., Collyar, D., Basila, D., Pinto, D., Watson, M. A., West, R., Davies, L., … Miller, S. (2024). Active monitoring with or without endocrine therapy for low- risk ductal carcinoma in situ: The COMET randomized clinical trial. JAMA: The Journal of the American Medical Association, 333(11), 972–980. https://doi.org/10.1001/jama.2024.26698
• Kleshchevnikov, V., Shmatko, A., Dann, E., Aivazidis, A., King, H. W., Li, T., Elmentaite, R., Lomakin, A., Kedlian, V., Gayoso, A., Jain, M. S., Park, J. S., Ramona, L., Tuck, E., Arutyunyan, A., Vento-Tormo, R., Gerstung, M., James, L., Stegle, O., … Bayraktar, O. A. (2022). Cell2location maps fine-grained cell types in spatial transcriptomics. Nature Biotechnology, 40(5), 661–671. https://doi.org/10.1038/s41587-021-01139-4
• Langlieb, J., Sachdev, N., Balderrama, K., Nadaf, N., Raj, M., Murray, E., Webber, J., Vanderburg, C., Gazestani, V., Tward, D., Mezias, C., Li, X., Cable, D., Norton, T., Mitra, P. P., Chen, F., & Macosko, E. (2023). The cell type composition of the adult mouse brain revealed by single cell and spatial genomics. bioRxiv.
• Larsson, L., Frisén, J., & Lundeberg, J. (2021). Spatially resolved transcriptomics adds a new dimension to genomics. Nature Methods, 18(1), 15–18. https://doi.org/10.1038/s41592-020-01038-7
• Lein, E. S., Hawrylycz, M. J., Ao, N., Ayres, M., Bensinger, A., Bernard, A., A. F. Boe, Boguski, M. S., Brockway, K. S., Byrnes, E. J., Chen, L., Chen, L., Chen, T.-M., Chin, M. Chi, Chong, J., Crook, B. E., Czaplinska, A., Dang, C. N., Datta, S., …Jones, A. R. (2007). Genome-wide atlas of gene expression in the adult mouse brain. Nature, 445(7124), 168–176. https://doi.org/10.1038/nature05453
• Levy-Jurgenson, A., Tekpli, X., Kristensen, V. N., & Yakhini, Z. (2020). Spatial transcriptomics inferred from pathology whole-slide images links tumor heterogeneity to survival in breast and lung cancer. Scientific Reports, 10(1), Article 18802. https://doi.org/10.1038/s41598-020-75708-z
• Li, Y., & Luo, Y. (2024). STdGCN: Spatial transcriptomic cell-type deconvolution using graph convolutional networks. Genome Biology, 25(1), 206. https://doi.org/10.1186/s13059-024-03353-0
• Li, B., Zhang, W., Guo, C., Xu, H., Li, L., Fang, M., Hu, Y., Zhang, X., Yao, X., Tang, M., Liu, K., Zhao, X., Lin, J., Cheng, L., Chen, F., Xue, T., & Qu, K. (2022). Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nature Methods, 19(6), 662–670. https://doi.org/10.1038/s41592-022-01480-9
• Li, H., Zhou, J., Li, Z., Chen, S., Liao, X., Zhang, B., Zhang, R., Wang, Y., Sun, S., & Gao, X. (2023). A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics. Nature Communications, 14(1), 1548. https://doi.org/10.1038/s41467-023-37168-7
• Liddelow, S. A., Guttenplan, K. A., Clarke, L. E., Bennett, F. C., Bohlen, C. J., Schirmer, L., Bennett, M. L., Münch, A. E., Chung, W.-S., Peterson, T. C., D. K. Wilton, Frouin, A., Napier, B. A., Panicker, N., Kumar, M., Buckwalter, M. S., Rowitch, D. H., Dawson, V. L., Dawson, T. M., …Barres, B. A. (2017). Neurotoxic reactive astrocytes are induced by activated microglia. Nature, 541(7638), 481–487. https://doi.org/10.1038/nature21029
• Liu, X., Tang, G., Chen, Y., Li, Y., Li, H., & Wang, X. (2025). SpatialDeX is a reference- free method for cell- type deconvolution of spatial transcriptomics data in solid tumors. Cancer Research, 85(1), 171–182. https://doi.org/10.1158/0008-5472.CAN-24-1472
• Liu, J., Tran, V., Vemuri, V. N. P., Byrne, A., Borja, M., Kim, Y. J., Agarwal, S., Wang, R., Awayan, K., Murti, A., Taychameekiatchai, A., Wang, B., Emanuel, G., He, J., Haliburton, J., Pisco, A. Oliveira, & Neff, N. F. (2023). Concordance of MERFISH spatial transcriptomics with bulk and single-cell RNA sequencing. Life Science Alliance, 6(1), e202201701. https://doi.org/10.26508/lsa.202201701
• Ma, Y., & Zhou, X. (2022). Spatially informed cell-type deconvolution for spatial transcriptomics. Nature Biotechnology, 40(9), 1349–1359. https://doi.org/10.1038/s41587-022-01273-7
• Mardia, K. (1988). Multi-dimensional multivariate Gaussian Markov random fields with application to image processing. Journal of Multivariate Analysis, 24(2), 265–284. https://doi.org/10.1016/0047-259X(88)90040-1
• Markram, H., Toledo-Rodriguez, M., Wang, Y., Gupta, A., Silberberg, G., & Wu, C. (2004). Interneurons of the neocortical inhibitory system. Nature Reviews Neuroscience, 5(10), 793–807. https://doi.org/10.1038/nrn1519
• Miao, Z., Balzer, M. S., Ma, Z., Liu, H., Wu, J., Shrestha, R., Aranyi, T., Kwan, A., Kondo, A., Pontoglio, M., Kim, J., Li, M., Kaestner, K. H., & Susztak, K. (2021). Single cell regulatory landscape of the mouse kidney highlights cellular differentiation programs and disease targets. Nature Communications, 12(1), 2277. https://doi.org/10.1038/s41467-021-22266-1
• Miller, B. F., Huang, F., Atta, L., Sahoo, A., & Fan, J. (2022). Reference-free cell type deconvolution of multi-cellular pixel-resolution spatially resolved transcriptomics data. Nature Communications, 13(1), 2339. https://doi.org/10.1038/s41467-022-30033-z
• Moncada, R., Barkley, D., Wagner, F., Chiodin, M., Devlin, J. C., Baron, M., Hajdu, C. H., Simeone, D. M., & Yanai, I. (2020). Integrating microarray-based spatial transcriptomics and single-cell RNA-seq reveals tissue architecture in pancreatic ductal adenocarcinomas. Nature Biotechnology, 38(3), 333–342. https://doi.org/10.1038/s41587-019-0392-8
• Morvaridi, S., Dhall, D., Greene, M. I., Pandol, S. J., & Wang, Q. (2015). Role of YAP and TAZ in pancreatic ductal adenocarcinoma and in stellate cells associated with cancer and chronic pancreatitis. Scientific Reports, 5(1), 16759. https://doi.org/10.1038/srep16759
• Murata, K., Kinoshita, T., Fukazawa, Y., Kobayashi, K., Kobayashi, K., Miyamichi, K., Okuno, H., Bito, H., Sakurai, Y., Yamaguchi, M., Mori, K., & Manabe, H. (2019). GABAergic neurons in the olfactory cortex projecting to the lateral hypothalamus in mice. Scientific Reports, 9(1), 7132. https://doi.org/10.1038/s41598-019-43580-1
• Mutiso, F., Pearce, J. L., Benjamin-Neelon, S. E., Mueller, N. T., Li, H., & Neelon, B. (2022). Bayesian negative binomial regression with spatially varying dispersion: Modeling COVID-19 incidence in Georgia. Spatial Statistics, 52, 100703. https://doi.org/10.1016/j.spasta.2022.100703
• Mutiso, F., Pearce, J. L., Benjamin-Neelon, S. E., Mueller, N. T., Li, H., & Neelon, B. (2024). A marginalized zero-inflated negative binomial model for spatial data: Modeling covid-19 deaths in Georgia. Biometrical Journal, 66(5), e202300182. https://doi.org/10.1002/bimj.v66.5
• Neelon, B., Wen, C.-C., & Benjamin-Neelon, S. E. (2023). A multivariate spatiotemporal model for tracking COVID-19 incidence and death rates in socially vulnerable populations. Journal of Applied Statistics, 50(8), 1812–1835. https://doi.org/10.1080/02664763.2022.2046713
• Nielsen, M. F. B., Mortensen, M. B., & Detlefsen, S. (2016). Key players in pancreatic cancer-stroma interaction: Cancer-associated fibroblasts, endothelial and inflammatory cells. World Journal of Gastroenterology, 22(9), 2678. https://doi.org/10.3748/wjg.v22.i9.2678
• Rao, A., Barkley, D., França, G. S., & Yanai, I. (2021). Exploring tissue architecture using spatial transcriptomics. Nature, 596(7871), 211–220. https://doi.org/10.1038/s41586-021-03634-9
• Shannon, P., Markiel, A., Ozier, O., Baliga, N. S., Wang, J. T., Ramage, D., Amin, N., Schwikowski, B., & Ideker, T. (2003). Cytoscape: A software nnvironment for integrated models of biomolecular interaction networks. Genome Research, 13(11), 2498–2504. https://doi.org/10.1101/gr.1239303
• Singh, R., Park, A. K., Hardison, R. C., Zhu, X., & Li, Q. (2023). RETROFIT: reference-free deconvolution of cell-type mixtures in spatial transcriptomics. bioRxiv.
• Ståhl, P. L., Salmén, F., Vickovic, S., Lundmark, A., Navarro, J. F., Magnusson, J., Giacomello, S., Asp, M., Westholm, J. O., Huss, M., Mollbrink, A., Linnarsson, S., Codeluppi, S., Borg, Å., Pontén, F., P. I. Costea, Sahlén, P., Mulder, J., Bergmann, O., …Frisén, J. (2016). Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science (New York, N.Y.), 353(6294), 78–82. https://doi.org/10.1126/science.aaf2403
• Teschendorff, A. E., Breeze, C. E., Zheng, S. C., & Beck, S. (2017). A comparison of reference-based algorithms for correcting cell-type heterogeneity in Epigenome-Wide Association Studies. BMC Bioinformatics, 18(1), 105. https://doi.org/10.1186/s12859-017-1511-5
• Tritsch, N. X., Ding, J. B., & Sabatini, B. L. (2012). Dopaminergic neurons inhibit striatal output through non-canonical release of GABA. Nature, 490(7419), 262–266. https://doi.org/10.1038/nature11466
• Vickovic, S., Eraslan, G., Salmén, F., Klughammer, J., Stenbeck, L., Schapiro, D., Äijö, T., Bonneau, R., Bergenstråhle, L., Navarro, J. F., Gould, J., Griffin, G. K., Borg, Å., Ronaghi, M., Frisén, J., Lundeberg, J., Regev, A., & Ståhl, P. L. (2019). High-definition spatial transcriptomics for in situ tissue profiling. Nature Methods, 16(10), 987–990. https://doi.org/10.1038/s41592-019-0548-y
• Williams, C. G., Lee, H. J., Asatsuma, T., Vento-Tormo, R., & Haque, A. (2022). An introduction to spatial transcriptomics for biomedical research. Genome Medicine, 14(1), 68. https://doi.org/10.1186/s13073-022-01075-1
• Yang, C. X., Sin, D. D., & Ng, R. T. (2024). SMART: Spatial transcriptomics deconvolution using marker-gene-assisted topic model. Genome Biology, 25(1), 304. https://doi.org/10.1186/s13059-024-03441-1
• Yu, K.-X., Yuan, W.-J., Wang, H.-Z., & Li, Y.-X. (2024). Extracellular matrix stiffness and tumor-associated macrophage polarization: New fields affecting immune exclusion. Cancer Immunology, Immunotherapy, 73(6), 115. https://doi.org/10.1007/s00262-024-03675-9
• Zaborszky, L., Den Pol, A. Van, & Gyengesi, E. (2012). The basal forebrain cholinergic projection system in mice. In The Mouse Nervous System (pp. 684–718). Academic Press.
• Zeisel, A., Hochgerner, H., Lönnerberg, P., Johnsson, A., Memic, F., J. Van Der Zwan, Häring, M., Braun, E., Borm, L. E., Manno, G. La, Codeluppi, S., Furlan, A., Lee, K., Skene, N., Harris, K. D., Hjerling-Leffler, J., Arenas, E., Ernfors, P., Marklund, U., … Linnarsson, S. (2018). Molecular architecture of the mouse nervous system. Cell, 174(4), 999–1014.e22. https://doi.org/10.1016/j.cell.2018.06.021
• Zhang, Y., Lin, X., Yao, Z., Sun, D., Lin, X., Wang, X., Yang, C., & Song, J. (2023). Deconvolution algorithms for inference of the cell-type composition of the spatial transcriptome. Computational and Structural Biotechnology Journal, 21, 176–184. https://doi.org/10.1016/j.csbj.2022.12.001

To cite this article: Xuan Li, Yincai Tang, Jingsi Ming & Xingjie Shi (08 May 2025): A Bayesian hierarchical model with spatially varying dispersion for reference-free cell type deconvolution in spatial transcriptomics, Statistical Theory and Related Fields, DOI: 10.1080/24754269.2025.2495651

To link to this article: https://doi.org/10.1080/24754269.2025.2495651