LDA\,算法在Mahout下的高效实现

华东师范大学学报(自然科学版) ›› 2013, Vol. 2013 ›› Issue (3): 118-130.

LDA\,算法在Mahout下的高效实现

许伯熹¹, 胡宁², 陈文斌¹, 高卫国¹, 程晋¹

1. 复旦大学~~数学科学学院, 上海 200433; 2. 上海聚胜万合广告有限公司, 上海 200070

收稿日期:2013-03-01 修回日期:2013-04-01 出版日期:2013-05-25 发布日期:2013-07-10

Efficient implementation for LDA in Mahout

XU Bo-xi¹, HU Ning², CHEN Wen-bin¹, GAO Wei-guo¹, CHENG Jin¹

1. School of Mathematical Sciences, Fudan University, Shanghai 200433, China; 2. Shanghai MediaV Advertising Corporation, Ltd., Shanghai} 200070, China

Received:2013-03-01 Revised:2013-04-01 Online:2013-05-25 Published:2013-07-10

摘要/Abstract

摘要： 通过对运用\,Gibbs\,采样的\,Latent Dirichlet Allocation
(LDA)\,算法和\,MapReduce\,计算框架的细致研究,
实现了\,LDA\,算法在\,Mahout 下的分布式并行计算.
详细地考察了该分布式并行计算程序的计算性能,
并深入地探讨了一些影响计算性能的关键问题.

关键词: Latent Dirichlet Allocation, Gibbs\,采样, Mahout, 分布式并行计算, MapReduce\,计算框架

Abstract: In a careful study of Latent Dirichlet Allocation (LDA)
using Gibbs sampling and the MapReduce framework, an efficient
implementation for LDA in Mahout was achieved. The experiments
showed the high performance of this distributed parallel LDA
program, and several issues about enhancing performance were
discussed.

Key words: Latent Dirichlet Allocation, Gibbs sampling, Mahout, distributed parallel computing, MapReduce framework

中图分类号:

O211

许伯熹, 胡宁, 陈文斌, 高卫国, 程晋. LDA\,算法在Mahout下的高效实现[J]. 华东师范大学学报(自然科学版), 2013, 2013(3): 118-130.

XU Bo-xi, HU Ning, CHEN Wen-bin, GAO Wei-guo, CHENG Jin. Efficient implementation for LDA in Mahout[J]. Journal of East China Normal University(Natural Sc, 2013, 2013(3): 118-130.

参考文献

{1}

BLEI D M, NG A Y, JORDAN M I.

Latent Dirichlet allocation[J].

Journal of Machine Learning Research, 2003 (3): 993-1022.

{2}

GRIFFITHS T L, STEYVERS M.

Finding scientific topics[J].

Proceedings of the National Academy of Sciences, 2004(101): 5228-5235.

{3}

VENNER J.

Pro Hadoop[M].

New York: Apress, 2009.

{4}

OWEN S, ANIL R, DUNNING T, FRIEDMAN E.

Mahout in Action[M].

New York: Manning Publications, 2010.

{5}

STEYVERS M, GRIFFITHS T.

Probabilistic topic models[M]//LANDAUER T,

MCNAMARA D, DENNIS S, et al. Latent Semantic Analysis: A Road to Meaning.[s.l.]:Routledge, 2007.

{6}

HEINRICH G.

Parameter estimation for text analysis[R].

Darmstadt: Fraunhofer IGD, 2004.

{7}

NEWMAN D, ASUNCION A, SMYTH P, WELLING M.

Distributed inference for latent Dirichlet allocation[J].

Proc Neural Information Processing Systems, 2007(20): 1081-1088.

{8}

WANG Y, BAI H J, STANTON M, et al.

PLDA: Parallel Latent Dirichlet Allocation for Large-Scale Applications[M].

Lecture Notes in Computer Science 5564. Berlin: Springer, 2009: 301-314.

{9}

GRIFFITHS T L, STEYVERS M.

A probabilistic approach to semantic representation[C]// Proceedings of the Twenty-Fourth Annual Conference of Cognitive Science Society,

2002.

{10}

LIU Z Y, ZHANG Y Z, CHANG E Y.

PLDA+: parallel latent Dirichlet allocation with data placement and pipeline processing[J].

ACM Transactions on Intelligent Systems and Technology, 2011(2): 26.

{11}

SMOLA A, NARAYANAMURTHY S.

An architecture for parallel topic models[J].

Proceedings of the VLDB Endowment, 2010(3): 703-710.

{12}

EKANAYAKE J, LI H, ZHANG B J, et al.

Twister: a runtime for iterative MapReduce[J].

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, 2010(1): 810-818.

{13}

BU Y Y, HOWE B, BALAZINSKA M, et al.

HaLoop: efficient iterative data processing on large clusters[J].

Proceedings of the VLDB Endowment, 2010(3): 285-296.

[1]	田德建, 江龙, 纪荣林. 二次交替容度的AVaR的表示定理[J]. 华东师范大学学报(自然科学版), 2014, 2014(3): 23-29.
[2]	宋文耀, 苗连英, 张淑洁. 不含3-圈的1-平面图的列表边染色与列表全染色[J]. 华东师范大学学报(自然科学版), 2014, 2014(3): 40-44.
[3]	张媛媛, 王文胜. 带常利率的二维风险模型的破产概率[J]. 华东师范大学学报(自然科学版), 2013, 2013(6): 22-31.
[4]	苏小囡, 王伟, 王文胜. Regime switching~模型下的幂式期权定价[J]. 华东师范大学学报(自然科学版), 2013, 2013(6): 32-39.
[5]	马海成, 李生刚. 一类图色等价的充分必要条件[J]. 华东师范大学学报(自然科学版), 2013, 2013(1): 1-6.
[6]	陆智萍, 陶勤英. 局部自相似过程的小波估计[J]. 华东师范大学学报(自然科学版), 2012, 2012(5): 76-84.
[7]	甄莉君, 张兴永, 牛成虎, 黎伟. 挂钩黄金理财产品定价的数值方法[J]. 华东师范大学学报(自然科学版), 2011, 2011(5): 25-32.