Parallel and Streaming Truth Discovery in Large-Scale Quantitative Crowdsourcing

Robin Wentao Ouyang, Lance Kaplan, Alice Toniolo, Mani Srivastava, Timothy J. Norman

Research output: Contribution to journalArticlepeer-review

23 Citations (Scopus)
72 Downloads (Pure)


To enable reliable crowdsourcing applications, it is of great importance to develop algorithms that can automatically discover the truths from possibly noisy and conflicting claims provided by various information sources. In order to handle crowdsourcing applications involving big or streaming data, a desirable truth discovery algorithm should not only be effective, but also be scalable. However, with respect to quantitative crowdsourcing applications such as object counting and percentage annotation, existing truth discovery algorithms are not simultaneously effective and scalable. They either address truth discovery in categorical crowdsourcing or perform batch processing that does not scale. In this paper, we propose new parallel and streaming truth discovery algorithms for quantitative crowdsourcing applications. Through extensive experiments on real-world and synthetic datasets, we demonstrate that 1) both of them are quite effective, 2) the parallel algorithm can efficiently perform truth discovery on large datasets, and 3) the streaming algorithm processes data incrementally, and can efficiently perform truth discovery both on large datasets and in data streams.
Original languageEnglish
Pages (from-to)2984-2997
Number of pages14
JournalIEEE Transactions on Parallel and Distributed Systems
Issue number10
Early online date6 Jan 2016
Publication statusPublished - 1 Oct 2016

Bibliographical note

This research is based upon work supported in part by the U.S. ARL and U.K. Ministry of Defense under Agreement Number W911NF-06-3-0001, and by the NSF under award CNS-1213140. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views or represent the official policies of the NSF, the U.S. ARL, the U.S. Government, the U.K. Ministry of Defense or the U.K. Government. The U.S. and U.K. Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon. This work was done when R. W. Ouyang was a postdoc at the University of California, Los Angeles, CA.


  • Crowdsourcing
  • truth discovery
  • quantitative task
  • big data
  • parallel algorithm
  • streaming algorithm


Dive into the research topics of 'Parallel and Streaming Truth Discovery in Large-Scale Quantitative Crowdsourcing'. Together they form a unique fingerprint.

Cite this