Journal articles on the topic 'Crowdsourced experiment'
To see the other types of publications on this topic, follow the link: Crowdsourced experiment.
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Crowdsourced experiment.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Usuba, Hiroki, Shota Yamanaka, Junichi Sato, and Homei Miyashita. "Predicting touch accuracy for rectangular targets by using one-dimensional task results." Proceedings of the ACM on Human-Computer Interaction 6, ISS (November 14, 2022): 525–37. http://dx.doi.org/10.1145/3567732.
Full textAbstract:
We propose a method that predicts the success rate in pointing to 2D rectangular targets by using 1D vertical-bar and horizontal-bar task results. The method can predict the success rates for more practical situations under fewer experimental conditions. This shortens the duration of experiments, thus saving costs for researchers and practitioners. We verified the method through two experiments: laboratory-based and crowdsourced ones. In the laboratory-based experiment, we found that using 1D task results to predict the success rate for 2D targets slightly decreases the prediction accuracy. In the crowdsourced experiment, this method scored better than using 2D task results. Thus, we recommend that researchers use the method properly depending on the situation.
2
Shiraishi, Yuhki, Jianwei Zhang, Daisuke Wakatsuki, Katsumi Kumai, and Atsuyuki Morishima. "Crowdsourced real-time captioning of sign language by deaf and hard-of-hearing people." International Journal of Pervasive Computing and Communications 13, no. 1 (April 3, 2017): 2–25. http://dx.doi.org/10.1108/ijpcc-02-2017-0014.
Full textAbstract:
Purpose The purpose of this paper is to explore the issues on how to achieve crowdsourced real-time captioning of sign language by deaf and hard-of-hearing (DHH) people, such that how a system structure should be designed, how a continuous task of sign language captioning should be divided into microtasks and how many DHH people are required to maintain a high-quality real-time captioning. Design/methodology/approach The authors first propose a system structure, including the new design of worker roles, task division and task assignment. Then, based on an implemented prototype, the authors analyze the necessary setting for achieving a crowdsourced real-time captioning of sign language, test the feasibility of the proposed system and explore its robustness and improvability through four experiments. Findings The results of Experiment 1 have revealed the optimal method for task division, the necessary minimum number of groups and the necessary minimum number of workers in a group. The results of Experiment 2 have verified the feasibility of the crowdsourced real-time captioning of sign language by DHH people. The results of Experiment 3 and Experiment 4 have shown the robustness and improvability of the captioning system. Originality/value Although some crowdsourcing-based systems have been developed for the captioning of voice to text, the authors intend to resolve the issues on the captioning of sign language to text, for which the existing approaches do not work well due to the unique properties of sign language. Moreover, DHH people are generally considered as the ones who receive support from others, but our proposal helps them become the ones who offer support to others.
3
Zhang, Xuanhui, Si Chen, Yuxiang Chris Zhao, Shijie Song, and Qinghua Zhu. "The influences of social value orientation and domain knowledge on crowdsourcing manuscript transcription." Aslib Journal of Information Management 72, no. 2 (December 24, 2019): 219–42. http://dx.doi.org/10.1108/ajim-08-2019-0221.
Full textAbstract:
Purpose The purpose of this paper is to explore how social value orientation and domain knowledge affect cooperation levels and transcription quality in crowdsourced manuscript transcription, and contribute to the recruitment of participants in such projects in practice. Design/methodology/approach The authors conducted a quasi-experiment using Transcribe-Sheng, which is a well-known crowdsourced manuscript transcription project in China, to investigate the influences of social value orientation and domain knowledge. The experiment lasted one month and involved 60 participants. ANOVA was used to test the research hypotheses. Moreover, inverviews and thematic analyses were conducted to analyze the qualitative data in order to provide additional insights. Findings The analysis confirmed that in crowdsourced manuscript transcription, social value orientation has a significant effect on participants’ cooperation level and transcription quality; domain knowledge has a significant effect on participants’ transcription quality, but not on their cooperation level. The results also reveal the interactive effect of social value orientation and domain knowledge on cooperation levels and quality of transcription. The analysis of the qualitative data illustrated the influences of social value orientation and domain knowledge on crowdsourced manuscript transcription in detail. Originality/value Researchers have paid little attention to the impacts of the psychological and cognitive factors on crowdsourced manuscript transcription. This study investigated the effect of social value orientation and the combined effect of social value orientation and domain knowledge in this context. The findings shed light on crowdsourcing transcription initiatives in the cultural heritage domain and can be used to facilitate participant selection in such projects.
4
Alyahya, Sultan. "Collaborative Crowdsourced Software Testing." Electronics 11, no. 20 (October 17, 2022): 3340. http://dx.doi.org/10.3390/electronics11203340.
Full textAbstract:
Crowdsourced software testing (CST) uses a crowd of testers to conduct software testing. Currently, the microtasking model is used in CST; in it, a testing task is sent to individual testers who work separately from each other. Several studies mentioned that the quality of test reports produced by individuals was a drawback because a large number of invalid defects were submitted. Additionally, individual workers tended to catch the simple defects, not those with high complexity. This research explored the effect of having pairs of collaborating testers working together to produce one final test report. We conducted an experiment with 75 workers to measure the effect of this approach in terms of (1) the total number of unique valid defects detected, (2) the total number of invalid defects reported, and (3) the possibility of detecting more difficult defects. The findings show that testers who worked in collaborating pairs can be as effective in detecting defects as an individual worker; the differences between them are marginal. However, CST significantly affects the quality of test reports submitted in two dimensions: it helps reduce the number of invalid defects and also helps detect more difficult defects. The findings are promising and suggest that CST platforms can benefit from new mechanisms that allow for the formation of teams of two individuals who can participate in doing testing jobs.
5
Miller, John, Yu (Marco) Nie, and Amanda Stathopoulos. "Crowdsourced Urban Package Delivery." Transportation Research Record: Journal of the Transportation Research Board 2610, no. 1 (January 2017): 67–75. http://dx.doi.org/10.3141/2610-08.
Full textAbstract:
Crowdsourced shipping presents an innovative shipping alternative that is expected to improve shipping efficiency, increase service, and decrease cost to the customer, and such shipping promises to enhance the sustainability of the transportation system. This study collected data on behavioral responses to choose from available crowdsourced shipping jobs. The goal of the study was to measure the potential willingness of individuals to change status from pure commuters to traveler–shippers. In particular, the study quantified potential crowdsourced shippers’ value of free time, or willingness to work (WTW) in a hypothetical scenario in which crowdsourced shipping jobs were available in a variety of settings. This WTW calculation is unique compared with the traditional willingness to pay (WTP) in that it measured the trade-off of making a profit and giving up time instead of spending money to save time. This work provides a foundation to analyze the application and effectiveness of crowdsourced shipping by exploring the WTW propensity of ordinary travelers. The analysis was based on a newly developed stated preference survey and analyzed choice across three potential shipping jobs and the option to choose none of the three (i.e., the status quo). Results showed that the experiment was successful in recovering reasonable WTW values that are higher than the normal WTP metrics. The results also identified many significant sociodemographic variables that could help crowdsourced shipping companies better target potential part-time drivers.
6
Luther, Kurt, Nathan Hahn, Steven Dow, and Aniket Kittur. "Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 3 (September 23, 2015): 110–19. http://dx.doi.org/10.1609/hcomp.v3i1.13239.
Full textAbstract:
Learning about a new area of knowledge is challenging for novices partly because they are not yet aware of which topics are most important. The Internet contains a wealth of information for learning the underlying structure of a domain, but relevant sources often have diverse structures and emphases, making it hard to discern what is widely considered essential knowledge vs. what is idiosyncratic. Crowdsourcing offers a potential solution because humans are skilled at evaluating high-level structure, but most crowd micro-tasks provide limited context and time. To address these challenges, we present Crowdlines, a system that uses crowdsourcing to help people synthesize diverse online information. Crowdworkers make connections across sources to produce a rich outline that surfaces diverse perspectives within important topics. We evaluate Crowdlines with two experiments. The first experiment shows that a high context, low structure interface helps crowdworkers perform faster, higher quality synthesis, while the second experiment shows that a tournament-style (parallelized) crowd workflow produces faster, higher quality, more diverse outlines than a linear (serial/iterative) workflow.
7
Lee, Tak Yeon, Casey Dugan, Werner Geyer, Tristan Ratchford, Jamie Rasmussen, N. Sadat Shami, and Stela Lupushor. "Experiments on Motivational Feedback for Crowdsourced Workers." Proceedings of the International AAAI Conference on Web and Social Media 7, no. 1 (August 3, 2021): 341–50. http://dx.doi.org/10.1609/icwsm.v7i1.14428.
Full textAbstract:
This paper examines the relationship between motivational design and its longitudinal effects on crowdsourcing systems. In the context of a company internal web site that crowdsources the identification of Twitter accounts owned by company employees, we designed and investigated the effects of various motivational features including individual / social achievements and gamification. Our 6-month experiment with 437 users allowed us to compare the features in terms of both quantity and quality of the work produced by participants over time. While we found that gamification can increase workers’ motivation overall, the combination of motivational features also matters. Specifically, gamified social achievement is the best performing design over a longer period of time. Mixing individual and social achievements turns out to be less effective and can even encourage users to game the system.
8
Andersen, David J., and Richard R. Lau. "Pay Rates and Subject Performance in Social Science Experiments Using Crowdsourced Online Samples." Journal of Experimental Political Science 5, no. 3 (2018): 217–29. http://dx.doi.org/10.1017/xps.2018.7.
Full textAbstract:
AbstractMechanical Turk has become an important source of subjects for social science experiments, providing a low-cost alternative to the convenience of using undergraduates while avoiding the expense of drawing fully representative samples. However, we know little about how the rates we pay to “Turkers” for participating in social science experiments affects their participation. This study examines subject performance using two experiments – a short survey experiment and a longer dynamic process tracing study of political campaigns – that recruited Turkers at different rates of pay. Looking at demographics and using measures of attention, engagement and evaluation of the candidates, we find no effects of pay rates upon subject recruitment or participation. We conclude by discussing implications and ethical standards of pay.
9
Bechtel, Benjamin, Matthias Demuzere, Panagiotis Sismanidis, Daniel Fenner, Oscar Brousse, Christoph Beck, Frieke Van Coillie, et al. "Quality of Crowdsourced Data on Urban Morphology—The Human Influence Experiment (HUMINEX)." Urban Science 1, no. 2 (May 9, 2017): 15. http://dx.doi.org/10.3390/urbansci1020015.
Full text
10
Ikeda, Kazushi, and Keiichiro Hoashi. "Utilizing Crowdsourced Asynchronous Chat for Efficient Collection of Dialogue Dataset." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 6 (June 15, 2018): 60–69. http://dx.doi.org/10.1609/hcomp.v6i1.13321.
Full textAbstract:
In this paper, we design a crowd-powered system to efficiently collect data for training dialogue systems. Conventional systems assign dialogue roles to a pair of crowd workers, and record their interaction on an online chat. In this framework, the pair is required to work simultaneously, and one worker must wait for the other when he/she is writing a message, which decreases work efficiency. Our proposed system allows multiple workers to create dialogues in an asynchronous manner, which relieves workers from time restrictions. We have conducted an experiment using our system on a crowdsourcing platform to evaluate the efficiency and the quality of dialogue collection. Results show that our system can reduce the necessary time to input a message by 68% while maintaining quality.
11
Auer, Elena M., Tara S. Behrend, Andrew B. Collmus, Richard N. Landers, and Ahleah F. Miles. "Pay for performance, satisfaction and retention in longitudinal crowdsourced research." PLOS ONE 16, no. 1 (January 20, 2021): e0245460. http://dx.doi.org/10.1371/journal.pone.0245460.
Full textAbstract:
In the social and cognitive sciences, crowdsourcing provides up to half of all research participants. Despite this popularity, researchers typically do not conceptualize participants accurately, as gig-economy worker-participants. Applying theories of employee motivation and the psychological contract between employees and employers, we hypothesized that pay and pay raises would drive worker-participant satisfaction, performance, and retention in a longitudinal study. In an experiment hiring 359 Amazon Mechanical Turk Workers, we found that initial pay, relative increase of pay over time, and overall pay did not have substantial influence on subsequent performance. However, pay significantly predicted participants' perceived choice, justice perceptions, and attrition. Given this, we conclude that worker-participants are particularly vulnerable to exploitation, having relatively low power to negotiate pay. Results of this study suggest that researchers wishing to crowdsource research participants using MTurk might not face practical dangers such as decreased performance as a result of lower pay, but they must recognize an ethical obligation to treat Workers fairly.
12
Kemmer, Ryan, Yeawon Yoo, Adolfo Escobedo, and Ross Maciejewski. "Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 8 (October 1, 2020): 73–82. http://dx.doi.org/10.1609/hcomp.v8i1.7465.
Full textAbstract:
There are many factors that affect the quality of data received from crowdsourcing, including cognitive biases, varying levels of expertise, and varying subjective scales. This work investigates how the elicitation and integration of multiple modalities of input can enhance the quality of collective estimations. We create a crowdsourced experiment where participants are asked to estimate the number of dots within images in two ways: ordinal (ranking) and cardinal (numerical) estimates. We run our study with 300 participants and test how the efficiency of crowdsourced computation is affected when asking participants to provide ordinal and/or cardinal inputs and how the accuracy of the aggregated outcome is affected when using a variety of aggregation methods. First, we find that more accurate ordinal and cardinal estimations can be achieved by prompting participants to provide both cardinal and ordinal information. Second, we present how accurate collective numerical estimates can be achieved with significantly fewer people when aggregating individual preferences using optimization-based consensus aggregation models. Interestingly, we also find that aggregating cardinal information may yield more accurate ordinal estimates.
13
Freire, Danilo, Manoel Galdino, and Umberto Mignozzetti. "Bottom-up accountability and public service provision: Evidence from a field experiment in Brazil." Research & Politics 7, no. 2 (April 2020): 205316802091444. http://dx.doi.org/10.1177/2053168020914444.
Full textAbstract:
Does local oversight improve public service delivery? We study the effect of a mobile phone application that allows citizens to monitor school construction projects in Brazilian municipalities. The app prompts users to submit data about construction sites, sends such crowdsourced information to independent engineers, and contacts the mayors’ offices about project delays. Our results show that the app has a null impact on school construction indicators. Additionally, we find that politicians are unresponsive to individual requests. The results question the impact of bottom-up monitoring on public service performance and suggest that interventions targeted at other groups, or focused on different issues, may produce better policy outcomes.
14
Rumi, Shakila Khan, Wei Shao, and Flora D. Salim. "Realtime Predictive Patrolling and Routing with Mobility and Emergency Calls Data." Proceedings of the International AAAI Conference on Web and Social Media 14 (May 26, 2020): 964–68. http://dx.doi.org/10.1609/icwsm.v14i1.7367.
Full textAbstract:
A well-planned patrol route plays a crucial role in increasing public security. Most of the existing studies designed the patrol route in a static manner. Situations when rerouting of patrol path are required due to the emergencies, e.g., an accident or ongoing homicide, are not considered. In this paper, we formulate the crime patrol routing problem jointly with dynamic crime event prediction, utilising crowdsourced check-in and real-time emergency call data. The extensive experiment on real-world datasets verifies the effectiveness of the proposed dynamic crime patrol route using different evaluation metrics.
15
Potsiou, C., N. Doulamis, N. Bakalos, M. Gkeli, and C. Ioannidis. "INDOOR LOCALIZATION FOR 3D MOBILE CADASTRAL MAPPING USING MACHINE LEARNING TECHNIQUES." ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences VI-4/W1-2020 (September 3, 2020): 159–66. http://dx.doi.org/10.5194/isprs-annals-vi-4-w1-2020-159-2020.
Full textAbstract:
Abstract. With the rapid global urbanization, several multi-dimensional complex infrastructures have emerged, introducing new challenges in the management of the vertically stratified buildings spaces. 3D indoor cadastral spaces consist a zestful research topic as their complexity and geometry alterations during time, prevents the assignment of the corresponding Rights, Restrictions and Responsibilities (RRR). In the absence of the necessary horizontal spatial data infrastructure/floor plans their determination is weak. In this paper a fit-for-purpose technical framework and a crowdsourced methodology for the implementation of 3D cadastral surveys focused on indoor cadastral spaces, is proposed and presented. As indoor data capturing tool, an open-sourced cadastral mobile application for Android devices, is selected and presented. An Indoor Positioning System based on Bluetooth technology is established while an innovative machine learning architecture is developed, in order to explore its potentials to automatically provide the position of the mobile device within an indoor environment, aiming to add more intelligence to the proposed 3D crowdsourced cadastral framework. A practical experiment for testing the examined technical solution is conducted. The produced results are assessed to be quite promising.
16
Yu, G., X. Zhou, D. Hou, and D. Wei. "ABNORMAL CROWDSOURCED DATA DETECTION USING REMOTE SENSING IMAGE FEATURES." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLIII-B4-2021 (June 30, 2021): 215–21. http://dx.doi.org/10.5194/isprs-archives-xliii-b4-2021-215-2021.
Full textAbstract:
Abstract. Quality is the key issue for judging the usability of crowdsourcing geographic data. While due to the un-professional of volunteers and the phenomenon of malicious labeling, there are many abnormal or poor quality objects in crowdsourced data. Based on this observation, an abnormal crowdsourced data detection method is proposed in this paper based on image features. This approach includes three main steps. 1) the crowdsourced vector data is used to segment the corresponding remote sensing imagery to get image objects with a priori information (e.g., shape and category) from vector data and spectral information from the images. Then, the sampling method is designed considering the spatial distribution and topographic properties of the objects, and the initial samples are obtained, although some samples are abnormal object or poor quality. 2) A feature contribution index (FCI) is defined based on information gain to select the optimal features, a feature space outlier index (FSOI) is presented to automatically identify outlier samples and changed objects. The initial samples are refined by an iteration procedure. After the iteration, the optimal features can be determined, and the refined samples with categories can be obtained; the imagery feature space is established using the optimal features for each category. 3) The abnormal objects are identified with the refined samples by calculating the FSOI values of image objects. In order to valid the effectiveness, an abnormal crowdsourced data detection prototype is developed using Visual Studio 2013 and C# programming, the above algorithms and methods are implemented and verified using water and vegetation categories as example, the OSM (OpenStreetMap) and corresponding imagery data of Changsha city as experiment data. The angular second moment (ASM), contrast, inverse difference moment (IDM), mean, variance, difference entropy, and normalized difference green index (NDGI) of vegetation, and the IDM, difference entropy and correlation and maximum band value of water are used to detect abnormal data after the selection of image optimal feature. Experimental results show that abnormal water and vegetation data in OSM can be effectively detected in this method, and the missed detection rate of the vegetation and water are all near to zero, and the positive detection rate reach 90.4% and 83.8%, respectively.
17
Hui, Julie, Amos Glenn, Rachel Jue, Elizabeth Gerber, and Steven Dow. "Using Anonymity and Communal Efforts to Improve Quality of Crowdsourced Feedback." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 3 (September 23, 2015): 72–82. http://dx.doi.org/10.1609/hcomp.v3i1.13229.
Full textAbstract:
Student entrepreneurs struggle to collect feedback on their product pitches in a classroom setting due to a lack of time, money, and access to motivated feedback providers. Online social networks present a unique opportunity for entrepreneurial students to quickly access feedback providers by leveraging their online social capital. In order to better understand how to improve crowdsourced online pitch feedback, we perform an experiment to test the effect of online anonymity on pitch feedback quality and quantity. We also test a communal feedback method—evenly distributing between teams feedback providers from the class’s collective online social networks—which would help all teams benefit from a useful amount of feedback rather than having some teams receive much more feedback than others. We found that feedback providers in the anonymous condition provided significantly more specific criticism and specific praise, which students rated as more useful. Furthermore, we found that the communal feedback method helped all teams receive sufficient feedback to edit their pitches. This research contributes an empirical investigation to the crowdsourcing community of how crowds through online social networks can help student entrepreneurs obtain authentic feedback to improve their work.
18
Haq, Ehsan-Ul, Yang K. Lu, and Pan Hui. "It's All Relative! A Method to Counter Human Bias in Crowdsourced Stance Detection of News Articles." Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (November 7, 2022): 1–25. http://dx.doi.org/10.1145/3555636.
Full textAbstract:
Using human intelligence to identify news articles' political stances is common in research and practical applications. But human judgement can be biased and prone to errors stemming from the comprehension of tasks and political alignment. This paper proposes a relative rating method based on news articles' stances relative to raters' own stances to avoid comprehension inconsistency and to control for human bias in crowdsourced stance detection of news articles. We also show how to use the relative ratings to construct a measure for raters' stances on a political topic and to identify raters whose ratings are of higher quality than others. We implement our proposed methods in an online experiment that recruits Amazon Mechanical Turk users as raters for news articles on Gun Control. Using the data from the experiment, we find evidence that raters' own stances on Gun Control significantly impact ratings of related news articles, both at the individual levels and at the aggregate levels. We also present evidence that our relative-rating-based stance measure captures more information about raters' actual stances than their self-reported stance does.
19
Lin, Christopher, Mausam Mausam, and Daniel Weld. "Dynamically Switching between Synergistic Workflows for Crowdsourcing." Proceedings of the AAAI Conference on Artificial Intelligence 26, no. 1 (September 20, 2021): 87–93. http://dx.doi.org/10.1609/aaai.v26i1.8121.
Full textAbstract:
To ensure quality results from unreliable crowdsourced workers, task designers often construct complex workflows and aggregate worker responses from redundant runs. Frequently, they experiment with several alternative workflows to accomplish the task, and eventually deploy the one that achieves the best performance during early trials. Surprisingly, this seemingly natural design paradigm does not achieve the full potential of crowdsourcing. In particular, using a single workflow (even the best) to accomplish a task is suboptimal. We show that alternative workflows can compose synergistically to yield much higher quality output. We formalize the insight with a novel probabilistic graphical model. Based on this model, we design and implement AGENTHUNT, a POMDP-based controller that dynamically switches between these workflows to achieve higher returns on investment. Additionally, we design offline and online methods for learning model parameters. Live experiments on Amazon Mechanical Turk demonstrate the superiority of AGENTHUNT for the task of generating NLP training data, yielding up to 50% error reduction and greater net utility compared to previous methods.
20
Gelino, Brett W., and Derek D. Reed. "Temporal discounting of tornado shelter-seeking intentions amidst standard and impact-based weather alerts: A crowdsourced experiment." Journal of Experimental Psychology: Applied 26, no. 1 (March 2020): 16–25. http://dx.doi.org/10.1037/xap0000246.
Full text
21
Landemore, Hélène. "When public participation matters: The 2010–2013 Icelandic constitutional process." International Journal of Constitutional Law 18, no. 1 (January 2020): 179–205. http://dx.doi.org/10.1093/icon/moaa004.
Full textAbstract:
Abstract Public participation in constitution-making is now both an established international norm and a widespread practice. But what does public participation really mean and when can it be said to matter? This article documents the case of public participation in constitution-making that took place in Iceland between 2010 and 2013. The Icelandic case is interesting both for the innovative ways in which the public was involved but also because public participation in the process can be shown to have made a causal difference to the resulting text. The quasi-natural experiment setup of the Icelandic constitutional process makes it possible to compare the textual output of expert groups and that of the non-professional politicians on the Constitutional Council who crowdsourced their work to the larger public. The comparison suggests that on some crucial aspects the more inclusively written text is marginally but significantly better than that written by experts alone.
22
Muñoz-Gil, Gorka, Alexandre Dauphin, Federica A. Beduini, and Alejandro Sánchez de Miguel. "Citizen Science to Assess Light Pollution with Mobile Phones." Remote Sensing 14, no. 19 (October 6, 2022): 4976. http://dx.doi.org/10.3390/rs14194976.
Full textAbstract:
The analysis of the colour of artificial lights at night has an impact on diverse fields, but current data sources have either limited resolution or scarce availability of images for a specific region. In this work, we propose crowdsourced photos of streetlights as an alternative data source: for this, we designed NightUp Castelldefels, a pilot for a citizen science experiment aimed at collecting data about the colour of streetlights. In particular, we extract the colour from the collected images and compare it to an official database, showing that it is possible to classify streetlights according to their colour from photos taken by untrained citizens with their own smartphones. We also compare our findings to the results obtained from one of the current sources for this kind of study. The comparison highlights how the two approaches give complementary information about artificial lights at night in the area. This work opens a new avenue in the study of the colour of artificial lights at night with the possibility of accurate, massive and cheap data collection.
23
Rogstadius, Jakob, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. "An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets." Proceedings of the International AAAI Conference on Web and Social Media 5, no. 1 (August 3, 2021): 321–28. http://dx.doi.org/10.1609/icwsm.v5i1.14105.
Full textAbstract:
Crowdsourced labor markets represent a powerful new paradigm for accomplishing work. Understanding the motivating factors that lead to high quality work could have significant benefits. However, researchers have so far found that motivating factors such as increased monetary reward generally increase workers’ willingness to accept a task or the speed at which a task is completed, but do not improve the quality of the work. We hypothesize that factors that increase the intrinsic motivation of a task – such as framing a task as helping others – may succeed in improving output quality where extrinsic motivators such as increased pay do not. In this paper we present an experiment testing this hypothesis along with a novel experimental design that enables controlled experimentation with intrinsic and extrinsic motivators in Amazon’s Mechanical Turk, a popular crowdsourcing task market. Results suggest that intrinsic motivation can indeed improve the quality of workers’ output, confirming our hypothesis. Furthermore, we find a synergistic interaction between intrinsic and extrinsic motivators that runs contrary to previous literature suggesting “crowding out” effects. Our results have significant practical and theoretical implications for crowd work.
24
Sonnenfeld, Alessandro, Aprajita Verma, Anupreeta More, Elisabeth Baeten, Christine Macmillan, Kenneth C. Wong, James H. H. Chan, et al. "Survey of Gravitationally-lensed Objects in HSC Imaging (SuGOHI)." Astronomy & Astrophysics 642 (October 2020): A148. http://dx.doi.org/10.1051/0004-6361/202038067.
Full textAbstract:
Context. Strong lenses are extremely useful probes of the distribution of matter on galaxy and cluster scales at cosmological distances, however, they are rare and difficult to find. The number of currently known lenses is on the order of 1000. Aims. The aim of this study is to use crowdsourcing to carry out a lens search targeting massive galaxies selected from over 442 square degrees of photometric data from the Hyper Suprime-Cam (HSC) survey. Methods. Based on the S16A internal data release of the HSC survey, we chose a sample of ∼300 000 galaxies with photometric redshifts in the range of 0.2 < zphot < 1.2 and photometrically inferred stellar masses of log M* > 11.2. We crowdsourced lens finding on this sample of galaxies on the Zooniverse platform as part of the Space Warps project. The sample was complemented by a large set of simulated lenses and visually selected non-lenses for training purposes. Nearly 6000 citizen volunteers participated in the experiment. In parallel, we used YATTALENS, an automated lens-finding algorithm, to look for lenses in the same sample of galaxies. Results. Based on a statistical analysis of classification data from the volunteers, we selected a sample of the most promising ∼1500 candidates, which we then visually inspected: half of them turned out to be possible (grade C) lenses or better. By including lenses found by YATTALENS or serendipitously noticed in the discussion section of the Space Warps website, we were able to find 14 definite lenses (grade A), 129 probable lenses (grade B), and 581 possible lenses. YATTALENS found half the number of lenses that were discovered via crowdsourcing. Conclusions. Crowdsourcing is able to produce samples of lens candidates with high completeness, when multiple images are clearly detected, and with higher purity compared to the currently available automated algorithms. A hybrid approach, in which the visual inspection of samples of lens candidates pre-selected by discovery algorithms or coupled to machine learning is crowdsourced, will be a viable option for lens finding in the 2020s, with forthcoming wide-area surveys such as LSST, Euclid, and WFIRST.
25
Ismail, Qatrunnada, Tousif Ahmed, Kelly Caine, Apu Kapadia, and Michael Reiter. "To Permit or Not to Permit, That is the Usability Question: Crowdsourcing Mobile Apps’ Privacy Permission Settings." Proceedings on Privacy Enhancing Technologies 2017, no. 4 (October 1, 2017): 119–37. http://dx.doi.org/10.1515/popets-2017-0041.
Full textAbstract:
Abstract Millions of apps available to smartphone owners request various permissions to resources on the devices including sensitive data such as location and contact information. Disabling permissions for sensitive resources could improve privacy but can also impact the usability of apps in ways users may not be able to predict. We study an efficient approach that ascertains the impact of disabling permissions on the usability of apps through large-scale, crowdsourced user testing with the ultimate goal of making recommendations to users about which permissions can be disabled for improved privacy without sacrificing usability. We replicate and significantly extend previous analysis that showed the promise of a crowdsourcing approach where crowd workers test and report back on various configurations of an app. Through a large, between-subjects user experiment, our work provides insight into the impact of removing permissions within and across different apps (our participants tested three apps: Facebook Messenger (N=218), Instagram (N=227), and Twitter (N=110)). We study the impact of removing various permissions within and across apps, and we discover that it is possible to increase user privacy by disabling app permissions while also maintaining app usability.
26
Musi, Elena, Elinor Carmi, Chris Reed, Simeon Yates, and Kay O’Halloran. "Developing Misinformation Immunity: How to Reason-Check Fallacious News in a Human–Computer Interaction Environment." Social Media + Society 9, no. 1 (January 2023): 205630512211504. http://dx.doi.org/10.1177/20563051221150407.
Full textAbstract:
To counter the fake news phenomenon, the scholarly community has attempted to debunk and prebunk disinformation. However, misinformation still constitutes a major challenge due to the variety of misleading techniques and their continuous updates which call for the exercise of critical thinking to build resilience. In this study we present two open access chatbots, the Fake News Immunity Chatbot and the Vaccinating News Chatbot, which combine Fallacy Theory and Human–Computer Interaction to inoculate citizens and communication gatekeepers against misinformation. These chatbots differ from existing tools both in function and form. First, they target misinformation and enhance the identification of fallacious arguments; and second, they are multiagent and leverage discourse theories of persuasion in their conversational design. After having described both their backend and their frontend design, we report on the evaluation of the user interface and impact on users’ critical thinking skills through a questionnaire, a crowdsourced survey, and a pilot qualitative experiment. The results shed light on the best practices to design user-friendly active inoculation tools and reveal that the two chatbots are perceived as increasing critical thinking skills in the current misinformation ecosystem.
27
Hotaling, Abigail, and James P. Bagrow. "Efficient crowdsourcing of crowd-generated microtasks." PLOS ONE 15, no. 12 (December 17, 2020): e0244245. http://dx.doi.org/10.1371/journal.pone.0244245.
Full textAbstract:
Allowing members of the crowd to propose novel microtasks for one another is an effective way to combine the efficiencies of traditional microtask work with the inventiveness and hypothesis generation potential of human workers. However, microtask proposal leads to a growing set of tasks that may overwhelm limited crowdsourcer resources. Crowdsourcers can employ methods to utilize their resources efficiently, but algorithmic approaches to efficient crowdsourcing generally require a fixed task set of known size. In this paper, we introduce cost forecasting as a means for a crowdsourcer to use efficient crowdsourcing algorithms with a growing set of microtasks. Cost forecasting allows the crowdsourcer to decide between eliciting new tasks from the crowd or receiving responses to existing tasks based on whether or not new tasks will cost less to complete than existing tasks, efficiently balancing resources as crowdsourcing occurs. Experiments with real and synthetic crowdsourcing data show that cost forecasting leads to improved accuracy. Accuracy and efficiency gains for crowd-generated microtasks hold the promise to further leverage the creativity and wisdom of the crowd, with applications such as generating more informative and diverse training data for machine learning applications and improving the performance of user-generated content and question-answering platforms.
28
A, Hemalatha. "Fake News Detection Using Machine Learning." International Journal for Research in Applied Science and Engineering Technology 10, no. 6 (June 30, 2022): 1472–78. http://dx.doi.org/10.22214/ijraset.2022.44048.
Full textAbstract:
Abstract: The role of social media in our day to day life has increased rapidly in recent years. Information quality in social media is an increasingly important issue, but web-scale data hinders experts’ ability to assess and correct much of the inaccurate content, or “fake news”, present in these platforms. It is now used not only for social interaction, but also as an important platform for exchanging information and news. Twitter, Facebook a micro-blogging service, connects millions of users around the world and allows for the real-time propagation of information and news. The fake news on social media and various other media is wide spreading and is a matter of serious concern due to its ability to cause a lot of social and national damage with destruction impacts. A lot of research is already focused on detecting it. A human being is unable to detect all these fake news. Detecting fake news is an important step. This process will result in feature extraction and vectorization; we propose using Python scikit-learn library to perform tokenization and feature extraction of text data, because this library contains useful tools like Count Vectorizer and Tiff Vectorizer. Then, we will perform feature selection methods, to experiment and choose the best fit features to obtain the highest precision, according to confusion matrix results. A feature analysis then identifies features that are most predictive for crowdsourced and journalistic accuracy assessments, results of which are consistent with prior work. We aim to provide the user with the ability to classify the news as “fake” or “real”.
29
Sarretta, A., and M. Minghini. "TOWARDS THE INTEGRATION OF AUTHORITATIVE AND OPENSTREETMAP GEOSPATIAL DATASETS IN SUPPORT OF THE EUROPEAN STRATEGY FOR DATA." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVI-4/W2-2021 (August 19, 2021): 159–66. http://dx.doi.org/10.5194/isprs-archives-xlvi-4-w2-2021-159-2021.
Full textAbstract:
Abstract. Digital transformation is at core of Europe’s future and the importance of data is well highlighted by the recently published European strategy for data, which envisions the establishment of so-called European data spaces enabling seamless data flows across actors and sectors to ultimately boost the economy and generate innovation. Integrating datasets produced by multiple actors, including citizen-generated data, is a key objective of the strategy. This study focuses on OpenStreetMap (OSM), the most popular crowdsourced geographic information project, and is the first step towards an exploration of pros and cons of integrating its open-licensed data with authoritative geospatial datasets from European National Mapping Agencies. In contrast to previous work, which has only tested data integration at the local or regional level, an experiment was presented to integrate the national address dataset published by the National Land Survey (NLS) of Finland with the corresponding dataset from OSM. The process included the analysis of the two datasets, a mapping between their data models and a set of processing steps – performed using the open source QGIS software – to transform and finally combine their content. The resulting dataset confirms that, while addresses from the NLS are in general more complete across Finland, in some areas OSM addresses provide a higher detail and more up-to-date information to usefully complement the authoritative one. Whilst the analysis confirms that an integration between OSM and authoritative geospatial datasets is technically and semantically feasible, future work is needed to evaluate enablers and barriers that also exist at the legal and organisational level.
30
Lioma, Christina, Birger Larsen, and Peter Ingwersen. "To Phrase or Not to Phrase – Impact of User versus System Term Dependence upon Retrieval." Data and Information Management 2, no. 1 (May 22, 2018): 1–14. http://dx.doi.org/10.2478/dim-2018-0001.
Full textAbstract:
Abstract When submitting queries to information retrieval (IR) systems, users often have the option of specifying which, if any, of the query terms are heavily dependent on each other and should be treated as a fixed phrase, for instance by placing them between quotes.In addition to such cases where users specify term dependence, automatic ways also exist for IR systems to detect dependent terms in queries. Most IR systems use both user and algorithmic approaches. It is not however clear whether and to what extent user-defined term dependence agrees with algorithmic estimates of term dependence, nor which of the two may fetch higher performance gains. Simply put, is it better to trust users or the system to detect term dependence in queries? To answer this question, we experiment with 101 crowdsourced search engine users and 334 queries (52 train and 282 test TREC queries) and we record 10 assessments per query. We find that (i) user assessments of term dependence differ significantly from algorithmic assessments of term dependence (their overlap is approximately 30%); (ii) there is little agreement among users about term dependence in queries, and this disagreement increases as queries become longer; (iii) the potential retrieval gain that can be fetched by treating term dependence (both user- and system-defined) over a bag of words baseline is reserved to a small subset (approximately 8%) of the queries, and is much higher for low-depth than deep precision measures. Points (ii) and (iii) constitute novel insights into term dependence.
31
Alamleh, Hosam, and Ali Abdullah S. AlQahtani. "A weighting system to build physical layer measurements maps by crowdsourcing data from smartphones." IAES International Journal of Robotics and Automation (IJRA) 9, no. 3 (September 1, 2020): 211. http://dx.doi.org/10.11591/ijra.v9i3.pp211-219.
Full textAbstract:
<p>Mobile devices can sense different types of radio signals. For example, broadcast signals. These broadcasted signals allow the device to establish a connection to the access point broadcasting it. Moreover, mobile devices can record different physical layer measurements. These measurements are an indication of the service quality at the point they were collected. These measurements data can be aggregated to form physical layer measurement maps. These maps are useful for several applications such as location fixing, navigation, access control, and evaluating network coverage and performance. Crowdsourcing can be an efficient way to create such maps. However, users in a crowdsourcing application tend to have different devices with different capabilities, which might impact the overall accuracy of the generated maps. In this paper, we propose a method to build physical layer measurements maps by crowdsourcing physical layer measurements, GPS locations, from participating mobile devices. The proposed system gives different weights to each data point provided by the participating devices based on the data source’s trustworthiness. Our tests showed that the different models of mobile devices return GPS location with different location accuracies. Consequently, when building the physical layer measurements maps our algorithm assigns a higher weight to data points coming from devices with higher GPS location accuracy. This allows accommodating a wide range of mobile devices with different capabilities in crowdsourcing applications. An experiment and a simulation were performed to test the proposed method. The results showed improvement in crowdsourced map accuracy when the proposed method is implemented.</p>
32
Andrade, Renato, Ana Alves, and Carlos Bento. "POI Mining for Land Use Classification: A Case Study." ISPRS International Journal of Geo-Information 9, no. 9 (August 20, 2020): 493. http://dx.doi.org/10.3390/ijgi9090493.
Full textAbstract:
The modern planning and management of urban spaces is an essential topic for smart cities and depends on up-to-date and reliable information on land use and the functional roles of the places that integrate urban areas. In the last few years, driven by the increased availability of geo-referenced data from social media, embedded sensors, and remote sensing images, various techniques have become popular for land use analysis. In this paper, we first highlight and discuss the different data types and methods usually adopted in this context, as well as their purposes. Then, based on a systematic state-of-the-art study, we focused on exploring the potential of points of interest (POIs) for land use classification, as one of the most common categories of crowdsourced data. We developed an application to automatically collect POIs for the study area, creating a dataset that was used to generate a large number of features. We used a ranking technique to select, among them, the most suitable features for classifying land use. As ground truth data, we used CORINE Land Cover (CLC), which is a solid and reliable dataset available for the whole European territory. It was used an artificial neural network (ANN) in different scenarios and our results reveal values of more than 90% for the accuracy and F-score in one experiment performed. Our analysis suggests that POI data have promising potential to characterize geographic spaces. The work described here aims to provide an alternative to the current methodologies for land use and land cover (LULC) classification, which are usually time-consuming and depend on expensive data types.
33
Bérubé, Caterina, Zsolt Ferenc Kovacs, Elgar Fleisch, and Tobias Kowatsch. "Reliability of Commercial Voice Assistants’ Responses to Health-Related Questions in Noncommunicable Disease Management: Factorial Experiment Assessing Response Rate and Source of Information." Journal of Medical Internet Research 23, no. 12 (December 20, 2021): e32161. http://dx.doi.org/10.2196/32161.
Full textAbstract:
Background Noncommunicable diseases (NCDs) constitute a burden on public health. These are best controlled through self-management practices, such as self-information. Fostering patients’ access to health-related information through efficient and accessible channels, such as commercial voice assistants (VAs), may support the patients’ ability to make health-related decisions and manage their chronic conditions. Objective This study aims to evaluate the reliability of the most common VAs (ie, Amazon Alexa, Apple Siri, and Google Assistant) in responding to questions about management of the main NCD. Methods We generated health-related questions based on frequently asked questions from health organization, government, medical nonprofit, and other recognized health-related websites about conditions associated with Alzheimer’s disease (AD), lung cancer (LCA), chronic obstructive pulmonary disease, diabetes mellitus (DM), cardiovascular disease, chronic kidney disease (CKD), and cerebrovascular accident (CVA). We then validated them with practicing medical specialists, selecting the 10 most frequent ones. Given the low average frequency of the AD-related questions, we excluded such questions. This resulted in a pool of 60 questions. We submitted the selected questions to VAs in a 3×3×6 fractional factorial design experiment with 3 developers (ie, Amazon, Apple, and Google), 3 modalities (ie, voice only, voice and display, display only), and 6 diseases. We assessed the rate of error-free voice responses and classified the web sources based on previous research (ie, expert, commercial, crowdsourced, or not stated). Results Google showed the highest total response rate, followed by Amazon and Apple. Moreover, although Amazon and Apple showed a comparable response rate in both voice-and-display and voice-only modalities, Google showed a slightly higher response rate in voice only. The same pattern was observed for the rate of expert sources. When considering the response and expert source rate across diseases, we observed that although Google remained comparable, with a slight advantage for LCA and CKD, both Amazon and Apple showed the highest response rate for LCA. However, both Google and Apple showed most often expert sources for CVA, while Amazon did so for DM. Conclusions Google showed the highest response rate and the highest rate of expert sources, leading to the conclusion that Google Assistant would be the most reliable tool in responding to questions about NCD management. However, the rate of expert sources differed across diseases. We urge health organizations to collaborate with Google, Amazon, and Apple to allow their VAs to consistently provide reliable answers to health-related questions on NCD management across the different diseases.
34
Liu, Xipei, and James Bagrow. "Autocompletion interfaces make crowd workers slower, but their use promotes response diversity." Human Computation 6 (June 2, 2019): 42–55. http://dx.doi.org/10.15346/hc.v6i1.89.
Full textAbstract:
Creative tasks such as ideation or question proposal are powerful applications of crowdsourcing, yet the quantity of workers available for addressing practical problems is often insufficient. To enable scalable crowdsourcing thus requires gaining all possible efficiency and information from available workers. One option for text-focused tasks is to allow assistive technology, such as an autocompletion user interface (AUI), to help workers input text responses. But support for the efficacy of AUIs is mixed. Here we designed and conducted a randomized experiment where workers were asked to provide short text responses to given questions. Our experimental goal was to determine if an AUI helps workers respond more quickly and with improved consistency by mitigating typos and misspellings. Surprisingly, we found that neither occurred: workers assigned to the AUI treatment were slower than those assigned to the non-AUI control and their responses were more diverse, not less, than those of the control. Both the lexical and semantic diversities of responses were higher, with the latter measured using word2vec. A crowdsourcer interested in worker speed may want to avoid using an AUI, but using an AUI to boost response diversity may be valuable to crowdsourcers interested in receiving as much novel information from workers as possible.
35
Mazzoleni, Maurizio, Vivian Juliette Cortes Arevalo, Uta Wehn, Leonardo Alfonso, Daniele Norbiato, Martina Monego, Michele Ferri, and Dimitri P. Solomatine. "Exploring the influence of citizen involvement on the assimilation of crowdsourced observations: a modelling study based on the 2013 flood event in the Bacchiglione catchment (Italy)." Hydrology and Earth System Sciences 22, no. 1 (January 17, 2018): 391–416. http://dx.doi.org/10.5194/hess-22-391-2018.
Full textAbstract:
Abstract. To improve hydrological predictions, real-time measurements derived from traditional physical sensors are integrated within mathematic models. Recently, traditional sensors are being complemented with crowdsourced data (social sensors). Although measurements from social sensors can be low cost and more spatially distributed, other factors like spatial variability of citizen involvement, decreasing involvement over time, variable observations accuracy and feasibility for model assimilation play an important role in accurate flood predictions. Only a few studies have investigated the benefit of assimilating uncertain crowdsourced data in hydrological and hydraulic models. In this study, we investigate the usefulness of assimilating crowdsourced observations from a heterogeneous network of static physical, static social and dynamic social sensors. We assess improvements in the model prediction performance for different spatial–temporal scenarios of citizen involvement levels. To that end, we simulate an extreme flood event that occurred in the Bacchiglione catchment (Italy) in May 2013 using a semi-distributed hydrological model with the station at Ponte degli Angeli (Vicenza) as the prediction–validation point. A conceptual hydrological model is implemented by the Alto Adriatico Water Authority and it is used to estimate runoff from the different sub-catchments, while a hydraulic model is implemented to propagate the flow along the river reach. In both models, a Kalman filter is implemented to assimilate the crowdsourced observations. Synthetic crowdsourced observations are generated for either static social or dynamic social sensors because these measures were not available at the time of the study. We consider two sets of experiments: (i) assuming random probability of receiving crowdsourced observations and (ii) using theoretical scenarios of citizen motivations, and consequent involvement levels, based on population distribution. The results demonstrate the usefulness of integrating crowdsourced observations. First, the assimilation of crowdsourced observations located at upstream points of the Bacchiglione catchment ensure high model performance for high lead-time values, whereas observations at the outlet of the catchments provide good results for short lead times. Second, biased and inaccurate crowdsourced observations can significantly affect model results. Third, the theoretical scenario of citizens motivated by their feeling of belonging to a community of friends has the best effect in the model performance. However, flood prediction only improved when such small communities are located in the upstream portion of the Bacchiglione catchment. Finally, decreasing involvement over time leads to a reduction in model performance and consequently inaccurate flood forecasts.
36
Shen, Zixuan, Zhihua Xia, and Peipeng Yu. "PLDP: Personalized Local Differential Privacy for Multidimensional Data Aggregation." Security and Communication Networks 2021 (January 28, 2021): 1–13. http://dx.doi.org/10.1155/2021/6684179.
Full textAbstract:
The collection of multidimensional crowdsourced data has caused a public concern because of the privacy issues. To address it, local differential privacy (LDP) is proposed to protect the crowdsourced data without much loss of usage, which is popularly used in practice. However, the existing LDP protocols ignore users’ personal privacy requirements in spite of offering good utility for multidimensional crowdsourced data. In this paper, we consider the personality of data owners in protection and utilization of their multidimensional data by introducing the notion of personalized LDP (PLDP). Specifically, we design personalized multiple optimized unary encoding (PMOUE) to perturb data owners’ data, which satisfies ϵ total -PLDP. Then, the aggregation algorithm for frequency estimation on multidimensional data under PLDP is developed, which is described in two situations. Experiments are conducted on four real datasets, and the results show that the proposed aggregation algorithm yields high utility. Moreover, case studies with four real datasets demonstrate the efficiency and superiority of the proposed scheme.
37
Sayin, Burcu, Evgeny Krivosheev, Jie Yang, Andrea Passerini, and Fabio Casati. "A review and experimental analysis of active learning over crowdsourced data." Artificial Intelligence Review 54, no. 7 (May 30, 2021): 5283–305. http://dx.doi.org/10.1007/s10462-021-10021-3.
Full textAbstract:
AbstractTraining data creation is increasingly a key bottleneck for developing machine learning, especially for deep learning systems. Active learning provides a cost-effective means for creating training data by selecting the most informative instances for labeling. Labels in real applications are often collected from crowdsourcing, which engages online crowds for data labeling at scale. Despite the importance of using crowdsourced data in the active learning process, an analysis of how the existing active learning approaches behave over crowdsourced data is currently missing. This paper aims to fill this gap by reviewing the existing active learning approaches and then testing a set of benchmarking ones on crowdsourced datasets. We provide a comprehensive and systematic survey of the recent research on active learning in the hybrid human–machine classification setting, where crowd workers contribute labels (often noisy) to either directly classify data instances or to train machine learning models. We identify three categories of state of the art active learning methods according to whether and how predefined queries employed for data sampling, namely fixed-strategy approaches, dynamic-strategy approaches, and strategy-free approaches. We then conduct an empirical study on their cost-effectiveness, showing that the performance of the existing active learning approaches is affected by many factors in hybrid classification contexts, such as the noise level of data, label fusion technique used, and the specific characteristics of the task. Finally, we discuss challenges and identify potential directions to design active learning strategies for hybrid classification problems.
38
Keuschnigg, Marc, Felix Bader, and Johannes Bracher. "Using crowdsourced online experiments to study context-dependency of behavior." Social Science Research 59 (September 2016): 68–82. http://dx.doi.org/10.1016/j.ssresearch.2016.04.014.
Full text
39
Giancola, Michael, Randy Paffenroth, and Jacob Whitehill. "Permutation-Invariant Consensus over Crowdsourced Labels." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 6 (June 15, 2018): 21–30. http://dx.doi.org/10.1609/hcomp.v6i1.13326.
Full textAbstract:
This paper introduces a novel crowdsourcing consensus model and inference algorithm — which we call PICA (Permutation-Invariant Crowdsourcing Aggregation) — that is designed to recover the ground-truth labels of a dataset while being invariant to the class permutations enacted by the different annotators. This is particularly useful for settings in which annotators may have systematic confusions about the meanings of different classes, as well as clustering problems (e.g., dense pixel-wise image segmentation) in which the names/numbers assigned to each cluster have no inherent meaning.The PICA model is constructed by endowing each annotator with a doubly-stochastic matrix (DSM), which models the probabilities that an annotator will perceive one class and transcribe it into another. We conduct simulations and experiments to show the advantage of PICA compared to two baselines (Majority Vote, and an "unpermutation" heuristic) for three different clustering/labeling tasks. We also explore the conditions under which PICA provides better inference accuracy compared to a simpler but related model based on right-stochastic matrices. Finally, we show that PICA can be used to crowdsource responses for dense image segmentation tasks, and provide a proof-of-concept that aggregating responses in this way could improve the accuracy of this labor-intensive task.
40
Li, Haifeng, Xin Dou, Chao Tao, Zhixiang Wu, Jie Chen, Jian Peng, Min Deng, and Ling Zhao. "RSI-CB: A Large-Scale Remote Sensing Image Classification Benchmark Using Crowdsourced Data." Sensors 20, no. 6 (March 12, 2020): 1594. http://dx.doi.org/10.3390/s20061594.
Full textAbstract:
Image classification is a fundamental task in remote sensing image processing. In recent years, deep convolutional neural networks (DCNNs) have experienced significant breakthroughs in natural image recognition. The remote sensing field, however, is still lacking a large-scale benchmark similar to ImageNet. In this paper, we propose a remote sensing image classification benchmark (RSI-CB) based on massive, scalable, and diverse crowdsourced data. Using crowdsourced data, such as Open Street Map (OSM) data, ground objects in remote sensing images can be annotated effectively using points of interest, vector data from OSM, or other crowdsourced data. These annotated images can, then, be used in remote sensing image classification tasks. Based on this method, we construct a worldwide large-scale benchmark for remote sensing image classification. This benchmark has large-scale geographical distribution and large total image number. It contains six categories with 35 sub-classes of more than 24,000 images of size 256 × 256 pixels. This classification system of ground objects is defined according to the national standard of land-use classification in China and is inspired by the hierarchy mechanism of ImageNet. Finally, we conduct numerous experiments to compare RSI-CB with the SAT-4, SAT-6, and UC-Merced data sets. The experiments show that RSI-CB is more suitable as a benchmark for remote sensing image classification tasks than other benchmarks in the big data era and has many potential applications.
41
Wan, Qiao, Xiaoqi Duan, Yue Yu, Ruizhi Chen, and Liang Chen. "Self-Calibrated Multi-Floor Localization Based on Wi-Fi Ranging/Crowdsourced Fingerprinting and Low-Cost Sensors." Remote Sensing 14, no. 21 (October 27, 2022): 5376. http://dx.doi.org/10.3390/rs14215376.
Full textAbstract:
Crowdsourced localization using geo-spatial big data has become an effective approach for constructing smart-city-based location services with the fast growing number of Internet of Things terminals. This paper presents a self-calibrated multi-floor indoor positioning framework using a combination of Wi-Fi ranging, crowdsourced fingerprinting and low-cost sensors (SM-WRFS). The localization parameters, such as heading and altitude biases, step-length scale factor, and Wi-Fi ranging bias are autonomously calibrated to provide a more accurate forward 3D localization performance. In addition, the backward smoothing algorithm and a novel deep-learning model are applied in order to construct an autonomous and efficient crowdsourced Wi-Fi fingerprinting database using the detected quick response (QR) code-based landmarks. Finally, the adaptive extended Kalman filter is adopted to combine the corresponding location sources using different integration models to provide a precise multi-source fusion based multi-floor indoor localization performance. The real-world experiments demonstrate that the presented SM-WRFS is proven to realize precise 3D indoor positioning under different environments, and the meter-level positioning accuracy can be acquired in Wi-Fi ranging supported indoor areas.
42
Foster, Joshua. "On the Relative Efficiency of Crowdsourced Rating Mechanisms: Experimental Evidence." Academy of Management Proceedings 2020, no. 1 (August 2020): 19583. http://dx.doi.org/10.5465/ambpp.2020.19583abstract.
Full text
43
Medina, Jose R., Ramadan Salim, B. Shane Underwood, and Kamil Kaloush. "Experimental Study for Crowdsourced Ride Quality Index Estimation Using Smartphones." Journal of Transportation Engineering, Part B: Pavements 146, no. 4 (December 2020): 04020070. http://dx.doi.org/10.1061/jpeodx.0000225.
Full text
44
Guo, Shikai, Rong Chen, Hui Li, Jian Gao, and Yaqing Liu. "Crowdsourced Web Application Testing Under Real-Time Constraints." International Journal of Software Engineering and Knowledge Engineering 28, no. 06 (June 2018): 751–79. http://dx.doi.org/10.1142/s0218194018500213.
Full textAbstract:
Crowdsourcing carried out by cyber citizens instead of hired consultants and professionals has become increasingly an appealing solution to test the feature rich and interactive web. Despite having various online crowdsourcing testing services, the benefits of exposure to a wider audience and harnessing the collective efforts of individuals remain uncertain, especially when the quality control is problematic in an open environment. The objective of this paper is to propose a real-time collaborative testing approach (RCTA) to create a productive crowdsourced testing on a dynamic Internet. We implemented a prototype crowdsourcing system XTurk, and carried out a case study, to understand the crowdsourced testers behavior, the trustworthiness, the execution time of test cases and accuracy of feedback. Several experiments are carried out and experimental results validate the quality, efficiency and reliability of the present approach and the positive testing feedback is are shown to outperform the previous methods.
45
Méndez Méndez, Ana Elisa, Mark Cartwright, Juan Pablo Bello, and Oded Nov. "Eliciting Confidence for Improving Crowdsourced Audio Annotations." Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (March 30, 2022): 1–25. http://dx.doi.org/10.1145/3512935.
Full textAbstract:
In this work we explore confidence elicitation methods for crowdsourcing "soft" labels, e.g., probability estimates, to reduce the annotation costs for domains with ambiguous data. Machine learning research has shown that such "soft" labels are more informative and can reduce the data requirements when training supervised machine learning models. By reducing the number of required labels, we can reduce the costs of slow annotation processes such as audio annotation. In our experiments we evaluated three confidence elicitation methods: 1) "No Confidence" elicitation, 2) "Simple Confidence" elicitation, and 3) "Betting" mechanism for confidence elicitation, at both individual (i.e., per participant) and aggregate (i.e., crowd) levels. In addition, we evaluated the interaction between confidence elicitation methods, annotation types (binary, probability, and z-score derived probability), and "soft" versus "hard" (i.e., binarized) aggregate labels. Our results show that both confidence elicitation mechanisms result in higher annotation quality than the "No Confidence" mechanism for binary annotations at both participant and recording levels. In addition, when aggregating labels at the recording level, results indicate that we can achieve comparable results to those with 10-participant aggregate annotations using fewer annotators if we aggregate "soft" labels instead of "hard" labels. These results suggest that for binary audio annotation using a confidence elicitation mechanism and aggregating continuous labels we can obtain higher annotation quality, more informative labels, with quality differences more pronounced with fewer participants. Finally, we propose a way of integrating these confidence elicitation methods into a two-stage, multi-label annotation pipeline.
46
Luger, Sarah. "The Effect of Text Length in Crowdsourced Multiple Choice Questions." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 3 (March 28, 2016): 16–19. http://dx.doi.org/10.1609/hcomp.v3i1.13268.
Full textAbstract:
Automated systems that aid in the development of Multiple Choice Questions (MCQs) have value for both educators, who spend large amounts of time creating novel questions, and students, who spend a great deal of effort both practicing for and taking tests. The current approach for measuring question difficulty in MCQs relies on models of how good pupils will perform and contrasts that with their lower-performing peers. MCQs can be difficult in many ways. This paper looks specifically at the effect of both the number of words in the question stem and in the answer options on question difficulty. This work is based on the hypothesis that questions are more difficult if the stem of the question and the answer options are semantically far apart. This hypothesis can be normalized, in part, with an analysis of the length of texts being compared. The MCQs used in the experiments were voluntarily authored by university students in biology courses. Future work includes additional experiments utilizing other aspects of this extensive crowdsourced data set.
47
Vo-Phamhi, Jenny M., Kevin A. Yamauchi, and Rafael Gómez-Sjöberg. "Validation and tuning of in situ transcriptomics image processing workflows with crowdsourced annotations." PLOS Computational Biology 17, no. 8 (August 9, 2021): e1009274. http://dx.doi.org/10.1371/journal.pcbi.1009274.
Full textAbstract:
Recent advancements in in situ methods, such as multiplexed in situ RNA hybridization and in situ RNA sequencing, have deepened our understanding of the way biological processes are spatially organized in tissues. Automated image processing and spot-calling algorithms for analyzing in situ transcriptomics images have many parameters which need to be tuned for optimal detection. Having ground truth datasets (images where there is very high confidence on the accuracy of the detected spots) is essential for evaluating these algorithms and tuning their parameters. We present a first-in-kind open-source toolkit and framework for in situ transcriptomics image analysis that incorporates crowdsourced annotations, alongside expert annotations, as a source of ground truth for the analysis of in situ transcriptomics images. The kit includes tools for preparing images for crowdsourcing annotation to optimize crowdsourced workers’ ability to annotate these images reliably, performing quality control (QC) on worker annotations, extracting candidate parameters for spot-calling algorithms from sample images, tuning parameters for spot-calling algorithms, and evaluating spot-calling algorithms and worker performance. These tools are wrapped in a modular pipeline with a flexible structure that allows users to take advantage of crowdsourced annotations from any source of their choice. We tested the pipeline using real and synthetic in situ transcriptomics images and annotations from the Amazon Mechanical Turk system obtained via Quanti.us. Using real images from in situ experiments and simulated images produced by one of the tools in the kit, we studied worker sensitivity to spot characteristics and established rules for annotation QC. We explored and demonstrated the use of ground truth generated in this way for validating spot-calling algorithms and tuning their parameters, and confirmed that consensus crowdsourced annotations are a viable substitute for expert-generated ground truth for these purposes.
48
Zhao, Yuan, Tieke He, and Zhenyu Chen. "A Unified Framework for Bug Report Assignment." International Journal of Software Engineering and Knowledge Engineering 29, no. 04 (April 2019): 607–28. http://dx.doi.org/10.1142/s0218194019500256.
Full textAbstract:
It is typically a manual, time-consuming, and tedious task of assigning bug reports to individual developers. Although some machine learning techniques are adopted to alleviate this dilemma, they are mainly focused on the open source projects, which use traditional repositories such as Bugzilla to manage their bug reports. With the boom of the mobile Internet, some new requirements and methods of software testing are emerging, especially the crowdsourced testing. Unlike the traditional channels, whose bug reports are often heavyweight, which means their bug reports are standardized with detailed attribute localization, bug reports tend to be lightweight in the context of crowdsourced testing. To exploit the differences of the bug reports assignment in the new settings, a unified bug reports assignment framework is proposed in this paper. This framework is capable of handling both the traditional heavyweight bug reports and the lightweight ones by (i) first preprocessing the bug reports and feature selections, (ii) then tuning the parameters that indicate the ratios of choosing different methods to vectorize bug reports, (iii) and finally applying classification algorithms to assign bug reports. Extensive experiments are conducted on three datasets to evaluate the proposed framework. The results indicate the applicability of the proposed framework, and also reveal the differences of bug report assignment between traditional repositories and crowdsourced ones.
49
Christoforou, Evgenia, Antonio Fernández Anta, and Angel Sánchez. "An experimental characterization of workers’ behavior and accuracy in crowdsourced tasks." PLOS ONE 16, no. 6 (June 16, 2021): e0252604. http://dx.doi.org/10.1371/journal.pone.0252604.
Full textAbstract:
Crowdsourcing systems are evolving into a powerful tool of choice to deal with repetitive or lengthy human-based tasks. Prominent among those is Amazon Mechanical Turk, in which Human Intelligence Tasks, are posted by requesters, and afterwards selected and executed by subscribed (human) workers in the platform. Many times these HITs serve for research purposes. In this context, a very important question is how reliable the results obtained through these platforms are, in view of the limited control a requester has on the workers’ actions. Various control techniques are currently proposed but they are not free from shortcomings, and their use must be accompanied by a deeper understanding of the workers’ behavior. In this work, we attempt to interpret the workers’ behavior and reliability level in the absence of control techniques. To do so, we perform a series of experiments with 600 distinct MTurk workers, specifically designed to elicit the worker’s level of dedication to a task, according to the task’s nature and difficulty. We show that the time required by a worker to carry out a task correlates with its difficulty, and also with the quality of the outcome. We find that there are different types of workers. While some of them are willing to invest a significant amount of time to arrive at the correct answer, at the same time we observe a significant fraction of workers that reply with a wrong answer. For the latter, the difficulty of the task and the very short time they took to reply suggest that they, intentionally, did not even attempt to solve the task.
50
Lin, Junhong, Bang Wang, Guang Yang, and Mu Zhou. "Indoor Localization Based on Weighted Surfacing from Crowdsourced Samples." Sensors 18, no. 9 (September 7, 2018): 2990. http://dx.doi.org/10.3390/s18092990.
Full textAbstract:
Fingerprinting-based indoor localization suffers from its time-consuming and labor-intensive site survey. As a promising solution, sample crowdsourcing has been recently promoted to exploit casually collected samples for building offline fingerprint database. However, crowdsourced samples may be annotated with erroneous locations, which raises a serious question about whether they are reliable for database construction. In this paper, we propose a cross-domain cluster intersection algorithm to weight each sample reliability. We then select those samples with higher weight to construct radio propagation surfaces by fitting polynomial functions. Furthermore, we employ an entropy-like measure to weight constructed surfaces for quantifying their different subarea consistencies and location discriminations in online positioning. Field measurements and experiments show that the proposed scheme can achieve high localization accuracy by well dealing with the sample annotation error and nonuniform density challenges.