False Belief Crowdsourcing Contest

  1. What percentage of gun deaths in 2016 were caused by assault-style rifles?

  2. Resumes with white-sounding names were how much more likely to get call-backs than the same resume with a black-sounding name?

Pilot testing has shown that Republicans are far more likely to give the correct answer to the first question (2%), whereas Democrats are far more likely to correctly answer the second question (50%). We are interested in developing a database of similar questions that tap into these differences between Republicans and Democrats.

Specifically, our project uses crowdsourcing to assemble trivia questions that reveal differences in cultural knowledge between Republicans and Democrats. As explained in the Overview section, there are two types of “False Belief” questions, and participants may submit as many questions as they like of each type. The best questions in each category will receive a $100 reward.

Contest Overview

We are accepting submissions on two types of trivia questions.

Republican “False Beliefs”

These are politically-relevant questions that a person who identifies as a Democrat is much more likely to get correct than a person who identifies as a Republican. Crucially, both Republican and Democrat respondents should have a strong hunch that their answer is right, while only one (easily verifiable) response will actually be correct. Because these questions are politically controversial, they should be selected and phrased in a way that the correct answer can be verified (citing specific statistics, for example).

Here are some pre-tested examples where the average Democrat is likely to guess closer to the correct answer than the average Republican:

What percentage of climate change scientists believe that climate change is primarily caused by human activity?

  • 97%
  • 88%
  • 62%
  • 26%

Source

Resumés submitted to job openings with White-sounding names were how much more likely to get a call back than the same resumés submitted with Black-sounding names?

  • 50%
  • 25%
  • 10%
  • <1%

Source

Of the 62 mass shootings between 1982 and 2012 how many were stopped by an armed civilian?

  • 0%
  • 9%
  • 15%
  • 25%

Source

Democrat “False Beliefs”

These are politically-relevant questions that a person who identifies as a Republican is much more likely to get correct than a person who identifies as a Democrat. Crucially, both Republican and Democrat respondents should have a strong hunch that their answer is right, while only one (easily verifiable) response will actually be correct. Because these questions are politically controversial, they should be selected and phrased in a way that the correct answer can be verified (citing specific statistics, for example). Here are some examples where the average Republican is likely to guess closer to the correct answer than the average Democrat:

Here are some examples where the average Republican is likely to guess closer to the correct answer than the average Democrat:

What is the safest form of energy, as determined by “mortality rate per kilowatt hour?”

  • Nuclear power
  • Solar power
  • Wind power
  • Hydro power

Source 1, Source 2

What percentage of gun deaths in 2016 were caused by assault-style rifles?

  • 2%
  • 14%
  • 35%
  • 56%

Source

Which religion had the largest number of hate crimes directed towards it in 2016?

  • Judaism
  • Islam
  • Christianity
  • Atheism

Source

To be clear: we are not looking for questions about which people in different political parties already have conflicting explicit beliefs. For example, a bad question would be, “Where was Barack Obama born?” While this question is likely to generate large differences in the pattern of answers between Republicans and Democrats, and both sides will have strong hunches (if not strong convictions) that their answer is correct, we are interested only in questions where Republicans and Democrats don’t already have a precise, pre-existing belief about the answer to a question. It should be possible for both sides to accept the right answer when it is revealed. This requires questions for which partisans have intuitive “hunches” guided by their politics, but not explicit, specific beliefs.

Answers to all questions must be backed by credible sources. We welcome submissions based on information in mainstream news articles. However, the news articles must reference a primary source for the information, such as a scientific journal article, institutional research report, or government report. Examples of non-credible sources include blog posts or opinion articles that do not reference primary sources. We reserve the right to determine which sources are credible.

Judging Criteria

Submission Criteria

  • Submissions must be in English.

  • Participants may submit as many questions as they like, but note that not all questions will pass our initial review. (See below for details). Submitting a large number of low-quality questions will not increase your chances of winning a prize.

  • All questions should be submitted with exactly four answer choices.

  • Prizes will be awarded to the top two submissions for each of the following categories: Democrat “False Beliefs” and Republican “False Beliefs” (see prizes tab for award amounts)

  • The deadline for submitting questions is October 20th, 2019 at 11:59 p.m.

Judging Criteria

To determine winners for each category of question, we will do the following:

  1. All questions will be subject to an initial review designed to screen out low-quality questions. (Given that we do not know how many questions we will receive, we cannot commit in advance to testing all submitted questions. It is our intention, however, to test all questions that could plausibly meet our criteria.)

  2. We will collect all of the questions that pass our initial review and ask a large sample of real Republican and Democrat respondents to answer them. If two or more participants submit what is substantively the same question, then only the participant who submitted the question first will be considered for a prize.

  3. To qualify as a high-quality “False Belief” question, participants from one major party should, on average, get the answer wrong more often than participants from the other major party. Additionally, both participants should have a hunch that their answer is right. Hunches will be gauged by asking participants to register the degree of confidence they have in their answers. We define a reasonable hunch as having 50% or more confidence in one’s answer.

  4. We will award prizes to the submissions that maximize the difference between the percentage of Republicans answering correctly and the percentage of Democrats answering correctly. (For the best questions we have developed so far, we have found that on average, the difference in the proportions of each party responding correctly exceeds 50%–for example, 76% of Republicans answer correctly, but only 24% of Democrats answer correctly.) We will award at least two prizes in each category, and we will reward any submission beyond the top two that we consider to be “high-quality.”

For Republican “False Belief” questions (questions Republicans are likely to get wrong), we compute the following:

$$\mid \text{∣% Correct for Democrats - % Correct for Republicans} \mid $$

For Democratic “False Belief” questions, we compute the following:

$$\mid \text{% Correct for Republicans - % Correct for Democrats} \mid $$

Awards will be given for the questions that maximize the above quantities. We reserve the right to use our discretion to reward questions that are particularly unique or clever, over and above the strict quantitative criteria described above.

We will award $100 to any high-quality “False Belief” question. In order to qualify as a “high-quality” question, the difference in the proportions of each party responding correctly should exceed 40%. So, for example, a Republican could answer the submitted question correctly 82% of the time, whereas a Democrat could answer correctly 40% of the time, resulting in an accuracy spread of 42%.

Moreover, the submitted question must satisfy the criteria described in the Overview section (e.g. they must be backed by credible sources; both parties should have a hunch that their answer is right; and the question should cover topics for which there is not precise, pre-existing beliefs about the answer to a question). In the event that a large number of high-quality questions are submitted, we will reward prizes to only the Top 25 “False Belief” questions.

Prizes

Prizes will be awarded as follows:

  • Two $100 prizes will be awarded for the top two questions in the “Democrat False Beliefs” category
  • Two $100 prizes will be awarded for the top two questions in the “Republican False Beliefs” category
  • $100 prizes will be awarded to any question that we determine to be of “high-quality” (see Judging page for details)

Submit

To submit to this contest please click here and fill out the survey. You are free to submit as many trivia questions as you like, although we may discard submissions from participants who submit a large number of questions that in our view have little chance of being high-quality.

The survey above will ask you to pre-specify how many questions you are submitting. For each question, you will be required to submit four multiple choice answers. Please bear in mind that high-quality questions must have well-designed incorrect answers, as the incorrect answers presented have a large effect on accuracy rates.

If you have already participated and would like to participate again, you may do so. But repeat participants will need to re-complete the demographic questions at the end so that we can match it to your previous submission.

About

Who are we?

This project is being managed by Evan DeFilippis (PhD student in Psychology and Organizational Behavior) and Joshua Greene (Professor of Psychology) at Harvard University.

Why are we doing this?

We are collecting questions that tap into differences in cultural knowledge and political belief between Republicans and Democrats. We will use these questions as testing materials in our research examining the causes of social/political beliefs and attitudes. These testing materials may eventually be made available to other researchers.

Why crowdsource?

“False Belief” questions are hard to design because it is difficult to know what one does not know. Furthermore, even if one can identify topics about which Republicans or Democrats are likely to be especially ignorant or ill-informed, it is difficult to identify specific questions that expose such differences. As such, we expect that the most talented and creative designers of these questions are not part of our research group. We hope that the best questions will likely come from one of you.

Contact