Belief elicitation in the presence of naïve respondents: An experimental study (original) (raw)
Related papers
Belief elicitation in experiments: is there a hedging problem?
2010
Belief Elicitation in Experiments: Is there a Hedging Problem? * Belief elicitation in economics experiments usually relies on paying subjects according to the accuracy of stated beliefs in addition to payments for other decisions. Such incentives, however, allow risk-averse subjects to hedge with their stated beliefs against adverse outcomes of other decisions in the experiment. This raises two questions: (i) can we trust the existing belief elicitation results, (ii) can we avoid potential hedging confounds? Our results instill confidence regarding both issues. We propose an experimental design that eliminates hedging opportunities, and use this to test for the empirical relevance of hedging effects in the lab. We find no evidence for hedging, comparing the standard "hedging-prone" belief elicitation treatment to a "hedging-proof" design in a sequential prisoners' dilemma game. Our findings are strengthened by the absence of hedging even in an additional non-belief elicitation treatment using a financial investment frame, where hedging arguably would be most natural.
Getting it right the first time: Belief elicitation with novice participants
2010
Abstract: The auction design literature makes clear that theoretically equivalent mechanisms can perform very differently in practice. Though of equal importance, much less is known about the empirical performance of theoretically equivalent mechanisms for belief elicitation. This is especially unfortunate given the increasing interest in eliciting beliefs from (often novice) respondents in large-scale surveys.
A Penny for Your Thoughts: A Survey of Methods for Belief Elicitation
Experimental Economics
Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons and give a new justification for the quadratic scoring rule. On the empirical side, we survey the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with a discussion of the trade-offs involved in using incentives for belief elicitation and some guidelines for implementation.
Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging
Accurate measurements of probabilistic beliefs have become increasingly important both in practice and in academia. Introduced by statisticians in the 1950s to promote truthful reports in simple environments, Proper Scoring Rules (PSR) are now arguably the most popular incentivized mechanisms to elicit an agent's beliefs. This paper generalizes the analysis of PSR to richer environments relevant to economists. More speci…cally, we combine theory and experiment to study how beliefs reported with a PSR may be biased when i) the PSR payments are increased, ii) the agent has a …nancial stake in the event she is predicting, and iii) the agent can hedge her prediction by taking an additional action. Our results reveal complex distortions of reported beliefs, thereby raising concerns about the ability of PSR to recover truthful beliefs in general economic environments.
Belief elicitation in the presence of na��ve respondents: An experimental study
2012
Abstract It is often of interest to elicit beliefs from populations that may include na��ve participants. Unfortunately, elicitation mechanisms are typically assessed by assuming optimal responses to incentives. Using laboratory experiments with a population that potentially includes na��ve participants, we compare the performance of two elicitation mechanisms proposed by Karni (Econometrica 77 (2): 603-606, 2009).
Eliciting beliefs in the laboratory
2006
Belief elicitation methods based on proper scoring rules such as the quadratic scoring rule provide the experimenter with little opportunity to control the incentives for optimal reporting behavior by experimental subjects. Since subjects are typically risk averse, distortions in belief reports should be expected from any kind of proper scoring rule. But how likely and how large are these distortions in practice? Can one correct for such distortions? And what impact will corrections have on the incentives for optimal behavior? We approach these questions theoretically and empirically. Our theoretical model shows how the widely used quadratic scoring rule can be generalized and represented as a contingent wealth opportunity set described by the indirect utility function of a CRRA agent in a competitive contingent claims market. This representation has a naturally-occurring form, much like the markets that sports betting agencies have developed. Optimal reporting behavior is logically equivalent to optimal pricing behavior against compensated demand functions of a consumer whose certainty equivalent, in a dual perspective, has the form of a CES utility function. The parameters of this utility function are the risk attitude (elasticity of substitution), distributional weights, and endowment location in the space of available experimental funds. Critically, these parameters are all under the control of the experimenter. We provide graphical examples to show how variations in the CRRA/CES parameters of this constraint impact on the incentives for optimal reporting of beliefs by subjects. The class of incentive functions we develop provide significantly stronger penalties for sub-optimal belief reporting behavior than the conventional quadratic scoring rule. The theory also suggests that belief reporting in the lab be framed for the subject as a pricing problem. Besides having strong conceptual and analytical foundations in De Finetti/Savage type Bayesian statistics where probabilities are viewed as prices, the idea of having a subject set odds or prices in a contingent claims market may help to overcome well known difficulties subjects face in understanding the language of frequency and probability. Their odds-setting or pricing behavior will reveal their beliefs even without them explicitly articulating their beliefs as probability distributions. We provide experimental evidence evaluating the performance of this new approach to belief elicitation.
Identification of Biased Beliefs in Games of Incomplete Information Using Experimental Data
SSRN Electronic Journal, 2017
This paper studies the identi…cation of players'preferences and beliefs in empirical applications of discrete choice games using experimental data. The experiment comprises a set of games with similar features (e.g., two-player coordination games) where each game has di¤erent values for the players'monetary payo¤s. Each game can be interpreted as an experimental treatment group. The researcher assigns randomly subjects to play these games and observes the outcome of the game as described by the vector of players' actions. Data from this experiment can be described in terms of the empirical distribution of players'actions conditional on the treatment group. The researcher is interested in the nonparametric identi…cation of players' preferences (utility function of money) and players' beliefs about the expected behavior of other players, without imposing restrictions such as unbiased or rational beliefs or a particular functional form for the utility of money. We show that the hypothesis of unbiased/rational beliefs is testable and propose a test of this null hypothesis. We apply our method to two sets of experiments conducted by Goeree and Holt (2001) and Heinemann, Nagel and Ockenfels (2009). Our empirical results suggest that in the matching pennies game, a player is able to correctly predict other player's behavior. In the public good coordination game, our test can reject the null hypothesis of unbiased beliefs when the payo¤ of the non-cooperative action is relatively low.
Theory and Decision, 2008
We develop an algorithm that can be used to approximate a decisionmaker's beliefs for a class of preference structures that includes, among others, α-maximin expected utility preferences, Choquet expected utility preferences, and, more generally, constant additive preferences. For both exact and statistical approximation, we demonstrate convergence in an appropriate sense to the true belief structure.
Opting-in: Participation bias in economic experiments
Journal of Economic Behavior & Organization, 2013
Assuming individuals rationally decide whether to participate or not to participate in lab experiments, we hypothesize several non-representative biases in the characteristics of lab participants. We test the hypotheses by first collecting survey and experimental data from a typical recruitment population and then inviting them to participate in a lab experiment. The results indicate that lab participants are not representative of the target population on almost all the hypothesized characteristics, including having lower income, working fewer hours, volunteering more often, and exhibiting behaviors correlated with interest in experiments and economics. The results reinforce the commonly understood limits of laboratory research to make quantitative inferences. We also discuss several methods for addressing non-representative biases to advance laboratory methods for improving quantitative inferences and consequently increasing confidence in qualitative conclusions.
2016
This paper studies the identification of players’preferences and beliefs in empirical applications of discrete choice games using experimental data. The experiment comprises a set of games with similar features (e.g., two-player coordination games) where each game has different values for the players’monetary payoffs. Each game can be interpreted as an experimental treatment group. The researcher assigns randomly subjects to play these games and observes the outcome of the game as described by the vector of players’actions. Data from this experiment can be described in terms of the empirical distribution of players’actions conditional on the treatment group. The researcher is interested in the nonparametric identification of players’preferences (utility function of money) and players’beliefs about the expected behavior of other players, without imposing restrictions such as unbiased or rational beliefs or a particular functional form for the utility of money. We show that the hypoth...