2.1. Quantile price lists

To make the discussion of our procedure more concrete, suppose that we want to learn about a participant’s belief about the distance between Los Angeles and San Diego. We have chosen this example to demonstrate that we can elicit subjective beliefs about events that are not inherently random. Ultimately, our methodology applies to eliciting beliefs about any quantity, random or not, as long as it can be observed or verified.Footnote ²

Let us start with a participant who is somewhat familiar with the geography of California. They know that these cities are both in southern California and are not that far apart. Suppose that we ask this participant if they would rather have $\$10$ with a $75\%$ chance or $\$10$ if the distance between the two cities is less than $1,000$ miles. This participant believes that the distance is almost certain to be less than $1,000$ miles and chooses the latter option, since they believe that it will yield the $\$10$ with a nearly $100\%$ chance. We can conclude that whatever the distribution of their belief F about the distance between these two cities, it is the case that $F(X\leq 1,000)\geq0.75$.

Now suppose that we continue asking questions like this, but where the second option pays conditional on the distance being below $900,800,700,600,500,400$, and 300, respectively. In each case, the participant chooses the second option. However, they conclude that the chance that the distance is below 200 miles is less than $75\%$ and, when asked to compare being paid $\$10$ with a $75\%$ chance and being paid conditional on the distance being below 200 miles, they now choose the first option. From these choices, we can conclude that for their belief F, $F(X\leq 300)\geq0.75$ but $F(X\leq 200)\leq0.75$. Thus, whatever number x solves $F(X\leq x)=0.75$ (the 75th quantile) must be between 200 and 300.Footnote ³ Gathering these questions into a list results in what we call a quantile price list, an example of which is shown in Figure 1.

Fig. 1. A quantile price list for the 0.75 quantile belief about the distance between Los Angeles and San Diego. The chosen options indicate the 0.75 quantile of the participants’ belief is between 200 and 300.

To find a different quantile, we repeat this exercise with a different objective lottery. For example, if the same participant were asked to compare $\$10$ with a $25\%$ chance and $\$10$ if the distance is below 200, they may well choose the latter option but switch to choosing the objective lottery when asked to compare it to $\$10$ if the distance is below 100. These pairs of choices imply that $F(X\leq 200)\geq0.25$ but $F(X\leq 100)\leq0.25$. We can conclude the 0.25 quantile of their subjective belief is between 100 and 200.

Notice that in these cases, we use the choices to bound the number x that would create an indifference between some objective lottery that pays with probability p and an act that pays if the random variable is below x. We interpret this x as being the p – th quantile of the participant’s belief. To interpret this indifference as a quantile of the belief, we need to assume a few things about preferences:

If a participant is indifferent between a $75\%$ chance of $\$10$ and $\$10$ if the random variable is below 250 but strictly prefers a $75\%$ chance of $\$20$ to $\$20$ if the random variable is below 250, then we cannot interpret 250 as the 0.75 quantile. This is because the number we would infer as the 0.75 quantile appears to depend on whether $\$10$ or $\$20$ is the outcome. In this case, we cannot reliably infer a belief using some arbitrarily chosen outcome for incentivizing the revelation of that belief. Thus, to interpret the outcome of our procedure as identifying the quantile of a belief, we need to assume that participants’ willingness to substitute between acts and objective lotteries does not depend on the arbitrarily chosen outcome. We refer to this assumption as the replacement axiom.

Another way our procedure can fail to deliver a result that can be interpreted as the quantile of a belief is if preferences over acts or objective lotteries are not monotone. For example, if a participant is indifferent between a $75\%$ chance of $\$10$ and $\$10$ if the random variable is below 250, but also indifferent between a $50\%$ chance of $\$10$ and $\$10$ if the random variable is below 300, then it is hard to interpret either of these indifferences as generated by a coherent belief, since it implies that F would have to be downward sloping. One way for this to happen is if preferences over acts do not respect the monotonicity of events, so that an act can be strictly preferred to another even if it pays conditional on an event that is “smaller” in terms of inclusion (such as $X\leq 250$ vs. $X\leq 300$). Another way to obtain preferences like this is if preferences over objective lotteries are not monotonic in probabilities (for instance, $\$10$ with $50\%$ chance is preferred to $\$10$ with $75\%$). On the other hand, if both of these monotonicity conditions are met, then the indifferences between acts that pay conditional on larger events (in terms of inclusion) will occur at higher objective probabilities, and thus these indifferences can be interpreted as belonging to some consistent distribution F. We refer to this pair of assumptions as the act monotonicity and objective lottery monotonicity axioms, respectively.

The replacement axiom and the two monotonicity axioms are enough to ensure that we can use indifferences to identify structure in preferences that can reasonably be interpreted as the quantiles of an underlying belief distribution. However, to actually find that indifference, we will need one more assumption, which has been latent in the discussion up until now. When we ask participants to choose their favorite option from the relevant pair and incentivize it by randomly implementing one of these choices, how do we know that their choice in each menu is their favorite? This requires weak structure on the participant’s preferences over compound lotteries for the types of acts and objective lotteries used in this procedure. This axiom is known as statewise monotonicity, which is an assumption required in any experimental economic procedure that asks participants to make multiple choices and then randomly chooses one to implement (Azrieli et al., Reference Azrieli, Chambers and Healy2018).

2.2. Formal results

This section makes use of the following notation. The set of outcomes is $\mathscr{X}=\left\{a,b\right\} $. Throughout, it is assumed $a\succ b$. X is a real-valued random variable with state space Ω. A tail event E_x is an event of the form $X\leq x$. Simple lotteries are objective lotteries of the form $S=(a\circ p,b\circ(1-p))$, where $p\in[0,1]$. Binary acts are acts of the form $\left(a\circ E,b\circ E^{c}\right)$ where E is an event in X and E ^c is the complement of that event. Simple mixtures are compound lotteries that mix (potentially) objective and subjective risk of the form $M=\left(L_{1}\circ p_{1},L_{2}\circ p_{2},...,L_{n}\circ p_{n}\right)$ where $p_{i}\in[0,1]$ with $\sum_{i=1}^{n}p_{i}=1$, and each L_i is a simple lottery or a binary act. The set of all simple mixtures is $\mathscr{M}$.

A quantile price list is a probability p (the quantile to be elicited), along with a sequence of n values $\left(x_{1},...,x_{n}\right)$ with $x_{i} \gt x_{i+1}$. The range of the quantile price list is $\left[x_{n},x_{1}\right]$. The price list is constructed by pairing a constant objective lottery $o_{p}=\left(a\circ p,b\circ1-p\right)$ with a sequence of increasing binary acts $A_{x_{i}}=(a\circ E_{x_{i}} ,b\circ E_{x_{i}}^c) $ (recall that tail events $E_{x_{i}}$ are events $X \lt x_i$) to create n menus of the form $\left\{o_{p},A_{x_{i}}\right\} $. To implement the price list, participants are asked to choose either the lottery or the act from each menu. A menu is randomly chosen (according to any fixed distribution), and the participant is rewarded with the lottery chosen from that menu.Footnote ⁴

Assume participants have preference relation $\succsim$ over $\mathscr{M}$ that is complete, transitive. We define these additional axioms.

Axiom (1): Objective Lottery Monotonicity:

$p\geq p'\Leftrightarrow\left(a\circ p,b\circ1-p\right)\succsim\left(a\circ p',b\circ1-p'\right)$.

Axiom (2): Act Monotonicity:

$E'\subseteq E\Rightarrow\left(a\circ E,b\circ E^c\right)\succsim\left(a\circ E',b\circ E'^c\right)$.

Axiom (3): Continuity:

$\forall E\in \Omega$, $\exists p\in[0,1]:$ $\left(a\circ E,b\circ E^c\right)\sim\left(a\circ p,b\circ 1-p\right)$.

Axiom (4): Replacement:

For all $a,a',b,b'$: $\left(a\circ E,b\circ E^{c}\right)\sim\left(a\circ p,b\circ1-p\right) \Leftrightarrow \\ \left(a'\circ E,b'\circ E^{c}\right)\sim\left(a'\circ p,b'\circ1-p\right)$

Axiom (5): Statewise Monotonicity:

$L_{i}^{*}\succ L_{i}\Leftrightarrow\left(L_{1}\circ p_{1},...,L_{i}^{*}\circ p_{i},...,L_{n}\circ p_{n}\right)\succ\left(L_{1}\circ p_{1},...,L_{i}\circ p_{i},...,L_{n}\circ p_{n}\right)$

We now formalize the discussion from the previous subsection. Under objective lottery monotonicity, act monotonicity, continuity, and replacement, participants behave “as if” their preferences are generated by a well-formed subjective distribution F.

Let F(x) be the CDF that solves $\left(a\circ F(x),b\circ 1-F(x)\right)\sim\left(a\circ E_x,b\circ E_x^c\right)$ under the conditions of Proposition 1. A p quantile of F(x) is any q that solves $F(q)\geq p$ and $F(q) \leq p$. Such a q always exists but may not be unique. However, since F(x) is increasing, the set of p quantiles is an interval $[q_l,q_h]$. On the other hand, if F(x) is strictly increasing every p quantile is unique ( $q_l=q_h=q$).

Call x_i a switching-point for the p quantile if the participant chooses $A_{x_{i}}$ from $\left\{o_{p},A_{x_{i}}\right\} $ but o_p from $\left\{o_{p},A_{x_{i+1}}\right\}$. Under the addition of statewise monotonicity, if a participant’s subjective distribution has a unique p quantile q_p within the range of the quantile price list, then the participant’s choices in the quantile price list will have a switching-point, and that switching-point reveals an interval that must contain q_p.

The fact that a p quantile may not be unique poses a minor complication for elicitation. If the set of p quantiles $[q_l,q_h]$ spans multiple values of $(x_1,...,x_n)$, then there is a range of values for which the participant is indifferent between being paid according to the objective lottery o_p and the binary act $A_{x_{i}}$. This can result in the multiple switching-points. In this case, however, any switching-point identifies a p quantile.

3.1. Characterization of the maximum entropy approximation

Suppose that a random variable X is known to be continuously distributed on $\left[\underline{q},\overline{q}\right]$. Furthermore, there are n quantile values $\{q_{p_1},q_{p_2},...,q_{p_n}\}$ associated with the probabilities $\{p_1,p_2,...,p_n\}$, with each quantile value $q_{p_i}$ constrained to be within an interval: $\left[l_{p_i},h_{p_i}\right]$. Without loss of generality, assume $p_i \gt p_{i-1}$. For convenience, let $p_0=0$ and $p_{n+1}=1$. The fact that the distribution has support $\left[\underline{q},\overline{q}\right]$ can be represented by $l_{p_0}=h_{p_0}=\underline{q}$ and $l_{p_{n+1}}=h_{p_{n+1}}=\overline{q}$. The other values of $l_{p_i},h_{p_i}$ are determined by our elicitation methodology.

We wish to find the distribution $\tilde{F}$ and the associated density $\tilde{f}$ that maximizes the entropy function $h(f)=\int_{\underline{q}}^{\overline{q}}-f(x)ln(f(x))dx$ subject to the n restrictions $l_{p_i}\leq q_{p_i} \leq h_{p_i}$. These constraints can be interpreted geometrically to restrict F to be in the set of all increasing functions from $\left[\underline{q},\overline{q}\right]$ to $\left[0,1\right]$ that pass through the horizontal line segments represented by each pair $[l_{p_i},h_{p_i}]$. This is shown in Figure 2.

Fig. 2. Constraints created by elicited quantiles. Each quantile is constrained to be within an interval represented by the horizontal line segments. The CDF of the participants subjective belief distribution must pass through these line segments

Of course, the maximum entropy distribution must have some value for each $q_{p_i}$. Proposition 3 shows that between these points, whatever they are, the maximum entropy distribution $\tilde{F}$ is a simple piecewise linear interpolation of the quantiles $q_{p_i}$. If the quantiles were known exactly, rather than constrained to be within some interval, this would be a full characterization of the maximum entropy distribution. An example of this is shown in Figure 3 for known quantiles $\{q_{0.25},q_{0.50},q_{0.75}\}$.

Fig. 3. An example of the maximum entropy CDF (dotted line) and true CDF (solid line) with known quantiles at $25\%,50\%, 75\%$

Notice that Proposition 3 does not fully characterize the maximum entropy distribution subject to the quantile intervals elicited by our methodology. However, it greatly simplifies the problem of finding this distribution. Although maximizing entropy in general is an infinite-dimensional optimization problem, Proposition 3 simplifies the problem to one of finding the values of ${q_{p_2},...,q_{p_{n-1}}}$ for which F constructed from the piecewise linear interpolation of the points $\Biggl(\Bigl(\underline{q},0\Bigr),\left(q_{p_1},p_1\right),\left(q_{p_2},p_2\right), ... , \left(q_{p_n},p_n\right),\Bigl(\overline{q},1\Bigr)\Biggr)$ maximizes the entropy function $h(f)=\int_{\underline{q}}^{\overline{q}}-f(x)ln(f(x))dx$.

The problem can be solved efficiently with generic nonlinear optimization packages. Figure 4 shows the maximum entropy distribution subject to the quantile intervals shown in Figure 2. In this case, q _0.25 and q _0.50 are maximized at corner solutions, while q _0.75 is maximized at an interior. The first-order conditions on each quantile ensure that whenever the maximum entropy function involves an interior solution for some quantile, the density is constant and equal on both sides of that quantile and, therefore, $\tilde{F}$ does not have a kink at that quantile.

Fig. 4. An example of the maximum entropy CDF through the constraints imposed by elicited quantile intervals for $q_{0.25},q_{0.50},q_{0.75}$

3.2. Approximating moments

Once we have approximated beliefs using the maximum entropy distribution $\tilde{F}$, it is possible to estimate properties of a belief that have not been observed. As an example, suppose that we want to calculate the belief about the mean of X. This can be done using the CDF and the fact that $E(X)=\int_{-\infty}^{\infty}1-F(x)dx$. Thus, the mean of the approximated distribution can be calculated by the following sum.

(1)\begin{equation} \begin{aligned} \int_{\underline{q}}^{q_{p_{1}}}x\left(\frac{p_{1}}{q_{p_{1}}-\underline{q}}\right)dx+\int_{q_{p_{1}}}^{q_{p_{2}}}x\left(\frac{p_{2}-p_{1}}{q_{p_{2}}-q_{p_{1}}}\right)dx+...\\ +\int_{q_{p_{n-1}}}^{q_{p_{n}}}x\left(\frac{p_{n}-p_{n-1}}{q_{p_{n}}-q_{p_{n-1}}}\right)dx+\int_{q_{p_{n}}}^{\overline{q}}x\left(\frac{1-p_{n}}{\overline{q}-q_{p_{n}}}\right)dx \end{aligned} \end{equation}

For our experiment, where we elicit q _0.25, q _0.50, and q _0.75, this sum simplifies to the following expression for the approximated mean in terms of $\underline{q},\overline{q}$ and the quantiles:

(2)\begin{equation} \frac{1}{8}\left(\underline{q}+\overline{q}\right)+\frac{1}{4}\left(q_{0.25}+q_{0.50}+q_{0.75}\right) \end{equation}

5.1. Experiment overview

The event that the participants considered in our experiment is the number of students who passed or failed a math task. We elicited three quantiles of the subjective belief distributions of the participants, 0.25, 0.50, and 0.75, as well as their belief about the probability that a randomly chosen student passed the math task. Under the assumption that a participant views the performance of each student on the math task as independent and has a point belief p about the probability that a randomly chosen student passed, their subjective belief about the number of students who passed should be binomial with mean $20*p$. We refer to this as their mean belief below.

Using the methodology presented in this paper, we can compare the directly elicited shape of participants’ subjective beliefs (using the quantiles) to the implied binomial shape. We emphasize that our goal is not to compare the performance of eliciting quantiles with the performance of eliciting a probability or mean. Instead, our experiment is intended to demonstrate how eliciting quantiles using our methodology can provide a more nuanced view of participants’ entire subjective belief and potentially shed light on the underlying assumptions implying a binomial subjective belief. In addition, we randomly varied whether we elicited quantiles or the mean first. Using the two treatments, we can determine whether reporting the mean belief before or after quantile beliefs leads to greater consistency across the four elicitations.

5.2. Experiment details

Before the experiment, 20 Ohio State students completed a math task similar to the task developed in Niederle and Vesterlund Reference Niederle and Vesterlund2007. Each student was shown a sequence of five two-digit numbers. The students had five minutes to correctly calculate the sum of as many sets of two-digit numbers as they could. For each correctly answered math problem, they received $\$1$. The students also received a $\$5$ show-up fee. The performance of these students on the math task is the basis of the main experiment.

We categorized each of the 20 students according to how many math problems they correctly answered. Table 1 shows the distribution of correctly answered math problems for the students. If a student correctly answered at least 13 math problems, we classified them as “passing” the math task. If they answered any number less than thirteen of the math problems correctly, we categorized them as “failing” the math task. We chose this categorization prior to running the experiment. All students who participated in the math task knew that their answers would be seen later by other participants, but did not know about the categorization of success. We chose not to disclose the categorization to the students in order to not change their performance goal on the task.

Table 1 Distribution of correctly answered math problems for the 20 students in our experiment.

For the main experiment, we used the number of students who passed the math task as our subjective event of interest. Participants in the main experiment were also recruited through the Ohio State Experimental Economics Laboratory. Anyone who participated in the previous math task was excluded from participation in the main experiment. We chose to do both the math task and the experiment in person at the Ohio State laboratory so that the participants in the experiment would have some baseline facts they could use to form beliefs about the students’ performance in the math task. Experiences in Ohio State math classes or general perceptions of the math abilities of university students are facts that could be used by our participants to form their beliefs about other students’ performances on the prior math task.

The participants were first informed about the math task and given an example of the math problems. The participants then told us their beliefs about the students’ performance on the math task. Their beliefs were elicited through their switching-points on four separate multiple price lists (MPLs). Three of the four MPLs were quantile price lists targeted at the 0.25, 0.50, and 0.75 quantiles. The fourth MPL elicited participants’ beliefs about the probability that a randomly chosen student passed. This MPL is not part of our methodology but has been used extensively in experimental economics. It is based on the logic of the BDM mechanism originally proposed to elicit willingness-to-pay by Becker et al. Reference Becker, DeGroot and Marschak1964. The concept of using this method for eliciting beliefs was described by Savage Reference Savage1971 and first implemented by Grether Reference Grether(1981). In this context and to our knowledge, Holt and Smith Reference Holt and Smith2016 were the first to represent this BDM in the multiple price list format.

In our experiment, each MPL consisted of 21 rows. In all MPLs, a single switching-point was enforced. In each row, there were two options, Option A and Option B. Participants decided which of the two options they preferred in each row. The right-hand side of each quantile price list, or Option B, was constant. For the 0.25 quantile, Option B was “ $\$10$ with a $25\%$ chance” in each row. For the 0.50 quantile, Option B was “ $\$10$ with a $50\%$ chance” in each row. For the 0.75 quantile, Option B was “ $\$10$ with a $75\%$ chance”. The left-hand side, or Option A changed in each row. Option A in the first row of all three quantile price lists was “ $\$10$ if ≤ 20 people passed.” In each row, the number of people who succeeded in the math task decreased by 1 down to “ $\$10$ if 0 people passed” in the 21st row. An example of the 0.25 quantile price list is shown in Figure 5.

Fig. 5. Screenshot of the 0.25 quantile price list used in our experiment

The fourth MPL elicited the participants’ beliefs about the probability that a randomly chosen student passed the math task. The right-hand side of this MPL, or Option B, was constant across all rows: “ $\$10$ if one randomly chosen subject passed.” The left-hand side, or Option A, changed in each row. Option A in the first row was “ $\$10$ with a $100\%$ chance.” In each row, the probability that the participants received $\$10$ in Option A decreased by $5\%$ down to “ $\$10$ with a $0\%$ chance” in the 21st row. The exact MPL used is shown in Figure 6. Since we use the probability elicited from this MPL to infer the mean beliefs of the participants, we refer to it as the mean MPL below.

Fig. 6. Screenshot of the MPL from our experiment used to elicit beliefs about the probability a randomly chosen participant passed the math task

Participants were randomly assigned to one of two treatments that differed only in the order of the four MPLs. In the mean-first treatment, the mean MPL was shown first. In the mean-last treatment, the three quantile price lists were shown first. In both treatments, the order of the three quantile price lists was randomized at the participant level.

One of the four MPLs was randomly chosen to determine payment. For the randomly chosen MPL, a row was then randomly chosen. The participant’s choice in that row was used to determine the final payment. If the chosen option was winning $\$10$ with a certain probability (Option B for quantile price list and Option A for mean MPL), a 100-sided die was rolled. If the number on the die was lower than or equal to the probability stated in that row, the participant won $\$10$. If the chosen option was winning $\$10$ depending on how the students performed in the math task (Option A for quantile price lists and Option B for mean MPL), the actual performance of the students was used to determine payment. For the three quantile price lists, if the number of participants who actually succeeded was less than or equal to the number stated in the row, the participant won $\$10$. For the mean MPL, a student was randomly chosen from the math task. If this student succeeded, the participant won $\$10$.

The math sessions took approximately 15 minutes, and the average payment was $\$7.40$. A total of 158 participants were recruited for the main experiment. Seventy-one participants were randomly assigned to mean-first and 87 participants were randomly assigned to mean-last. The experiment took 10 minutes, and the participants received a $\$5.00$ show-up fee plus an average payment of $\$3.29$. The full experimental instructions can be found in the Online Appendix.