Not that Irrational

This post is from Judgment and Decision Making, Vol. 11, No. 6, November 2016, pp. 601–610, and is based on the paper:  “The irrational hungry judge effect revisited: Simulations reveal that the magnitude of the effect is overestimated,” written by Andreas Glöckner. Danziger, Levav and Avnaim-Pesso analyzed legal rulings of Israeli parole boards concerning the effect of serial order in which cases are presented within ruling sessions. DLA analyzed 1,112 legal rulings of Israeli parole boards that cover about 40% of the parole requests of the country. They assessed the effect of the serial order in which cases are presented within a ruling session and took advantage of the fact that the ruling boards work on the cases in three sessions per day, separated by a late morning snack and a lunch break. They found that the probability of a favorable decision drops from about 65% to 5% from the first ruling to the last ruling within each session. This is equivalent to an odds ratio of 35. The authors argue that these findings provide support for extraneous factors influencing judicial decisions and speculate that the effect might be driven by mental depletion. Glockner notes that the article has attracted attention and the supposed order effect is considerably cited in psychology.

We all know that individuals do not strictly adhere to standards of rationality, in that judgments and choices are influenced by many irrelevant factors such as changes in presentation format, the presence of random anchors, and many more. Glockner notes that legal realism has a long history and many facets but it is often characterized by the phrase that “justice is what the judge ate for breakfast”. However, it is quite a jump from agreeing that judicial decision making is influenced to some degree by extraneous factors and the 35 odds ratio that the Danziger et al research finds. It seems to be fashionable to emphasize the irrationality of humans and to forget that our rationality has been adaptive.

Glockner explains that Danziger et al take the concern of non-random order of cases into account by including reasonable control variables for substantive factors that might influence both ordering and rulings. They show that the results remain robust when controlling statistically for severity of offense, previous imprisonment, months served, participation in a rehabilitation program, and proportion of previous favorable decisions.  One argument is that the downward trend might be due to the fact that, within each session, unrepresented prisoners usually go last and are less likely to be granted parole than prisoners who are represented by attorneys. Danziger et al have asserted that the effect continues after controlling for this, but have not mentioned its size.

A  more subtle, concern is that results might be driven by factors that systematically influence judges’ decisions to take a break.  Judges might aim to finish a set of cases (e.g., to complete all cases from one prison) within a session. This indicates that some organizational planning occurs.

Glockner provides an innovative examination of the Danziger et al (DLA) findings by simulating the rulings of an ideal judge who makes choices without errors and biases. Glockner assumed that the judge has a rough time limit for each session and works on cases until recognizing that a case would go over this limit. The case that would be too long would not be solved anymore in the current session, but it would be the first case in the next session.

The results of Glockner’s simulations indicate that, following the approach by DLA, a rational judge working on cases that are presented in random order would show a strongly decreasing probability of a favorable decision towards the end of the session. Even the shape of the curve and the magnitude of the effect are comparable to that observed by DLA. Glockner notes that the planning described here is merely organizational and does not require any foresight concerning how the case will be decided. All it requires is that the judges have a rough estimate, whether the next case will be quick or take longer. Thus the observed influence of order can be alternatively explained by a statistical artifact resulting from favorable rulings taking longer than unfavorable ones as opposed to mental depletion. More generally, the analysis shows that sometimes there is a not so obvious rational basis for irrational-looking behavior.

My personal experience working with about one thousand development/zoning permission agendas over several decades provides me some opinions on this sort of situation. There is definitely some  “mental depletion” going on especially after four or five hours. However, I would agree with Andreas Glockner that the “irrational” pattern of decisions was largely due to other things than “mental depletion”.  There were systematic reasons, but there was another thing that I noticed.

There seemed to me an institutional expectation that members of the body felt that would be similar to a parole board. For the parole board, the expectation is that most applications will not be approved, but that some will be. Some situation such as an overcrowded prison system could dramatically change expectations.  Similarly for development/zoning permission but in reverse, the expectation is that most applications will be approved, but that some will not be. Where the community perception is that more development is needed, the expectation would be more applications be approved and fewer denied. This seems to act similarly to the gambler’s fallacy. The gambler’s fallacy is the mistaken belief that, if something happens more frequently than normal during some period, it will happen less frequently in the future, or that, if something happens less frequently than normal during some period, it will happen more frequently in the future. The boards seemed to expect the “normal” percentage of approvals versus denials to be expressed at each meeting. So if 15% was the average denial rate, a meeting with a 40% denial rate was unlikely even if as a staff member, I believed that it was appropriate.  I recall knowing that I was in trouble when several bad applications walked in the door. If all were on a single agenda, it would be nearly impossible to get them all denied. They would start to be measured against each other.

As a staff member and knowing this tendency, you could counteract it to some extent. In setting such agendas, there tends to be more flexibility than might be claimed or admitted. In the parole board situation, if  few parolees would be coming up that session, but some that the staff considered approvable, the staff might allow parole seekers who had messed up the paperwork or be in some gray area of eligibility to be placed on the agenda. Then the parole board could reject an appropriate portion of the applications they saw, while still approving applications considered worthwhile.