Hogarth on Simulation

scm1This post is a contination of the previous blog post Hogarth on Description. Hogarth and Soyer suggest that the information humans use for probabilistic decision making has two distinct sources: description of the particulars of the situations involved and through experience of past instances. Most decision aiding has focused on exploring effects of different problem descriptions and, as has been shown, is important because human judgments and decisions are so sensitive to different aspects of descriptions. However, this very sensitivity is problematic in that different types of judgments and decisions seem to need different solutions. To find methods with more general application, Hogarth and Soyer suggest exploiting the well-recognized human ability to encode frequency information, by building a simulation model that can be used to generate “outcomes” through a process that they call “simulated experience”.

Simulated experience essentially allows a decision maker to live actively through a decision situation as opposed to being presented with a passive description. The authors note that the difference between resolving problems that have been described as opposed to experienced is related to Brunswik’s distinction between the use of cognition and perception. In the former, people can be quite accurate in their responses but they can also make large errors. I note that this is similar to Hammond’s correspondence and coherence. With perception and correspondence, they are unlikely to be highly accurate but errors are likely to be small. Simulation, perception, and correspondence tend to be robust.

Advantages of simulated experience as related by Hogarth and Soyer

  • Simulation technology today allows for the construction of simulations for virtually any decision scenario and can function quite rapidly.
  • People relate easily to the task of simulating sequential outcomes and interact seamlessly with simulations. Moreover, as problems grow more complex, they prefer their experiential intuitions over their analytical calculations.
  • Simulated experience can be employed even when the decision maker has limited knowledge of statistics.
  • If the experience produced by the simulation is kind (i.e., abundant, unbiased and immediate),  then the experiential approach provides the decision maker with a complete picture of the outcomes of a process rather than a frame that nudges her to a particular choice. (See post Learning, Feedback and Intuition for  kind and wicked environments.)
  • Simulated experience essentially takes description out of the picture.  This is an advantage because description always involves a frame, and thus different descriptions may lead to different perceptions and decisions. Second, for some problems that incorporate uncertainties or complex structures, descriptions might be hard to construct or obscure a crucial part of the story. Where there are uncertainties such as outcomes subject to regression toward the mean (see post Regression to the Mean), descriptions lead people to overestimate the predictability of the dependent variable. The main reason behind this illusion of predictability is that description frames the question mainly around average effects. In problems with complex probabilistic structures, descriptions may not only be hard to decipher, they could also mislead one’s analysis of the situation. Consider for instance the birthday problem(i.e., What is the probability that two or more people have the same birthday in a group of N people?). People find it difficult to understand the problem structure. Simulations  let them live through the problems many times(learning the birthdays of multiple groups of N people in the birthday problem). In this way, users learn about the outcome probabilities through experience, without resorting to any bias-prone analyses.

Disadvantages of simulated experience

  • Sample size–You may choose a small sample that is unrepresentative since simulated experience allows decision makers to determine the number of trials they wish to experience. Hence, sample size becomes an important factor in the precision of judgments. In experiments, statistically sophisticated people sampled consistently more information, which, in turn, led to better judgments. Simulation training should include information on the law of large numbers.
  • Rare events–Decisions based on experience are consistent with underweighting small probabilities which can result in less risk aversion. By definition, rare events will not be experienced often in simulations with the consequence that decision makers might not pay much attention to these when making judgments and decisions. One way to overcome this problem would be to use conditional simulations: the probability of having a disease, given a positive result in a medical test; or the possibility of a loss beyond a certain amount, given the occurrence of a natural disaster.
  • A weakness of simulation technology could be that the decision maker does not gain insight into the problem structure that can be generalized. On the other hand, in the experiments of Hogarth and Soyer, they observed that, when able to consult both a description and a simulation, people’s analytical calculations improved. Their experiences provided a means to check the accuracy of their analysis.

Clearly simulation can not take the place of description in every situation involving probabilistic decisions. People want descriptions and they are used to them, even though the researchers found situations in which descriptions added to simulations produced poorer decisions than simulations alone. One tool that merges descriptions with simulation is a graph of simulated outcomes that result from the described model. Such a plot would follow the description of the problem and include individual level data (simulated or real) that makes visible the uncertainty inherent in the outcomes. According to Hogarth and Soyer the challenge to be faced is to develop understanding of when information should be presented as description, experience, or both.

Hogarth and Soyer use the example of Fibonacci’s advice to adopt the Hindu-Arabic numerical system over Roman numerals in the early 13th century. The Hindu-Arabic system conveyed information more intuitively, but most of its advantages were analytical and in coherence. Using simulation allows us to utilize our robust correspondence intelligence to help us avoid the big error (think Hubbell telescope). The current explosion of “big data” owes much to having an appropriate numerical system and much will be gained by those who can interpret this new trove of information. However, since interpretation will always be a human activity, future progress will depend on how we cope with this bottleneck. Exploiting people’s ability to process simulated experience is one way to enhance the use of intuition.

That simulation is superior to description for learning seems clear. This also tracks with the concept of predictive processing and embodied cognition. Predictive processing (See post The Prediction Machine) is to a large extent simulation which seems to say that is what our brains and neurons do –simulate. Well worn neural paths will do a better job of simulating. Simulation allows us to experience.  Years ago sensitivity analysis struck me as helpful. Change the value of the variables in reasonable ways and see what kind of answers you get. It allows you to experience variance one simulation at a time. However, when you start to aggregate these simulations without at least your predictive processing system having experienced each one, you lose quite a bit of the advantage. Investment strategy simulations may tell you that your saving and investments give you a 90% certainty of meeting an investment goal.  Knowing the distribution of 100% of the results is valuable. If I have 10% chance of being dead broke and I only have one life to live, I may need to adjust my investment goals.