CSCS 530 | Main / VerificationAndValidationReadings

Readings related to verification, validation sensitivity analysis.

Tools and techniques for developing policies for complex and uncertain systems.
Stephen C. Bankes. PNAS 2002 99 Suppl. 3: 7263-66
Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology.
Grimm etal. Science 310(5750): 987- (2005).
Axtell and Epstein. "Agent-Based Modeling: Understanding Our Creations." From SFI Bulletin, Winter 1994.
Copy at eReserves site.
Simulation model Verification and Validation: Increasing the User's Confidence.
Stewart Robinson. Proc. 1997 Winter Simulation Conf..
Techniques to Understand Computer Simulations: Markov Chain Analysis.
Izquierdo etal, Journal of Artificial Societies and Social Simulation vol. 12, no. 1 6 (2009).
Di Paolo, Nobe and Bullock. "Simulation Models as Opaque Thought Experiments."
(Postscript Version or PDF Version) Artificial Life VII:

  The Seventh International Conference on the Simulation and Synthesis of Living Systems, August, 2000.

Highly recommended:

A standard protocol for describing individual-based and agent-based models. V. Grimm etal. Ecological Modelling 198(1): 115-126 (2006).

Optional Readings:

Stylised Facts and the Contribution of Simulation to the Economic Analysis of Budgeting.
Bernd-O. Heine, Matthias Meyer and Oliver Strangfeld. JASSS 4(4).
Grimm and Railsback. Chapter 3, Pattern Oriented Modeling
from their book Individual-based Modeling and Ecology.
From Interesting Details to dynamical relevance: toward more effective use of empirical insights in theory construction. Oswald J. Schmitz. Oikos 94(1): 39-50 (2001).
Errors and Artefacts in Agent-Based Modelling.
Galan etal (2009). [[Journal of Artificial Societies and Social Simulation vol. 12, no. 1 1

Path dependence and the validation of agent-based spatial models of land use.
Brown, D.G. etal. International Journal of Geographical Information Science, 19(2): 153-174 (2005)
Note: This addresses some of the difficulties with validation of models of CAS.
(Also listed under March 27th).

Required Readings:

Confronting Surprise. Robert Lempert, Steven Popper, Steven Bankes.
Soc.Sci.Computer Review 20(4): 420-440 (2002)
(Or try this source for the same paper).
John Miller. "Active Nonlinear Tests (ANTs) of Complex Simulation Models." Management Science 44(6): 820-30 (June, 1998). (Available here) A postscript version of the working paper is available via the SFI web site.
Bankes, Steve -- "Exploratory Modeling for Policy Analysis," Operations Research, 41(3), 435-449. 1993. (Available online here)

This one is cited a lot, so you should read it sometime (i don't like it!):

Verification, Validation and Confirmation of Numerical Models in the Earth Sciences.
Oreskes, Naomi, Shrader-Frechette, Kristin and Belitz, Kenneth. Science 263 (5147): 641-646 (1994).
Note Well: This article uses "verification" in a way that most modellers call "validation," i.e., does the model "validly" model the part of the world it is meant model.

Optional Readings:

A couple of oft-cited papers about the effects of synchronous vs asynchronous updating:
- Huberman, B. and Glance, N.. "Evolutionary games and computer simulations."
  Proc. Natl. Acad. Sciences (PNAS USA) 90 (16): 7716-18 (1993)
- Scan this next one for their answer to Huberman and Glance): Nowak, Bonfoeffer and May. "Spatial games and the maintenace of cooperation." PNAS (USA) v91(11): pp4877-4881. 1994.
"The poor quality of random numbers" and "Monte Carlo physics: a cautionary lesson."
Ferrenberg, Alan M., Landau, D.P., and Wong, Y.Joanna. Physical Review Letters 69(23). Dec 1992.
See also News and Views in Nature Vol 372, 1 Dec 1994.
Jonathan Bart. "Acceptance Criteria for Using Individual-Based Models to Make Management Decisions." Ecological Applications 5 (2): 411-420 MAY 1995. Go to browse this journal to find it.
Edward J. Rykiel, Jr. "Testing Ecological Models: the meaning of validation." In Ecological Modelling, 90 (3): 229-244 (1996). Go to and find the journal.
Individuality in modelling: a simplifying assumption too far? David R. Cope. Nonlinear Analysis: Real World Applications 6(4): 691-704 (2005)

A nice tutorial:

Stewart Robinson. "Simulation Verifcation, Validation and Confidence: A Tutorial."
Transactions of the Society for Computer Simulation International, 16 (No.2), pp. 63-69.

A good description and example applications of UA/SA methods:

A Methodology for performing global uncertainty analysis and sensitivity analysis in systems biology.
Marino, Simeone etal. J. Theoretical Biology 254: 178-196 (2008) Δ.

A number of more technical papers, but you should definitely read these at some point if you are going to do "real" computational modelling:

Evaluating prediction uncertainty in simulation models. Michael D. McKay, John D. Morrison and
Stephen C. Upton. Computer Physics Communications 117(1-2): 44-51 (1999).
Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems.
J. C. Helton and F. J. Davis. Reliability Engineering & System Safety 81(1): 23-69 (2003).
Blower S, Dowlatabadi H, May R, Anderson R. Sensitivity and Uncertainty Analysis of complex models of
disease transmission: an HIV model, as an example. Int. Statist Rev. 1994;62:229-243.
Helton, J.C., Johnson, J.D., Sallaberry, C.J. and Storlie, C.B. (2006).
Survey of sample-based methods for uncertainty and sensitivity analysis.
Reliability Engineering and System Safety 91(10-11): 1175-1209.
Iman R, Helton J. An Investigation of Uncertainty and Sensitivity Analysis Techniques for
Computer Models. Risk Analysis 1980;8:71-90.
Stochastic modelling of ecological processes using hybrid Gibbs samplers.
D.M.Walker etal. Ecological Modelling 198: 40-52 (2006).
E.L.Ionides, C.Breto, A.A.King. Inference for Nonlinear dynamical systems.
PNAS 103(49): 18438-43 (Dec 5 2006).
Selecting Parameters for calibration via sensitivity analysis: An inidividual-based model
of mosquitofish population dynamics. Beaudouin etal (2008). Ecological Modeling 218: 29-48.