NDSSL Seminars > McKay, 09 Apr, 2007
The Network Dynamics and Simulation Science Laboratory
Complexity Science Seminar Series - Abstracts

Title: Introduction to Statistical Methods for Understanding Prediction Uncertainty in Simulation Models

Presenter: Dr. Michael McKay, Los Alamos National Laboratory (ret.)

Date and Time: Monday, April 09, 2007, 3:00-4:00 PM

Note: This is the first seminar in a series of three to be held by Dr. Michael McKay during the week of April 9, 2007. The other two seminars are:
  • Sensitivity Analysis When Model Outputs are Functions (Wednesday April 11 at 10:00 AM)
  • Calibrated Prediction (Friday April 13 at 10:00 AM)

    • Abstract:
    Quantifying and understanding uncertainty of predictions made by combining computer simulations, i.e., theoretical predictions, with physical data involve many statistical issues. Generally, relevant statistical methods can be grouped into categories related to (1) designs and sampling plans for computer experiments, (2) measures of model-input importance, (3) model calibration and prediction, (4) model evaluation (testing) and (5) model validation.

    This talk focuses on assessing model-input importance and the role of uncertainty in input values in prediction uncertainty. Input uncertainty refers to incomplete knowledge of "correct" values of model inputs, including model parameters. When it is characterized by a probability distribution, prediction uncertainty is characterized by the induced prediction distribution. Comparison of a model predictor based on a subset of model inputs to that of the full model leads to a natural decomposition of the prediction variance and to Pearson's (1903) correlation ratio as a measure of importance. Additive variance decompositions similar to those from classical analysis of variance (ANOVA) have been used by several authors to evaluate computer models, including, Cukier, Levine and Shuler (1978), Cox (1982) and Sobol' (1990). A most general decomposition due to Panjer (1973), and discussed by McKay (1999), provides a (not unique) variance decomposition useful when model inputs cannot be treated as statistically independent. The correlation ratio has been used by Iman and Hora (1990), Krzykacz (1990), McKay (1995, 1997) and in Saltelli et al. (2000).

    Elements of this lecture include a mathematical background for uncertainty quantification and uncertainty importance, as well as a scheme for estimation of the correlation ratio. Concepts are demonstrated in an application with a compartmental model that simulates the movement of contaminants through an ecosystem. The 84 model inputs represent parameters in differential equations as well as initial conditions. The model is studied with a set of 343 computer runs, selected using Orthogonal array sampling (OAS). Addition examples are used to demonstrate interpretation of sample values of R^2, used as an estimate of the correlation ratio. For these examples, replicated Latin hypercube sampling (rLHS) is used.
    References
  • Cox, D. C. (1982). An analytical method for uncertainty analysis of nonlinear output functions, with application to fault-tree analysis. IEEE Transactions on Reliability, R-31, 5: 265-68.
  • Cukier, R. I., Levine, H. B. and Shuler, K. E. (1978). Nonlinear sensitivity analysis of multiparameter model systems. Journal of Computational Physics, 26: 1-42.
  • Iman, R. L. and Hora, S. C. (1990). A robust measure of uncertainty importance for use in fault tree system analysis. Risk Analysis, 10(3): 401-406.
  • Krzykacz, B. (1990), SAMOS: A computer program for the derivation of empirical sensitivity measures of results from large computer models. GRS-A-1700, Gesellschaft fur Reaktorsicherheit (GRS) mbH, Garching, Republic of Germany.
  • McKay, M. D. (1995). Evaluating prediction uncertainty. NUREG/CR-6311, U.S. Nuclear Regulatory Commission and Los Alamos National Laboratory Report.
  • McKay, M. D. (1997). Nonparametric variance-based methods of assessing uncertainty importance. Reliability Engineering and System Safety, 57: 267-279.
  • McKay, M. D., Morrison, J. D. and Upton S. C. (1999). Evaluating prediction uncertainty in simulation models. Computer Physics Communications, 117: 44-51.
  • Panjer, H. H. (1973). On the decomposition of moments by conditional moments. The American Statistician, 27: 170-171.
  • Pearson, K. (1903). Mathematical contributions to the theory of evolution. Proceedings of the Royal Society of London, 71: 288-313.
  • Saltelli, A., Chan, K. and Scott E. M. (Eds.) (2000). Sensitivity Analysis, John Wiley and Sons, Ltd.
  • Sobol', I. M. (1990). Sensitivity estimates for non-linear mathematical models. Matematicheskoe Modelirovanie, 2: 112-118 [Russian] and (1993) Mathematical Modeling for Computer Experiments, 1: 407-414 [English].
    • Seminar Location: The seminars are held at:
    Virginia Tech, Corporate Research Center
    1880 Pratt Drive, Building XV
    Seminar Room 2018, Second Floor
    Directions: Map (PDF)

    Back to: NDSSL Seminar Page
    (Last updated: Mon Mar 24 23:43:04 EST 2008)