|
|
Title: Introduction to Statistical Methods for Understanding Prediction Uncertainty in Simulation Models
Presenter: Dr. Michael McKay
Los Alamos National Laboratory (ret.)
Date and Time: Monday, April 09, 2007, 3:00-4:00 PM
Note: This is the first seminar in a series of three to be held
by Dr. Michael McKay during the week of April 9, 2007. The other two seminars are:
Sensitivity Analysis When Model Outputs are Functions (Wednesday April 11 at 10:00 AM)
Calibrated Prediction (Friday April 13 at 10:00 AM)
Abstract:
Quantifying and understanding uncertainty of predictions made by
combining computer simulations, i.e., theoretical predictions, with
physical data involve many statistical issues. Generally, relevant
statistical methods can be grouped into categories related to (1)
designs and sampling plans for computer experiments, (2) measures of
model-input importance, (3) model calibration and prediction, (4)
model evaluation (testing) and (5) model validation.
This talk focuses on assessing model-input importance and the role of
uncertainty in input values in prediction uncertainty. Input
uncertainty refers to incomplete knowledge of "correct" values of
model inputs, including model parameters. When it is characterized by
a probability distribution, prediction uncertainty is characterized by
the induced prediction distribution. Comparison of a model predictor
based on a subset of model inputs to that of the full model leads to a
natural decomposition of the prediction variance and to Pearson's
(1903) correlation ratio as a measure of importance. Additive
variance decompositions similar to those from classical analysis of
variance (ANOVA) have been used by several authors to evaluate
computer models, including, Cukier, Levine and Shuler (1978), Cox
(1982) and Sobol' (1990). A most general decomposition due to Panjer
(1973), and discussed by McKay (1999), provides a (not unique)
variance decomposition useful when model inputs cannot be treated as
statistically independent. The correlation ratio has been used by Iman
and Hora (1990), Krzykacz (1990), McKay (1995, 1997) and in Saltelli
et al. (2000).
Elements of this lecture include a mathematical background for
uncertainty quantification and uncertainty importance, as well as a
scheme for estimation of the correlation ratio. Concepts are
demonstrated in an application with a compartmental model that
simulates the movement of contaminants through an ecosystem. The 84
model inputs represent parameters in differential equations as well as
initial conditions. The model is studied with a set of 343 computer
runs, selected using Orthogonal array sampling (OAS). Addition
examples are used to demonstrate interpretation of sample values of
R^2, used as an estimate of the correlation ratio. For these examples,
replicated Latin hypercube sampling (rLHS) is used.
References
Cox, D. C. (1982). An analytical method for uncertainty analysis of
nonlinear output functions, with application to fault-tree
analysis. IEEE Transactions on Reliability, R-31, 5: 265-68.
Cukier, R. I., Levine, H. B. and Shuler, K. E. (1978). Nonlinear
sensitivity analysis of multiparameter model systems. Journal of
Computational Physics, 26: 1-42.
Iman, R. L. and Hora, S. C. (1990). A robust measure of uncertainty
importance for use in fault tree system analysis. Risk Analysis,
10(3): 401-406.
Krzykacz, B. (1990), SAMOS: A computer program for the derivation of
empirical sensitivity measures of results from large computer
models. GRS-A-1700, Gesellschaft fur Reaktorsicherheit (GRS) mbH,
Garching, Republic of Germany.
McKay, M. D. (1995). Evaluating prediction uncertainty. NUREG/CR-6311,
U.S. Nuclear Regulatory Commission and Los Alamos National Laboratory
Report.
McKay, M. D. (1997). Nonparametric variance-based methods of assessing
uncertainty importance. Reliability Engineering and System Safety,
57: 267-279.
McKay, M. D., Morrison, J. D. and Upton S. C. (1999). Evaluating
prediction uncertainty in simulation models. Computer Physics
Communications, 117: 44-51.
Panjer, H. H. (1973). On the decomposition of moments by conditional
moments. The American Statistician, 27: 170-171.
Pearson, K. (1903). Mathematical contributions to the theory of
evolution. Proceedings of the Royal Society of London, 71: 288-313.
Saltelli, A., Chan, K. and Scott E. M. (Eds.) (2000). Sensitivity
Analysis, John Wiley and Sons, Ltd.
Sobol', I. M. (1990). Sensitivity estimates for non-linear
mathematical models. Matematicheskoe Modelirovanie, 2: 112-118
[Russian] and (1993) Mathematical Modeling for Computer Experiments,
1: 407-414 [English].
Seminar Location: The seminars are held at:
Virginia Tech, Corporate Research Center
1880 Pratt Drive, Building XV
Seminar Room 2018, Second Floor
Directions: Map (PDF)
Back to: NDSSL Seminar Page
|
