NASA-STD-7009NASA TECHNICALSTANDARDApproved: 07-11-2008 National Aeronautics and Space AdministrationWashington, DC 20546-0001Expiration Date: 07-10-2013Superseding NASA-STD-(I)-7009 STANDARD FOR MODELS AND SIMULATIONSMEASUREMENT SYSTEM IDENTIFICATION:NOT MEASUREMENT SENSITIVEDOCUMENT HISTORY LOGApproval Date Description Status DocumentRevisionInterim 12-01-2006 Interim Baseline ReleaseRelease Baseline 07-11-2008 BaselineFOREWORDThis standard is published by the National Aeronautics and Space Administration (NASA) to provide uniform engineering and technical requirements for processes, procedures, practices, and methods that have been endorsed as standard for models and simulations (M&S) developed and used in NASA programs and projects, including requirements for selection, application, and design criteria of an item. This standard was specifically developed to respond to Action 4 from the 2004 report “A Renewed Commitment to Excellence,” with consideration also given to related findings as identified in the Columbia Accident Investigation Board (CAIB) Report.This standard is approved for use by NASA Headquarters and NASA Centers, including Component Facilities.This standard covers the development and operation (or execution) of M&S, as well as the analysis and presentation of the results from M&S. This also includes the proper training ofM&S practitioners and the identification of recommended practices, while ensuring thecredibility of the results from M&S is assessed and properly conveyed to those making critical decisions.Requests for information, corrections, or additions to this standard should be submitted via “Feedback” in the NASA Technical Standards System at .Original Signed By July 11, 2008Michael G. Ryschkewitsch Approval DateNASA Chief EngineerSECTION TABLE OF CONTENTSPAGEDOCUMENT HISTORY LOG (2)FOREWORD (3)TABLE OF CONTENTS (4)LIST OF FIGURES (6)LIST OF TABLES (6)1. SCOPE (7)1.1 Purpose (7)1.2 Applicability (8)1.3 Focus (9)2. APPLICABLEDOCUMENTS (9)2.1 General (9)2.2 GovernmentDocuments (9)2.3 Non-GovernmentDocuments (9)2.4 Order of Precedence (9)3. ACRONYMSANDDEFINITIONS (10)3.1 Acronyms and Abbreviations (10)3.2 Definitions (10)4. REQUIREMENTS (14)4.1 Programmatics (15)4.2 Models (16)4.3 Simulations and Analyses (18)4.4 Verification, Validation, and Uncertainty Quantification (19)4.5 Identification and Use of Recommended Practices (21)4.6 Training (22)4.7 Assessing the Credibility of M&S Results (23)4.8 Reporting Results to Decision Makers (24)5.GUIDANCE (26)5.1 ReferenceDocuments (26)5.2 KeyWordListing (28)TABLE OF CONTENTS, continuedSECTION PAGE Appendix A M&S Risk Assessment (29)Appendix B Credibility Assessment Scale (31)Appendix C Compliance Matrix (51)LIST OF FIGURESFigure Title Page 1 Sample M&S Risk Assessment Matrix (30)Scale (31)2 CredibilityAssessmentWeights (45)3 Subfactor4 Bar Chart and Radar Plot for Factor Scores (47)Illustration (49)Flag5 DeficiencyFlagIllustration (50)6 Exceedance7 Sufficiency Thresholds and Color Coding on Bar Chart and RadarPlot for Factor Scores (50)LIST OF TABLESTable Title Page1 Key Aspects of Credibility Assessment Levels (33)2 Level Definitions for Evidence Subfactors in the M&SDevelopment Category (37)3 Level Definitions for Evidence Subfactors in the M&S OperationsCategory (40)4 Level Definitions for Factors in the Supporting Evidence Category.. 425 Level Definitions for the Technical Review Subfactors (44)6 Roll-up of Subfactor Scores to Factor Score (46)7 Roll-up of Factor Scores to Overall Score (47)8 Roll-up of Factor Scores to Category Score (48)STANDARD FOR MODELS AND SIMULATIONS1. SCOPE1.1 PurposeThis standard was developed in response to Action 4 from the 2004 report “A Renewed Commitment to Excellence,” which stated the following:“Develop a standard for the development, documentation, and operation of models and simulationsa.Identify best practices to ensure that knowledge of operations is captured in the userinterfaces (e.g., users are not able to enter parameters that are out of bounds),b.Develop process for tool verification and validation, certification, reverification,revalidation, and recertification based on operational data and trending,c.Develop standard for documentation, configuration management, and qualityassurance,d.Identify any training or certification requirements to ensure proper operationalcapabilities,e.Provide a plan for tool management, maintenance, and obsolescence consistent withmodeling/simulation environments and the aging or changing of the modeledplatform or system,f.Develop a process for user feedback when results appear unrealistic or defyexplanation.”Subsequently, in 2006, the NASA Chief Engineer provided the following further guidance:g.“Include a standard method to assess the credibility of the models and simulationspresented to the decision maker when making critical decisions (i.e., decisions thateffect human safety or mission success) using results from models and simulations,h.Assure that the credibility of models and simulations meet the project requirements.” Each of the requirements and recommendations in this standard can be traced to one or more of the eight objectives listed above. The traceability matrix of the requirements in this standard to the eight objectives can be found online upon accessing this standard at URL; refer to “Requirements Traceability Matrix.” Some of these objectives are met by recommendations rather than by requirements as a result of either (a) the practical impossibility of satisfying the requirement in all cases, or (b) further guidance received from NASA Headquarters.These eight objectives are encapsulated in the overall goal for this standard, which is to ensure that the credibility of the results from M&S is properly conveyed to those making critical decisions. Critical decisions based on M&S results, as defined by this standard, are those technical decisions related to design, development, manufacturing, ground, or flight operations that may impact human safety or program/project-defined mission success criteria. The intent is to reduce the risks associated with critical decisions. This standard covers the development and operation (or execution) of M&S as well as the processes of analysis and presentation of the results from the M&S.This standard addresses aspects of M&S that are common across NASA activities. Discipline-specific details of M&S should be addressed in future documents, such as Recommended Practices (usually entitled “Handbooks” in the NASA document hierarchy), and are not included in this standard.The scope of this standard covers the development and maintenance of models, the operation of simulations, the analysis of the results, training, recommended practices, the assessment of the M&S credibility, and the reporting of the M&S results. Some of the key features of this standard are requirements and recommendations for verification, validation, uncertainty quantification, training, credibility assessment, and reporting to decision makers; also included are the cross-cutting areas of documentation and configuration management (CM).The requirements/recommendations in sections 4.7 and 4.8 are the culmination of the standard. The requirements/recommendations in sections 4.1 – 4.6 are intended to support the requirements in sections 4.7 and 4.8. This is accomplished by ensuring that sufficient details of the M&S process along with intermediate results are available to support the requirements in sections 4.7 and 4.8 and to respond to in-depth queries by the decision maker. Appendix A provides guidance for assessing the risk of using M&S in engineering decisions. Appendix B provides details related to some of the requirements/recommendations in sections 4.7 and 4.8. Appendix C contains a template for a compliance matrix.1.2ApplicabilityThis standard applies to M&S used by NASA and its contractors for critical decisions in design, development, manufacturing, ground operations, and flight operations. (Guidance for determining which particular M&S are in scope is provided in section 4.1 and Appendix A.) This standard also applies to use of legacy as well as commercial-off-the-shelf (COTS), government-off-the-shelf (GOTS), and modified-off-the-shelf (MOTS) M&S to support critical decisions. Generally, for such M&S, particular attention may need to be paid to defining the limits of operation and to verification, validation, and uncertainty quantification. Programs and projects are encouraged to apply this standard to M&S, if the M&S results may impact future critical decisions.This standard does not apply to M&S that are embedded in control software, emulation software, and stimulation environments. However, Center implementation plans for NPR 7150.2, NASA Software Engineering Requirements, should specifically cover embedded M&S, and addresssuch M&S-specific issues as numerical accuracy, uncertainty analysis, sensitivity analysis, M&S verification, and M&S validation.This standard may be cited in contract, program, and other Agency documents as a technical requirement. Requirements are indicated by the word “shall”; explanatory or guidance text is indicated in italics.1.2.1Tailoring for application to a specific program or project shall be formally documented as part of program or project requirements and approved by the Technical Authority.1.3 FocusIn general, standards may focus on engineering/technical requirements, processes, procedures, practices, or methods. This standard focuses on requirements and recommendations. Hence, this standard specifies what shall or should be done; it does not prescribe how the requirements are to be met, nor does it specify who is the responsible party for complying with the requirements.DOCUMENTS2. APPLICABLE2.1 GeneralThe documents listed in this section contain provisions that constitute requirements of this standard as cited in the text of section 4.2.1.1The latest issuances of cited documents shall be used unless otherwise approved by the assigned Technical Authority.The applicable documents are accessible via the NASA Online Directives Information System at and the NASA Technical Standards System at, or may be obtained directly from the Standards Developing Organizations or other document distributors.2.2Government DocumentsNone.2.3Non-Government DocumentsNone.2.4 Order of PrecedenceThis document establishes requirements and guidance for models and simulations but does not supersede nor waive established Agency requirements found in other documentation.2.4.1Conflicts between this standard and other requirements documents shall be resolved by the responsible Technical Authority.3. ACRONYMS AND DEFINITIONS3.1Acronyms and AbbreviationsAIAA American Institute of Aeronautics and AstronauticsASME American Society of Mechanical EngineersCAIB Columbia Accident Investigation BoardScaleCAS CredibilityAssessmentManagementCM ConfigurationCOTS Commercial-Off-The-ShelfCPIAC Chemical Propulsion Information Analysis CenterDMSO Defense Modeling and Simulation OfficeGOTS Government-Off-The-ShelfIEEE Institute of Electrical and Electronics EngineersISG Implementation Study GroupOrganization for StandardizationISO InternationalJANNAF Joint Army-Navy-NASA-Air ForceSimulationsandM&S ModelsMOTS Modified-Off-The-ShelfNASA National Aeronautics and Space AdministrationNASTRAN NASA Structural AnalysisNPR NASA Procedural RequirementsPMBA Primary Mirror Backplane AssemblyReq. RequirementGuideRPG RecommendedPracticesInteroperability Standards OrganizationSISO SimulationSTD StandardValidationV&V Verification&Validation, and AccreditationVV&A Verification,3.2 DefinitionsThe definitions listed below are those used in this document. Wherever possible, these definitions have been taken from official NASA documents. In some cases, after reviewing definitions of interest in the International Organization for Standardization (ISO), the Defense Modeling and Simulation Office (DMSO), professional society publications, and English language dictionaries, some of these definitions were taken or adapted from these sources to achieve the goal and objectives stated in section 1.1. Some definitions may have alternate meanings in other documents and disciplines.Abstraction: The process of selecting the essential aspects of a reference system to be represented in a model or simulation while ignoring those aspects that are not relevant to the purpose of the model or simulation (adapted from Fidelity ISG Glossary, Vol. 3.0).Accuracy: The difference between a parameter or variable (or a set of parameters or variables) within a model, simulation, or experiment and the true value or the assumed true value.Analysis: Any post-processing or interpretation of the individual values, arrays, files of data, or suites of executions resulting from a simulation.Artifact: Any tangible product that is produced by the project team, i.e., requirements documents, help systems, code, executables, test documentation, test results, diagrams, etc.Calibration: The process of adjusting numerical or modeling parameters in the model to improve agreement with a referent.Model: The numerical representation of the mathematical model.ComputationalConceptualModel: The collection of abstractions, assumptions, and descriptions of physical processes representing the behavior of the reality of interest from which the mathe-matical model or validation experiments can be constructed (adapted from ASME V&V 10).Configuration Management (CM): A management discipline applied over the product's life cycle to provide visibility into and to control changes to performance, functional, and physical characteristics (NPR 7120.5D, NASA Space Flight Program and Project Management Requirements).Credibility: The quality to elicit belief or trust in M&S results.Decision: Those technical decisions related to design, development,Criticalmanufacturing, ground, or flight operations that may impact human safety or mission success, as measured by program/project-defined criteria.Emulation: The use of an M&S to imitate another system, so that the M&S behaves like or appears to be the other system.Endorsement: A formal assurance that a product, process, or service conforms to specified characteristics. (Examples of endorsement include “accreditation”—the official acceptance of a model or simulation and its associated data to use for a specific purpose—and “certification,” which is similar to accreditation, but often applies to a class of purposes or a general domain and generally implies an independent and/or third-party certifier.)HumanSafety: The condition of being protected from death, permanently disabling injury, severe injury, and several occupational illnesses. In the NASA context this refers to safety of the public, astronauts, pilots and the NASA workforce (adapted from NPR 8000.4 and the NASA Safety Hierarchy).Limits of Operation: The boundary of the set of parameters for which an M&S result is acceptable based on the program/project-required outcomes of verification, validation, and uncertainty quantification.Model: The mathematical equations, boundary values, initial conditions,Mathematicaland modeling data needed to describe the conceptual model (ASME V&V 10).Mission Success Criteria: Standards against which the program or project will be deemed a success. Mission success criteria may be both qualitative and quantitative, and may cover mission cost, schedule, and performance results as well as actual mission outcomes (NPR 7120.5C, NASA Program and Project Management Processes and Requirements).Model: A description or representation of a system, entity, phenomena, or process (adapted from Banks, J., ed. (1998). Handbook of Simulation. New York: John Wiley & Sons).(A model may be constructed from multiple sub-models; the sub-models and the integrated sub-models are all considered models. Likewise, any data that goes into a model is considered part of the model. A model of a model (commonly called a metamodel), e.g., a response surface constructed from the results of M&S, is considered a model).Referent: Data, information, knowledge, or theory against which simulation results can be compared (adapted from ASME V&V 10).Risk: The combination of the probability that a program or project will experience an undesired event and the consequences, impact, or severity of the undesired event, if it were to occur. Both the probability and consequences may have associated uncertainties (adapted from NPR 7120.5D).SensitivityAnalysis: The study of how the variation in the output of a model can be apportioned to different sources of variation in the model input and parameters (adapted from Saltelli and others, 2000).Simulation: The imitation of the characteristics of a system, entity, phenomena, or process using a computational model.Stimulation: The description of a type of simulation whereby artificially generated signals are provided to real equipment in order to trigger it to produce the result required for verification of a real-world system, training, maintenance, or for research and development.Subject Matter Expert: An individual having education, training, or experience in a particular technical or operational discipline, system, or process and who participates in an aspect of M&S requiring his/her expertise.Tailoring: The documentation and approval of the adaptation of the processes and approach to complying with requirements according to the purpose, complexity, and scope of a NASA program or project. (NPR 7123.1A, NASA Systems Engineering Processes and Requirements).Uncertainty: (1) The estimated amount or percentage by which an observed or calculated value may differ from the true value (The American Heritage Dictionary of the English Language, 4th ed.). (2) A broad and general term used to describe an imperfect state of knowledge or a variability resulting from a variety of factors including, but not limited to, lack of knowledge, applicability of information, physical variation, randomness or stochastic behavior, indeterminacy, judgment, and approximation (adapted from NPR 8715.3B, NASA General Safety Program Requirements).Quantification: The process of identifying all relevant sources ofUncertaintyuncertainties, characterizing them in all models, experiments, and comparisons of M&S results and experiments, and of quantifying uncertainties in all relevant inputs and outputs of the simulation or experiment.Validation: The process of determining the degree to which a model or a simulation is an accurate representation of the real world from the perspective of the intended uses of the model or the simulation.Verification: The process of determining that a computational model accurately represents the underlying mathematical model and its solution from the perspective of the intended uses of M&S.Waiver: A documented authorization intentionally releasing a program or project from meeting a requirement (NPR 7120.5D). Deviations and exceptions are considered special cases of waivers.4. REQUIREMENTSThis standard establishes a minimum set of requirements and recommendations for the use of M&S to support critical decisions.For decisions based on results from M&S, the risk assumed by the decision maker is often misestimated due to inadequate assessment of uncertainties within M&S development, verification, validation, execution, analysis, and reporting. This standard establishes practicesto enable a more accurate assessment of this risk by making M&S credibility more apparent tothe decision maker. This standard emphasizes documentation and CM of M&S to enforce transparency, repeatability, and traceability; and it requires that key M&S personnel receive appropriate training.The requirements and recommendations are generic in nature because of their broad applicability to all types of M&S. Implementation details of the M&S requirements should be addressed in discipline-specific Recommended Practices, project/program management plans, etc.The following organizational structure is employed in this standard:4.1 Programmatics4.2 Models4.3 Simulations and Analyses4.4 Verification, Validation, and Uncertainty Quantification4.5 Identification and Use of Recommended Practices4.6 Training4.7 Assessing the Credibility of M&S Results4.8 Reporting Results to Decision MakersIn many instances, the modeling, simulation, and analysis activities are interwoven, particularly during the development, verification, and validation phases. This standard is intended to be inclusive of all these possibilities.Many of the requirements in this standard require documentation. With the exception of the documentation required for reports to decision makers (section 4.8), this documentation may consist of a reference to other existing documents, such as a journal article, a technical report, or a program/project document, provided that all the required details are contained in the referenced document(s).4.1 ProgrammaticsCritical decisions that are based entirely or partially on M&S are usually made within the context of a program or project. Program and project management have the responsibility to identify and document the parties responsible for complying with the requirements in this standard.The actual person identified by program and project management to fulfill the role of the “responsible party” in specific requirements will likely vary depending upon the context of the requirement; for example, the responsible party might be the lead, or another supporting person associated with the model development, operation, analysis, and/or reporting of results to decision makers.Program and project management in collaboration with the Technical Authority have the responsibility to identify and document the extent and level of formality of documentation needed to meet the documentation requirements in this standard. Some requirements, in particular, 4.1.5,4.2.6, 4.2.8, 4.3.6, 4.4.1, 4.4.2, 4.4.4, 4.4.5, 4.4.6, 4.4.7, 4.4.8, and 4.4.9, are to be interpreted as meaning that the activity in question is not required per se, but that whatever was done is to be documented, and if nothing was done a clear statement to that effect is to be documented. Program and project management in collaboration with the Technical Authority have the responsibility to identify and document the critical decisions to be addressed with M&S and to determine which M&S are in scope.The latter determination should be based upon the risk posed by the anticipated use of the M&S. Appendix A describes a representative M&S risk assessment matrix for this purpose.Furthermore, the Technical Authority has the particular responsibility to assure appropriate outcomes of Req. 4.1.3.The responsible party performs the following:Req. 4.1.1 – Shall document the risk assessment for any M&S used in critical decisions,Req. 4.1.2 – Shall identify and document those M&S that are in scope.Req. 4.1.3 – Shall define the objectives and requirements for M&S products including the following:a. The acceptance criteria for M&S products, including any endorsement for theM&S.b. The rationale for the weights used for the subfactors in the CredibilityAssessment Scale (see Appendix B.4).c. I ntended use.d. M etrics (programmatic and technical).e. V erification, validation, and uncertainty quantification (see section 4.4).f. Reporting of M&S information for critical decisions (see section 4.8).g.CM (artifacts, timeframe, processes) of M&S.(The acceptance criteria in 4.1.3 (a) includes specification of what constitutes a favorable comparison for the Verification Evidence, Validation Evidence, Input Pedigree Evidence, andUse History level definitions in the Credibility Assessment Scale (see Appendix B).)Req. 4.1.4– Shall develop a plan (including identifying the responsible organization(s)) for the acquisition, development, operation, maintenance, and/or retirement of the M&S. Req. 4.1.5 – Shall document any technical reviews performed in the areas of Verification, Validation, Input Pedigree, Results Uncertainty, and Results Robustness (seeAppendix B).Req. 4.1.6 – Shall document M&S waiver processes.Req. 4.1.7 – Shall document the extent to which an M&S effort exhibits the characteristics of work product management, process definition, process measurement, processcontrol, process change, and continuous improvement, including CM and M&Ssupport and maintenance.4.2 ModelsThe processes of developing conceptual, mathematical, or computational models are all considered to be modeling activities. Empirically adjusting the results of a simulation in an attempt to improve correlation is considered a modeling activity.For models, the responsible party performs the following:Req. 4.2.1 – Shall document the assumptions and abstractions underlying the conceptual model, including their rationales.Req. 4.2.2 – Shall document the basic structure and mathematics of the model (e.g., reality modeled, equations solved, behaviors modeled, conceptual models).(For COTS, GOTS, MOTS, and legacy M&S, some of the documentation required in 4.2.1 and 4.2.2 may be available in published user guides; a reference to the user guides will suffice for this part of the documentation.)Req. 4.2.3 – Shall document data sets and any supporting software used in model development and input preparation.Req. 4.2.4 – Shall document required units and vector coordinate frames (where applicable) for all input/output variables in the M&S.Req. 4.2.5 – Shall document the limits of operation of models.Req. 4.2.6 – Shall document any methods of uncertainty quantification and the uncertainty in any data used to develop the model or incorporated into the model.Req. 4.2.7 – Shall document guidance on proper use of the model.(Guidance on proper use of a model includes descriptions of appropriate practices for set-up, execution, and analysis of results.)Req. 4.2.8 – Shall document any parameter calibrations and the domain of calibration.Req. 4.2.9 – Shall document updates of models (e.g., solution adjustment, change ofparameters, calibration, and test cases) and assign unique version identifier,description, and the justification for the updates.Req. 4.2.10 – Shall document obsolescence criteria and obsolescence date of the model. (Obsolescence refers to situations where changes to the real system invalidate the model—see item (e) of Diaz Action #4.)Req. 4.2.11 – Shall provide a feedback mechanism for users to report unusual results to model developers or maintainers.Req. 4.2.12 – Shall maintain (conceptual, mathematical, and computational) models and associated documentation in a controlled CM system.Req. 4.2.13 – Shall maintain the data sets and supporting software referenced in Req. 4.2.3 and the associated documentation in a controlled CM system.。
