Evaluation and diagnostics ED

Scien**tific Working Group on Evaluation and Diagnostics**

The Scien**tific Working Group on Evaluation and Diagnostics** exists to foster activities that help describe, document and understand the performance of integrated assessment models so as to improve model performance and reliability and to make recommendations in the areas of priorities for community activities and standards of best practice.

Integrated assessment modelers have and will continue to be asked by potential users and critics why they should have confidence in the results of IAMs, or what level of confidence they should have in the results. Like all models, IAMs describe a critical set of relationships between assumed values for key input assumptions, which lie outside the model, and variables of interest that the model predicts, albeit a contingent prediction. For IAMs exogenous assumptions typically include demographic characteristics, economic growth, technology characteristics and availability, and the policy environment. IAMs produce contingent predictions of such variables as energy supply, demand, transformation, trade and prices, agricultural supply, demand, land use, land cover, and prices, carbon and other greenhouse gas prices, and the economic cost of policy interventions. Determining the usefulness of a given model for a given purpose is both art and science. The purpose of the model is critical to determining usefulness. It is also critical to determining the right set of test to apply in the evaluation and diagnosis of a given model. A variety of techniques have been applied to evaluate and diagnose models. While the SWG will not undertake model evaluation, diagnostics or validation, the SWG can undertake activities that help shape and facilitate such activities. In recent years, a number of community activities have undertaken model documentation, diagnostics and evaluation activities. Workshops on model evaluation and diagnostics have been conducted by the US DOE-funded PIAMDDI and PCHES (2010-present) and the EU FP7 funded AMPERE projects (2011-2014) that helped to enhance our understanding about the various ways contributing to model evaluation and the analogies to the approaches in other communities such as the operations research and climate modelling communities. Both projects developed diagnostic methods and indicators to better understand underlying reasons for similarities and differences in model behavior. The EU FP7 funded ADVANCE project (2013-2016) developed standards for model documentation (see common IAM documentation wiki) and continued work on IAM diagnostics and evaluation (see Wilson et al. in Climatic Change volume 166, Article number: 3 (2021)). In the framework of the EU H2020 NAVIGATE project (2019-2023), an extension of AMPERE project, a selected set of well-defined indicators as a community standard, to systematically and routinely assess IAM behaviour, similar to metrics used for other modeling communities such as climate models has been built (see Harmsen et al. in Environmental Research Letters, Vol 16 No. 15, 2021 "Integrated assessment model diagnostics: key indicators and model evolution".