The PEN model-data comparison workgroup: Initial notes from the 1-4 September 2015 Crewe Hall Conference (07/09/2015 LCS)

Included page "clone:modeldatacomparison" does not exist (create it now)

Navlogo.png PEN is here: http://www.pastearth.net/index.html

Quick start guide on how to contribute to this wiki

Who at PEN?

21 initial members – about 10 identified as primarily focused on statistics. 3 initial members at least largely identified as being focused on data / the analysis of data. 8 initial members at least largely identified as being focused on modelling (GCMS/EMICS).

We decided that a private Wiki like this might be a good way to collaborate remotely. [Thanks to Peter Green for initiating this page!]

Data-model comparison bibliography
And our first funded project

Discussion on challenges

1. How do we make data and models comparable? The representation of uncertainty due to differences in spatial and temporal scales.
(i) Linking models with data when the models are run at differing spatial scales.
(ii) Can single data points, based on individual cores, be representative of model values within a grid cell?
(iii) Which properties do we wish to compare? For example, how do we do model-data comparisons for the probability of extremes (e.g. extreme precipitation events).
(iv) The use of proxy-enabled models.

2. Some problems holding up our learning re: the mismatch of GCMs with data
(i) The replication of GCM simulations/replications/ensembles. Linked to that was how do we specify how many model simulations are required.
(ii) How far away are we from generating PDFs of model output?
(iii) The associated problem of capturing abrupt/unexpected events in model PDFs.

3. The use of model output for the generation of pseudo/test datasets to help test statistical approaches to data.
(i) What type of pseudo-proxy studies have been performed? Which statistical approaches should be tested?
(ii) How can proxy system models be integrated and tested using GCM output and proxy data?

4. Timeseries analysis/Events?
(i) The Last Millennium / 2K. Links to PAGES and PMIP3 WGs
(ii) The last deglaciation. Links particularly to the Ruza's PMIP3 deglaciation WG. How do we compare large sets of paleodata with transient deglacial GCM simulations. What stat framework? Equivalence of problem with Ander’s previous 1/2K work.
(iii) Glacial/interglacial cycles and changes therein (MPT?)
(iv) The exploration of deep time?
Links to other WGs? Choice of specific events?

5. The use of conceptual models
Tim Lenton's comments - would require that we invite more conceptual modellers

Ideas for feasibility studies

1. The development of a statistical framework to assess whether an model-data error has significance/meaning to the community of GCM developers e.g. the MetOffice.
2. The development of a standard statistical framework/approach to apply to standard model output and data. This could be applied to the problem of whether PMIP3 model output is an improvement on PMIP2 output.
3. The idea of linking/pairing GCM modellers with statisticians. Could PEN set up a cross-institute visiting program? [Needs to have an application procedure - and steering committee approval.] This could fits with the idea of PEN identifying which inter-group collaborations might be most beneficial.
4. Collaboration via a Wiki page. [Started!] Collaborations however need to be rewarding for statisticians. Do we need to think about whether all are sufficiently rewarded to ensure adequate buy-in for collaborations?


1. We discussed two options for WG3 workshops. Either individual small workshops to address specific problems e.g. the deglaciation. Or, as an alternative one larger workshop with, for example, half a day on each problem. It was felt that it would be better if this/these occurred sooner rather than later.
2. The group suggested that collaboration via the wiki might help us select (and formulate) problems for individual sessions/workshops. Do we then need a PEN call for session/workshop ideas?

Notes on other discussion points

1. We had a longer discussion of whether data may be used twice i.e. once in the tuning of GCM parameters and once in model-data evaluation. Users of GCMs felt that this was not a problem for paleoclimate since, at least within the UK, paleoclimate data is never used to tune UK GCMs.
2. On the issue of CMIP-type model-data comparisons. Ensemble statistics/ boundary condition specification is necessary - replication? To do large scale replication/large ensembles we likely need to use older GCMs since they are faster/cheaper. However there are problems associated with model support for older model. Use of HadCM3 may be an exception? Application of emulator/ensemble stat technique to assess CMIP5/6 output?
3. It was emphasised a few times that statistics are not a sticking plaster. We should ideally work together at the start of the study. Modeller/observationalists may need to better formulate their problems - which may itself need close collaborations between PEN model-data statistical experts.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License