In Demo 2 we will be looking at how PEcAn can use information about parameter uncertainty to perform three automated analyses:
Return to the main menu for the PEcAn web interface: URL > Run Models
Repeat the steps for site selection and run specification from Demo 01, but also click on “Advanced setup”, then click Next.
By clicking Advanced setup, PEcAn will first show an Analysis Menu, where we are going to specify new settings.
For an ensemble analysis, increase the number of runs in the ensemble, in this case set Runs to 50. In practice you would want to use a larger ensemble size (100-5000) than we are using in the demo. The ensemble analysis samples parameters from their posterior distributions to propagate this uncertainty into the model output.
PEcAn’s sensitivity analysis holds all parameters at their median value and then varies each parameter one-at-a-time based on the quantiles of the posterior distribution. PEcAn also includes a handy shortcut, which is the default behavior for the web interface, that converts a specified standard deviation into its Normal quantile equivalent (e.g. 1 and -1 are converted to 0.157 and 0.841). In this example set Sensitivity to -2,-1,1,2 (the median value, 0, occurs by default).
We also can tell PEcAn which variable to run the sensitivity on. Here, set Variables to NEE, so we can compare against flux tower NEE observations.
The PEcAn workflow will take considerably longer to complete since we have just asked for over a hundred model runs. Once the runs are complete you will return to the output visualization page were there will be a few new outputs to explore, as well as outputs that were present earlier that we’ll explore in greater details:
While the sensitivity and ensemble analyses synthesize across runs, you can also select individual runs from the Run ID menu. You can use the Graphs menu to visualize each individual run, or open individual runs in Shiny
This menu shows the contents of /run which lets you look at and download:
This menu shows the contents of /out. A number of files generated by the underlying ecosystem model are archived and available for download. These include:
This menu shows the contents of /pft. There is a wide array of outputs available that are related to the process of estimating the model parameters and running sensitivity/uncertainty analyses for a specific Plant Functional Type.
names(trait.data)as this will tell you what variables were extracted,
names(trait.data$Amax)will tell you the names of the columns in the Amax table, and
summary(trait.data$Amax)will give you summary data about the Amax values.
*.bug: The evaluation of the meta-analysis is done using a Bayesian statistical software package called JAGS that is called by the R code. For each trait, the R code will generate a [trait].model.bug file that is the JAGS code for the meta-analysis itself. This code is generated on the fly, with PEcAn adding or subtracting the site, treatment, and greenhouse terms depending upon the presence of these effects in the data itself. If the <random.effects> tag is set to FALSE then all random effects will be turned off even if there are multiple sites.
meta-analysis.logcontains a number of diagnostics, including the summary statistics of the model, an assessment of whether the posterior is consistent with the prior, and the status of the Gelman-Brooks-Rubin convergence statistic (which is ideally 1.0 but should be less than 1.1).
ma.summaryplots.*.pdfare collections of diagnostic plots produced in R after the above JAGS code is run that are useful in assessing the statistical model. Open up one of these pdfs to evaluate the shape of the posterior distributions (they should generally be unimodal), the convergence of the MCMC chains (all chains should be mixing well from the same distribution), and the autocorrelation of the samples (should be low).
traits.mcmc.Rdatacontains the raw output from the statistical code. This includes samples from all of the parameters in the meta-analysis model, not just those that feed forward to the ecosystem, but also the variances, fixed effects, and random effects.
post.distns.Rdatastores a simple tables of the posterior distributions for all model parameters in terms of the name of the distribution and its parameters.
posteriors.pdfprovides graphics showing, for each model parameter, the prior distribution, the data, the smoothed histogram of the posterior distribution (labeled post), and the best-fit analytical approximation to that smoothed histogram (labeled approx). Open posteriors.pdf and compare the posteriors to the priors and data
sensitivity.analysis.[RunID].[Variable].[StartYear].[EndYear].pdfshows the raw data points from univariate one-at-a-time analyses and spline fits through the points. Open this file to determine which parameters are most and least sensitive
variance.decomposition.[RunID].[Variable].[StartYear].[EndYear].pdf, contains three columns, the coefficient of variation (normalized posterior variance), the elasticity (normalized sensitivity), and the partial standard deviation of each model parameter. Open this file for BOTH the soil and conifer PFTS and answer the following questions:
This menu shows the contents of the root workflow folder that are not in one of the folders indicated above. It mostly contains log files from the PEcAn workflow that are useful if the workflow generates an error, and as metadata & provenance (a detailed record of how data was generated).
STATUSgives a summary of the steps of the workflow, the time they took, and whether they were successful
pecan.*.xmlare PEcAn settings files
workflow.Ris the workflow script
workflow.Routis the corresponding log file
samples.Rdatacontains the parameter values used in the runs. This file contains two data objects, sa.samples and ensemble.samples, that are the parameter values for the sensitivity analysis and ensemble runs respectively
sensitivity.output.[RunID].[Variable].[StartYear].[EndYear].Rdatacontains the object sensitivity.output which is the model outputs corresponding to the parameter values in sa.samples.
ensemble.Rdatacontains contains the object ensemble.output, which is the model predictions at the parameter values given in ensemble.samples.
ensemble.analysis.[RunID].[Variable].[StarYear].[EndYear].pdfcontains the ensemble prediction as both a histogram and a boxplot.
ensemble.ts.[RunID].[Variable].[StartYear].[EndYear].pdfcontains a time-series plot of the ensemble mean, median, and 95% CI
Navigate to URL/shiny/global-sensitivity.
This app uses the output from the ENSEMBLE runs to perform a global Monte Carlo sensitivity analysis. There are three modes controlled by Output type:
In all of these analyses, the app also fits a linear regression to these scatterplots and reports a number of summary statistics. Among these, the slope is an indicator of global sensitivity and the R2 is an indicator of the contribution to global uncertainty
The next set of tutorials will focus on the process of data assimilation and parameter estimation. The next two steps are in “.Rmd” files which can be viewed online.
Explore how model error changes as a function of parameter value (i.e. data assimilation ‘by hand’)
Explore Bayesian MCMC concepts using the photosynthesis module
Additional information about specific tasks (adding sites, models, data; software updates; etc.) and analyses (e.g. data assimilation) can be found in the PEcAn documentation
If you encounter a problem with PEcAn that’s not covered in the documentation, or if PEcAn is missing functionality you need, please search known bugs and issues, submit a bug report, or ask a question in our chat room. Additional questions can be directed to the project manager