NEW C ESM EP v2 code: content and organization (original) (raw)

This page documents the content and organization of the new C-ESM-EP_v2 code.

This is what you get when you clone the default C-ESM-EP (following 'The C-ESM-EP in a nutshell'): (path_to_your_cesmep_workdir is where you cloned the C-ESM-EP)

${path_to_your_cesmep_workdir}/C-ESM-EP/

• run_C-ESM-EP.py The job launcher of the C-ESM-EP. No need to edit (only for advanced users/developers)
• main_C-ESM-EP.py The core script of the C-ESM-EP. This is where we run the diagnostics and build the html page of each atlas. If you want to see how to add your own diagnostic, check this page.
• setenv_C-ESM-EP.sh Source this script to setup your environment properly to use the C-ESM-EP (automatically sourced before submission of a cesmep job).
• locations.py This is where you set up the paths to the output directories for the atlases, the url of the web servers, depending on where you work.
• standard_comparison/ standard_comparison is a typical example of a comparison of simulations/models that we do with the C-ESM-EP. It also serves as testing when you install the C-ESM-EP for the first time. See here how to run it.
  • datasets_setup.py Specify the datasets (simulations/models) you want to compare in this file (details here and even more details here)
  • job_C-ESM-EP.sh Job script used by run_C-ESM-EP.py to submit the C-ESM-EP jobs (one per atlas)
  • and the standard atlas directories (see here the details on those atlas directories)
• set_available_period_ts_clim.py Finds the available time periods for clim_period and ts_period once, which discards the automatic period manager for all the diagnostics that can be time consuming (documentation here)
• cesmep_simu_finder.py A tool to explore your existing datasets and check if your dataset specification (in datasets_setup.py) is correct.
• custom_plot_params.py An interface to specify your own plot parameters per variable. Documentation TBD.
• clean_out_error.sh A script to clean your jobs executions outputs and errors files, and the climaf.log and last.out files. Documentation here
• share/ Contains the code of the diagnostics that are not directly written in main_C-ESM-EP.py (cesmep_modules), optional atlases (optional_atlas) and additional scripts used for certain diagnostics (scientific_packages), default settings (default), and some html templates (fp_template)
  • cesmep_diagnostics/ (new feature in v2)
    * diagnostics_${component}.py : the diagnostics files contain the code that both compute the diagnostics and add the results to the html page; the diagnostics files are read by main_C-ESM-EP.py together with params_${component}.py for each atlas/component. For each component, main_C-ESM-EP.py will first check if a diagnostic file is located in the component directory of the comparison directory (${comparison}/${component}/, for instance standard_comparison/AtlasExplorer). If no diagnostic file is found, then main_C-ESM-EP.py uses the diagnostic files in share/cesmep_diagnostics
  • cesmep_modules/
    * CM_atlas/ The 'Coupled Model Atlas'. Contains generic functions like the Atlas Explorer functions, the Time and Color Mangers, and other Coupled Model specific functions (ENSO, ORCHIDEE, WaterBudegt)
    * time_manager.py you find here the functions used to translate clim_period='last_10Y' and ts_period='full' in actual real dates available for the simulation: get_period_manager, period_for_diag_manager, frequency_manager_for_diag, find_common_period, and base_variable_of_derived_variable
    * colors_manager.py colors_manager is a common function to allocate colors to a set of datasets, taking into account the potential colors specified by the user manually; it returns standard hash colors, understood by any software (python, R...)
    * plot_CM_atlas.py The basic functions of Atlas Explorer are in this file:
    * section_2D_maps, section_climato_2D_maps, section_zonal_profiles and section_2D_maps_climobs_bias_modelmodeldiff return an html section of plots; each line is a variable (season, projection...) and each column is a dataset (reference, simulation, model)
    * plot_climato, plot_diff, plot_zonal_profile _these are the basic functions used in the section functions
    * safe_mode_cfile_plot returns the plot if successful, and a blank image if the plot failed
    * start_line starts a line of plot
    * build_plot_title is used for the plots; it returns the customname, or the model name for the CMIP5 project, or the simulation name for the other projects, and adds the reference in the string if you provide one (second argument)
    * replace_keywords_with_values typically useful for customname: if the user sets customname='${model} ${simulation}' it will return a string with the model and simulation values
    * get_realization_simulation_kw returns the realization keyword if it exists, or the simulation otherwise
    * build_period_str returns the period of a dataset, after analyzing whether it should use period or clim_period
    * parallel_CM_atlas.py
    * parallel_section_2D_maps is based on section_2D_maps and uses the joblib and multiprocessing python libraries to execute the code in parallel; it uses section_2D_maps(..., do_cfile=False) first to return a python list with the CRS of all the plots; then, it executes the plots in parallel by providing this list to joblib; and finally, it re-uses section_2D_maps(..., do_cfile=True) to actually build the html section (taking advantage of the smart cache)
    * plot_ENSO_atlas.py contains all the functions needed by the ENSO atlas; see the documentation on the scientific content of the ENSO CLIVAR atlas for the details of what these functions do
    * ENSO_ts_ssta
    * ENSO_std_ssta
    * ENSO_pr_clim
    * ENSO_tauu_clim
    * ENSO_linreg_tauuA_on_SSTANino3
    * ENSO_linreg_rsds_on_SSTANino3
    * find_common_period
    * plot_ENSO_annual_cycles
    * plot_ZonalWindStress_long_profile
    * plot_WaterBudget_IPSL.py in progress
    * section_WaterBudget
    * ORCHIDEE_function.py only contains:
    * derive_var_PFT that is used to automatically create a derived variable from a variable that has the PFT dimension
    * init.py
    * LMDZ_SE_atlas/ is the set of functions for the LMDZ SE atlas; we do not use them directly in the C-ESM-EP
    * plot_LMDZ_SE_atlas.py
    * plot_atmos_climato
    * plot_atmos_bias
    * init.py
    * NEMO_atlas_v2/
    * plot_NEMO_CERFACS.py
    * vertical_profile
    * moc_slice
    * moc_profile_vs_obs
    * maxmoc_time_serie
    * hovmoller_drift_profile
    * index_timeserie
    * zonal_mean_slice2 and zonal_mean_slice
    * whichORCAGrid
    * nav_lat_zovarbasin_file
    * build_coordinates_zovarbasin
    * title_region
    * region2basin
    * set_fixed_fields
    * SurfaceFluxes_GB2015.py
    * get_product
    * ref_ensemble_GB2015
    * stat_ref_ensemble_GB2015
    * plot_bias_TurbFlux_vs_GB2015
    * plot_climato_TurbFlux_GB2015
    * plot_NEMO_atlas.py
    * plot_curl
    * plot_sic_climato_with_ref
    * plot_SIV
    * init.py
    * PMP_MG/
    * MetricsGardenv2.py
    * PMP_MG_time_manager.py
    * frequency_manager_for_diag_type
    * split_periods_in_models
    * get_keys_for_PMP_MG
    * build_metric_outpath
    * build_input_climatology_filename
    * init.py
    * reference/
    * reference.py
    * variable2reference
    * init.py
  • default/
    * default_atlas_settings.py
  • fp_template/
    * cesmep_atlas_style_css
    * C-ESM-EP_template.html
    * edit_error_html.py
    * Running_template.html
    * CESMEP_bandeau.png
    * copy_html_error_page.sh
    * error_template.html
  • optional_atlas/
    * see details here
  • scientific_packages/
    * Hotelling_Test/
    * parallel_coordinates/
    * TuningMetrics/
    * UEVE_otorres/
    * WaterBudgetSketch
• Documentation
• README.md