WRF - Regression Testing

top_left_banner SimCon logo

WRF - Static Analysis Regression Testing

The Aim

Whenever the WRF code is changed, the fpt regression tests may be re-run to verify that no new detectable errors have been injected into the code, and (hopefully) that existing problems have been eliminated.

Availability

The fpt distribution contains a free community licence for regression testing of WRF.

The distribution also contains the fsp scripts and supporting files for running fpt on different WRF configurations. The first configurations for which scripts have been released are:

em serial without nesting
em serial with simple nesting
em sm with simple nesting
em dm with simple nesting
em sm+dm with simple nesting

Running the Tests

Install fpt from http://simconglobal.com as described here.

Change directory to the main WRF directory where WRF has been configured and built. Run the csh script WRF_fpt_regression (This is in your PATH after fpt has been installed).

$ cd WRF $ WRF_fpt_regression

The process is completely automated. A typical run is shown below:

john@gemsbok:~/EHU/WRF/WRFV4.2/UCAR_em_serial/WRF-master$ WRF_fpt_regression WRF fpt Regression Test Copying regression files *** NOTE *** The following errors have been suppressed 2277 Semicolon, ';', statement separators have been removed. 2493 INTENT declared OUT but argument is read: 3085 Objects of .EQ., .NE. .GT. etc. are of different data types 1273 Fortran auxiliary keyword(s) used as identifier(s). 4133 \ character at EOL changed to & and treated as a continuation 3089 Comparison of REAL or COMPLEX values which differ in precision 3087 REAL or COMPLEX quantity tested for exact equality/inequality 1269 Fortran keyword(s) used as local identifier(s). 3291 String shorter than parameter in parameter assignment 1967 PARAMETER was not evaluated by fpt 1271 Non-standard Fortran intrinsic(s) used as local identifier(s) 3295 Parameter will not have the expected precision 2345 Data type of PARAMETER does not match that of the expression 1267 Fortran auxiliary keyword(s) used as identifier(s). 3271 Include file contains executable code 3425 No code was found in this file 2939 Same name is public in two used modules - access will be denied 3437 Mixed real or complex sizes in expression - loss of precision 3921 This symbol is already declared to be SAVE or STATIC 2451 Unused primary files encountered 2449 Unused sub-programs encountered 4141 Unsafe REAL NaN value may be changed by the system 1463 Not recognised as a directive, assumed to be a comment. 3073 Danger of overflow due to different integer sizes 3067 Loss of precision in real or complex assignment 3069 The left-hand-side variable will not have the expected precision 2865 REAL value assigned to INTEGER variable - danger of overflow 4209 VARIANT: No argument matches the specification <Many diagnostics issued by fpt, ending with> ------------------------------------------------------------------------------ !!! WARNING !!! Number 3637 Severity 1 (Worst 2) Line 1532 File: WRF-master/phys/module_cumulus_driver.F ,psfc = psfc,z = z,z_at_w = z_at_w,ht = ht,tsk = tsk &!CuP, wig 5-Oct-2006 ^ Unguarded use of an OPTIONAL argument which is always PRESENT ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ !!! WARNING !!! Number 3637 Severity 1 (Worst 2) Line 1870 File: WRF-master/phys/module_pbl_driver.F ,z = z,t_phy = t_phy,qc_curr = qc_curr,qi_curr = qi_curr,z_at_w = z_at_w & ^ Unguarded use of an OPTIONAL argument which is always PRESENT ------------------------------------------------------------------------------ File: ../fpt/WRF regression test ================================ Run on Wed 24 Jun 11:51:44 BST 2020 The left file is the reference listing file, the right is the new file WRF-master/ has been changed to WRF/ in your listing (fpl) file The dates and times in the file headers SHOULD be different Output generated by diff -w fpl_ref/wrf_em_serial.fpl ./wrf_em_serial.fpl 3c3 < !H! Output by fpt 4.1-b Intel-Linux On 16-JUN-20 At 15:46:38 Input files: --- > !H! Output by fpt 4.1-c Intel-Linux On 24-JUN-20 At 11:50:59 Input files: 5c5 < !H! Licensee: SimCon: Development version. --- > !H! Licensee: WRF Community Licence <End of file> john@gemsbok:~/EHU/WRF/WRFV4.2/UCAR_em_serial/WRF-master$

What does WRF_fpt_regression do?

The script WRF_fpt_regression copies directories named fpt, modified_source and xx_yy into the parent directory of WRF.

It then identfies which of the configurations listed above was used to build WRF and runs the appropriate static analysis on the *.F and *.F90 files.

The results of the analysis are written to an fpt listing file (an .fpl file) in the ../fpt directory, e.g. to wrf_em_serial.fpl for the em serial case. You may examine this file to see the fpt diagnostics generated and the reports of the tests carried out. The file is compared to a reference fpl file in the directory ../fpt/fpl_ref and the results are written to ../fpt/WRF_regression.txt. This file is listed to the screen.

If problems have been corrected you will see diagnostics in the reference file which are not in the .fpl file generated in the test. If new problems have been generated you will see new diagnostics in the test output.

Please

Tell us if you:

Find a problem in fpt (It is only a computer program!).
Publish a change in the master branch which changes a reference fpl file.

Some Notes

What does fpt test?

fpt analyses the code in the same way as a Fortran compiler and issues diagnostics when problems are detected. Many of the less important diagnostics are suppressed in the scripts in the distribution. Suppression of diagnostics is controlled by the file fpt/wrf_diagnostics.fsp and the diagnostics which have been suppressed are shown at the beginning of the listing file.

fpt then checks the WRF code for:

Ambiguous program flow: This can cause problems when the order of function evaluation is undefined and varies between compilers. Please see check ambiguous flow.
Variables used before they are assigned, assigned but never used and declared but never used: Please see check usage.
Mis-matched subroutine and function arguments: This check has been filtered to ignore the lengths of character arguments and the ranks of arrays. Cases where arrays are passed to scalar formal arguments and vice-versa have not been suppressed because we have seen problems where this occurs. Templates are provided for the c sub-programs in the file fpt/wrf_c_templates.ftm. Some arguments are used only to pass aggregate objects to the inter-process communication routines and these arguments are excluded from the checks by the variant declarations in the file fpt/wrf_checks.fsp. Please see check arguments.
EQUIVALENCE relationships which are potentially unsafe: See check equivalence.
Un-subscripted whole array assignments from scalar variables: These occasionally cause errors when array subscripts are omitted in error. Please see check whole array assignments.
Failed overloaded assignments: This is the situation where an assignment of a derived type is overloaded, usually to handle type components which are pointers, but the overload is not exported successfully to the code which needs it. There is an example in WRF in ESMF_time. Please see check overloaded assignments.
Optional arguments which are not protected from access by IF PRESENT constructs: See check optional arguments.

The listing (.fpl) file is very long. To find these reports edit the file and search for the strings:

     "ambiguous program flow"
     "check symbol usage"
     "check arguments"
     "check equivalence"
     "check whole array assignments"
     "check overloaded assignments"
     "check optional arguments"

If you think any of the reports are incorrect, please tell us.

Comparing the listing files

To compare fpl files we recommend beyond compare. The script WRF_fpt_regression simply uses diff.

Investigating problems in WRF

You can run fpt interactively to investigate problems. The command is, e.g.:

$ fpt wrf_em_dm.fsp %i

The %i means run interactively. You can use search and show commands to find and investigate symbols. Please see running_fpt.html for a description.

The modified_source directories

The modified_source directories contain:

A modified Makefile for the frame/ directory.
A very small number of files which we have corrected to avoid processing problems in fpt. In each case the corrections are noted in the file headers.
Copies of netcdf.inc and mpif.h because fpt will not know where they are on your system.

Setting up new regression tests

To set up your own regression tests for a configuration we haven't set up yet, you need to capture the symbols passed to cpp (and sometimes to the compiler) in WRF/compile. Please examine WRF_fpt_regression to see how this has been done.