1. Overview of METviewer
1.1. Purpose and organization of the User’s Guide
The goal of this User’s Guide is to provide basic information for users of the METviewer database and display system to enable users to create plots from their MET output statistics.
The METviewer User’s Guide is organized as follows.
1.2. The Developmental Testbed Center (DTC)
METviewer has been developed, and will be maintained and enhanced, by the Developmental Testbed Center (DTC ). The main goal of the DTC is to serve as a bridge between operations and research, to facilitate the activities of these two important components of the numerical weather prediction (NWP) community. The DTC provides an environment that is functionally equivalent to the operational environment in which the research community can test model enhancements; the operational community benefits from DTC testing and evaluation of models before new models are implemented operationally. METviewer serves both the research and operational communities in this way - offering capabilities for researchers to test their own enhancements to models and providing a capability for the DTC to evaluate the strengths and weaknesses of advances in NWP prior to operational implementation.
The METviewer package will also be available to DTC visitors and to the modeling community for testing and evaluation of new model capabilities, applications in new environments, and so on.
1.3. METviewer goals and design philosophy
The METviewer tool reads MET verification statistics output from a database and creates plots using the R statistical package. The tool includes a web application that can be accessed from a web browser to create a single plot. The specification for each plot is built using a series of controls which are then serialized into XML. The controls are found in the left half, lower panel, and lower right quadrant of the METviewer browser. Details about these controls are found in METviewer User Specifications. For each plot, METviewer generates output in the form of a SQL query, an R script to create the plot, a flat file containing the data that will be plotted and the plot itself. This output populates the upper right quadrant of the METviewer database.
The METviewer code and documentation is maintained by the DTC in Boulder, Colorado. The MET package is freely available to the modeling, verification, and operational communities, including universities, governments, the private sector, and operational modeling and prediction centers.
1.4. METviewer Requirements
METviewer requires installation of Java, Apache / Tomcat, MySQL, and R statistics.
1.5. METviewer User Specifications
The top menu allows the user to select a database and type for the plot.
The first section is where the dependable variable (Y-axis) , statistic of interest, and series variables of interest are selected. The Y2 axis can be used to plot a Base Rate.
The fixed variables section controls the constraints for the plot (what period of data is used, what thresholds to include, interpolation method, as well as domains). Fixed values are for stratifications. For example, select one or more thresholds to be aggregated, or one or more initialization times, or a date range based on valid times or cycle time.
Each of the above selections can have one or multiple items. To add variables, series, or fixed values, select the button with the ‘+’ directly below the desired item. Then additional selections will appear in a new dropdown menu. To delete these extras, select the trash icon to the left of the item.
The independent variable section is where the X-axis is configured. As an example, forecast lead hour often serves as the x-axis for a series plot.
This section is reserved for line labels and formatting for the individual series confidence intervals: Normal Bootstrap Standard error (median or mean) – configurable.
Summary – scores computed per each combination of fixed values and independent variable then mean or median taken.
Aggregate statistics – accumulates SL1L2 lines or CTC counts prior to calculating the statistics.
Confidence intervals - METviewer supports two statistical derivation methods… (1) plot the mean of the daily stats with confidence intervals defined by the “standard error”… that uses the standard deviation to add confidence intervals and assumes normality (2) plot the aggregated statistics with confidence intervals defined by bootstrapping that aggregation.
Pairwise Differencing – computes the difference in statistics for two models on a pairwise basis (same forecast hour, same initialization, same region, etc…). Confidence intervals may be applied to determine statistical significance.
Event Equalization – performs the filtering on two datasets to produce a “homogeneous” intersection between the samples (same forecast hour, same initialization, same region, etc…). Use of this method is recommended when trying to compare the skill of one or more predictions.
1.6. METviewer Output
METviewer output all appears in the tabbed boxes at the top right quadrant of the web browser. The plot will show by default, but the user can click any of the tabs along the top to see the other windows. The tabs include:
XML - This window contains all the code to reproduce the plot. It is recommended that the user save it. It can be reloaded using the ‘Load XML’ button at the top right of the METviewer web page.
Log - Contains some runtime information regarding input data, run time, and which scripts were called.
R script - Contains the R code used to organize the data and create the plot. This can be easily modified directly by the user to create slightly different plots for alternate purposes.
R data - It is essential that all users look at the contents of this tab. The data contained under the R data tab is the data used to create the plot. Users should check to see if it is correct. Often, a user with limited knowledge of the database may select data in such a way as to combine information in an unintended way. For example, statistics for different event thresholds should not be combined, but if the user fails to select the threshold(s) correctly in the ‘fixed values’ section of METviewer, the plot will show statistics for the combined data.
SQL - This tab contains the SQL code used to gather the data for the plot.
Y1 Points - This tab will be blank for several of the plot types.
Y2 Points - See above.
The rightmost tab is a small square. Clicking here will save the contents of the current window to the default (usually ‘Downloads’) folder. It is recommended that the user click through at least the plot and xml windows and save this output.
1.7. METviewer Plot Formatting Specifications
METviewer plot formatting options appear in the tabbed boxes at the bottom right quadrant of the web browser. The “Titles & Labels” tab will show by default, but the user can click any of the tabs along the top to see the other windows. The tabs include:
Titles and Labels: This menu contains boxes for the user to specify a plot title, caption, and axis labels.
Common: This tab contains multiple check boxes and a confidence level specification box. The check boxes add X2 axes, stagger points from different axes, print points.
Formatting: Plot, title and grid formatting options are available here. Options include image type, height, width, and resolution. Margins, offsets, text size, line widths, etc. can all be set in units of text point size (e.g. 8 = 8 pt font)
X1, X2, Y1, and Y2 - These tabs specify the axis labels and values. The text size, weight, alignment and offset can be chosen. For Y1 and Y2 axes, limits and buffers can also be set.
Legend & Caption: Options for the Legend and Caption include text size, position, alignment, color, offset. For details about setting these specifications, see the R-help pages for the functions “legend” and / or “title”.
Reset: This button, shown to the far right with a circular arrow icon, will reset all options to the defaults.
1.8. METviewer Series Formatting Specifications
At the bottom of the METviewer window is the series formatting box. It allows the user to control the minute details of each line on a plot, including title, color, symbol, line type, line width, legend text.
Four radio buttons along the bottom of this menu allow users to create a derived curve or remove one, apply default values or lock the formatting in place (so subsequent plots are identical in format).
1.10. METviewer Release Notes
When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature: https://github.com/dtcenter/METviewer/issues
1.10.1. Version 4.1.0 release notes
1.10.2. beta3 release (20211007)
- New Features:
new SSIDX STAT line type and statistic SS_INDEX (#327)
python plot is not created if plot margin values are doubles (#331)
1.10.3. beta2 release (20210901)
- New Features:
The loading module produces an error when parsing <date_list> element with <date_offset> (#319)
1.10.4. beta1 release (20210706)
Fixed loading XML for the ROC diagram to the UI #266
Fixed a bug when XML with variable indep plot_vals doesn’t get loaded to UI correctly #257
Fixed Event Equalisation for ROC diagram with multiple series #256
Plot all series with different number of i_values on the Hist plot #254
Fixed bugs in Docker file: build of ‘pingouin’ module, permissions for the executables, mysql user name
Fixed the UI bug when the plot formatting menu is inaccessible when the Series formatting menu is pulled up and then down
Load the additional climatology column in the ORANK line type updated for met-10.0.0 #261
Load the additional climatology column in the ECNT line type updated for met-10.0.0 #259
Add ‘Equalize fixed var” option to scorecard #235
Add ‘cov_thresh’ to the menu lists in GUI #206
LODDS, ODDS, ORSS, SEDI, SEDS, EDI, EDS statistics are available for aggregation logic #252
Support to reverse the connection order in ROC curve
Use the circular bootstrap as a default method instead of the regular
New Equivalence Testing Bounds plot
In the scorecard XML allow users to specify the label that will be used instead of CSV list for <plot_fix> fcst_init_beg
In the scorecard add support for the weights.
Python implementation of Reliability diagram
Add nbrctc data to the Performance diagram
Plotly/Python implementation of the Series plot(#201)
Interactive plots in GUI (currently is implemented for Series plots only - will add other plots later)
Using Github actions to build documentation #274
Restructure the “Conf interval” dropdown list #255
METdatadb is available in METviewer
Thw warning message will be displayed if no plots were produced on AWS
HTML plot is scaled to the size of the window in GUI
The version number is added to the scripts INFO messages
<met_version> tag is discontinued and removed from METviewer
Updated 3rd-party libraries
Revise computation of No-Skill reference line on Reliability diagram
Check-box to add/remove No-Skill line (#209)
1.11. Future development plans
METviewer is an evolving verification database and display software package. New capabilities are planned in controlled, successive version releases. Bug fixes and user-identified problems will be addressed as they are found. Plans are also in place to incorporate many new capabilities and options in future releases of METviewer. Additionally, updates to accommodate new features of the MET software are often required.
1.12. Code support
METviewer support is provided through the METplus GitHub Discussions Forum. We will endeavor to respond to requests for help in a timely fashion.
We welcome comments and suggestions for improvements to METviewer, especially information regarding errors. In addition, comments on this document would be greatly appreciated. While we cannot promise to incorporate all suggested changes, we will certainly take all suggestions into consideration.
METviewer is a “living” application. The goal is to continually enhance it and add to its capabilities.