14. Performance Diagrams

14.1. Description

Performance diagrams are used to show the relationship between categorical statistics, with axes representing detection and success (1 - false alarm) rates (Roebber, 2009). Frequency bias and the critical success index (CSI) are also shown on the diagram. The simplest input to the performance diagram is the MET contingency table statistics (CTS) output. This output can be produced by many of the MET tools (Point-Stat, Grid-Stat, etc.). In METviewer, contingency tables can be accumulated over some time period but kept separate for different models, lead times, observation sources, domains, etc.

There are several reference lines on the performance diagram. The dashed lines that radiate from the origin are lines of equal frequency bias. Labels for the frequency bias amount are at the end of each line in the margins. The diagonal represents a perfect frequency bias score of 1. Curves of equal Critical Success Index (CSI) connect the top of the plot to the right side. CSI amounts are listed to the right side of the plot, with better values falling closer to the top.

14.2. Line Type

Performance requires categorical statistics generated by either Point-Stat or Grid-Stat. These are:

  • CTC

  • NBRCTC

  • CTS

14.3. How-To

Selection of options to produce the reliability diagram proceeds approximately counter-clockwise around the METviewer window.

  1. Select the desired database from the “Select databases” pulldown menu at the top margin of the METviewer window.

  2. There are a number of tabs just under the database pulldown menu. Select the ‘Perf’ tab.

  3. Select the desired series variable to calculate for in the “Series Variables” tab. Press the “+ Series Variable” button to reveal two pulldown menus. The first pulldown menu in the lists the categories available in the selected dataset. The second pulldown menu allows the user to select the value of that category.

  4. It usually does not make sense to mix statistics for different groups. The desired group to calculate statistics over can be specified in the “Fixed Values” section. For a performance diagram, the forecast variable (“FCST_VAR”) must be selected. In the example below, the forecast variable is “APCP_03”. A single domain (category: “VX_MASK”, value: “CONUS”) is chosen. If multiple domains or thresholds were chosen, the statistics would be a summary of all of those cases together, which may not always be desired.

  5. Select the independent variable in the “Independent Variable” section.

  6. Select the type of statistics summary by selecting either “Summary” or “Aggregation Statistics” button in the “Statistics” section. Aggregated statistics may be selected for certain varieties of statistics. The selection can be made from the leftmost dropdown menu in the “Statistics” section. By default, the median value of all statistics will be plotted. Using the dropdown menu, the mean or sum may be selected instead. Choosing this option will cause a single statistic to be calculated from the individual database lines.

  7. Now enough information has been entered to produce a graph. To do this, click the “Generate Plot” button at the top of the METviewer window (this is in red text). Typically, if a plot is not produced, it is because the database selected does not contain the correct type of data. Also, it is imperative to check the data used for the plot by selecting the “R data” tab on the right hand side, above the plot area. The data from the database that is being used to calculate the statistics is listed in this tab. This tab should be checked to avoid the accidental accumulation of inappropriate database lines. For example, it does not make sense to accumulate statistics over different domains, thresholds, models, etc.

There are many other options for plots, but these are the basics.

14.4. Example

The reliability diagram shown below is for 3 hour accumulated precipitation over the CONUS for 8 ensemble members at the 18 hour forecast lead time. In the figure, the ensemble members fall closer to the lower left area of the diagram, indicating that the ensemble has a low performance for forecasting 3-hr accumulated precipitation at the 18 hour forecast lead time.

../_images/perfdiag_plot.png

Figure 14.1 Example performance diagram.

Here is the associated xml for this example. It can be copied into an empty file and saved to the desktop then uploaded into the system by clicking on the “Load XML” button in the upper-right corner of the GUI. This XML can be downloaded from this link: perfdiag_xml.xml.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plot_spec>
    <connection>
        <host>mohawk</host>
        <database>mv_hrrr_sseo_test</database>
        <user>******</user>
        <password>******</password>
        <management_system>mariadb</management_system>
    </connection>
    <rscript>/usr/local/R/bin/Rscript</rscript>
    <folders>
        <r_tmpl>/opt/vxwww/tomcat/webapps/metviewer//R_tmpl</r_tmpl>
        <r_work>/opt/vxwww/tomcat/webapps/metviewer//R_work</r_work>
        <plots>/d2/www/dtcenter/met/metviewer_output//plots</plots>
        <data>/d2/www/dtcenter/met/metviewer_output//data</data>
        <scripts>/d2/www/dtcenter/met/metviewer_output//scripts</scripts>
    </folders>
    <plot>
        <template>performance.R_tmpl</template>
        <series1>
            <field name="model">
                <val>HRRR_mem0_ctrl_hrconus</val>
                <val>HRRR_mem1_hrconus</val>
                <val>HRRR_mem2_hrconus</val>
                <val>HRRR_mem3_hrconus</val>
                <val>HRRR_mem4_hrconus</val>
                <val>HRRR_mem5_hrconus</val>
                <val>HRRR_mem6_hrconus</val>
                <val>HRRR_mem7_hrconus</val>
            </field>
        </series1>
        <plot_fix>
            <field equalize="false" name="fcst_var">
                <set name="fcst_var_0">
                    <val>APCP_03</val>
                </set>
            </field>
            <field equalize="false" name="vx_mask">
                <set name="vx_mask_1">
                    <val>CONUS</val>
                </set>
            </field>
        </plot_fix>
        <plot_cond/>
        <annotation_template/>
        <indep name="fcst_lead">
            <val label="18" plot_val="">180000</val>
        </indep>
        <plot_stat>median</plot_stat>
        <tmpl>
            <data_file>plot_20200921_180859.data</data_file>
            <plot_file>plot_20200921_180859.png</plot_file>
            <r_file>plot_20200921_180859.R</r_file>
            <title>3hr APCP, 18 hr Lead, CONUS</title>
            <x_label/>
            <y1_label/>
            <y2_label/>
            <caption/>
            <job_title/>
            <keep_revisions>false</keep_revisions>
            <listdiffseries1>list()</listdiffseries1>
            <listdiffseries2>list()</listdiffseries2>
        </tmpl>
        <execution_type>Rscript</execution_type>
        <event_equal>false</event_equal>
        <vert_plot>false</vert_plot>
        <x_reverse>false</x_reverse>
        <num_stats>false</num_stats>
        <indy1_stag>false</indy1_stag>
        <indy2_stag>false</indy2_stag>
        <grid_on>true</grid_on>
        <sync_axes>false</sync_axes>
        <dump_points1>false</dump_points1>
        <dump_points2>false</dump_points2>
        <log_y1>false</log_y1>
        <log_y2>false</log_y2>
        <varianceinflationfactor>true</varianceinflationfactor>
        <plot_type>png16m</plot_type>
        <plot_height>8.5</plot_height>
        <plot_width>11</plot_width>
        <plot_res>72</plot_res>
        <plot_units>in</plot_units>
        <mar>c(8,4,5,4)</mar>
        <mgp>c(1,1,0)</mgp>
        <cex>1</cex>
        <title_weight>2</title_weight>
        <title_size>1.4</title_size>
        <title_offset>-2</title_offset>
        <title_align>0.5</title_align>
        <xtlab_orient>1</xtlab_orient>
        <xtlab_perp>-0.75</xtlab_perp>
        <xtlab_horiz>0.5</xtlab_horiz>
        <xtlab_freq>0</xtlab_freq>
        <xtlab_size>1</xtlab_size>
        <xlab_weight>1</xlab_weight>
        <xlab_size>1</xlab_size>
        <xlab_offset>2</xlab_offset>
        <xlab_align>0.5</xlab_align>
        <ytlab_orient>1</ytlab_orient>
        <ytlab_perp>0.5</ytlab_perp>
        <ytlab_horiz>0.5</ytlab_horiz>
        <ytlab_size>1</ytlab_size>
        <ylab_weight>1</ylab_weight>
        <ylab_size>1</ylab_size>
        <ylab_offset>-2</ylab_offset>
        <ylab_align>0.5</ylab_align>
        <grid_lty>3</grid_lty>
        <grid_col>#cccccc</grid_col>
        <grid_lwd>1</grid_lwd>
        <grid_x>listX</grid_x>
        <x2tlab_orient>1</x2tlab_orient>
        <x2tlab_perp>1</x2tlab_perp>
        <x2tlab_horiz>0.5</x2tlab_horiz>
        <x2tlab_size>0.8</x2tlab_size>
        <x2lab_size>0.8</x2lab_size>
        <x2lab_offset>-0.5</x2lab_offset>
        <x2lab_align>0.5</x2lab_align>
        <y2tlab_orient>1</y2tlab_orient>
        <y2tlab_perp>0.5</y2tlab_perp>
        <y2tlab_horiz>0.5</y2tlab_horiz>
        <y2tlab_size>1</y2tlab_size>
        <y2lab_size>1</y2lab_size>
        <y2lab_offset>1</y2lab_offset>
        <y2lab_align>0.5</y2lab_align>
        <legend_box>o</legend_box>
        <legend_inset>c(0, -.25)</legend_inset>
        <legend_ncol>3</legend_ncol>
        <legend_size>0.8</legend_size>
        <caption_weight>1</caption_weight>
        <caption_col>#333333</caption_col>
        <caption_size>0.8</caption_size>
        <caption_offset>3</caption_offset>
        <caption_align>0</caption_align>
        <ci_alpha>0.05</ci_alpha>
        <plot_ci>c("none","none","none","none","none","none","none","none")</plot_ci>
        <show_signif>c(FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE,FALSE)</show_signif>
        <plot_disp>c(TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,TRUE)</plot_disp>
        <colors>c("#ff0000FF","#ffb600FF","#92ff00FF","#00ff24FF","#00ffdbFF","#006dffFF","#2400ffFF","#7f00ffFF")</colors>
        <pch>c(20,20,20,20,20,20,20,20)</pch>
        <type>c("b","b","b","b","b","b","b","b")</type>
        <lty>c(1,1,1,1,1,1,1,1)</lty>
        <lwd>c(1,1,1,1,1,1,1,1)</lwd>
        <con_series>c(1,1,1,1,1,1,1,1)</con_series>
        <order_series>c(1,2,3,4,5,6,7,8)</order_series>
        <plot_cmd/>
        <legend>c("","","","","","","","")</legend>
        <y1_lim>c()</y1_lim>
        <x1_lim>c()</x1_lim>
        <y1_bufr>0.04</y1_bufr>
        <y2_lim>c()</y2_lim>
    </plot>
</plot_spec>