12. Receiver Operating Characteristic (ROC) Plots
12.1. Description
MET produces a hit rate (Probability of Detection; POD) and Probability of False Detection (POFD) values for each user-specified threshold of a forecast or from each category of a probabilistic forecast. This information can be used to create a scatter plot of POFD vs. POD. When the points are connected, the plot is generally referred to as the Receiver Operating Characteristic (ROC) curve (also called the “Relative Operating Characteristic” curve). An ROC plot is shown for an example set of forecasts, with a solid line connecting the points for six user specified thresholds. The diagonal dashed line indicates no skill, while the dash-dot line shows the ROC for a perfect forecast. An ROC curve shows how well the forecast discriminates between two outcomes, so it is a measure of resolution. The ROC is invariant to linear transformations of the forecast and is thus unaffected by bias. An unbiased (i.e., well-calibrated) forecast can have the same ROC as a biased forecast, though most would agree that an unbiased forecast is “better”. Since the ROC is conditioned on the observations, it is often paired with the reliability diagram, which is conditioned on the forecasts.
12.2. Line Type
ROC requires probability statistics generated by either Point-Stat or Grid-Stat. These line types are:
PRC
PCT
CTC
12.3. How-To
Selection of options to produce the ROC plot proceeds approximately counter-clockwise around the METviewer window.
Select the desired database from the “Select databases” pulldown menu at the top margin of the METviewer window.
There are a number of tabs just under the database pulldown menu. Select the ‘Roc’ tab.
“Stat” is the only acceptable statistic type for a ROC plot as this type of diagram can only use information from the CTC and PCT line types available in the “Stat” output. For details about these types of statistics from MET, please see the most recent version of the MET User’s Guide.
Select the desired series variable to calculate statistics for in the “Series Variables” tab. Press the “+ Series Variable” button to reveal two pulldown menus. The first pulldown menu lists the categories available in the selected dataset. The second pulldown menu allows the selection of the value of that category.
It usually does not make sense to mix statistics for different groups. The desired group to calculate statistics over can be specified in the “Specialized Plot Fixed Values” section. For a ROC plot, the forecast variable (“FCST_VAR”) must be selected. In the example below, the forecast variable is “APCP_03_ENS_FREQ_gt2.540”. A single domain (category: “VX_MASK”, value: “CONUS”) is chosen. If multiple domains or thresholds were chosen, the statistics would be a summary of all of those cases together, which may not always be desired.
Select the desired method of ROC calculation in the “ROC Calculations” section. This can be done in one of two ways. In both cases, the observation threshold must be identical for all points on the plot. Otherwise, this is not an ROC plot at all.
PCT: The most common way to produce an ROC plot is using the Nx2 Probabilistic Contingency Table (PCT) counts from the PCT line in MET. MET can also write out the PSTD line type which includes the ROC_AUC column: a number between 0 and 1 indicating the area under the ROC curve, with 1 being the best. MET calculates this value and it gets loaded into METviewer. METviewer can plot the ROC_AUC value but it does not compute it.
CTC: The less common way of generating a ROC curve is using several 2x2 contingency table lines. This would be the FHO or CTC line types from MET. They need to be multiple forecast thresholds all computed for the same observation threshold. However, METviewer does not currently compute the area under the ROC curve, thus there is no way for METviewer to plot that statistic.
Now enough information has been entered to produce a graph. To do this, click the “Generate Plot” button at the top of the METviewer window (this is in red text). Typically, if a plot is not produced, it is because the database selected does not contain the correct type of data. Also, it is imperative to check the data used for the plot by selecting the “R data” tab on the right hand side, above the plot area. The data from the database that is being used to calculate the statistics is listed in this tab. This tab should be checked to avoid the accidental accumulation of inappropriate database lines. For example, it does not make sense to accumulate statistics over different domains, thresholds, models, etc.
There are many other options for plots, but these are the basics.
12.4. Example
The plot shown below is for two different probability of precipitation forecasts. The dashed diagonal line represents the no-skill line. Ideal forecasts would have a point at the top left of the graphic, thus better forecasts are closer to the top left of the plot. In this example, the red line is closer than the purple line to ideal. However, this small difference is probably not statistically significant.
Figure 12.1 Example ROC diagram showing POD vs. POFD for two different POP forecasts at thresholds of (0%, 25%, 50%, and 75%).
Here is the associated xml for this example. It can be copied into an empty file and saved to the desktop then uploaded into the system by clicking on the “Load XML” button in the upper-right corner of the GUI. This XML can be downloaded from this link: roc_xml.xml.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plot_spec>
<connection>
<host>mohawk</host>
<database>mv_hrrr_sseo_test</database>
<user>******</user>
<password>******</password>
<management_system>mariadb</management_system>
</connection>
<rscript>/usr/local/R/bin/Rscript</rscript>
<folders>
<r_tmpl>/opt/vxwww/tomcat/webapps/metviewer//R_tmpl</r_tmpl>
<r_work>/opt/vxwww/tomcat/webapps/metviewer//R_work</r_work>
<plots>/d2/www/dtcenter/met/metviewer_output//plots</plots>
<data>/d2/www/dtcenter/met/metviewer_output//data</data>
<scripts>/d2/www/dtcenter/met/metviewer_output//scripts</scripts>
</folders>
<plot>
<template>roc.R_tmpl</template>
<series1>
<field name="model">
<val>HRRR_ens_mean_ctrl_hrconus</val>
<val>HRRR_ens_mean_smois_pbl_hrconus</val>
</field>
</series1>
<plot_fix>
<field equalize="false" name="fcst_var">
<set name="fcst_var_0">
<val>APCP_03_ENS_FREQ_gt2.540</val>
</set>
</field>
<field equalize="false" name="vx_mask">
<set name="vx_mask_1">
<val>CONUS</val>
</set>
</field>
</plot_fix>
<roc_calc>
<roc_pct>true</roc_pct>
<roc_ctc>false</roc_ctc>
</roc_calc>
<summary_curve/>
<add_point_thresholds>true</add_point_thresholds>
<tmpl>
<data_file>plot_20200918_215659.data</data_file>
<plot_file>plot_20200918_215659.png</plot_file>
<r_file>plot_20200918_215659.R</r_file>
<title>CONUS: APCP 03hr >2.540 PCT </title>
<x_label>POFD (1-PODn)</x_label>
<y1_label>PODy</y1_label>
<y2_label/>
<caption/>
<job_title/>
<keep_revisions>false</keep_revisions>
<listdiffseries1>list()</listdiffseries1>
<listdiffseries2>list()</listdiffseries2>
</tmpl>
<execution_type>Rscript</execution_type>
<event_equal>false</event_equal>
<vert_plot>false</vert_plot>
<x_reverse>false</x_reverse>
<num_stats>false</num_stats>
<indy1_stag>false</indy1_stag>
<indy2_stag>false</indy2_stag>
<grid_on>true</grid_on>
<sync_axes>false</sync_axes>
<dump_points1>false</dump_points1>
<dump_points2>false</dump_points2>
<log_y1>false</log_y1>
<log_y2>false</log_y2>
<varianceinflationfactor>false</varianceinflationfactor>
<plot_type>png16m</plot_type>
<plot_height>8.5</plot_height>
<plot_width>11</plot_width>
<plot_res>72</plot_res>
<plot_units>in</plot_units>
<mar>c(8,4,5,4)</mar>
<mgp>c(1,1,0)</mgp>
<cex>1</cex>
<title_weight>2</title_weight>
<title_size>1.4</title_size>
<title_offset>-2</title_offset>
<title_align>0.5</title_align>
<xtlab_orient>1</xtlab_orient>
<xtlab_perp>-0.75</xtlab_perp>
<xtlab_horiz>0.5</xtlab_horiz>
<xtlab_freq>0</xtlab_freq>
<xtlab_size>1</xtlab_size>
<xlab_weight>1</xlab_weight>
<xlab_size>1</xlab_size>
<xlab_offset>2</xlab_offset>
<xlab_align>0.5</xlab_align>
<ytlab_orient>1</ytlab_orient>
<ytlab_perp>0.5</ytlab_perp>
<ytlab_horiz>0.5</ytlab_horiz>
<ytlab_size>1</ytlab_size>
<ylab_weight>1</ylab_weight>
<ylab_size>1</ylab_size>
<ylab_offset>-2</ylab_offset>
<ylab_align>0.5</ylab_align>
<grid_lty>3</grid_lty>
<grid_col>#cccccc</grid_col>
<grid_lwd>1</grid_lwd>
<grid_x>listX</grid_x>
<x2tlab_orient>1</x2tlab_orient>
<x2tlab_perp>1</x2tlab_perp>
<x2tlab_horiz>0.5</x2tlab_horiz>
<x2tlab_size>0.8</x2tlab_size>
<x2lab_size>0.8</x2lab_size>
<x2lab_offset>-0.5</x2lab_offset>
<x2lab_align>0.5</x2lab_align>
<y2tlab_orient>1</y2tlab_orient>
<y2tlab_perp>0.5</y2tlab_perp>
<y2tlab_horiz>0.5</y2tlab_horiz>
<y2tlab_size>1</y2tlab_size>
<y2lab_size>1</y2lab_size>
<y2lab_offset>1</y2lab_offset>
<y2lab_align>0.5</y2lab_align>
<legend_box>o</legend_box>
<legend_inset>c(0, -.25)</legend_inset>
<legend_ncol>2</legend_ncol>
<legend_size>0.8</legend_size>
<caption_weight>1</caption_weight>
<caption_col>#333333</caption_col>
<caption_size>0.8</caption_size>
<caption_offset>3</caption_offset>
<caption_align>0</caption_align>
<ci_alpha>0.05</ci_alpha>
<plot_ci>c("none","none")</plot_ci>
<show_signif>c(FALSE,FALSE)</show_signif>
<plot_disp>c(TRUE,TRUE)</plot_disp>
<colors>c("#ff0000FF","#8000ffFF")</colors>
<pch>c(20,20)</pch>
<type>c("b","b")</type>
<lty>c(1,1)</lty>
<lwd>c(1,1)</lwd>
<con_series>c(1,1)</con_series>
<order_series>c(1,2)</order_series>
<plot_cmd/>
<legend>c("","")</legend>
<y1_lim>c()</y1_lim>
<x1_lim>c()</x1_lim>
<y1_bufr>0.04</y1_bufr>
<y2_lim>c()</y2_lim>
</plot>
</plot_spec>