18. Economic Cost/Loss Value Plots

18.1. Description

The Economic Cost Loss Value statistic is sometimes also called the Relative value score (Richardson, 2000; Wilks, 2001). This plot produces the relative value curve for deterministic forecasts based on counts in a 2x2 contingency table along with the expected cost-to-loss ratio.

This information can help the user decide, for a cost/loss ratio C/L for taking action based on a forecast, what the relative improvement is in economic value between climatological and perfect information. The relative value is a skill score based on expected cost, with (sample) climatology as the reference forecast. Because the cost/loss ratio is different for different users of forecasts, the value is plotted as a function of cost to loss.

The ECLV score can range from -\(\infty\) to 1.

Like ROC diagrams, it gives information that can be used in decision making.

18.2. Line Type

ECLV requires the ECLV line type generated by either Point-Stat or Grid-Stat.

18.3. How-To

Selection of options to produce the ECLV plot proceeds approximately counter-clockwise around the METviewer window.

  1. Select the desired database from the “Select databases” pulldown menu at the top margin of the METviewer window.

  2. There are a number of tabs just under the database pulldown menu. Select the ‘Eclv’ tab.

  3. Select the desired series variable to calculate statistics for in the “Series Variables” tab. Press the “+ Series Variable” button to reveal two pulldown menus. The first pulldown menu lists the categories available in the selected dataset. The second pulldown menu allows the selection of the value of that category.

  4. For a ECLV plot, the forecast variable (“FCST_VAR”) must be selected. This is found in the “Specialized Plot Fixed Values” section. In the example below, the forecast variable is 6-hour accumulated precipitation “APCP_06”.

  5. It usually does not make sense to mix statistics for different groups. The desired group to calculate statistics over can be specified in the “Specialized Plot Fixed Values” section. A single domain (category: “VX_MASK”, value: “FULL”), a single accumulation time (category: “FCST_LEV”, value: “A6”), and an observation type (category: “OBTYPE”, value: “METAR_SYNOP”) are chosen. If multiple domains or thresholds were chosen, the statistics would be a summary of all of those cases together, which may not always be desired.

  6. Select the type of statistics summary by selecting either the “Summary” or “Aggregation Statistics” button in the “Statistics” section. Aggregated statistics may be selected for certain varieties of statistics. The selection can be made from the leftmost dropdown menu in the “Statistics” section. By default, the median value of all statistics will be plotted. Using the dropdown menu, the mean or sum may be selected instead. Choosing this option will cause a single statistic to be calculated from the individual database lines.

  7. Now enough information has been entered to produce a graph. To do this, click the “Generate Plot” button at the top of the METviewer window (this is in red text). Typically, if a plot is not produced, it is because the database selected does not contain the correct type of data. Also, it is imperative to check the data used for the plot by selecting the “R data” tab on the right hand side, above the plot area. The data from the database that is being used to calculate the statistics is listed in this tab. This tab should be checked to avoid the accidental accumulation of inappropriate database lines. For example, it does not make sense to accumulate statistics over different domains, thresholds, models, etc.

18.4. Example

The figure below shows an ECLV plot. In this example, three different forecasting systems are used to predict precipitation at two different thresholds. The economic value peaks at about 0 for all forecasts. Values of the ECLV are negative for the majority of the cost to loss ratio. Low values of C/L indicate either a very small cost to protect, very high losses, or both. In these cases, it probably makes sense to protect regardless of the forecast. At the other end, the cost to protect nears the amount of the potential loss. In that case, it probably makes sense to do nothing, regardless of the forecast, so the economic value of the forecast is negative. Between those extremes, each user can determine their own C/L ratio, risk tolerance, etc. to determine the best forecasting system for their needs. In the example below, many of the forecasts are quite similar, so a user may select from a grouping based on other criteria, such as forecast latency or computational requirements.

../_images/eclv_plot.png

Figure 18.1 Example ECLV plot for three models using two different thresholds.

Here is the associated xml for this example. It can be copied into an empty file and saved to the desktop then uploaded into the system by clicking on the “Load XML” button in the upper-right corner of the GUI. This XML can be downloaded from this link: eclv_xml.xml.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plot_spec>
    <connection>
        <host>mohawk</host>
        <database>mv_skymet</database>
        <user>******</user>
        <password>******</password>
        <management_system>mariadb</management_system>
    </connection>
    <rscript>/usr/local/R/bin/Rscript</rscript>
    <folders>
        <r_tmpl>/opt/vxwww/tomcat/webapps/metviewer//R_tmpl</r_tmpl>
        <r_work>/opt/vxwww/tomcat/webapps/metviewer//R_work</r_work>
        <plots>/d2/www/dtcenter/met/metviewer_output//plots</plots>
        <data>/d2/www/dtcenter/met/metviewer_output//data</data>
        <scripts>/d2/www/dtcenter/met/metviewer_output//scripts</scripts>
    </folders>
    <plot>
        <template>eclv.R_tmpl</template>
        <series1>
            <field name="model">
                <val>exp01</val>
                <val>exp02</val>
                <val>exp03</val>
            </field>
            <field name="fcst_thresh">
                <val>&gt;0.1</val>
                <val>&gt;2.5</val>
            </field>
        </series1>
        <plot_fix>
            <field equalize="false" name="fcst_var">
                <set name="fcst_var_0">
                    <val>APCP_06</val>
                </set>
            </field>
            <field equalize="false" name="vx_mask">
                <set name="vx_mask_1">
                    <val>FULL</val>
                </set>
            </field>
            <field equalize="false" name="fcst_lev">
                <set name="fcst_lev_2">
                    <val>A6</val>
                </set>
            </field>
            <field equalize="false" name="obtype">
                <set name="obtype_3">
                    <val>METAR_SYNOP</val>
                </set>
            </field>
        </plot_fix>
        <plot_stat>median</plot_stat>
        <tmpl>
            <data_file>plot_20201001_190659.data</data_file>
            <plot_file>plot_20201001_190659.png</plot_file>
            <r_file>plot_20201001_190659.R</r_file>
            <title>Economic Value for 6-hr APCP</title>
            <x_label>Cost/Loss Ratio</x_label>
            <y1_label>Economic Value</y1_label>
            <y2_label/>
            <caption/>
            <job_title/>
            <keep_revisions>false</keep_revisions>
            <listdiffseries1>list()</listdiffseries1>
            <listdiffseries2>list()</listdiffseries2>
        </tmpl>
        <execution_type>Rscript</execution_type>
        <event_equal>false</event_equal>
        <vert_plot>false</vert_plot>
        <x_reverse>false</x_reverse>
        <num_stats>false</num_stats>
        <indy1_stag>false</indy1_stag>
        <indy2_stag>false</indy2_stag>
        <grid_on>true</grid_on>
        <sync_axes>false</sync_axes>
        <dump_points1>false</dump_points1>
        <dump_points2>false</dump_points2>
        <log_y1>false</log_y1>
        <log_y2>false</log_y2>
        <varianceinflationfactor>false</varianceinflationfactor>
        <plot_type>png16m</plot_type>
        <plot_height>8.5</plot_height>
        <plot_width>11</plot_width>
        <plot_res>72</plot_res>
        <plot_units>in</plot_units>
        <mar>c(8,4,5,4)</mar>
        <mgp>c(1,1,0)</mgp>
        <cex>1</cex>
        <title_weight>2</title_weight>
        <title_size>1.4</title_size>
        <title_offset>-2</title_offset>
        <title_align>0.5</title_align>
        <xtlab_orient>1</xtlab_orient>
        <xtlab_perp>-0.75</xtlab_perp>
        <xtlab_horiz>0.5</xtlab_horiz>
        <xtlab_freq>0</xtlab_freq>
        <xtlab_size>1</xtlab_size>
        <xlab_weight>1</xlab_weight>
        <xlab_size>1</xlab_size>
        <xlab_offset>2</xlab_offset>
        <xlab_align>0.5</xlab_align>
        <ytlab_orient>1</ytlab_orient>
        <ytlab_perp>0.5</ytlab_perp>
        <ytlab_horiz>0.5</ytlab_horiz>
        <ytlab_size>1</ytlab_size>
        <ylab_weight>1</ylab_weight>
        <ylab_size>1</ylab_size>
        <ylab_offset>-2</ylab_offset>
        <ylab_align>0.5</ylab_align>
        <grid_lty>3</grid_lty>
        <grid_col>#cccccc</grid_col>
        <grid_lwd>1</grid_lwd>
        <grid_x>listX</grid_x>
        <x2tlab_orient>1</x2tlab_orient>
        <x2tlab_perp>1</x2tlab_perp>
        <x2tlab_horiz>0.5</x2tlab_horiz>
        <x2tlab_size>0.8</x2tlab_size>
        <x2lab_size>0.8</x2lab_size>
        <x2lab_offset>-0.5</x2lab_offset>
        <x2lab_align>0.5</x2lab_align>
        <y2tlab_orient>1</y2tlab_orient>
        <y2tlab_perp>0.5</y2tlab_perp>
        <y2tlab_horiz>0.5</y2tlab_horiz>
        <y2tlab_size>1</y2tlab_size>
        <y2lab_size>1</y2lab_size>
        <y2lab_offset>1</y2lab_offset>
        <y2lab_align>0.5</y2lab_align>
        <legend_box>o</legend_box>
        <legend_inset>c(0, -.25)</legend_inset>
        <legend_ncol>3</legend_ncol>
        <legend_size>0.8</legend_size>
        <caption_weight>1</caption_weight>
        <caption_col>#333333</caption_col>
        <caption_size>0.8</caption_size>
        <caption_offset>3</caption_offset>
        <caption_align>0</caption_align>
        <ci_alpha>0.05</ci_alpha>
        <plot_ci>c("none","none","none","none","none","none")</plot_ci>
        <show_signif>c(FALSE,FALSE,FALSE,FALSE,FALSE,FALSE)</show_signif>
        <plot_disp>c(TRUE,TRUE,TRUE,TRUE,TRUE,TRUE)</plot_disp>
        <colors>c("#ff0000FF","#ff0000FF","#0000ffFF","#0000ffFF","#008000FF","#008000FF")</colors>
        <pch>c(20,20,20,20,20,20)</pch>
        <type>c("b","b","b","b","b","b")</type>
        <lty>c(1,2,1,2,1,2)</lty>
        <lwd>c(1,1,1,1,1,1)</lwd>
        <con_series>c(1,1,1,1,1,1)</con_series>
        <order_series>c(1,2,3,4,5,6)</order_series>
        <plot_cmd/>
        <legend>c("","","","","","")</legend>
        <y1_lim>c()</y1_lim>
        <x1_lim>c()</x1_lim>
        <y1_bufr>0.04</y1_bufr>
        <y2_lim>c()</y2_lim>
    </plot>
</plot_spec>