1) Objectives:

• Recommend verification metrics and datasets for assessing forecast quality of S2S forecasts
• Provide guidance for a potential centralized verification effort for comparing forecast quality of different S2S forecast systems, including the comparison of multi-model and individual ensemble systems and consider linkages with users and applications

The S2S verification science plan provides more detailed information about this sub-project.

2) Membership:

Caio Coelho (CPTEC/INPE, Brazil)

Andrew Robertson (IRI, USA)

Richard Graham (UKMO, UK)

Yuhei Takaya (JMA, Japan)

Debra Hudson (CAWCR, Australia)

Joanne Robbins (UKMO, UK)

3) List of published literature on verification methods of relevance to S2S verification

Below in sections 3.1 and 3.2 is a selected list of published literature (including books, technical reports and scientific papers) on verification methodologies of relevance for S2S forecast verification. A more comprehensive list and additional information on forecast verification is available at http://www.cawcr.gov.au/projects/verification/ a website of the WWRP/WGNE Joint Working Group on Forecast Verification Research.

Please note that further down on this wiki page Section 4 provides a list of published literature on S2S verification including in section 4.1 papers on the assessment of S2S systems forecast skill, in section 4.2 papers on the assessment of MJO/ISO forecast skill, in section 4.3 papers on the assessment of monsoon systems forecast skill and associated characteristics, in section 4.4 papers on applications and in section 4.5 on seamless verification.

3.1) Books and technical reports

Jolliffe IT, Stephenson DB (2012) Forecast Verification: A Practitioner's Guide in Atmospheric Science. 2nd Edition.  Wiley and Sons Ltd, 274 pp.

Stanski HR, Wilson LJ, Burrows WR (1989) Survey of common verification methods in meteorology. World Weather Watch Tech. Rept. No.8, WMO/TD No.358, WMO, Geneva, 114 pp. Available here.

Wilks DS (2011) Statistical Methods in the Atmospheric Sciences. 3rd Edition.  Elsevier, 676 pp.

3.2) Scientific papers

Bradley AA, Hashino T, Schwartz SS (2003) Distributions-oriented verification of probability forecasts for small data samples. Wea. Forecasting, 18, 903-917. http://dx.doi.org/10.1175/1520-0434(2003)018%3C0903:DVOPFF%3E2.0.CO;2

Bradley AA, Schwartz SS, Hashino T (2008) Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Wea. Forecasting, 23, 992-1006. doi: http://dx.doi.org/10.1175/2007WAF2007049.1

Brier GW (1950) Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1-3. doi: http://dx.doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

Epstein ES (1969) A scoring system for probability forecasts of tanked categories. J. App. Met. Vol 8. No 6, 985-987. doi: http://dx.doi.org/10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2

Ferro CAT (2007) Comparing probabilistic forecasting systems with the Brier score. Weather and Forecasting 22, 1076-1088. doi: http://dx.doi.org/10.1175/WAF1034.1

Ferro CAT, Richardson DS, Weigel AP (2008) On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorol. Appl., 15, 19-24. http://onlinelibrary.wiley.com/doi/10.1002/met.45/epdf

Ferro CAT, Stephenson DB (2011) Extremal Dependence Indices: improved verifiation measures for deterministic forecasts of rare binary events. Wea. Forecasting, 26, 699-713. doi: http://dx.doi.org/10.1175/WAF-D-10-05030.1

Ferro CAT, Fricker TE (2012) A bias-corrected decomposition of the Brier score. Quarterly Journal of the Royal Meteorological Society, 138, 1954-1960, doi:10.1002/qj.1924. pdf

Ferro CAT (2014) Fair scores for ensemble forecasts. Quarterly Journal of the Royal Meteorological Society, 140, 1917-1923, http://dx.doi.org/10.1002/qj.2270

Gneiting T,  Raftery AE (2007) Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102, Issue 477,  359-378. doi: 10.1198/016214506000001437 http://www.eecs.harvard.edu/cs286r/courses/fall12/papers/Gneiting07.pdf

Hamill TM (2001) Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550-560. doi: http://dx.doi.org/10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2

Hersbach H (2000) Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting, 15, 559-570. doi: http://dx.doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2

Hsu W-R., Murphy AH (1986) The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecasting, 2, 285-293. http://dx.doi.org/10.1016/0169-2070(86)90048-8

Jolliffe IT (2007) Uncertainty and inference for verification measures. Wea. Forecasting, 22, 637-650. doi: http://dx.doi.org/10.1175/WAF989.1

Jupp TE, Lowe R, Coelho CAS, Stephenson DB (2012): On the visualization, verification and recalibration of ternary probabilistic forecasts. Phil. Trans. R. Soc. A, 370, 1100-1120. doi:10.1098/rsta.2011.0350

Mason I (1982) A model for assessment of weather forecasts. Aust. Met. Mag., 30, 291-303. http://www.nssl.noaa.gov/users/brooks/public_html/feda/papers/mason82.pdf

Mason SJ, Weigel AP (2009) A generic forecast verification framework for administrative purposes. Mon. Wea. Rev., 137, 331-349. doi: http://dx.doi.org/10.1175/2008MWR2553.1

Mason SJ, Graham NE (2002) Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society Volume 128, Issue 584, pages 2145–2166, July 2002 Part B. DOI: 10.1256/003590002320603584 http://onlinelibrary.wiley.com/doi/10.1256/003590002320603584/pdf

Mason SJ (2008) Understanding forecast verification statistics. Meteorol. Appl., 15., 31-34. DOI: 10.1002/met.51 http://onlinelibrary.wiley.com/doi/10.1002/met.51/abstract

Murphy AH (1970) The ranked probability score and the probability score: A comparison. UDC 551.509.314. Vol. 98, No. 12. 917-924 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.395.1780&rep=rep1&type=pdf

Murphy AH (1973) A new vector partition of the probability score. J. Appl. Meteor., 12, 595-600. doi: http://dx.doi.org/10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2

Murphy AH (1988) Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev., 116, 2417-2424. doi: http://dx.doi.org/10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2

Murphy AH (1993) What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281-293. doi: http://dx.doi.org/10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2

Murphy AH (1995) The coefficients of correlation and determination as measures of performance in forecast verification. Wea. Forecasting, 10, 681-688. doi: http://dx.doi.org/10.1175/1520-0434(1995)010<0681:TCOCAD>2.0.CO;2

Murphy AH (1996) General decompositions of MSE-based skill scores: Measures of some basic aspects of forecast quality. Mon. Wea. Rev., 124, 2353-2369. doi: http://dx.doi.org/10.1175/1520-0493(1996)124<2353:GDOMBS>2.0.CO;2

Murphy AH, Epstein ES (1989) Skill scores and correlation coefficients in model verification. Mon. Wea. Rev., 117, 572-581. doi: http://dx.doi.org/10.1175/1520-0493(1989)117<0572:SSACCI>2.0.CO;2

Richardson DS (2000) Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Royal Met. Soc., 126, 649-667. DOI: 10.1002/qj.49712656313 http://onlinelibrary.wiley.com/doi/10.1002/qj.49712656313/abstract

Roulston MS, Smith LA (2002) Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 1653-1660. DOI: http://dx.doi.org/10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2

Stephenson DB, Casati C, Ferro CAT, Wilson CA (2008) The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meteorol. Appl., 15, 41-50. DOI: 10.1002/met.53 http://onlinelibrary.wiley.com/doi/10.1002/met.53/abstract

Stephenson DB, Coelho CAS, Jolliffe IT (2008) Two extra components in the Brier Score Decomposition, Wea. Forecasting, 23, pp 752-757. doi: http://dx.doi.org/10.1175/2007WAF2006116.1

Taylor KE (2001) Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106 (D7), 7183-7192. DOI: 10.1029/2000JD900719 http://onlinelibrary.wiley.com/doi/10.1029/2000JD900719/abstract

Weigel AP, Liniger MA, Appenzeller C (2007) The discrete Brier and ranked probability skill scores. Mon. Wea. Rev., 135, 118–124. doi: http://dx.doi.org/10.1175/MWR3280.1

Weigel AP, Liniger MA, Appenzeller C (2007) Generalization of the Discrete Brier and Ranked Probability Skill Scores for Weighted Multimodel Ensemble Forecasts. Mon. Wea. Rev., 135, 2778-2785. doi: http://dx.doi.org/10.1175/MWR3428.1

Weigel AP, Mason SJ (2011) The generalized discrimination Score for ensemble forecasts. Mon. Wea. Rev.,, 139, 3069-3074. doi: http://dx.doi.org/10.1175/MWR-D-10-05069.1

Wilson LJ, Burrows WR, Lanzinger A (1999) A strategy for verification of weather element forecasts from an ensemble prediction system. Mon. Wea. Rev., 127, 956-970. doi: http://dx.doi.org/10.1175/1520-0493(1999)127<0956:ASFVOW>2.0.CO;2

4) List of published literature on S2S verification

4.1) Assessment of S2S systems forecast skill

Hudson D, Alves O, Hendon HH, Marshall AG (2011) Bridging the gap between weather and seasonal forecasting: Intraseasonal forecasting for Australia. Quart. J. Roy. Meteor. Soc., 137,673–689, doi:10.1002/qj.769 http://onlinelibrary.wiley.com/doi/10.1002/qj.769/abstract

Hudson D, Marshall AG, Yin Y, Alves O, Hendon HH (2013) Improving intraseasonal prediction with a new ensemble generation strategy. Mon Wea Rev, 141, 4429-4449. doi: http://dx.doi.org/10.1175/MWR-D-13-00059.1

Hudson D, Marshall AG, Alves O, Shi L, Young G (2015) Forecasting upcoming extreme heat on multi-week to seasonal timescales: POAMA experimental forecast products. Bureau Research Report, No. 1. Bureau of Meteorology, Australia (http://www.bom.gov.au/research/research-reports.shtml).

Hudson D, Marshall AG, Alves O, Young G, Jones D, Watkins A (2015) Forewarned is forearmed: Extended range forecast guidance of recent extreme heat events in Australia. Weather and Forecasting. doi: http://dx.doi.org/10.1175/WAF-D-15-0079.1

Jung T, Miller MJ, Palmer TN (2010) Diagnosing the Origin of Extended-Range Forecast Errors. Mon. Wea. Rev., 138, 2434–2446. doi: http://dx.doi.org/10.1175/2010MWR3255.1

Koster RD, Mahanama SPP, Yamada TJ, Balsamo G, Berg AA, Boisserie M, Dirmeyer PA, Doblas-Reyes FJ, Drewitt G, Gordon CT, Guo Z, Jeong J.-H, Lawrence DM, Lee W.-S, Li Z, Luo L, Malyshev S, Merryfield WJ, Seneviratne SI, Stanelle T, van den Hurk BJJM, Vitart F, Wood EF (2010) Contribution of land surface initialization to subseasonal forecast skill: First results from a multi-model experiment. Geophysical Research Letters, Vol. 37, L02402, doi:10.1029/2009GL041677, 2010 http://onlinelibrary.wiley.com/doi/10.1029/2009GL041677/full

Kumar A, Chen M, Wang W (2011) An analysis of prediction skill of monthly mean climate variability. Clim. Dyn., 37, 1119-1131. http://link.springer.com/article/10.1007%2Fs00382-010-0901-4

Li S, Robertson AW (2015) Evaluation of Submonthly Precipitation Forecast Skill from Global Ensemble Prediction Systems. Monthly Weather Review 143:7, 2871-2889. doi:http://dx.doi.org/10.1175/MWR-D-14-00277.1

Marshall AG, Hudson D, Wheeler MC, Hendon HH, Alves O. (2012) Simulation and prediction of the Southern Annular Mode and its influence on Australian intra-seasonal climate in POAMA. Climate Dynamics. 38:2483-2502, doi:10.1007/s00382-011-1140-z. http://link.springer.com/article/10.1007%2Fs00382-011-1140-z

Marshall AG, Hudson D, Wheeler MC, Alves O, Hendon HH, Pook MJ, Risbey JS (2013) Intra-seasonal drivers of extreme heat over Australia in observations and POAMA-2. Climate Dynamics, doi: 10.1007/s00382-013-2016-1 http://link.springer.com/article/10.1007%2Fs00382-013-2016-1

Marshall AG, Hudson D, Hendon HH, Pook MJ, Alves O, Wheeler MC (2014) Simulation and prediction of blocking in the Australian region and its influence on intra-seasonal rainfall in POAMA-2. Clim. Dyn., 42, 3271-3288, DOI: 10.1007/s00382-013-1974-7 http://link.springer.com/article/10.1007%2Fs00382-013-1974-7

Mastrangelo D, Malguzzi P, Rendina C, Drofa O, Buzzi A (2012) First Outcomes from the CNR-ISAC monthly forecasting system Adv. Sci. Res., 8, 77–82, 2012 www.adv-sci-res.net/8/77/2012/ doi:10.5194/asr-8-77-2012

Vitart F (2014): Evolution of ECMWF sub-seasonal forecast skill scores. Quarterly Journal of the Royal Meteorological Society. Volume 140, Issue 683, pages 1889–1899, July 2014 Part B. DOI: 10.1002/qj.2256 http://onlinelibrary.wiley.com/doi/10.1002/qj.2256/abstract

Vitart F (2013) Evolution of ECMWF sub-seasonal forecast skill scores over the past 10 years . ECMWF Technical report http://www.ecmwf.int/sites/default/files/elibrary/2013/12932-evolution-ecmwf-sub-seasonal-forecast-skill-scores-over-past-10-years.pdf

Weigel A, Baggenstos D, Liniger MA, Vitart F, Appenzeller C. (2008) Probabilistic verification of monthly temperature forecasts.  Mon. Weather Review 136: 5162–5182. doi: http://dx.doi.org/10.1175/2008MWR2551.1

White CJ, Hudson D, Alves O (2013) ENSO, the IOD and the intraseasonal prediction of heat extremes across Australia using POAMA-2. Climate Dynamics. doi:10.1007/s00382-013-2007-2 http://link.springer.com/article/10.1007%2Fs00382-013-2007-2

4.2) Assessment of MJO/ISO forecast skill

Gottschalck J, Wheeler M, Weickmann K, Vitart F, Savage N, Lin H, Hendon H, Waliser D, Sperber K, Nakagawa M, Prestrelo C, Flatau M, Higgins W (2010) A Framework for Assessing Operational Madden–Julian Oscillation Forecasts: A CLIVAR MJO Working Group Project. Bull. Amer. Meteor. Soc., 91, 1247–1258. doi: http://dx.doi.org/10.1175/2010BAMS2816.1

Jiang X, Waliser DE, Wheeler  MC, Jones C, Lee M.-I, Schubert SD (2008) Assessing the skill of an all-season statistical forecast model for the Madden-Julian oscillation. Mon. Wea. Rev., 136, 1940-1956.
 doi: http://dx.doi.org/10.1175/2007MWR2305.1

Jones C, Waliser DE, Schemm JKE, Lau WKM (2000) Prediction skill of the Madden and Julian oscillation in dynamical extended-range forecasts. Clim. Dyn. 16: 273–289. http://link.springer.com/article/10.1007/s003820050327

Kim H, Webster PJ, Toma VE, Kim D (2014) Predictability and prediction skill of the MJO in two operational forecasting systems, J. Climate, 27 (14), 5364‐5378. doi: http://dx.doi.org/10.1175/JCLI-D-13-00480.1

Lee S-S, Wang B, Waliser DE, Neena JM, Lee J-Y (2015) Predictability and prediction skill of the boreal summer intraseasonal oscillation in the Intraseasonal Variability Hindcast Experiment. Climate Dynamics 45, 2123-2135. http://link.springer.com/article/10.1007%2Fs00382-014-2461-5

Liess S, Waliser DE, Schubert S. 2005. Predictability studies of the intraseasonal oscillation with the ECHAM5 GCM. J. Atmos. Sci. 62: 3320–3336. doi:  http://dx.doi.org/10.1175/JAS3542.1

Lin H, Brunet G, Derome J (2008) Forecast Skill of the Madden–Julian Oscillation in Two Canadian Atmospheric Models. Mon. Wea. Rev., 136, 4130–4149. doi: http://dx.doi.org/10.1175/2008MWR2459.1

Lin H, Brunet G, Fontecilla JS (2010) Impact of the Madden-Julian Oscillation on the intraseasonal forecast skill of the North Atlantic Oscillation. Geophys. Res. Lett.,37,L19803, doi:10.1029/2010GL044315 http://onlinelibrary.wiley.com/doi/10.1029/2010GL046131/pdf

MacLachlan, C., Arribas, A., Peterson, K. A., Maidens, A., Fereday, D., Scaife, A. A., Gordon, M., Vellinga, M., Williams, A., Comer, R. E., Camp, J., Xavier, P. and Madec, G. (2015) Global Seasonal forecast system version 5 (GloSea5): a high-resolution seasonal forecast system. Q.J.R. Meteorol. Soc., 141: 1072–1084. doi: 10.1002/qj.2396 http://onlinelibrary.wiley.com/doi/10.1002/qj.2396/abstract

Maharaj EA, Wheeler MC (2005) Forecasting an index of the Madden-Oscillation. Int. J. Climatol. 25: 1611–1618 (2005) DOI: 10.1002/joc.1206 http://onlinelibrary.wiley.com/doi/10.1002/joc.1206/full

Marshall AG, Hudson D, Wheeler MC, Hendon HH, Alves O (2010) Assessing the simulation and prediction of rainfall associated with the MJO in the POAMA seasonal forecast system. Climate Dynamics, 37, 2129-2141, doi: 10.1007/s00382-010-0948-2 http://link.springer.com/article/10.1007%2Fs00382-010-0948-2

Marshall AG, Hudson D, Wheeler MC, Alves O, Hendon HH, Pook MJ, Risbey JS (2014) Intra-seasonal drivers of extreme heat over Australia in observations and POAMA-2. Climate Dynamics, 43, 1915-1937, DOI: 10.1007/s00382-013-2016-1 http://link.springer.com/article/10.1007%2Fs00382-013-2016-1

Neena JM, Lee JY, Waliser D, Wang B, Jiang X (2014): Predictability of the Madden–Julian Oscillation in the Intraseasonal Variability Hindcast Experiment (ISVHE). J. Climate, 27, 4531–4543. doi: http://dx.doi.org/10.1175/JCLI-D-13-00624.1

Rashid HA, Hendon HH, Wheeler MC, Alves O (2011) Prediction of the Madden–Julian oscillation with the POAMA dynamical prediction system. Climate Dynamics. Volume 36, Issue 3-4, pp 649-661 http://link.springer.com/article/10.1007%2Fs00382-010-0754-x

Vitart F, Molteni F (2010) Simulation of the Madden- Julian Oscillation and its teleconnections in the ECMWF forecast system. Quarterly Journal of the Royal Meteorological Society 136:10.1002/qj.v136:649, 842-855. http://onlinelibrary.wiley.com/doi/10.1002/qj.623/pdf

4.3) Assessment of monsoon systems forecast skill and associated characteristics

Jones C, Carvalho LMV, Liebmann B (2012) Forecast Skill of the South American Monsoon System. J. Climate, 25, 1883–1889. doi: http://dx.doi.org/10.1175/JCLI-D-11-00586.1

Jones C, Carvalho LMV (2002) Active and Break Phases in the South American Monsoon System. J. Climate,15, 905–914 doi: http://dx.doi.org/10.1175/1520-0442(2002)015<0905:AABPIT>2.0.CO;2

Lo F, Wheeler MC, Meinke H, Donald A (2007) Probabilistic Forecasts of the Onset of the North Australian Wet Season. Mon. Wea. Rev., 135, 3506–3520. doi: http://dx.doi.org/10.1175/MWR3473.1

Moron V, Robertson AW, Boer R (2009) Spatial Coherence and Seasonal Predictability of Monsoon Onset over Indonesia. J. Climate, 22, 840–850. doi: http://dx.doi.org/10.1175/2008JCLI2435.1

Vellinga M, Arribas A, Graham R (2013) Seasonal forecasts for regional onset of the West African monsoon. Climate Dynamics. Volume 40, Issue 11, pp 3047-3070. http://link.springer.com/article/10.1007%2Fs00382-012-1520-z

Drosdowsky W, Wheeler MC (2014): Predicting the onset of the north Australian wet season with the POAMA dynamical prediction system. Wea. Forecasting, 29, 150-161. doi: http://dx.doi.org/10.1175/WAF-D-13-00091.1

4.4) Applications

Calanca P, Bolius D, Weigel AP, Liniger MA (2011) Application of long-range weather forecasts to agricultural decision problems in Europe. The Journal of Agricultural Science 149, 15-22. http://dx.doi.org/10.1017/S0021859610000729

Hirschi M, Spirig C, Weigel AP, Calanca P, Samietz J, Rotach MW (2012) Monthly Weather Forecasts in a Pest Forecasting Context: Downscaling, Recalibration, and Skill Improvement. J. Appl. Meteor. Climatol., 51, 1633–1638. doi: http://dx.doi.org/10.1175/JAMC-D-12-082.1

Lynch KJ, Brayshaw DJ, Charlton-Perez A (2014) Verification of European Subseasonal Wind Speed Forecasts. Mon. Wea. Rev., 142, 2978–2990.
doi: http://dx.doi.org/10.1175/MWR-D-13-00341.1

Spillman CM, Hartog JR, Hobday AJ, Hudson DA (2015) Predicting environmental drivers for prawn aquaculture production to aid improved farm management. Aquaculture. 447: 56-65. doi:10.1016/j.aquaculture.2015.02.008

4.5) Seamless verification

 Zhu H, Wheeler MC, Sobel AH, Hudson D (2014) Seamless Precipitation Prediction Skill in the Tropics and Extratropics from a Global Model. Mon. Wea. Rev., 142, 1556–1569. doi: http://dx.doi.org/10.1175/MWR-D-13-00222.1