How Accurate and Reliable are USDA Production Forecasts?

How Accurate and Reliable are USDA Production Forecasts?

While a brief summary of forecast accuracy and reliability is presented at the end of each WASDE report, there is an extensive body of literature that explores the accuracy and efficiency of USDA forecasts. It is important to be aware of the USDA report shortcomings with respect to accuracy and efficiency in order to take them into account in interpreting forecasted information for decision making.  A quick comparison across tables 1, 2 and 3 shows that the accuracy of crop production forecasts (shown in table 1) have received the most scrutiny, followed by WASDE price forecasts (table 2), while the evaluation of other forecasts (table 3) received less attention.  This probably is not surprising given that crop production forecasts tend to cause strong market reactions, as discussed in the previous sections.

A number of the studies of the USDA crop production forecasts, included in table 1, focus on both the methodology and accuracy associated with these estimates.  For example, Good and Irwin (2006), Good and Irwin (2011), Good and Irwin (2013), Irwin, Good and Sanders (2015), and Irwin and Good (2016) explain the methodology behind USDA crop yield forecasts and argue that in order to build trust and support across producers and encourage them to participate in the surveys on which these forecasts are built, USDA should “open the black box” behind their forecasts and become more transparent about their methods and any changes to their approaches.  These studies also argued that that WAOB corn and soybean yield forecasts presented in May-July WASDE reports did not have a substantial bias and remained consistently accurate over time (1993-2012).  NASS corn and soybean yield forecasts (presented in August-November WASDE) were also unbiased, but there was some evidence of improvement in corn yield forecasts, while soybean yield forecasts have become more conservative over time with an increasing tendency to under-estimate final yields during 2004-2012.  Furthermore, Irwin, Good and Sanders (2014) argued that the accuracy of USDA corn yield forecasts has improved over time, particularly since 2011.  Thus, the general consensus of these studies is that USDA crop production forecasts are accurate and unbiased but USDA needs to do a better job communicating their methods and procedures to the public to maintain and improve survey response rates.

Another issue that received considerable attention in the literature is whether USDA crop production forecasts were smoothed, resulting in big crops getting bigger and small crops getting smaller.  This issue was first raised by Isengildina, Irwin and Good (2006), who showed that revisions of NASS corn and soybean production forecasts over 1970-2004 were sometimes positively correlated and directionally consistent. This pattern of predictability in production forecast revisions is consistent with the concept of “smoothing,” which may be due to a conservative bias in farm operators’ assessments of yield potential and in the procedure for translating enumerator’s information about plant fruit counts into objective yield estimates.  The authors argued that losses in forecast accuracy due to smoothing were statistically and economically significant. Xie, Isengildina-Massa and Sharp (2016) developed a statistical procedure for correction of smoothing in corn, soybean, wheat and cotton production forecasts and demonstrated potential improvements in accuracy resulting from this correction. However, in a follow up study (Isengildina, Irwin and Good, 2013) found that although the pattern of smoothing may appear obvious to market analysts in hindsight, it is difficult to anticipate.  In other words, one would need to know that we are expecting a big crop to apply the pattern of “big crop getting bigger” to crop production revisions.  Irwin, Good and Newton (2014) updated and extended this analysis to show that historically not all big crops got bigger and the challenges with anticipating the size of 2014 crop during August.  Nevertheless, Isengildina-Massa, Karali and Irwin (2017) showed that market participants appear to be aware of smoothing and adjust for it in forming their price expectations.

Several other studies explored how additional information can be used to improve yield forecasts.  Irwin and Hubbs (2020) found that several alternative crop weather models generated errors that were larger than WAOB and NASS corn yield forecasts.  Irwin and Hubbs (2018) argued that while crop condition ratings provide useful information for yield forecasting, considerable amount of uncertainty exists in this measure during the growing season. Specifically, historical model errors for mid-June are associated with about 10bu/acre errors relative to the final crop yield estimate for corn and 2.5 bu/acre errors for soybeans, indicating they should be used with caution.  Tack et al. (2019) explored whether non-random private farm level data available from precision agriculture sources can be used to generate accurate forecasts of corn yield. Their findings revealed that these non-representative samples of data result in biased yield estimates on both regional and national levels.  “To the extent that large farms are the early adopters of precision technology, our results suggest that, if not corrected for bias, data from those farms could introduce inaccuracy relative to a representative national sample.” (p. 680).