Space weather forecast study turns table of effective predictions on its head

Solar flares are sudden and dramatic releases of energy from the Sun's atmosphere in the form of radiation and electrically charged particles. These eruptions are associated with many aspects of 'space weather', which can damage satellites and interfere with communications, navigations and power systems. In our technology-dependent society, accurate advanced warning of solar flare occurrence is an area of increasing concern.

"The most important aspect of any type of forecast is how it performs," said Bloomfield. "If we always say, 'flare expected today', we will have successfully predicted all flares. However, we would be crying wolf and be completely wrong on most days, as flares can occur quite far apart in time. We need be accurate in both our predictions of when flares will occur and when they won't for this to be of real value to society."

Bloomfield and his colleagues compared seven different systems for predicting solar flares: two of these used images showing the structure of sunspots; the other five used a variety of parameters relating to the magnetic field, including maximum field strength, total flux and strengths of gradients. The source data came from a range of ground based and space based solar telescopes, including SOHO.

In the past, solar flare forecasts have generally been tested over periods of time containing different amounts of flare activity. Bloomfield and his colleagues found that the previous standard system for rating flare forecasts was very sensitive to the relative activity levels between testing periods. The team suggest that the True Skill Statistic (TSS) should be adopted as the new standard rating for solar flare prediction systems. TSS is calculated from the fraction of correct flare forecasts out of all flares observed, minus the fraction of false alarms out of all non-flares observed.

"The benefit of the TSS over other ratings methods is that it is not changed by the number of flares or non-flares observed. We can make a proper comparison of forecast systems, regardless whether they have made 50 or 5000 predictions. Even so, surveys with small data sets are still prone to noise and their results must be considered less reliable," said Bloomfield.

Bloomfield and his colleagues found that according to the TSS score, adaptive systems including artificial neural networks, did not actually appear to perform much better than simple predictions based on the shape of sunspots. However, the researchers were not able to include several forecasting systems in their study as not all the data needed to calculate the TSS was available.

"If we are to move forward in developing a standard ratings system for flare predictions that produces meaningful results, we need to encourage solar forecasters to be more open about publishing their results. As well as the number of flares correctly predicted, we need to know numbers of correct non-flare predictions, false alarms and missed flares. If these differences in flare statistics are not taken into account properly, some methods can appear to perform better than others when in reality they are the same or worse," said Bloomfield.