Glossary of Nowcasting Competition Terms
DISCLAIMER: The aim of this glossary is to provide persons considering whether to compete for the European Statistics Awards for Nowcasting with an overview of some of its essential elements to allow them to assess whether it would be in their interest to form a team to submit entries. It does not constitute authoritative information on the rules, terms and conditions for the competitions of the European Statistics Awards Programme. Those rules, terms and conditions will be made available to prospective teams once a competition opens for registration to allow them to take an informed decision.
Accuracy Awards
All entries are assigned an accuracy score. The team with the entry having the best accuracy score will win the 1st accuracy award (The 2nd and 3rd accuracy awards will be granted similarly).
Accuracy score
For each entry, the accuracy score is calculated as the sum of the five best country scores of that entry. In case there are fewer than five valid country scores for the entry, it is not eligible to compete for any awards.
Code
In order to compete for the Reproducibility Award, teams will be required to submit their programming code which was executed to predict the point estimates for a particular entry.
Completeness
of documentation
This is an eligibility criterion for the Reproducibility Award. Incomplete entries will not be evaluated with respect to the Reproducibility Award criteria.
The evaluation panel will check the documentation of each entry and assess the completeness of its individual key elements, including:
- procedure description
- input data
- code (including inline comments)
and other documentation.
Country score
The country score of an entry is based on
- the volatility index for the country multiplied by
- the MSRE for that country across all monthly submissions of that entry.
An entry will receive a better country score if it has a small MSRE.
The volatility index (which has a lower value the more volatile the indicator in a particular country) allows teams to select more volatile countries (which will naturally result in higher MSRE values) and remain competitive for the Accuracy Award.
There must be a mean square relative error (MSRE) of less than 15% across all monthly submissions of an entry in order for a valid country score to be generated for that country for the entry in question. In some specific situations like the overall poor performance of models for a particular date, the evaluation panel may decide to increase this cut-off threshold.
Only countries for which there are at least 6 consecutive submissions are taken into account. In case there are more than 6 monthly submissions made for a country, the six consecutive submissions that generate the best country score will be used for the calculation.
Documentation
For entries competing exclusively for the Accuracy Awards, it suffices to submit a short description of the approach used to arrive at the submitted point estimates. For an entry to also be in the running for the Reproducibility Award, complete documentation of the entry is required at the end of the competition.
Entry
An entry is made of a submission of point estimates (i.e. nowcasted values) for at least 5 countries using a given approach.
Each team may submit up to 5 different entries, and thus submit 5 different nowcasts. Teams will have to take care to keep their entries separate, since each entry is evaluated in isolation.
Integrity
of the approach
This is one of the five award criteria evaluated for the Reproducibility Award.
For an entry to be eligible for evaluation, at least 6 separate monthly submissions must be made for that entry for 6 consecutive months. Furthermore, to score high on the integrity criterion, the same approach should be consistently used for all submissions of an entry.
An adaptative approach based on the adjustment of (super)parameters taking contextual information into account is permissible. The compliance with the integrity criterion (and the extent to which model parameters change can be linked to contextual information) will be assessed by the panel when evaluating entries for this criterion.
Teams that are interested in competing for the Reproducibility Award should therefore foresee different entries when implementing different approaches.
Interpretability
of the approach
This is one of the five award criteria evaluated for the Reproducibility Award.
For each entry, the evaluation panel will assess the approach’s interpretability, i.e. the extent to which a human could understand and articulate the relationship between the approach’s predictors and its outcome.
Leaderboard
The monthly leaderboard will keep track of the performance of the different entries of the teams for each month over the course of the competition. For each indicator, the monthly leaderboard will be updated once the official release is available for all countries (thus with a considerable lag in relation to the reference period).
Mean Square Relative Error (MSRE)
Mean square relative error (MSRE): average of the square of the relative differences between the point estimate (i.e. the nowcasted value) (Yi) and the first official release (Ri). Better nowcasting approaches are those where the MSRE is closer to zero.
C is the country in the provided entry and n is the number of different estimation periods.
In case there are more than 6 monthly submissions made for a country, the six consecutive submissions that generate the best country score will be used for the calculation.
In case of revision of the official release, only the first official release for a given period is used.
Nowcast
A nowcast is a point estimate of a time series indicator, provided before the end of the reference period that the value refers to.
Openness
of the input data
This is one of the five award criteria evaluated for the Reproducibility Award.
The evaluation panel will assess the possible external data used for each entry with respect to their openness, availability, coverage (for instance geographical and time coverage) and consistency.
The general task of the evaluation panel when providing a score for this criterion is to assess whether there would be any major obstacles to scaling up an entry (typically made for a few countries and a certain time period) to all (or most) countries of the European Statistical System.
Originality
of the approach
This is one of the five award criteria evaluated for the Reproducibility Award.
For each entry, the evaluation panel will compare the approach used to the state-of-the-art, i.e. those pre-existing approaches that are closest to the approach applied for the entry, and the extent to which the entry represents an improvement over these pre-existing approaches.
The evaluation panel will assign higher marks to entries that are bringing novelty with respect to existing approaches, and lower marks to entries that rely on "baseline state-of-the-art" approaches.
Panel
of evaluators
For each European Statistics Awards competition, a panel of independent expert evaluators has been appointed. The evaluation panel is responsible for rating each entry with regard to its accuracy score and reproducibility score.
On the basis of the scores, the European Commission (Eurostat) decides on the awarding of the Reproducibility Awards and Accuracy Awards.
The evaluation panel is at liberty to set the following additional criteria during the competition:
- In the case that official indicators are not published for a particular country in due time, the evaluation panel is at liberty to set a cut-off date for the beginning of evaluation. If official indicators have not been published by the cut-off date, all entries containing a point estimate for that country will be treated as invalid for that month. Teams are therefore encouraged to submit point estimates for more than 5 countries as well as make more than 6 consecutive submissions, thereby reducing the risk that their submissions become invalid due to a lack of publication of an official indicator.
- Based on all the entries received by the final deadline, the evaluation panel may choose to increase the 15% mean square relative error (MSRE) cut-off threshold required for a country score to be valid.
Reproducibility Awards
Eurostat is assigning great importance to approaches having a potential to be scaled up for use in European statistics production. Therefore, entries for which teams submit additional documentation will be in the running for the Reproducibility Award.
The entry with the highest reproducibility score will receive the Reproducibility Award.
However, for each indicator, only those entries are considered that:
- are in the best quartile with regard to their accuracy score (since there is expected to be little interest in reproducing underperforming models);
- have a reproducibility score above the minimum cut-off – both overall and for each individual reproducibility criterion. The minimum cut-off is provided in the Evaluation section of each competition.
If there are no such entries (best quartile) for a given indicator, then no reproducibility prize is awarded.
Reproducibility Score
The reproducibility score of an entry will be evaluated by a panel of expert evaluators, who will rate the entry according to its integrity, openness, originality, interpretability, and simplicity.
The key eligibility criterion for evaluation is that the team provides complete documentation, as stated in the Evaluation section of each competition.
Simplicity
of the approach
This is one of the five award criteria evaluated for the Reproducibility Award.
For each entry, the evaluation panel will evaluate the approach used with respect to its predictive validity and assess the distinction between simple, comprehensible approaches and more complicated, non-linear approaches.
A higher score will be assigned to approaches which use parameters that are explainable and can be linked to assumptions made when developing the approach.
Submission
of nowcasts
For each entry, at least six (6) monthly consecutive submissions of nowcasts over eight (8) months are required. Each submission consists of a single point estimate for at least 5 countries for the indicator concerned – to be submitted via the competition platform ahead of the deadline set in the rules.
Team
Participation takes place in the form of a team of 1, 2, 3, 4 or 5 individuals. There are no restrictions on how participants organise themselves: private persons, persons working for private companies and persons working for public and/or research institutions are equally welcome to participate. Persons who work for organisations that are part of the European Statistical System (including Other National Authorities) can participate with certain limitations, which can be found in the Terms and Conditions section of each competition.
Volatility Index
A volatility index for a given country and a given indicator is a normalised measure of the volatility score. It is used to take into account the difficulty of nowcasting an indicator associated with the corresponding countries.
For each indicator, the volatility index for the countries is obtained by means of sorting the volatility scores, and normalising them by using a scaling function that maps the scores into a (semi-)linear order. The function used to scale is:
where vC is the country’s volatility score, Vmin and Vmax are the minimum and maximum country volatility scores, respectively, f is the scaling function, and Xmin and Xmax are the maximum and minimum values in which we map the country volatility scores. For all competitions, we use Xmin = 0.5 and Xmax = 2, and the natural logarithm is used as the scaling function f.
Note that the scaling function returns smaller weights for countries with higher volatility scores; this is to ensure (1) some level of normalisation of countries with high and low volatility rates, and (2) that teams take countries with higher volatility scores into consideration when preparing their entries.
For each country and each indicator, the volatility index will be fixed for the entire duration of the competition round and provided in the Evaluation section of each competition.
Volatility Score
The volatility score for a given country and a given indicator is a measure of the variability of the indicator’s associated time series calculated using historical data.
For each indicator, a country’s volatility score vC is computed based on the historical variability of its time series. Official available data from the Eurostat data portal were downloaded and used to calculate the volatility scores:
- For tourism, the number of nights spent at tourist accommodation establishments were used (eurobase code: TOUR_OCC_NIM). The dataset is available here.
- For production volume in industry the PVI dataset was used (eurobase code: STS_INPR_M). The dataset is available here.
- For producer prices in industry (domestic market), the PPI dataset was used (eurobase code: STS_INPPD_M). The dataset is available here.
To compute the country’s volatility score for an indicator, we base the scores on the GARCH(1,1) model applied to the available data from the period 2010-2019. The model is able to take the seasonality of the time series to adjust the calculated score. In case the volatility score cannot be calculated with a minimum assurance e.g. too short time series available, the country series is considered not in scope for the competition.
