Forecast with forecasts: diversity matters

---

# Forecast with forecasts:

# Diversity matters

#### *Yanfei Kang | ISF2020 | 26 Oct 2020*

---
class: inverse, center, middle
#

> "In combining the results of these two methods, one can obtain a result whose probability law of error will be more rapidly decreasing."

> — *Pierre-Simon Laplace*

---
# Model combination

Model combination helps improve the overall performance in

- **Regression** (e.g., Mendes-Moreira, Soares, Jorge, and Sousa, 2012)

- **Classification** (e.g., Kuncheva and Whitaker, 2003)

- **Anomaly detection** (e.g., Perdisci, Gu, and Lee, 2006)

- **Forecasting** (e.g., Thomson, Pollock, Onkal, and Gonul, 2019)

---
class: inverse, center, middle

# A look back at forecast combination

---
# A look back at forecast combination

- The seminal work:  (e.g., Bates and Granger, 1969)

- Forecast combination can improve forecasting accuracy, provided that the sets of forecasts contain some independent information. 
    
- Thereafter a variety of weighting methods
  
    - Error-based approach (e.g., Winkler and Makridakis, 1983)
    - Regression-based approach (e.g., Granger and Ramanathan, 1984)
    - Bayesian averaging (e.g., Granger and Ramanathan, 1984)
    - etc.
    
---
# Feature-based, **OF COURSE!**

- Pioneer studies: rule-based methods (e.g., Wang, Smith-Miles, and Hyndman, 2009)

- More recently,

- `FFORMA` (Montero-Manso, Athanasopoulos, Hyndman, and Talagala, 2020):  42 features +  `XGBoost` 
    - `gratis` (Kang, Hyndman, and Li, 2020): time series generation + 26 features + nonlinear regression models
    - Forecasting with imaging (Li, Kang, and Li, 2020): imaging features + `XGBoost`
    - `fuma` (Wang, Kang, Petropoulos, and Li, 2019) : 42 features + `gam` + **interval forecasting**

---
# Challenges

- Choice and estimation of time series features.

- Time series features vary from tens to thousands (Fulcher and Jones, 2014).

- Features are extracted using the historical, observed data of each time series - they are doomed to *change*, though.

- Feature extraction might not be robust in the case of limited historical data.

- Large number of chosen features increases the computational time.

---
class: inverse, center, middle

# Historical data `$\Rightarrow$` Produced forecasts

# Forecast with forecasts

# Diversity matters

---

# Diversity matters?

.pull-left[
<img src="figure/unnamed-chunk-3-1.svg" width="576" style="display: block; margin: auto;" />
]

```
#> # A tibble: 9 x 2
#> Method MASE
#> <chr> <dbl>
#> 1 auto_arima_forec 0.363
#> 2 nnetar_forec 0.4 
#> 3 tbats_forec 0.408
#> 4 snaive_forec 0.412
#> 5 ets_forec 0.435
#> 6 naive_forec 0.53 
#> 7 rw_drift_forec 0.603
#> 8 thetaf_forec 0.744
#> 9 stlm_ar_forec 0.844
```
]

---
# Combining two methods

.pull-left[
<div id="htmlwidget-1863cc1ee34b68bbca83" style="width:504px;height:504px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-1863cc1ee34b68bbca83">{"x":{"data":[{"x":[0.009,0,0.007,0.142,0.008,0.019,0.003,0.011,0.008,0.001,0.143,0.013,0.021,0.01,0.008,0.006,0.152,0.005,0.015,0.002,0.011,0.141,0.011,0.02,0.008,0.008,0.211,0.256,0.186,0.133,0.002,0.001,0.02,0.006,0.031,0.016],"y":[0.382,0.38,0.381,0.365,0.482,0.551,0.445,0.363,0.402,0.42,0.446,0.503,0.573,0.467,0.414,0.398,0.358,0.502,0.571,0.465,0.381,0.431,0.495,0.564,0.458,0.402,0.32,0.303,0.336,0.45,0.672,0.567,0.481,0.636,0.551,0.445],"text":["div: 0.009 mase: 0.382 colour: Other","div: 0.000 mase: 0.380 colour: Other","div: 0.007 mase: 0.381 colour: Other","div: 0.142 mase: 0.365 colour: Other","div: 0.008 mase: 0.482 colour: Other","div: 0.019 mase: 0.551 colour: Other","div: 0.003 mase: 0.445 colour: Other","div: 0.011 mase: 0.363 colour: Other","div: 0.008 mase: 0.402 colour: Other","div: 0.001 mase: 0.420 colour: Other","div: 0.143 mase: 0.446 colour: Other","div: 0.013 mase: 0.503 colour: Other","div: 0.021 mase: 0.573 colour: Other","div: 0.010 mase: 0.467 colour: Other","div: 0.008 mase: 0.414 colour: Other","div: 0.006 mase: 0.398 colour: Other","div: 0.152 mase: 0.358 colour: Other","div: 0.005 mase: 0.502 colour: Other","div: 0.015 mase: 0.571 colour: Other","div: 0.002 mase: 0.465 colour: Other","div: 0.011 mase: 0.381 colour: Other","div: 0.141 mase: 0.431 colour: Other","div: 0.011 mase: 0.495 colour: Other","div: 0.020 mase: 0.564 colour: Other","div: 0.008 mase: 0.458 colour: Other","div: 0.008 mase: 0.402 colour: Other","div: 0.211 mase: 0.320 colour: Other","div: 0.256 mase: 0.303 colour: Other","div: 0.186 mase: 0.336 colour: Other","div: 0.133 mase: 0.450 colour: Other","div: 0.002 mase: 0.672 colour: Other","div: 0.001 mase: 0.567 colour: Other","div: 0.020 mase: 0.481 colour: Other","div: 0.006 mase: 0.636 colour: Other","div: 0.031 mase: 0.551 colour: Other","div: 0.016 mase: 0.445 colour: Other"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(205,133,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(205,133,0,1)"}},"hoveron":"points","name":"Other","legendgroup":"Other","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.009,0,0.007,0.142,0.008,0.019,0.003,0.011,0.008,0.001,0.143,0.013,0.021,0.01,0.008,0.006,0.152,0.005,0.015,0.002,0.011,0.141,0.011,0.02,0.008,0.008,0.211,0.256,0.186,0.133,0.002,0.001,0.02,0.006,0.031,0.016],"y":[0.382,0.38,0.381,0.365,0.482,0.551,0.445,0.363,0.402,0.42,0.446,0.503,0.573,0.467,0.414,0.398,0.358,0.502,0.571,0.465,0.381,0.431,0.495,0.564,0.458,0.402,0.32,0.303,0.336,0.45,0.672,0.567,0.481,0.636,0.551,0.445],"text":["auto_arima_forec + ets_forec div: 0.009 mase: 0.382 colour: Other","auto_arima_forec + nnetar_forec div: 0.000 mase: 0.380 colour: Other","auto_arima_forec + tbats_forec div: 0.007 mase: 0.381 colour: Other","auto_arima_forec + stlm_ar_forec div: 0.142 mase: 0.365 colour: Other","auto_arima_forec + rw_drift_forec div: 0.008 mase: 0.482 colour: Other","auto_arima_forec + thetaf_forec div: 0.019 mase: 0.551 colour: Other","auto_arima_forec + naive_forec div: 0.003 mase: 0.445 colour: Other","auto_arima_forec + snaive_forec div: 0.011 mase: 0.363 colour: Other","ets_forec + nnetar_forec div: 0.008 mase: 0.402 colour: Other","ets_forec + tbats_forec div: 0.001 mase: 0.420 colour: Other","ets_forec + stlm_ar_forec div: 0.143 mase: 0.446 colour: Other","ets_forec + rw_drift_forec div: 0.013 mase: 0.503 colour: Other","ets_forec + thetaf_forec div: 0.021 mase: 0.573 colour: Other","ets_forec + naive_forec div: 0.010 mase: 0.467 colour: Other","ets_forec + snaive_forec div: 0.008 mase: 0.414 colour: Other","nnetar_forec + tbats_forec div: 0.006 mase: 0.398 colour: Other","nnetar_forec + stlm_ar_forec div: 0.152 mase: 0.358 colour: Other","nnetar_forec + rw_drift_forec div: 0.005 mase: 0.502 colour: Other","nnetar_forec + thetaf_forec div: 0.015 mase: 0.571 colour: Other","nnetar_forec + naive_forec div: 0.002 mase: 0.465 colour: Other","nnetar_forec + snaive_forec div: 0.011 mase: 0.381 colour: Other","tbats_forec + stlm_ar_forec div: 0.141 mase: 0.431 colour: Other","tbats_forec + rw_drift_forec div: 0.011 mase: 0.495 colour: Other","tbats_forec + thetaf_forec div: 0.020 mase: 0.564 colour: Other","tbats_forec + naive_forec div: 0.008 mase: 0.458 colour: Other","tbats_forec + snaive_forec div: 0.008 mase: 0.402 colour: Other","stlm_ar_forec + rw_drift_forec div: 0.211 mase: 0.320 colour: Other","stlm_ar_forec + thetaf_forec div: 0.256 mase: 0.303 colour: Other","stlm_ar_forec + naive_forec div: 0.186 mase: 0.336 colour: Other","stlm_ar_forec + snaive_forec div: 0.133 mase: 0.450 colour: Other","rw_drift_forec + thetaf_forec div: 0.002 mase: 0.672 colour: Other","rw_drift_forec + naive_forec div: 0.001 mase: 0.567 colour: Other","rw_drift_forec + snaive_forec div: 0.020 mase: 0.481 colour: Other","thetaf_forec + naive_forec div: 0.006 mase: 0.636 colour: Other","thetaf_forec + snaive_forec div: 0.031 mase: 0.551 colour: Other","naive_forec + snaive_forec div: 0.016 mase: 0.445 colour: Other"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(205,133,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(205,133,0,1)"}},"hoveron":"points","name":"Other","legendgroup":"Other","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":23.3059360730594,"r":7.30593607305936,"b":37.2602739726027,"l":43.1050228310502},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-0.0128,0.2688],"tickmode":"array","ticktext":["0.0","0.1","0.2"],"tickvals":[0,0.1,0.2],"categoryorder":"array","categoryarray":["0.0","0.1","0.2"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"Diversity","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[0.28455,0.69045],"tickmode":"array","ticktext":["0.3","0.4","0.5","0.6"],"tickvals":[0.3,0.4,0.5,0.6],"categoryorder":"array","categoryarray":["0.3","0.4","0.5","0.6"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"MASE","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":-0.2,"orientation":"h","x":0.1},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"dd6457e1cd3":{"x":{},"y":{},"colour":{},"type":"scatter"},"dd65a968449":{"text":{},"x":{},"y":{},"colour":{}}},"cur_data":"dd6457e1cd3","visdat":{"dd6457e1cd3":["function (y) ","x"],"dd65a968449":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
]

---
# Combining two methods

.pull-left[
<div id="htmlwidget-4cebd437c8bc960f8f87" style="width:504px;height:504px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-4cebd437c8bc960f8f87">{"x":{"data":[{"x":[0.009,0,0.007,0.142,0.008,0.019,0.003,0.011,0.008,0.001,0.143,0.013,0.021,0.01,0.008,0.006,0.152,0.005,0.015,0.002,0.011,0.141,0.011,0.02,0.008,0.008,0.211,0.256,0.186,0.133,0.002,0.001,0.02,0.006,0.031,0.016],"y":[0.382,0.38,0.381,0.365,0.482,0.551,0.445,0.363,0.402,0.42,0.446,0.503,0.573,0.467,0.414,0.398,0.358,0.502,0.571,0.465,0.381,0.431,0.495,0.564,0.458,0.402,0.32,0.303,0.336,0.45,0.672,0.567,0.481,0.636,0.551,0.445],"text":["div: 0.009 mase: 0.382 colour: Other","div: 0.000 mase: 0.380 colour: Other","div: 0.007 mase: 0.381 colour: Other","div: 0.142 mase: 0.365 colour: Other","div: 0.008 mase: 0.482 colour: Other","div: 0.019 mase: 0.551 colour: Other","div: 0.003 mase: 0.445 colour: Other","div: 0.011 mase: 0.363 colour: Other","div: 0.008 mase: 0.402 colour: Other","div: 0.001 mase: 0.420 colour: Other","div: 0.143 mase: 0.446 colour: Other","div: 0.013 mase: 0.503 colour: Other","div: 0.021 mase: 0.573 colour: Other","div: 0.010 mase: 0.467 colour: Other","div: 0.008 mase: 0.414 colour: Other","div: 0.006 mase: 0.398 colour: Other","div: 0.152 mase: 0.358 colour: Other","div: 0.005 mase: 0.502 colour: Other","div: 0.015 mase: 0.571 colour: Other","div: 0.002 mase: 0.465 colour: Other","div: 0.011 mase: 0.381 colour: Other","div: 0.141 mase: 0.431 colour: Other","div: 0.011 mase: 0.495 colour: Other","div: 0.020 mase: 0.564 colour: Other","div: 0.008 mase: 0.458 colour: Other","div: 0.008 mase: 0.402 colour: Other","div: 0.211 mase: 0.320 colour: Other","div: 0.256 mase: 0.303 colour: Other","div: 0.186 mase: 0.336 colour: Other","div: 0.133 mase: 0.450 colour: Other","div: 0.002 mase: 0.672 colour: Other","div: 0.001 mase: 0.567 colour: Other","div: 0.020 mase: 0.481 colour: Other","div: 0.006 mase: 0.636 colour: Other","div: 0.031 mase: 0.551 colour: Other","div: 0.016 mase: 0.445 colour: Other"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(205,133,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(205,133,0,1)"}},"hoveron":"points","name":"Other","legendgroup":"Other","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.009,0,0.007,0.142,0.008,0.019,0.003,0.011,0.008,0.001,0.143,0.013,0.021,0.01,0.008,0.006,0.152,0.005,0.015,0.002,0.011,0.141,0.011,0.02,0.008,0.008,0.211,0.256,0.186,0.133,0.002,0.001,0.02,0.006,0.031,0.016],"y":[0.382,0.38,0.381,0.365,0.482,0.551,0.445,0.363,0.402,0.42,0.446,0.503,0.573,0.467,0.414,0.398,0.358,0.502,0.571,0.465,0.381,0.431,0.495,0.564,0.458,0.402,0.32,0.303,0.336,0.45,0.672,0.567,0.481,0.636,0.551,0.445],"text":["auto_arima_forec + ets_forec div: 0.009 mase: 0.382 colour: Other","auto_arima_forec + nnetar_forec div: 0.000 mase: 0.380 colour: Other","auto_arima_forec + tbats_forec div: 0.007 mase: 0.381 colour: Other","auto_arima_forec + stlm_ar_forec div: 0.142 mase: 0.365 colour: Other","auto_arima_forec + rw_drift_forec div: 0.008 mase: 0.482 colour: Other","auto_arima_forec + thetaf_forec div: 0.019 mase: 0.551 colour: Other","auto_arima_forec + naive_forec div: 0.003 mase: 0.445 colour: Other","auto_arima_forec + snaive_forec div: 0.011 mase: 0.363 colour: Other","ets_forec + nnetar_forec div: 0.008 mase: 0.402 colour: Other","ets_forec + tbats_forec div: 0.001 mase: 0.420 colour: Other","ets_forec + stlm_ar_forec div: 0.143 mase: 0.446 colour: Other","ets_forec + rw_drift_forec div: 0.013 mase: 0.503 colour: Other","ets_forec + thetaf_forec div: 0.021 mase: 0.573 colour: Other","ets_forec + naive_forec div: 0.010 mase: 0.467 colour: Other","ets_forec + snaive_forec div: 0.008 mase: 0.414 colour: Other","nnetar_forec + tbats_forec div: 0.006 mase: 0.398 colour: Other","nnetar_forec + stlm_ar_forec div: 0.152 mase: 0.358 colour: Other","nnetar_forec + rw_drift_forec div: 0.005 mase: 0.502 colour: Other","nnetar_forec + thetaf_forec div: 0.015 mase: 0.571 colour: Other","nnetar_forec + naive_forec div: 0.002 mase: 0.465 colour: Other","nnetar_forec + snaive_forec div: 0.011 mase: 0.381 colour: Other","tbats_forec + stlm_ar_forec div: 0.141 mase: 0.431 colour: Other","tbats_forec + rw_drift_forec div: 0.011 mase: 0.495 colour: Other","tbats_forec + thetaf_forec div: 0.020 mase: 0.564 colour: Other","tbats_forec + naive_forec div: 0.008 mase: 0.458 colour: Other","tbats_forec + snaive_forec div: 0.008 mase: 0.402 colour: Other","stlm_ar_forec + rw_drift_forec div: 0.211 mase: 0.320 colour: Other","stlm_ar_forec + thetaf_forec div: 0.256 mase: 0.303 colour: Other","stlm_ar_forec + naive_forec div: 0.186 mase: 0.336 colour: Other","stlm_ar_forec + snaive_forec div: 0.133 mase: 0.450 colour: Other","rw_drift_forec + thetaf_forec div: 0.002 mase: 0.672 colour: Other","rw_drift_forec + naive_forec div: 0.001 mase: 0.567 colour: Other","rw_drift_forec + snaive_forec div: 0.020 mase: 0.481 colour: Other","thetaf_forec + naive_forec div: 0.006 mase: 0.636 colour: Other","thetaf_forec + snaive_forec div: 0.031 mase: 0.551 colour: Other","naive_forec + snaive_forec div: 0.016 mase: 0.445 colour: Other"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(205,133,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(205,133,0,1)"}},"hoveron":"points","name":"Other","legendgroup":"Other","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256],"y":[0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303],"text":"div[min_index]: 0.256 mase[min_index]: 0.303 colour: theta + stlm_ar div: 0.256 mase: 0.303 colour: theta + stlm_ar","type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(255,0,0,1)","opacity":1,"size":15.1181102362205,"symbol":"circle-open","line":{"width":1.88976377952756,"color":"rgba(255,0,0,1)"}},"hoveron":"points","name":"theta + stlm_ar","legendgroup":"theta + stlm_ar","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":23.3059360730594,"r":7.30593607305936,"b":37.2602739726027,"l":43.1050228310502},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-0.0128,0.2688],"tickmode":"array","ticktext":["0.0","0.1","0.2"],"tickvals":[0,0.1,0.2],"categoryorder":"array","categoryarray":["0.0","0.1","0.2"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"Diversity","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[0.28455,0.69045],"tickmode":"array","ticktext":["0.3","0.4","0.5","0.6"],"tickvals":[0.3,0.4,0.5,0.6],"categoryorder":"array","categoryarray":["0.3","0.4","0.5","0.6"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"MASE","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":-0.2,"orientation":"h","x":0.1},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"dd63c1018c4":{"x":{},"y":{},"colour":{},"type":"scatter"},"dd644ca0aaa":{"text":{},"x":{},"y":{},"colour":{}},"dd62c82422e":{"x":{},"y":{},"colour":{},"x.1":{},"y.1":{},"colour.1":{}}},"cur_data":"dd63c1018c4","visdat":{"dd63c1018c4":["function (y) ","x"],"dd644ca0aaa":["function (y) ","x"],"dd62c82422e":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
]

---
# Combining two methods

.pull-left[
<div id="htmlwidget-e78427d13f2bdea304c8" style="width:504px;height:504px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-e78427d13f2bdea304c8">{"x":{"data":[{"x":[0.009,0,0.007,0.142,0.008,0.019,0.003,0.011,0.008,0.001,0.143,0.013,0.021,0.01,0.008,0.006,0.152,0.005,0.015,0.002,0.011,0.141,0.011,0.02,0.008,0.008,0.211,0.256,0.186,0.133,0.002,0.001,0.02,0.006,0.031,0.016],"y":[0.382,0.38,0.381,0.365,0.482,0.551,0.445,0.363,0.402,0.42,0.446,0.503,0.573,0.467,0.414,0.398,0.358,0.502,0.571,0.465,0.381,0.431,0.495,0.564,0.458,0.402,0.32,0.303,0.336,0.45,0.672,0.567,0.481,0.636,0.551,0.445],"text":["div: 0.009 mase: 0.382 colour: Other","div: 0.000 mase: 0.380 colour: Other","div: 0.007 mase: 0.381 colour: Other","div: 0.142 mase: 0.365 colour: Other","div: 0.008 mase: 0.482 colour: Other","div: 0.019 mase: 0.551 colour: Other","div: 0.003 mase: 0.445 colour: Other","div: 0.011 mase: 0.363 colour: Other","div: 0.008 mase: 0.402 colour: Other","div: 0.001 mase: 0.420 colour: Other","div: 0.143 mase: 0.446 colour: Other","div: 0.013 mase: 0.503 colour: Other","div: 0.021 mase: 0.573 colour: Other","div: 0.010 mase: 0.467 colour: Other","div: 0.008 mase: 0.414 colour: Other","div: 0.006 mase: 0.398 colour: Other","div: 0.152 mase: 0.358 colour: Other","div: 0.005 mase: 0.502 colour: Other","div: 0.015 mase: 0.571 colour: Other","div: 0.002 mase: 0.465 colour: Other","div: 0.011 mase: 0.381 colour: Other","div: 0.141 mase: 0.431 colour: Other","div: 0.011 mase: 0.495 colour: Other","div: 0.020 mase: 0.564 colour: Other","div: 0.008 mase: 0.458 colour: Other","div: 0.008 mase: 0.402 colour: Other","div: 0.211 mase: 0.320 colour: Other","div: 0.256 mase: 0.303 colour: Other","div: 0.186 mase: 0.336 colour: Other","div: 0.133 mase: 0.450 colour: Other","div: 0.002 mase: 0.672 colour: Other","div: 0.001 mase: 0.567 colour: Other","div: 0.020 mase: 0.481 colour: Other","div: 0.006 mase: 0.636 colour: Other","div: 0.031 mase: 0.551 colour: Other","div: 0.016 mase: 0.445 colour: Other"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(205,133,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(205,133,0,1)"}},"hoveron":"points","name":"Other","legendgroup":"Other","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.009,0,0.007,0.142,0.008,0.019,0.003,0.011,0.008,0.001,0.143,0.013,0.021,0.01,0.008,0.006,0.152,0.005,0.015,0.002,0.011,0.141,0.011,0.02,0.008,0.008,0.211,0.256,0.186,0.133,0.002,0.001,0.02,0.006,0.031,0.016],"y":[0.382,0.38,0.381,0.365,0.482,0.551,0.445,0.363,0.402,0.42,0.446,0.503,0.573,0.467,0.414,0.398,0.358,0.502,0.571,0.465,0.381,0.431,0.495,0.564,0.458,0.402,0.32,0.303,0.336,0.45,0.672,0.567,0.481,0.636,0.551,0.445],"text":["auto_arima_forec + ets_forec div: 0.009 mase: 0.382 colour: Other","auto_arima_forec + nnetar_forec div: 0.000 mase: 0.380 colour: Other","auto_arima_forec + tbats_forec div: 0.007 mase: 0.381 colour: Other","auto_arima_forec + stlm_ar_forec div: 0.142 mase: 0.365 colour: Other","auto_arima_forec + rw_drift_forec div: 0.008 mase: 0.482 colour: Other","auto_arima_forec + thetaf_forec div: 0.019 mase: 0.551 colour: Other","auto_arima_forec + naive_forec div: 0.003 mase: 0.445 colour: Other","auto_arima_forec + snaive_forec div: 0.011 mase: 0.363 colour: Other","ets_forec + nnetar_forec div: 0.008 mase: 0.402 colour: Other","ets_forec + tbats_forec div: 0.001 mase: 0.420 colour: Other","ets_forec + stlm_ar_forec div: 0.143 mase: 0.446 colour: Other","ets_forec + rw_drift_forec div: 0.013 mase: 0.503 colour: Other","ets_forec + thetaf_forec div: 0.021 mase: 0.573 colour: Other","ets_forec + naive_forec div: 0.010 mase: 0.467 colour: Other","ets_forec + snaive_forec div: 0.008 mase: 0.414 colour: Other","nnetar_forec + tbats_forec div: 0.006 mase: 0.398 colour: Other","nnetar_forec + stlm_ar_forec div: 0.152 mase: 0.358 colour: Other","nnetar_forec + rw_drift_forec div: 0.005 mase: 0.502 colour: Other","nnetar_forec + thetaf_forec div: 0.015 mase: 0.571 colour: Other","nnetar_forec + naive_forec div: 0.002 mase: 0.465 colour: Other","nnetar_forec + snaive_forec div: 0.011 mase: 0.381 colour: Other","tbats_forec + stlm_ar_forec div: 0.141 mase: 0.431 colour: Other","tbats_forec + rw_drift_forec div: 0.011 mase: 0.495 colour: Other","tbats_forec + thetaf_forec div: 0.020 mase: 0.564 colour: Other","tbats_forec + naive_forec div: 0.008 mase: 0.458 colour: Other","tbats_forec + snaive_forec div: 0.008 mase: 0.402 colour: Other","stlm_ar_forec + rw_drift_forec div: 0.211 mase: 0.320 colour: Other","stlm_ar_forec + thetaf_forec div: 0.256 mase: 0.303 colour: Other","stlm_ar_forec + naive_forec div: 0.186 mase: 0.336 colour: Other","stlm_ar_forec + snaive_forec div: 0.133 mase: 0.450 colour: Other","rw_drift_forec + thetaf_forec div: 0.002 mase: 0.672 colour: Other","rw_drift_forec + naive_forec div: 0.001 mase: 0.567 colour: Other","rw_drift_forec + snaive_forec div: 0.020 mase: 0.481 colour: Other","thetaf_forec + naive_forec div: 0.006 mase: 0.636 colour: Other","thetaf_forec + snaive_forec div: 0.031 mase: 0.551 colour: Other","naive_forec + snaive_forec div: 0.016 mase: 0.445 colour: Other"],"type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(205,133,0,1)","opacity":1,"size":5.66929133858268,"symbol":"circle","line":{"width":1.88976377952756,"color":"rgba(205,133,0,1)"}},"hoveron":"points","name":"Other","legendgroup":"Other","showlegend":false,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256,0.256],"y":[0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303,0.303],"text":"div[min_index]: 0.256 mase[min_index]: 0.303 colour: theta + stlm_ar div: 0.256 mase: 0.303 colour: theta + stlm_ar","type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(255,0,0,1)","opacity":1,"size":15.1181102362205,"symbol":"circle-open","line":{"width":1.88976377952756,"color":"rgba(255,0,0,1)"}},"hoveron":"points","name":"theta + stlm_ar","legendgroup":"theta + stlm_ar","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null},{"x":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"y":[0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38,0.38],"text":"div[which(combn_names == twobest)]: 0 mase[which(combn_names == twobest)]: 0.38 colour: arima + nnetar div: 0 mase: 0.38 colour: arima + nnetar","type":"scatter","mode":"markers","marker":{"autocolorscale":false,"color":"rgba(0,205,0,1)","opacity":1,"size":15.1181102362205,"symbol":"circle-open","line":{"width":1.88976377952756,"color":"rgba(0,205,0,1)"}},"hoveron":"points","name":"arima + nnetar","legendgroup":"arima + nnetar","showlegend":true,"xaxis":"x","yaxis":"y","hoverinfo":"text","frame":null}],"layout":{"margin":{"t":23.3059360730594,"r":7.30593607305936,"b":37.2602739726027,"l":43.1050228310502},"plot_bgcolor":"rgba(235,235,235,1)","paper_bgcolor":"rgba(255,255,255,1)","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187},"xaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[-0.0128,0.2688],"tickmode":"array","ticktext":["0.0","0.1","0.2"],"tickvals":[0,0.1,0.2],"categoryorder":"array","categoryarray":["0.0","0.1","0.2"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"y","title":{"text":"Diversity","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"yaxis":{"domain":[0,1],"automargin":true,"type":"linear","autorange":false,"range":[0.28455,0.69045],"tickmode":"array","ticktext":["0.3","0.4","0.5","0.6"],"tickvals":[0.3,0.4,0.5,0.6],"categoryorder":"array","categoryarray":["0.3","0.4","0.5","0.6"],"nticks":null,"ticks":"outside","tickcolor":"rgba(51,51,51,1)","ticklen":3.65296803652968,"tickwidth":0.66417600664176,"showticklabels":true,"tickfont":{"color":"rgba(77,77,77,1)","family":"","size":11.689497716895},"tickangle":-0,"showline":false,"linecolor":null,"linewidth":0,"showgrid":true,"gridcolor":"rgba(255,255,255,1)","gridwidth":0.66417600664176,"zeroline":false,"anchor":"x","title":{"text":"MASE","font":{"color":"rgba(0,0,0,1)","family":"","size":14.6118721461187}},"hoverformat":".2f"},"shapes":[{"type":"rect","fillcolor":null,"line":{"color":null,"width":0,"linetype":[]},"yref":"paper","xref":"paper","x0":0,"x1":1,"y0":0,"y1":1}],"showlegend":true,"legend":{"bgcolor":"rgba(255,255,255,1)","bordercolor":"transparent","borderwidth":1.88976377952756,"font":{"color":"rgba(0,0,0,1)","family":"","size":11.689497716895},"y":-0.2,"orientation":"h","x":0.1},"hovermode":"closest","barmode":"relative"},"config":{"doubleClick":"reset","showSendToCloud":false},"source":"A","attrs":{"dd67423d839":{"x":{},"y":{},"colour":{},"type":"scatter"},"dd65d48c1c0":{"text":{},"x":{},"y":{},"colour":{}},"dd653a85418":{"x":{},"y":{},"colour":{},"x.1":{},"y.1":{},"colour.1":{}},"dd650291e90":{"x":{},"y":{},"colour":{},"x.1":{},"y.1":{},"colour.1":{}},"dd63b95a90d":{"x":{},"y":{},"colour":{},"x.1":{},"y.1":{},"colour.1":{}}},"cur_data":"dd67423d839","visdat":{"dd67423d839":["function (y) ","x"],"dd65d48c1c0":["function (y) ","x"],"dd653a85418":["function (y) ","x"],"dd650291e90":["function (y) ","x"],"dd63b95a90d":["function (y) ","x"]},"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script>
]

---
# Yes, diversity matters!

---
# Yes, diversity matters!

---
class: inverse, center, middle

# How to "forecast with forecasts"?

---
# How to use diversity to forecast?

- Input:  the forecasts from a pool of models.

- Measure their diversity, a feature that has been identified as a decisive factor in forecast combination (Thomson, Pollock, Onkal, et al., 2019; Lichtendahl and Winkler, 2020).

- Through meta-learning, we link the diversity of the forecasts with their out-of-sample performance to fit combination models based on diversity.

---
# Measuring diversity

.small[
$$
`\begin{aligned}
    MSE_{comb} & = \frac{1}{H}\sum_{h = 1}^HMSE_{ch} \\
    & = \frac{1}{H}\sum_{h = 1}^H (\frac{1}{M}\sum_{i = 1}^Mf_{ih} - y_{T+h})^2\\
    & = \frac{1}{H} \frac{1}{M} \sum_{h=1}^{H} \sum_{i=1}^{M}(f_{ih}-y_{T+h})^2 - \frac{1}{H} \frac{1}{M^2} \sum_{h=1}^{H} \sum_{i=1}^{M-1} \sum_{j=1,j>i}^{M}(f_{ih}-f_{jh})^2 \\
    & = \frac{1}{M} \sum_{i=1}^{M} MSE_i - \frac{1}{M^2} \sum_{i=1}^{M-1} \sum_{j=1,j>i}^{M}Div_{i,j},
\end{aligned}`
$$
`$H$` is the forecasting horizon, `$M$` is the number of forecasting methods, `$T$` is the historical length (Thomson, Pollock, Onkal, et al., 2019). 
]

---
# Measuring diversity

$$
`\begin{aligned}
    MSE_i &= \frac{1}{H} \sum_{i=1}^{H} MSE_{ch} =  \frac{1}{H} \sum_{i=1}^{H} (f_{ch}-y_{T+h})^2,\\
    Div_{i,j}& = \frac{1}{H} \sum_{i=1}^{H} (f_{ih}-f_{jh})^2, \\
    \\
    \\
   sDiv_{i,j}& = \frac{\frac{1}{H}\sum\limits_{h=1}^H(f_{ih}-f_{jh})^2}{(\frac{1}{T}\sum\limits_{t=1}^T|y_t|)^2}.
\end{aligned}`
$$
---
# Diversity for forecast combination

---
# The forecasting framework

---
class: inverse, center, middle

# Does it work?

---
# Data used: M4

---
# Methods used

---
# Forecasting with logistic regression

---
# Forecasting with XGBoost

---
# Does it work?

- We empirically show that this single feature is enough.

- Moreover, we show that expanding the list of other time-series features to include the diversity further enhances the forecasting performance.

- Computational time:  1 min for diversity features; 44 mins to extract 42 features for M4 (32 cores).

- We found that the proposed method replies on the length of forecasting horizon.

---
# Conclusion

- Facing  the challenge of selecting an appropriate set of time series features.

- Features are unreliable when historical data is limited.

- We propose to forecast with forecasts, without manually choosing time series features, yielding comparable performance.

- The proposed diversity-based forecast combination automatically controls the combination via measuring the pairwise diversity between forecasts and linking them, via a meta-learner, to the accuracy on a test set.

- Our approach does not rely on particular families of forecasting models. It can equally be applied on both statistical and judgmental forecasts.

---
class: center, middle

# Thanks!

## `Web`: *http://yanfei.site*
## `Lab`: *http://kllab.org*