Fitting a forecasting method on earlier data and measuring its error only on later data it has not seen.
A hold-out test checks a forecasting or predictive method on data that was kept back while it was fitted. Any method looks good on the data it was built from; only the error on the held-back periods says how it will do on the next month. The test period should be the most recent one, because that is the closest to the future being predicted.
Split the history by date. Fit the method on the earlier part, forecast the held-back part, and compare. A common error measure is the sum of absolute errors ÷ the sum of actuals over the test periods.
With 36 months of sales, fit on the first 30 and forecast the last 6. Actuals over those 6 months total $600,000 and absolute errors total $54,000, a hold-out error of 9%, even if the fit on the first 30 months showed 4%.
Shuffling rows into training and test sets, which lets the method see the future. Tuning repeatedly against the same hold-out until it stops being unseen. The full guide is predictive analytics.