Machine learning: a forecast with a fair test

State the forecast task precisely

This exercise predicts demand for the next hourly interval at each successive forecast origin. It uses a complete synthetic series of 56 days and compares a regularised linear model with yesterday’s and last week’s values at the same hour. The aim is a defensible evaluation workflow, not a claim that machine learning will improve every utility forecast.

The final fourteen days are reserved for evaluation. Forecasts are assessed as rolling one-hour-ahead decisions: observations from earlier hours become available as time advances. This differs from issuing all fourteen days of predictions at the beginning of the holdout. That distinction determines which lagged inputs are legitimately available.

Prepare the environment and data

1. Use an approved Python 3 environment with pandas and scikit-learn installed. A separate virtual environment helps keep the project dependencies reproducible.
2. Extract the complete practice pack. The script expects demand_hourly_clean.csv in its own folder.
3. Run python forecast_demand.py, or use the equivalent Python executable for your environment.
4. Retain the installed package versions with the run record, for example using python -m pip freeze in the active environment.
5. Inspect forecast_results.csv and compare the console results with the values below.

The clean file has 1,344 continuous hourly records, in UTC, with one demand value in m³/hour per interval. It contains no measured utility data. The separate data-audit tutorial shows why a raw series should be checked before fitting a model. Do not interpolate a missing value using future observations without considering whether those observations would exist at prediction time.

Create only features available at the forecast origin

The example uses demand 24 hours earlier, demand 168 hours earlier, hour of day and day of week. The first week cannot supply the weekly lag and is removed from model fitting. Each subsequent row pairs the target with information available before its target interval.

The model uses simple numeric calendar features to keep the implementation readable. A more flexible calendar encoding or a different model might improve a real application, but added complexity must earn its place through validation. The two lagged demands already carry much of the synthetic daily and weekly structure.

Weather features require particular care. A forecast issued tomorrow can use tomorrow’s weather forecast as available today; it cannot legitimately use tomorrow’s eventual observed temperature. Keep extraction timestamps or forecast vintages where those distinctions affect an operational evaluation.

Separate model selection from final evaluation

After lag creation, the last fourteen days form the untouched holdout. Earlier data is used for three expanding chronological validation splits, each with a seven-day validation block. The script compares three regularisation strengths using only these splits, then fits the selected model on the full training period.

A pipeline fits feature scaling within each training fold. Fitting a scaler or an imputer on the entire dataset before validation would allow future distribution information into the training process. Randomly mixing time-series rows can also make a model appear more capable than it is when used on future periods.

A gap between fitting and validation may be needed for overlapping labels, delayed information or a longer forecast horizon. This exercise uses already-available lag features for a one-hour target, so no additional gap is applied. Record the reasoning instead of treating one validation configuration as universally correct.

Read the complete executable experiment

The script rejects duplicate, missing or non-hourly data, creates features, selects the model, evaluates the final holdout and exports row-level predictions. No external dataset or service is needed once the dependencies are installed.

"""Rolling one-hour-ahead evaluation with observations available at each origin."""
from pathlib import Path
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error
root=Path(__file__).resolve().parent
frame=pd.read_csv(root/'demand_hourly_clean.csv',parse_dates=['timestamp_utc']).set_index('timestamp_utc').sort_index()
if not frame.index.is_unique:raise ValueError('Duplicate timestamps')
if not (frame.index.to_series().diff().dropna()==pd.Timedelta(hours=1)).all():
 raise ValueError('Expected a continuous hourly series')
if frame.isna().any().any():raise ValueError('Missing data')
frame['lag24']=frame['demand_m3_h'].shift(24)
frame['lag168']=frame['demand_m3_h'].shift(168)
frame['hour']=frame.index.hour
frame['weekday']=frame.index.dayofweek
frame=frame.dropna()
features=['lag24','lag168','hour','weekday']
train=frame.iloc[:-14*24];test=frame.iloc[-14*24:]
cv=TimeSeriesSplit(n_splits=3,test_size=7*24)
scores={}
for alpha in [0.1,1.0,10.0]:
 errors=[]
 for fit_idx,val_idx in cv.split(train):
  model=make_pipeline(StandardScaler(),Ridge(alpha=alpha))
  model.fit(train.iloc[fit_idx][features],train.iloc[fit_idx]['demand_m3_h'])
  pred=model.predict(train.iloc[val_idx][features])
  errors.append(mean_absolute_error(train.iloc[val_idx]['demand_m3_h'],pred))
 scores[alpha]=sum(errors)/len(errors)
best=min(scores,key=scores.get)
model=make_pipeline(StandardScaler(),Ridge(alpha=best))
model.fit(train[features],train['demand_m3_h'])
pred=model.predict(test[features])
observed=test['demand_m3_h']
print('Chosen alpha:',best)
for name,forecast in [('Yesterday',test['lag24']),('Last week',test['lag168']),('Ridge',pred)]:
 print(name,'holdout MAE m3/h:',round(mean_absolute_error(observed,forecast),4))
peak=observed>=train['demand_m3_h'].quantile(.90)
print('Ridge peak-period MAE m3/h:',round(mean_absolute_error(observed[peak],pred[peak]),4))
print('Negative predictions:',int((pred<0).sum()))
test.assign(prediction_m3_h=pred)[['demand_m3_h','prediction_m3_h','lag24','lag168']].to_csv(root/'forecast_results.csv')

Interpret the metrics without overstating them

The synthetic signal was constructed with regular daily and weekly behaviour and a simple trend, so a very low error is not evidence of field performance. A real demand series contains meter faults, restrictions, industrial events and behaviour shifts. Preserve that distinction when presenting the result.

Mean absolute error summarises typical magnitude in the original units. Also inspect bias, large errors and periods important to the operational decision. This example defines a peak threshold from the training data’s 90th percentile and reports holdout error for observations above that threshold. It does not tune the threshold on the holdout.

Connect forecast error to the operating decision

A forecast can have a low average error while underpredicting the few hours that govern a reservoir operating decision. Pass representative forecast errors through a storage or hydraulic assessment and examine consequences. A numerical accuracy gain that does not change a useful decision may not justify added maintenance and monitoring.

Point predictions do not express the full uncertainty. If intervals are developed, evaluate their coverage and width on chronological data and inspect high-demand conditions separately. An interval that contains the observation most of the time can still be poorly calibrated when the system is stressed.

Maintain the baseline forecast as a fallback. If inputs are missing, a model is stale or the observed behaviour shifts, the operational process needs a known response. Forecasting authority and actual control authority should be established separately.

Move from an exercise to a credible pilot

• Use a representative dataset covering relevant seasons, operating regimes and events.
• Define the prediction horizon, input availability and decision owner before choosing features.
• Compare seasonal baselines and simple statistical models with more complex candidates.
• Retain chronological holdouts and report peak-period and event performance.
• Evaluate in shadow mode, monitor drift and define retraining and fallback triggers.
• Check that improved forecasts produce an engineering benefit without violating physical or operational constraints.

Continue to demand-forecast applications and integration with a connected model. A trained predictor remains one component in a larger engineering system.

Sources & further reading

• TimeSeriesSplit ↗scikit-learn · Living documentation
  External source · Checked 25 September 2026
• Python documentation ↗Python Software Foundation · Living documentation
  External source · Checked 24 September 2026
• Improving short-term water demand forecasting using evolutionary algorithms ↗Stańczyk, Kajewska-Szkudlarek, Lipiński & Rychlikowski · 8 August 2022
  External source · Checked 24 September 2026
• AI Risk Management Framework ↗National Institute of Standards and Technology · 2023 framework; 2024 generative AI profile
  External source · Checked 24 September 2026

Source findings are distinguished from editorial interpretation. Apply current local criteria and project evidence when making engineering decisions.