Time Series Preprocessing (Verbose)

Core Patterns

Establishing a Regular Index

Almost every downstream bug traces back to an index that was assumed regular and was not. Make the grid explicit before anything else.

df = (df.drop_duplicates(subset="ts", keep="last")
        .set_index("ts")
        .sort_index()
        .asfreq("h"))          # absent hours become rows of NaN, not silent gaps

asfreq is the load-bearing call. Without it, shift(1) means "the previous row", which may be an hour ago or a week ago depending on what the source system happened to emit. With it, shift(1) means "one hour ago" — which is what every lag feature actually assumes.

Store timestamps in UTC. A series in local time gains a duplicated hour each autumn and loses one each spring; asfreq will either raise on the duplicate or silently produce a NaN row, and seasonal features built on local hour-of-day drift by one for half the year. Convert to local time only for calendar features (is_weekend, hour_of_day), and derive those from the localized view rather than storing the whole series that way.

Causal Feature Engineering

The governing rule: a feature for time t may use only data timestamped strictly before t, and available before t in the production system. Those are two different constraints and both bite.

g = df.groupby("store_id")["sales"]

df["lag_1"]   = g.shift(1)
df["lag_7"]   = g.shift(7)
df["lag_364"] = g.shift(364)                      # same weekday last year

# Rolling windows: shift first, then roll.
df["ma_7"]    = g.shift(1).rolling(7,  min_periods=7).mean()
df["ma_28"]   = g.shift(1).rolling(28, min_periods=28).mean()
df["std_28"]  = g.shift(1).rolling(28, min_periods=28).std()

# Ratio features generalize better than levels across stores of different sizes.
df["ma_ratio"] = df["ma_7"] / df["ma_28"]

pandas rolling is right-aligned: at row t the window covers t-6 .. t inclusive. So an unshifted rolling(7).mean() contains y[t] — one seventh of the target is literally inside the predictor. This produces beautiful validation curves and a model that collapses the moment it runs on data where y[t] is not yet known. The .shift(1) before .rolling() is not stylistic; it is the whole correctness argument.

min_periods equal to the window prevents a second, subtler issue: partially filled early windows have different variance than full ones, so the model learns a spurious relationship between series age and noise level.

The availability constraint is the one that survives code review and then fails in production. A daily sales total for day t-1 may not land in the warehouse until 06:00 on day t; if you score at 02:00, lag_1 is not there. Lag features to what is available at scoring time, not to what is timestamped before now.

Resampling and Target Definition

agg = df.resample("D").agg(
    sales_total=("sales", "sum"),
    sales_peak=("sales", "max"),
    price_close=("price", "last"),
    n_obs=("sales", "size"),        # keep the count — it exposes partial periods
)

Source → target	Aggregation	Why
Counts/volume	sum	Additive quantity
Prices, levels, gauges	last	Endpoint is the state
Rates, utilization	mean	Time-weighted average
Capacity planning	max / quantile(0.95)	Peaks drive the decision
Presence	size	Detects incomplete periods

Downsampling is not a neutral cleanup step — it changes the question. Predicting hourly demand and predicting daily demand are different problems with different noise floors, different seasonality, and different naive baselines. Recompute the baseline after resampling; a model that beat the hourly naive forecast may lose to the daily one, because aggregation removed exactly the noise the model was exploiting.

Always carry n_obs. The final period of a resampled series is almost always partial — the day is not over — and a partial sum looks like a dramatic drop. Drop or mask incomplete periods rather than letting the model learn a phantom end-of-series collapse.

Missing Data

df["gap_len"] = df["value"].isna().groupby(df["value"].notna().cumsum()).transform("size")
df["was_imputed"] = df["value"].isna()
df["value"] = df["value"].ffill(limit=3)      # causal, bounded
df = df[df["gap_len"] <= 24]                  # drop windows built over long outages

Method	Causal	Use when
ffill(limit=n)	Yes	Slowly varying state; short sensor dropouts
Rolling median of the past	Yes	Noisy series where the last value is unreliable
Seasonal lag (shift(168))	Yes	Strong weekly cycle, gaps under a week
interpolate()	No	Offline visualization and reporting only
bfill()	No	Never in a training set
Leave NaN	Yes	LightGBM/XGBoost handle it natively — often best

Leaving NaN is underrated. Gradient boosting learns a default direction for missing values, which lets the model treat "sensor was down" as the signal it frequently is. Imputation manufactures a plausible value and destroys that signal.

Validation Splits

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, test_size=30, gap=7)
for tr, te in tscv.split(X):
    ...   # tr always precedes te; 7 rows discarded between them

gap must be at least the forecast horizon. Predicting seven days ahead with gap=0 means the last training row's target window overlaps the first validation row's — the model sees part of the answer.

Expanding-window splits (the sklearn default, where the training set grows) match production retraining. Sliding-window splits, where training length is fixed, better estimate performance under regime change and are the honest choice when the data-generating process is known to drift. Pick one deliberately and say which.

Group by entity when panel data has one series per store, device, or user: leakage across time within an entity is the obvious risk, but a random split also puts the same store on both sides, which inflates scores for anything the model can memorize per entity.

Common Anti-Patterns

❌ df["ma"] = df["y"].rolling(7).mean() — the window includes y[t]. ✅ df["y"].shift(1).rolling(7).mean().

❌ interpolate() or bfill() before splitting. Future values are dragged backwards into training rows. ✅ Forward fill with a limit, a seasonal lag, or leave NaN.

❌ train_test_split(X, y, shuffle=True). Trains on the future to predict the past. ✅ TimeSeriesSplit, or a fixed date cutoff.

❌ Scaler, target encoder, or PCA fitted on the full series. Test-period statistics leak into every training row. ✅ Fit inside a Pipeline so each fold refits on its own training portion.

❌ Lagging to what is timestamped before now, ignoring ingestion delay. Passes backtest, fails at 02:00 when yesterday's batch has not landed. ✅ Lag to what is available at scoring time; verify against the actual SLA.

❌ Reporting the last, partial period. A half-finished day reads as a crash. ✅ Track n_obs per period and mask incomplete ones.

❌ Feeding a trending series to gradient boosting undifferenced. Trees cannot extrapolate; forecasts saturate at the training maximum. ✅ Difference, detrend, or model the ratio to a rolling baseline.

❌ Naive outlier clipping before the split. Quantile thresholds computed over all data encode future extremes. ✅ Compute clipping bounds on training data only — and see anomaly-detection-techniques before deciding an extreme value is noise.

Time Series Preprocessing Checklist

• Timestamps stored in UTC; duplicates resolved; index sorted
• asfreq applied so gaps are explicit NaN rows
• Every rolling / expanding / ewm feature preceded by .shift(1)
• min_periods set equal to the window length
• Lags reflect ingestion delay, not just timestamp order
• Aggregation function chosen from the downstream decision, not by default
• Naive baseline recomputed after any resampling
• Incomplete final period masked or dropped
• Imputation is causal and bounded; was_imputed flag retained
• Split is time-ordered with gap ≥ forecast horizon
• Expanding vs sliding window chosen deliberately and documented
• All fitted transforms inside a per-fold Pipeline
• Stationarity checked before using an ARIMA-family model
• Trending series differenced before tree-based models