DS ML Pipelines

A machine learning pipeline links every step, from raw input to prediction, into a single object. Filling gaps, encoding categories, scaling numbers, and training a model all run in a fixed order. Pipelines prevent mistakes, shorten code, and make models easy to save and reuse.

A car factory line offers a good picture. Each station performs one job, and the product moves along the same path every time.

Why Pipelines Matter

Problem Without Pipelines	Fix With Pipelines
Scaler learns from test data (leakage)	Steps fit on training data only
Forgetting a step at prediction time	One object repeats every step
Long, messy notebooks	Short, readable code
Hard to save the whole process	Save one file

Pipeline Structure

 Raw data
    |
 [ Impute missing values ]
    |
 [ Encode categories / Scale numbers ]
    |
 [ Select features (optional) ]
    |
 [ Model ]
    |
 Prediction

A Simple Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression())
])

pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))

Calling fit runs each step in order on the training data. Calling predict applies the learned scaling to new data automatically.

Different Steps for Different Columns

Numeric and text columns need different handling. ColumnTransformer applies the right steps to the right columns.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier

num_cols = ["age", "income"]
cat_cols = ["city", "plan"]

num_steps = Pipeline([
    ("fill", SimpleImputer(strategy="median")),
    ("scale", StandardScaler())
])

cat_steps = Pipeline([
    ("fill", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore"))
])

prep = ColumnTransformer([
    ("num", num_steps, num_cols),
    ("cat", cat_steps, cat_cols)
])

pipe = Pipeline([
    ("prep", prep),
    ("model", RandomForestClassifier(n_estimators=200, random_state=0))
])

              +--> numeric columns --> fill median --> scale --+
 Raw table ---+                                                 +--> Model
              +--> text columns -----> fill common --> encode -+

Tune the Whole Pipeline

from sklearn.model_selection import GridSearchCV

params = {
    "model__n_estimators": [100, 200],
    "model__max_depth": [5, 10, None]
}

search = GridSearchCV(pipe, params, cv=5, scoring="accuracy")
search.fit(X_train, y_train)
print(search.best_params_)
print(search.best_score_)

The double underscore reaches inside a step. Cross-validation refits the preprocessing for every fold, which keeps the evaluation honest.

Save and Reuse the Pipeline

import joblib

joblib.dump(search.best_estimator_, "customer_pipeline.pkl")
loaded = joblib.load("customer_pipeline.pkl")

new_row = pd.DataFrame([{"age": 34, "income": 52000,
                         "city": "Pune", "plan": "basic"}])
print(loaded.predict(new_row))

The saved file carries every cleaning and encoding rule. A web service only loads the file and calls predict.

Custom Steps

from sklearn.preprocessing import FunctionTransformer
import numpy as np

log_step = FunctionTransformer(np.log1p)

FunctionTransformer wraps any function as a pipeline step. Write custom classes with fit and transform methods for logic that needs learning.

Best Practices

• Split data before any step that learns from data.
• Name steps clearly, such as prep and model.
• Set handle_unknown to ignore, so new categories at prediction time do not crash the code.
• Keep a fixed random_state for repeatable results.
• Store the library versions beside the saved file.

Example: Predicting Customer Cancellations

A subscription company stores age, income, city, and plan type for every customer. Some ages are blank, and city names are text. Without a pipeline, the analyst writes separate cleaning code for training and for new customers, and the two versions drift apart over months. With a pipeline, one object fills blank ages, converts cities into columns of zeros and ones, scales income, and predicts cancellation risk. The same object serves training, testing, and live predictions. New team members read one short block of code and understand the whole workflow.

Common Errors and Fixes

Error	Cause	Fix
Unknown category at prediction time	New city never seen in training	Set handle_unknown to ignore in the encoder
Column not found	New data uses different column names	Match names exactly, and validate inputs before prediction
Score too good to be true	Preprocessing fit on all data	Place every learning step inside the pipeline
Different results after loading	Library versions changed	Pin versions in requirements.txt

Pipelines in Production

Teams often schedule pipelines to retrain on fresh data each week. Larger systems use workflow tools that run each step as a separate task, log results, and retry failures. Scikit-learn pipelines cover the modeling part, and workflow tools cover the scheduling part. Learning the small version first makes the larger tools easy to grasp.

Key Points

• A pipeline chains preprocessing and modeling into one object.
• ColumnTransformer sends each column type through suitable steps.
• Tuning and saving the whole pipeline prevents leakage and deployment mistakes.