DS End to End Project

A full project joins every skill from earlier topics. This project predicts house prices from size, rooms, and age. The same steps apply to sales forecasts, delivery times, and many other business questions. Work through each stage in a notebook and keep short notes beside the code.

Project Map

 1 Define goal --> 2 Load data --> 3 Clean --> 4 Explore --> 5 Engineer features
                                                                   |
 10 Share  <-- 9 Save model <-- 8 Evaluate <-- 7 Train <-- 6 Split data

Folder Layout

house-price-project/
 |-- data/
 |     |-- raw/houses.csv
 |     |-- clean/houses_clean.csv
 |-- notebooks/
 |     |-- 01_explore.ipynb
 |     |-- 02_model.ipynb
 |-- models/house_model.pkl
 |-- README.md
 |-- requirements.txt

A tidy folder lets teammates find files fast and rerun your work.

Stage 1: Define the Goal

Write the goal in one sentence: predict the selling price of a house in thousands of dollars. Choose a success measure before modeling. This project uses mean absolute error, which reports the average miss in price units, plus R-squared, which shows how much variation the model explains.

Stage 2: Load the Data

import pandas as pd

df = pd.read_csv("data/raw/houses.csv")
print(df.shape)
print(df.head())
print(df.info())

Check row counts, column names, and data types. Compare them with the data description to confirm that nothing broke during download.

Stage 3: Clean the Data

df = df.drop_duplicates()
print(df.isnull().sum())

df["area"] = df["area"].fillna(df["area"].median())
df["age"] = df["age"].fillna(df["age"].median())
df = df[df["price"] > 0]

Remove repeated rows, fill gaps with medians, and drop rows with impossible prices. Write down each rule.

Stage 4: Explore the Data

import seaborn as sns
import matplotlib.pyplot as plt

sns.histplot(df["price"], kde=True)
plt.show()

sns.scatterplot(data=df, x="area", y="price")
plt.show()

sns.heatmap(df.corr(numeric_only=True), annot=True)
plt.show()

Expect a right-skewed price chart and a clear upward trend between area and price. The heatmap shows which columns link most strongly with price.

Stage 5: Engineer Features

df["price_per_area"] = df["price"] / df["area"]
df["is_new"] = (df["age"] < 5).astype(int)
df["room_size"] = df["area"] / df["rooms"]
df = pd.get_dummies(df, columns=["neighborhood"])

Remove price_per_area before training, since it contains the answer and causes leakage. Keep is_new, room_size, and the neighborhood columns.

Stage 6: Split the Data

from sklearn.model_selection import train_test_split

X = df.drop(columns=["price", "price_per_area"])
y = df["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

The test set stays locked away until the final check.

Stage 7: Train and Compare Models

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score

models = {
    "Linear": LinearRegression(),
    "Forest": RandomForestRegressor(n_estimators=200, random_state=42)
}

for name, m in models.items():
    m.fit(X_train, y_train)
    pred = m.predict(X_test)
    print(name, round(mean_absolute_error(y_test, pred), 1),
          round(r2_score(y_test, pred), 3))

Start with a simple model as a baseline. A complex model earns its place only by beating the baseline.

Stage 8: Evaluate Honestly

Model	Mean Absolute Error	R-squared
Linear Regression	Higher	Lower
Random Forest	Lower	Higher

Plot predicted prices against real prices. Points near a diagonal line show good predictions. Examine the largest misses to learn where the model struggles, such as luxury homes with few examples.

Stage 9: Save the Model

import joblib
joblib.dump(models["Forest"], "models/house_model.pkl")

Stage 10: Share the Results

• Write a README with the goal, data source, steps, and final scores.
• Add two or three clear charts, and label axes and units.
• State limits plainly, such as the date range and regions covered.
• List ideas for improvement, such as more neighborhoods or extra features.
• Publish the project on a code-sharing site to build a portfolio.

Checklist Before You Finish

Question	Done?
Is the goal written in one sentence?	Yes or No
Did cleaning rules get recorded?	Yes or No
Does any feature leak the answer?	Yes or No
Did you test on unseen data?	Yes or No
Can a stranger rerun the notebook?	Yes or No

Skills Practiced in This Project

Each stage trains a different habit. Loading and cleaning build patience with messy files. Exploration builds curiosity about odd values. Feature work builds creativity, and evaluation builds honesty about weak results. Sharing builds the skill of explaining numbers to people who never saw the code.

Ideas to Extend the Project

• Tune the forest with a grid search over depth and tree count.
• Add a map-based chart that colors neighborhoods by average price.
• Wrap the saved model in a small web API.
• Try a gradient boosting model and compare the scores.

Key Points

• A project moves through goal, data, cleaning, exploring, features, modeling, evaluation, and sharing.
• Baselines, honest testing, and leakage checks protect the quality of results.
• Clear notes and tidy folders turn a notebook into a portfolio piece.