📌 Introduction

Welcome to my Week 15 learning journey in Advanced Regression Techniques! As part of my machine learning specialization, this week was all about moving beyond simple linear regression into the wild world of Polynomial Regression, Support Vector Regression (SVR), and Decision Tree Regression.

I’ll be honest—I started this week overwhelmed. Terms like kernel tricks, feature scaling, and tree pruning sounded like jargon from a sci-fi movie. But by the end, I not only built models—I understood why they worked, when to use them, and how to debug them.

Here’s my story, complete with code snippets, errors, fixes, and key learnings. You can follow along with my Google Colab notebook here.

📁 The Datasets

We were given a neatly organized folder (as per the Data Dictionary) with:

• Task Datasets (for practice)

• Assignment Datasets (for comparison)

• Assessment Dataset (final project)

Each dataset was specifically designed to highlight the strengths and weaknesses of different regression techniques.

🧪 Task 1: Polynomial Regression

The Goal

Compare Linear Regression vs Polynomial Regression on task1_polynomial_data.csv.

The Struggle

I started by fitting a simple linear model. The results were… okay. But when I plotted it, I immediately saw the problem:

python

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Load data
data = pd.read_csv('task1_polynomial_data.csv')
X = data[['Experience_Years']].values
y = data['Salary'].values

# Linear regression
lin_reg = LinearRegression()
lin_reg.fit(X, y)

# Plot
plt.scatter(X, y, color='blue', label='Actual')
plt.plot(X, lin_reg.predict(X), color='red', linewidth=2, label='Linear Fit')
plt.title("Linear Regression on Non-Linear Data")
plt.xlabel('Experience Years')
plt.ylabel('Salary')
plt.legend()
plt.show()

The line was too rigid. Clearly, the relationship wasn't linear. My R² score was only 0.72—not terrible, but not great.

The Fix

I used PolynomialFeatures from sklearn to transform the features:

python

from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import mean_squared_error, r2_score

# Try different degrees
degrees = [2, 3, 4]
results = []

for degree in degrees:
    poly = PolynomialFeatures(degree=degree)
    X_poly = poly.fit_transform(X)

    poly_reg = LinearRegression()
    poly_reg.fit(X_poly, y)

    y_pred = poly_reg.predict(X_poly)
    mse = mean_squared_error(y, y_pred)
    r2 = r2_score(y, y_pred)

    results.append((degree, mse, r2))

    print(f"Degree {degree}: MSE = {mse:.2f}, R² = {r2:.4f}")

⚠️ The Error I Made

I initially tried degree=6 without cross-validation and got perfect training fit but realized I was overfitting. The model was memorizing noise!

Key Learning

• Degree matters: Too low → underfitting, too high → overfitting.

• Always visualize the fit against the actual data.

• Cross-validate polynomial degrees using cross_val_score.

Best result: Degree 3 gave the best balance (R² = 0.96).

🔥 Task 2: Support Vector Regression (SVR)

The Goal

Predict ice cream sales from temperature using SVR with an RBF kernel on task2_svr_data.csv.

The Error That Broke Me

I forgot to scale the features. SVR is sensitive to scale! My model's predictions were completely off:

python

from sklearn.svm import SVR

# WRONG: Not scaling
svr = SVR(kernel='rbf')
svr.fit(X, y)  # Disaster!

# Error message didn't show, but predictions were terrible
# MSE was in the millions instead of thousands

The Fix

I added StandardScaler:

python

from sklearn.preprocessing import StandardScaler
import numpy as np

# Scale features
sc_X = StandardScaler()
sc_y = StandardScaler()

X_scaled = sc_X.fit_transform(X)
y_scaled = sc_y.fit_transform(y.reshape(-1, 1)).ravel()  # Important: ravel()

# Now SVR works
svr = SVR(kernel='rbf', C=100, gamma=0.1)
svr.fit(X_scaled, y_scaled)

# Don't forget to inverse transform predictions!
y_pred_scaled = svr.predict(X_scaled)
y_pred = sc_y.inverse_transform(y_pred_scaled.reshape(-1, 1))

⚠️ Another Gotcha

The gamma parameter! Too high → overfitting, too low → underfitting. I used GridSearchCV to find optimal values:

python

from sklearn.model_selection import GridSearchCV

param_grid = {
    'C': [0.1, 1, 10, 100],
    'gamma': [0.01, 0.1, 1, 'scale'],
    'kernel': ['rbf', 'linear']
}

grid = GridSearchCV(SVR(), param_grid, cv=5, scoring='r2')
grid.fit(X_scaled, y_scaled)
print(f"Best parameters: {grid.best_params_}")

Key Learning

• Always scale features for SVR (and many other algorithms).

• The C parameter controls tolerance for errors (regularization).

• gamma controls the influence of individual points.

• Inverse transform your predictions back to original scale.

  

🌳 Task 3: Decision Tree Regression

The Goal

Model study hours vs exam scores using a Decision Tree Regressor on task3_decision_tree_data.csv.

The "Overfitting" Trap

My first tree was too deep. It memorized the training data:

python

from sklearn.tree import DecisionTreeRegressor

# WRONG: No depth limit
tree_reg = DecisionTreeRegressor(random_state=42)
tree_reg.fit(X, y)

# Training score was perfect (R² = 1.0)
# But this was obviously overfitting

The Visualization That Saved Me

I plotted the decision tree's predictions across a high-resolution range:

python

# Create high-res X for smooth plot
X_grid = np.arange(min(X), max(X), 0.01).reshape(-1, 1)

plt.scatter(X, y, color='red', label='Actual')
plt.plot(X_grid, tree_reg.predict(X_grid), color='blue', label='Tree Prediction')
plt.title("Decision Tree Regression (Overfitting)")
plt.xlabel('Hours Studied')
plt.ylabel('Exam Score')
plt.legend()
plt.show()

The plot showed a jagged, stair-step pattern that was clearly memorizing noise.

The Fix

I tuned max_depth and used min_samples_split:

python

# Better: Regularized tree
tree_reg = DecisionTreeRegressor(
    max_depth=3,
    min_samples_split=5,
    min_samples_leaf=2,
    random_state=42
)
tree_reg.fit(X, y)

# Visualize tree structure (optional but helpful)
from sklearn.tree import plot_tree
plt.figure(figsize=(12, 8))
plot_tree(tree_reg, filled=True, feature_names=['Hours_Studied'])
plt.show()

Key Learning

• Trees can overfit easily if not regularized.

• Visualize the tree with plot_tree() to understand splits.

• Use max_depth, min_samples_split, and min_samples_leaf for regularization.

• Decision trees don't need feature scaling (unlike SVR).

📊 Assignment 1: Salary Prediction

Here's where things got real. I had to compare all three techniques on assignment1_salary_prediction.csv.

My Comparison Script

python

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Load data
data = pd.read_csv('assignment1_salary_prediction.csv')
X = data[['Position_Level']].values
y = data['Salary'].values

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

results = {}

# 1. Linear Regression
lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)
results['Linear'] = {
    'MSE': mean_squared_error(y_test, y_pred),
    'R2': r2_score(y_test, y_pred)
}

# 2. Polynomial Regression (degree=6)
poly = PolynomialFeatures(degree=6)
X_poly_train = poly.fit_transform(X_train)
X_poly_test = poly.transform(X_test)

poly_reg = LinearRegression()
poly_reg.fit(X_poly_train, y_train)
y_pred = poly_reg.predict(X_poly_test)
results['Polynomial (deg=6)'] = {
    'MSE': mean_squared_error(y_test, y_pred),
    'R2': r2_score(y_test, y_pred)
}

# 3. SVR
scaler_X = StandardScaler()
scaler_y = StandardScaler()

X_train_scaled = scaler_X.fit_transform(X_train)
y_train_scaled = scaler_y.fit_transform(y_train.reshape(-1, 1)).ravel()
X_test_scaled = scaler_X.transform(X_test)

svr = SVR(kernel='rbf', C=100, gamma=0.1)
svr.fit(X_train_scaled, y_train_scaled)

y_pred_scaled = svr.predict(X_test_scaled)
y_pred = scaler_y.inverse_transform(y_pred_scaled.reshape(-1, 1))
results['SVR (RBF)'] = {
    'MSE': mean_squared_error(y_test, y_pred),
    'R2': r2_score(y_test, y_pred)
}

# 4. Decision Tree
tree = DecisionTreeRegressor(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)
results['Decision Tree'] = {
    'MSE': mean_squared_error(y_test, y_pred),
    'R2': r2_score(y_test, y_pred)
}

# Display results
for model, metrics in results.items():
    print(f"{model}: MSE = {metrics['MSE']:.2f}, R² = {metrics['R2']:.4f}")

My Comparison Results

Winner: Polynomial Regression (degree=3).

Reflection

• Polynomial regression excelled because the salary trend was exponential.

• SVR was close but needed careful tuning of C and gamma.

• Decision trees were interpretable but slightly less accurate.

• Important: I learned to always split data before comparing models!

🧩 Assignment 2: Multi-Feature Regression

Now with 4 features in assignment2_energy_efficiency.csv, I practiced feature importance and hyperparameter tuning.

Key Code Snippet: Feature Importance

python

# Train decision tree
tree = DecisionTreeRegressor(max_depth=5, random_state=42)
tree.fit(X_train, y_train)

# Get feature importance
importance = tree.feature_importances_
features = ['Temperature', 'Humidity', 'Wind_Speed', 'Solar_Radiation']

for feature, imp in zip(features, importance):
    print(f"{feature}: {imp:.4f}")

⚠️ The Mistake

I initially trained SVR without tuning on the multi-feature data and got poor results. Lesson: GridSearchCV is essential for SVR with multiple features.

Key Takeaway

For multi-feature non-linear problems:

• SVR with scaling + RBF kernel + GridSearch works well.

• Decision Trees give you feature importance "for free".

• Solar_Radiation was the most important predictor of energy consumption.

📈 Assignment 3: Time-Series Prediction

Stock price prediction is tricky! I used assignment3_stock_prices.csv with 90 days of trading data.

What I Tried

1. Polynomial Regression on Day vs Closing_Price

2. Decision Tree with multiple features

3. Feature Engineering: Added moving averages

The Hard Truth I Learned

python

# WARNING: Time-series requires special treatment
X = data[['Day', 'Volume', 'Opening_Price']]
y = data['Closing_Price']

# DON'T randomly shuffle time-series data!
# Instead, use time-based split
train_size = int(len(data) * 0.8)
X_train, X_test = X[:train_size], X[train_size:]
y_train, y_test = y[:train_size], y[train_size:]

Key Learning

• Polynomial regression can capture trends but not volatility.

• Decision trees struggle with sequential data unless lag features are added.

• Never use random shuffle for time-series data!

• Financial prediction requires specialized models (ARIMA, LSTM).

🏁 The Grand Finale: Assessment

The car_price_prediction.csv dataset had mixed data types, categorical variables, and complex interactions.

My End-to-End Pipeline

python

# 1. Load and explore
data = pd.read_csv('car_price_prediction.csv')
print(data.info())
print(data.describe())

# 2. Handle categorical variables
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

# Label encode binary categoricals
le = LabelEncoder()
data['Accident_History'] = le.fit_transform(data['Accident_History'])
data['Service_Records'] = le.fit_transform(data['Service_Records'])

# One-hot encode multi-category
data = pd.get_dummies(data, columns=['Brand', 'Fuel_Type', 'Transmission'], drop_first=True)

# 3. Split data
X = data.drop('Price', axis=1)
y = data['Price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 4. Scale features (for SVR)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# 5. Train multiple models
models = {
    'Linear Regression': LinearRegression(),
    'Polynomial Regression': Pipeline([
        ('poly', PolynomialFeatures(degree=2)),
        ('linear', LinearRegression())
    ]),
    'SVR': SVR(kernel='rbf', C=100, gamma=0.1),
    'Decision Tree': DecisionTreeRegressor(max_depth=5, random_state=42)
}

results = {}
for name, model in models.items():
    if name == 'SVR':
        # Scale y for SVR
        scaler_y = StandardScaler()
        y_train_scaled = scaler_y.fit_transform(y_train.values.reshape(-1, 1)).ravel()
        model.fit(X_train_scaled, y_train_scaled)
        y_pred_scaled = model.predict(X_test_scaled)
        y_pred = scaler_y.inverse_transform(y_pred_scaled.reshape(-1, 1))
    elif name == 'Polynomial Regression':
        model.fit(X_train, y_train)
        y_pred = model.predict(X_test)
    else:
        model.fit(X_train, y_train)
        y_pred = model.predict(X_test)

    results[name] = {
        'MSE': mean_squared_error(y_test, y_pred),
        'R2': r2_score(y_test, y_pred)
    }

⚠️ Critical Error Encountered

When one-hot encoding, I initially didn't use drop_first=True, which led to multicollinearity (dummy variable trap). Linear regression failed with singular matrix error!

Business Insights Generated

• Year and Mileage were top predictors (from decision tree feature importance)

• Accident_History reduced price by ~15%

• Electric cars had a premium of ~20%

• BMW and Mercedes held value better than other brands

Final Model Performance

Winner: Decision Tree Regressor (with tuning).

🛠 Troubleshooting Checklist I Created

Here's a quick reference I built for myself during this journey:

📚 Resources That Saved Me

1. Scikit-learn Documentation - My bible

2. Stack Overflow - For every error message

3. Google Colab GPU - For faster SVR training

4. My own error log - Seriously, keep a debugging journal!

🎯 Final Takeaways

1. Start simple — always fit a linear baseline first

2. Visualize everything — plots reveal what metrics hide

3. Scale for SVR — non-negotiable

4. Trees overfit — regularize with depth and sample limits

5. Polynomial is powerful but dangerous beyond degree 4

6. Real-world data is messy — preprocessing is 80% of the work

7. Cross-validate everything — don't trust training scores

8. Interpretability matters — sometimes a simpler model is better

💡 Pro Tip for Learners Like me

Save every version of your notebook with descriptive names:

• week15_regression_v1_errors.ipynb

• week15_regression_v2_fixed_scaling.ipynb

• week15_regression_final.ipynb

This way, you can trace your learning journey and never lose working versions.

If you're also learning regression, what was your biggest "aha" moment? Share in the comments! 👇

Follow my journey as I document my path from ML beginner to practitioner.