Medical Cost Personal Prediction
A machine learning regression analysis pipeline predicting personal healthcare charges, featuring rigorous outlier detection and data preprocessing.
// 01Project Overview
An academic machine learning project utilizing regression models to forecast individual medical costs based on demographic and lifestyle factors. The project covers the end-to-end data science lifecycle, from Exploratory Data Analysis (EDA) and robust outlier removal to model evaluation and residual analysis.
The Problem
Medical expense forecasting is heavily influenced by skewed distributions and extreme outliers (e.g., severe acute treatments), which distort standard regression models if not properly handled.
The Solution
Implemented a comprehensive preprocessing pipeline with rigorous outlier detection and feature transformation to train and evaluate generalized regression models.
My Individual Contribution
Led the statistical analysis, outlier detection, data sanitation, and feature preprocessing pipeline.
Key Features & Technical Responsibilities I Implemented:
- Conducted distribution analysis on financial target variables and demographic features using Pandas and NumPy.
- Implemented statistical outlier detection techniques (Interquartile Range - IQR) to isolate and treat anomalies.
- Constructed Scikit-learn transformers for one-hot encoding categorical variables and standardizing continuous features.
- Prepared cleaned train/test splits and assisted in running baseline regression model evaluations.
// 02System Capabilities & Features
- Exploratory Data Analysis (EDA) uncovering correlations between smoking, BMI, age, and charges
- Statistical outlier detection and treatment to prevent model distortion
- Categorical encoding and feature normalization pipelines using Scikit-learn
- Comparative regression model training across baseline and regularized estimators
- Evaluation reporting supporting RMSE, MAE, and R² evaluation metrics
// 04Engineering Challenges & Solutions
Challenge: Balancing outlier removal without discarding legitimate high-cost medical scenarios.
Solution: Applied segmented IQR boundaries conditioned on key risk categories (e.g., smoker status) to retain valid high-risk distributions.
Key Engineering Takeaways
- Hands-on mastery of data preprocessing techniques and their critical impact on downstream model generalization.
- Practical experience evaluating regression performance with Scikit-learn.
- Appreciation for data hygiene and ethical handling of medical insurance datasets.