Breast Cancer Wisconsin Diagnostic — ML Analysis

A fine needle aspiration (FNA) is a quick biopsy in which a thin needle draws cells from a breast lump. The cells are photographed under a microscope and software measures each nucleus: its size, shape and texture. This project asks whether a machine-learning model can use those measurements to tell malignant (cancerous) from benign masses, and which measurements matter most.

Dataset Overview

The Breast Cancer Wisconsin (Diagnostic) dataset contains 569 samples with 30 numerical features computed from digitized images of Fine Needle Aspiration (FNA) biopsies of breast masses. For each cell nucleus, ten real-valued features are computed (radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry, fractal dimension), and three statistics are reported: mean, standard error (SE), and worst (largest of the three values), yielding 30 features in total.

The target variable is diagnosis: M (malignant, encoded as 1) or B (benign, encoded as 0).

Clinical Relevance

Breast cancer is the most commonly diagnosed cancer in women worldwide. Early and accurate detection is critical — distinguishing malignant from benign masses using FNA biopsy analysis can inform timely treatment decisions. A classifier on these nuclear measurements could, after proper clinical validation, offer pathologists a reproducible second opinion.

In a screening context, minimizing false negatives (missed malignancies) is paramount — it is clinically safer to recall a benign case than to miss a cancer. In a confirmatory context, precision becomes more important to avoid unnecessary invasive follow-up procedures.

Citation

Wolberg, W., Mangasarian, O., Street, N. & Street, W. (1993). Breast Cancer Wisconsin (Diagnostic). UCI Machine Learning Repository. https://doi.org/10.24432/C5DW2B

The first step is simply to open the data and check it: how many samples there are, what each column means and whether anything is missing. Each row is one biopsy and each column one measurement, reported as the mean, the standard error (how much it varies) and the “worst” value across the nuclei in the image.

Libraries loaded successfully

Shape: (569, 31)
Columns: ['diagnosis', 'radius_mean', 'texture_mean', 'perimeter_mean', 'area_mean', 'smoothness_mean', 'compactness_mean', 'concavity_mean', 'concave points_mean', 'symmetry_mean', 'fractal_dimension_mean', 'radius_se', 'texture_se', 'perimeter_se', 'area_se', 'smoothness_se', 'compactness_se', 'concavity_se', 'concave points_se', 'symmetry_se', 'fractal_dimension_se', 'radius_worst', 'texture_worst', 'perimeter_worst', 'area_worst', 'smoothness_worst', 'compactness_worst', 'concavity_worst', 'concave points_worst', 'symmetry_worst', 'fractal_dimension_worst']

Data types:
diagnosis                      str
radius_mean                float64
texture_mean               float64
perimeter_mean             float64
area_mean                  float64
smoothness_mean            float64
compactness_mean           float64
concavity_mean             float64
concave points_mean        float64
symmetry_mean              float64
fractal_dimension_mean     float64
radius_se                  float64
texture_se                 float64
perimeter_se               float64
area_se                    float64
smoothness_se              float64
compactness_se             float64
concavity_se               float64
concave points_se          float64
symmetry_se                float64
fractal_dimension_se       float64
radius_worst               float64
texture_worst              float64
perimeter_worst            float64
area_worst                 float64
smoothness_worst           float64
compactness_worst          float64
concavity_worst            float64
concave points_worst       float64
symmetry_worst             float64
fractal_dimension_worst    float64
dtype: object

Missing values: 0

Class distribution:
  B: 357 (62.7%)
  M: 212 (37.3%)

Class ratio (M/B): 0.594
Dataset is moderately imbalanced (63% benign, 37% malignant).
This makes Recall, F1, and ROC-AUC more informative than raw Accuracy.

Exploratory data analysis (EDA) means looking at the data with charts before building any model. It shows how many cases are malignant versus benign, how each measurement differs between the two groups, and which measurements move together (correlation). Patterns found here help explain later results and catch problems early.

We visualize the class distribution, feature distributions by class, the correlation structure among features, and a pairplot of the most discriminative features. These plots reveal that malignant tumours tend to have larger, less regular nuclei — which aligns with known pathological characteristics of cancer cells.

Class distribution: 357 Benign (62.7%), 212 Malignant (37.3%)

Many highly correlated feature groups (e.g., radius/perimeter/area).
This multi-collinearity will penalize linear models but is handled well by tree ensembles.

Strong linear separation visible in radius_mean vs concave_points_mean.
This suggests linear models (Logistic Regression) may perform well.

Models must be tested on data they have never seen, otherwise their scores are misleadingly high. The data are split into a training set (to build the model) and a test set (to evaluate it), keeping the same share of malignant cases in both. Scaling puts all measurements on a comparable range; doing it with information from the test set would be a form of “data leakage” that inflates results.

Train/Test Split

We use an 80/20 stratified split to preserve the class ratio in both sets. With 569 samples, this gives 455 training and 114 test instances.

Preventing Data Leakage

The StandardScaler is fitted exclusively on the training set and then applied to both train and test sets. Fitting on the full dataset — or the test set — would constitute data leakage: the scaler would encode information about the test distribution (mean, variance) into the feature transformation, causing the model to have implicitly "seen" test data during training. This leads to overly optimistic evaluation metrics that do not generalise to truly unseen data. By fitting only on training data, we simulate a realistic deployment scenario where future data is genuinely unknown at training time.

Training set: (455, 30) | Test set: (114, 30)
Train class balance: 0.374 malignant
Test class balance:  0.368 malignant
Stratification successful: class ratio preserved in both splits.

Four common algorithms are compared, from a simple, transparent one (logistic regression, which weighs each measurement) to ensembles that combine many decision trees. Hyperparameter tuning means trying different settings for a model and keeping the best, judged by cross-validation: the training data are split into five parts and each part takes a turn as a mini test set.

Model Selection Rationale

We train four classifiers representing different algorithmic families:

Hyperparameter Tuning

The two ensemble models (Random Forest and Gradient Boosting) are tuned with GridSearchCV using 5-fold cross-validation and ROC-AUC as the scoring metric, which is appropriate for an imbalanced binary classification problem in a clinical context.

Logistic Regression: Acc=0.9649, F1=0.9512, AUC=0.9960
Decision Tree: Acc=0.9298, F1=0.9048, AUC=0.9246

Random Forest: Acc=0.9737, F1=0.9630, AUC=0.9929

Gradient Boosting: Acc=0.9649, F1=0.9500, AUC=0.9947

Baseline Model Performance:

Best RF params: {'max_depth': None, 'min_samples_split': 5, 'n_estimators': 200}
Best RF CV ROC-AUC: 0.9906

Best GB params: {'learning_rate': 0.1, 'max_depth': 3, 'n_estimators': 200}
Best GB CV ROC-AUC: 0.9893

Random Forest (Tuned): Acc=0.9737, F1=0.9630, AUC=0.9950
Gradient Boosting (Tuned): Acc=0.9649, F1=0.9500, AUC=0.9954

All Models — Complete Metrics Table:

Best model by ROC-AUC: Logistic Regression
ROC-AUC: 0.9960
Best model saved to outputs/models/best_model.pkl

Several metrics are needed because errors have different costs. Recall (sensitivity) is the share of cancers the model catches; precision is the share of “cancer” predictions that are correct. ROC-AUC summarises how well the model ranks malignant above benign cases across all thresholds: 0.5 is chance, 1.0 is perfect. In cancer detection, missing a tumour is usually worse than a false alarm, so recall is watched closely.

We evaluate all six models (4 baseline + 2 tuned) using five metrics:

• Accuracy: overall correctness
• Precision: proportion of positive predictions that are truly positive (minimises false positives)
• Recall: proportion of true positives correctly identified (minimises false negatives — critical in cancer screening)
• F1: harmonic mean of precision and recall
• ROC-AUC: discrimination ability across all thresholds; robust to class imbalance

For clinical applications, Recall and ROC-AUC are the primary metrics of interest.

Complete Metrics — All Models (Baseline + Tuned):

Confusion matrices saved.

A model outputs a probability, and a threshold turns it into a yes/no decision. The usual cut-off is 0.5, but it can be moved. A lower threshold catches more cancers at the cost of more false alarms, which suits screening; a higher one avoids unnecessary procedures, which suits confirmation before surgery. This section shows how that trade-off plays out.

Clinical Context for Threshold Selection

The default 0.5 classification threshold optimises for overall accuracy, but in clinical practice the threshold should reflect the cost asymmetry between error types:

• Screening context (community-level testing): A false negative (missed cancer) has severe consequences — delayed treatment. We lower the threshold to 0.3, accepting more false positives (benign recalls for follow-up) to catch virtually all malignancies. The motto: "When in doubt, flag it."

• Confirmatory context (specialist review before invasive procedure): A false positive causes unnecessary biopsy with associated risk and cost. A higher threshold (e.g., 0.6–0.7) may be appropriate to increase specificity.

We analyse how precision and recall trade off as the threshold varies, and demonstrate the effect of a clinically motivated threshold of 0.3 on the confusion matrix.

=== Logistic Regression ===

Default threshold (0.5):
  Accuracy: 0.9649
  Precision: 0.9750
  Recall: 0.9286
  F1: 0.9512
  ROC-AUC: 0.9960

Screening threshold (0.3):
  Accuracy: 0.9825
  Precision: 0.9762
  Recall: 0.9762
  F1: 0.9762
  ROC-AUC: 0.9960

At threshold 0.3: Recall improves at the cost of slightly lower Precision.
Recommended for population screening to minimise missed malignancies.

A good score is not enough in medicine: clinicians need to know why a model makes a decision. SHAP is a method that measures how much each feature pushed a given prediction towards “malignant” or “benign”. Averaged across patients, it ranks which measurements the model relies on most, and these can be checked against what pathologists already know.

To understand which features drive predictions, we use SHAP (SHapley Additive exPlanations), a game-theoretic framework that assigns each feature a contribution to the model's output for every prediction.

Why SHAP?

SHAP values are model-agnostic in principle but have efficient implementations for tree-based models (TreeExplainer) and linear models (LinearExplainer). They satisfy important axioms (local accuracy, missingness, consistency) that make them a theoretically grounded interpretability method.

Clinical Interpretation

Features with large mean |SHAP| values are the most influential predictors of malignancy. We expect concave_points_worst, radius_worst, and perimeter_worst to dominate — larger, more irregular nuclei are hallmarks of malignant cells in cytology.

Computing SHAP values for: Logistic Regression
SHAP values shape: (114, 30)

SHAP beeswarm: each dot is one test sample. Red = high feature value, blue = low.
Positive SHAP value -> pushes prediction toward Malignant.

Top 10 features by mean |SHAP| value:
texture_worst           1.2507
radius_se               0.7611
symmetry_worst          0.7406
concave points_mean     0.7228
radius_worst            0.7125
concavity_worst         0.6937
compactness_se          0.6887
area_worst              0.6846
concave points_worst    0.5749
perimeter_worst         0.5614

Clinical interpretation:
  - concave_points_worst / concavity_worst: measure nuclear irregularity — higher in malignant cells
  - radius_worst / perimeter_worst / area_worst: nuclear size — malignant nuclei are larger
  - texture_worst: nuclear texture (standard deviation of grayscale values) — coarser in cancer
These findings are consistent with established cytological criteria for malignancy.

This section brings the results together: which model performed best, why a simple model can beat more complex ones on this kind of data, and what the findings do and do not mean. The dataset is small and comes from a single source, so the results show a sound method rather than a tool ready for clinical use.

Key Findings

All four classifiers achieved strong performance on this dataset, reflecting the high signal-to-noise ratio of the WDBC features. The complete results are summarised below:

Why Logistic Regression Wins

Logistic Regression achieved the highest ROC-AUC (0.9960), surpassing even tuned ensemble methods. This outcome is explained by the dataset's characteristics:

1. Near-linear separability: The pairplot and EDA showed that malignant and benign samples are largely linearly separable in the feature space — particularly in concave_points_mean vs radius_mean. Logistic Regression is optimally designed for this structure.

2. L2 regularisation handles multicollinearity: Many features are highly correlated (e.g., radius, perimeter, area). Logistic Regression's L2 penalty effectively distributes weight across correlated features, resulting in stable and well-calibrated probability estimates — which directly improves AUC.

3. Sample size: With only 455 training samples, complex models like Gradient Boosting can overfit the training distribution. Logistic Regression's simplicity is a virtue at this scale.

4. Well-calibrated probabilities: Tree ensembles produce probabilities that are often poorly calibrated (scores cluster near 0 and 1). Logistic Regression produces better-calibrated scores, leading to a smoother and higher ROC curve.

Threshold choice

In this dataset, a threshold of 0.3 maximises Recall (catching more malignancies at the cost of some additional false positives), the trade-off a screening setting would favour. In a confirmatory / pre-surgical setting, 0.5 or higher would reduce false positives and help minimise unnecessary procedures.

Most Predictive Features (SHAP)

The SHAP analysis confirms that concave_points_worst, radius_worst, and perimeter_worst are the strongest predictors of malignancy. These correspond directly to well-established cytological criteria: malignant cells have larger, more irregular, and more concave nuclei.

Limitations

• Dataset size: 569 samples is relatively small; larger multi-site datasets are needed before clinical deployment.
• Single modality: The dataset is based solely on FNA measurements; integration with imaging (mammography, ultrasound) and clinical features would improve diagnostic value.
• No prospective validation: Results are based on a retrospective dataset; prospective validation is required.
• Class imbalance: While moderate (63/37), a real-world screening population would have a much lower malignancy prevalence, which would affect precision substantially.

Next Steps

1. Calibration: Apply Platt scaling or isotonic regression to improve probability calibration for the ensemble models.
2. Cross-dataset validation: Validate on independent FNA datasets from different institutions.
3. Feature engineering: Explore ratios and polynomial interactions between nuclear measurements.
4. Cost-sensitive learning: Incorporate explicit misclassification costs (e.g., 5x cost for false negatives) directly into model training.
5. Fairness analysis: Examine whether model performance is consistent across demographic subgroups.

============================================================
FINAL RESULTS SUMMARY
============================================================
Best model: Logistic Regression
Test set metrics:
  Accuracy: 0.9649
  Precision: 0.9750
  Recall: 0.9286
  F1: 0.9512
  ROC-AUC: 0.9960

All outputs saved to outputs/plots/ and outputs/models/
Best model pickle: outputs/models/best_model.pkl
============================================================