Overview
CatBoost Intrusion Detection Model is a final research requirement for our machine learning in cybersecurity course. Our research was about Categorical Boosting and whether its native categorical feature handling will help with dataset generalization.
We trained models with the UNSW-NB15 and CICIoT2023 datasets, which represent traditional network attacks and Internet of Things (IoT) attacks, respectively. Because the datasets contained different feature schemas, we aligned the datasets with overlapping features which mostly consisted of basic network attributes.
Results showed that CatBoost achieved the strongest overall performance among the evaluated models. The feature alignment also enabled multiple ensemble models to generalize effectively despite the significant differences between the datasets.
Results
Models Trained on UNSW-NB15
| Model | Test: UNSW F1 | Test: CICIoT F1 | Combined F1 |
|---|---|---|---|
| CatBoost | 92.84% | 85.00% | 88.70% |
| Random Forest | 89.79% | 84.44% | 86.93% |
| Logistic Regression | 78.04% | 50.32% | 65.03% |
| Decision Tree | 92.23% | 58.09% | 76.48% |
| Gradient Boosting | 91.32% | 75.06% | 83.12% |
| SVM | 78.06% | 50.32% | 65.04% |
Models Trained on CICIoT2023
| Model | Test: UNSW F1 | Test: CICIoT F1 | Combined F1 |
|---|---|---|---|
| CatBoost | 71.91% | 99.29% | 86.11% |
| Random Forest | 68.21% | 99.07% | 84.35% |
| Logistic Regression | 64.20% | 91.82% | 77.60% |
| Decision Tree | 82.59% | 99.14% | 90.84% |
| Gradient Boosting | 79.98% | 99.09% | 89.47% |
| SVM | 64.23% | 94.10% | 78.90% |
Models Trained on the UNSW-NB15 + CICIoT2023
| Model | Test: UNSW F1 | Test: CICIoT F1 | Combined F1 |
|---|---|---|---|
| CatBoost | 92.78% | 99.28% | 96.10% |
| Random Forest | 90.26% | 99.04% | 94.78% |
| Logistic Regression | 78.54% | 89.58% | 84.07% |
| Decision Tree | 92.26% | 99.20% | 95.81% |
| Gradient Boosting | 90.64% | 99.09% | 94.98% |
| SVM | 78.61% | 89.93% | 84.30% |
My Contribution
I developed the models and feature engineered the two datasets.
Responsibilities included:
- Aligning the features of UNSW-NB15 and CICIoT2023
- Training, evaluating, and benchmarking CatBoost against baseline machine learning models
- Analyzing cross-dataset generalization performance and documenting research findings.
Technologies Used
- Python
- CatBoost
- Sci-kit learn
- Pandas
- NumPy
- MatPlotLib
- Seaborn
Challenges
The main challenge I encountered was the alignment of the two datasets. UNSW-NB15 and CICIoT2023 were designed for different environments and exposed substantially different feature sets. To conduct meaningful cross-dataset experiments. I reduced both datasets to just 12 features, 5 of which didnt exist on UNSW-NB15 and were derived from related features.
Key Takeaways
To my surprise, the feature alignment/engineering of the two datasets actually retained a lot of the necessary information for intrusion detection. I also learned that CatBoost is a very powerful model which requires minimal tuning to be performant.
Future Improvements
We only really tested the model with the two + concatenated datasets and a good next step would be to test it on real-world network and deploy it to an IDS pipeline.