AN OPTIMIZED XGBOOST FOR FALSE POSITIVE REDUCTION IN A NETWORK INTRUSION DETECTION

Abstract

Cybersecurity operations are increasingly challenged by large volume of false alerts produced by Intrusion Detection Systems (IDS) which leads to analyst fatigue and increases the likelihood of missing real threats. This study proposes an optimized eXtreme Gradient Boosting (XGBoost) model designed to reduce false positives and improve operational reliability of IDS using University of New South Wales-Network Intrusion Detection System-15 (UNSW-NB15) dataset for validation of the model. The optimization included systematic hyperparameter tuning of key parameters such as learning rate, maximum tree depth, gamma, subsampling ratio, and L1/L2 regularization to balance model complexity and generalization. The performance of the model was evaluated against reproduced benchmark ensemble classifier under identical conditions. The benchmark achieved False Positive Rate (FPR) of 17.69%, while the proposed XGBoost model reduced it to 5.85%, representing a 66.9% improvement and 2,925 fewer false alerts on the test set. In real world deployment, this substantial deduction would significantly lower alert fatigue and enable timely and effective responses to genuine attacks. The most significant gain was observed in the classification of legitimate “Normal” traffic where the FPR decreased from 9.22% in the benchmark model to 0.12%. The results demonstrate that a single well-tuned XGBoost model can provide high accuracy (94.15%) while substantially improving operational dependability. This study shows that prioritizing false positive reduction offers a practical path toward building deployment-ready IDS solutions. The novelty of this research is in its emphasis on minimizing the false positive rate (FPR) over accuracy as the main performance metric.