[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"tool-Western-OC2-Lab--Intrusion-Detection-System-Using-Machine-Learning":3,"similar-Western-OC2-Lab--Intrusion-Detection-System-Using-Machine-Learning":69},{"id":4,"github_repo":5,"name":6,"description_en":7,"description_zh":8,"ai_summary_zh":8,"readme_en":9,"readme_zh":10,"quickstart_zh":11,"use_case_zh":12,"hero_image_url":13,"owner_login":14,"owner_name":15,"owner_avatar_url":16,"owner_bio":17,"owner_company":18,"owner_location":18,"owner_email":18,"owner_twitter":18,"owner_website":19,"owner_url":20,"languages":21,"stars":26,"forks":27,"last_commit_at":28,"license":29,"difficulty_score":30,"env_os":31,"env_gpu":31,"env_ram":31,"env_deps":32,"category_tags":44,"github_topics":47,"view_count":30,"oss_zip_url":18,"oss_zip_packed_at":18,"status":64,"created_at":65,"updated_at":66,"faqs":67,"releases":68},5146,"Western-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning","Intrusion-Detection-System-Using-Machine-Learning","Code for IDS-ML: intrusion detection system development using machine learning algorithms (Decision tree, random forest, extra trees, XGBoost, stacking, k-means, Bayesian optimization..)","Intrusion-Detection-System-Using-Machine-Learning 是一个专注于利用机器学习算法开发入侵检测系统（IDS）的开源项目。它旨在解决车联网（IoV）及自动驾驶网络中面临的拒绝服务、欺骗和嗅探等网络安全威胁，帮助开发者构建能够高效识别异常流量和各类网络攻击的智能防御模型。\n\n该项目非常适合网络安全研究人员、数据科学家以及从事智能交通系统开发的工程师使用。其核心亮点在于集成了多种先进的机器学习技术：不仅涵盖了决策树、随机森林、XGBoost、LightGBM 等基于树的算法，还引入了 K-means 无监督学习、Stacking 集成学习以及自定义的 LCCDE 框架。此外，项目特别结合了贝叶斯优化技术进行超参数调优，在确保高检测率的同时有效降低了计算成本。作为一系列已发表学术论文的代码实现，它为构建通用型入侵检测和异常发现应用提供了坚实的技术基础和参考范例。","# Intrusion-Detection-System-Using-Machine-Learning\n\nThis repository contains the code for the project \"IDS-ML: Intrusion Detection System Development Using Machine Learning\". The code and proposed Intrusion Detection System (IDSs) are general models that can be used in any IDS and anomaly detection applications. In this project, three papers have been published:  \n* L. Yang, A. Moubayed, I. Hamieh and A. Shami, \"[Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.08635.pdf),\" in 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1-6, doi: 10.1109\u002FGLOBECOM38437.2019.9013892.  \n* L. Yang, A. Moubayed, and A. Shami, “[MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2105.13289.pdf),” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 616-632, Jan.1, 2022, doi: 10.1109\u002FJIOT.2021.3084796.\n* L. Yang, A. Shami, G. Stevens, and S. DeRusett, “[LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2208.03399.pdf),\" in 2022 IEEE Global Communications Conference (GLOBECOM), 2022, pp. 1-6, doi: 10.1109\u002FGLOBECOM48099.2022.10001280.\n\n\nThe code introduction of this repository is publicly available at:  \n* L. Yang, and A. Shami, “[IDS-ML: An open source code for Intrusion Detection System development using Machine Learning](https:\u002F\u002Fwww.sciencedirect.com\u002Fscience\u002Farticle\u002Fpii\u002FS2665963822001300),\" Software Impacts, vol. 14, pp. 1-4, 2022, doi: 10.1016\u002Fj.simpa.2022.100446.\n\nThis repository proposed three **intrusion detection systems** by implementing many **machine learning** algorithms, including tree-based algorithms (**decision tree, random forest, XGBoost, LightGBM, CatBoost etc.**), unsupervised learning algorithms (**k-means**), ensemble learning algorithms (**stacking, proposed LCCDE**), and hyperparameter optimization techniques (**Bayesian optimization**)**.\n\n- Another **intrusion detection system development code** using **convolutional neural networks (CNNs)** and **transfer learning** techniques can be found in: [Intrusion-Detection-System-Using-CNN-and-Transfer-Learning](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-CNN-and-Transfer-Learning)\n\n- A comprehensive **hyperparameter optimization** tutorial code can be found in: [Hyperparameter-Optimization-of-Machine-Learning-Algorithms](https:\u002F\u002Fgithub.com\u002FLiYangHart\u002FHyperparameter-Optimization-of-Machine-Learning-Algorithms)\n\n\n## Paper Abstract\n### Paper 1:  Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles\n  The use of autonomous vehicles (AVs) is a promising technology in Intelligent Transportation Systems (ITSs) to improve safety and driving efficiency. Vehicle-to-everything (V2X) technology enables communication among vehicles and other infrastructures. However, AVs and Internet of Vehicles (IoV) are vulnerable to different types of cyber-attacks such as denial of service, spoofing, and sniffing attacks. An intelligent IDS is proposed in this paper for network attack detection that can be applied to not only Controller Area Network (CAN) bus of AVs but also on general IoVs. The proposed IDS utilizes tree-based ML algorithms including decision tree (DT), random forest (RF), extra trees (ET), and Extreme Gradient Boosting (XGBoost). The results from the implementation of the proposed intrusion detection system on standard data sets indicate that the system has the ability to identify various cyber-attacks in the AV networks. Furthermore, the proposed ensemble learning and feature selection approaches enable the proposed system to achieve high detection rate and low computational cost simultaneously.\n\n**\u003Cp align=\"center\">Figure 1: The overview of the tree-based IDS model.\u003C\u002Fp>**\n\u003Cp align=\"center\">\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_readme_ccd787f96fed.jpg\" width=\"280\" \u002F>\n\u003C\u002Fp>\n\n### Paper 2:  MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles\n  Modern vehicles, including connected vehicles and autonomous vehicles, nowadays involve many electronic control units connected through intra-vehicle networks to implement various functionalities and perform actions. Modern vehicles are also connected to external networks through vehicle-to-everything technologies, enabling their communications with other vehicles, infrastructures, and smart devices. However, the improving functionality and connectivity of modern vehicles also increase their vulnerabilities to cyber-attacks targeting both intra-vehicle and external networks due to the large attack surfaces. To secure vehicular networks, many researchers have focused on developing intrusion detection systems (IDSs) that capitalize on machine learning methods to detect malicious cyber-attacks. In this paper, the vulnerabilities of intra-vehicle and external networks are discussed, and a multi-tiered hybrid IDS that incorporates a signature-based IDS and an anomaly-based IDS is proposed to detect both known and unknown attacks on vehicular networks. Experimental results illustrate that the proposed system can accurately detect various types of known attacks on the CAN-intrusion-dataset representing the intra-vehicle network data and the CICIDS2017 dataset illustrating the external vehicular network data.  \n  The proposed MTH-IDS framework consists of two traditional ML stages (data pre-processing and feature engineering) and four tiers of learning models: \n1. Four tree-based supervised learners — decision tree (DT), random forest (RF), extra trees (ET), and extreme gradient boosting (XGBoost) — used as multi-class classifiers for known attack detection; \n2. A stacking ensemble model and a Bayesian optimization with tree Parzen estimator (BO-TPE) method for supervised learner optimization; \n3. A cluster labeling (CL) k-means used as an unsupervised learner for zero-day attack detection; \n4. Two biased classifiers and a Bayesian optimization with Gaussian process (BO-GP) method for unsupervised learner optimization. \n\n**\u003Cp align=\"center\">Figure 2: The overview of the MTH-IDS model.\u003C\u002Fp>**\n\u003Cp align=\"center\">\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_readme_deca41da326e.png\" width=\"700\" \u002F>\n\u003C\u002Fp>\n\n\n### Paper 3:  LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles\n  Modern vehicles, including autonomous vehicles and connected vehicles, have adopted an increasing variety of functionalities through connections and communications with other vehicles, smart devices, and infrastructures. However, the growing connectivity of the Internet of Vehicles (IoV) also increases the vulnerabilities to network attacks. To protect IoV systems against cyber threats, Intrusion Detection Systems (IDSs) that can identify malicious cyber-attacks have been developed using Machine Learning (ML) approaches. To accurately detect various types of attacks in IoV networks, we propose a novel ensemble IDS framework named Leader Class and Confidence Decision Ensemble (LCCDE). It is constructed by determining the best-performing ML model among three advanced ML algorithms (XGBoost, LightGBM, and CatBoost) for every class or type of attack. The class leader models with their prediction confidence values are then utilized to make accurate decisions regarding the detection of various types of cyber-attacks. Experiments on two public IoV security datasets (Car-Hacking and CICIDS2017 datasets) demonstrate the effectiveness of the proposed LCCDE for intrusion detection on both intra-vehicle and external networks. \n\n**\u003Cp align=\"center\">Figure 3: The overview of the LCCCDE IDS model.\u003C\u002Fp>**\n\u003Cp align=\"center\">\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_readme_43a10d8127c9.jpg\" width=\"800\" \u002F>\n\u003C\u002Fp>\n\n\n## Implementation \n### Dataset \nCICIDS2017 dataset, a popular network traffic dataset for intrusion detection problems\n* Publicly available at: https:\u002F\u002Fwww.unb.ca\u002Fcic\u002Fdatasets\u002Fids-2017.html  \n* For the purpose of displaying the experimental results in Jupyter Notebook, the sampled subsets of CICIDS2017 is used in the sample code. The subsets are in the \"data\" folder.\n\nCAN-intrusion dataset, a benchmark network security dataset for intra-vehicle intrusion detection\n* Publicly available at: https:\u002F\u002Focslab.hksecurity.net\u002FDatasets\u002FCAN-intrusion-dataset  \n* Can be processed using the same code\n\n### Code  \n* [Tree-based_IDS_GlobeCom19.ipynb](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning\u002Fblob\u002Fmain\u002FTree-based_IDS_GlobeCom19.ipynb): code for the paper \"Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles\"  \n* [MTH_IDS_IoTJ.ipynb](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning\u002Fblob\u002Fmain\u002FMTH_IDS_IoTJ.ipynb): code for the paper \"MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles\"  \n* [LCCDE_IDS_GlobeCom22.ipynb](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning\u002Fblob\u002Fmain\u002FLCCDE_IDS_GlobeCom22.ipynb): code for the paper \"LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles\"  \n\n### Machine Learning Algorithms  \n* Decision tree (DT)\n* Random forest (RF)\n* Extra trees (ET)\n* XGBoost  \n* LightGBM  \n* CatBoost  \n* Stacking\n* K-means\n\n### Hyperparameter Optimization Methods  \n* Bayesian Optimization with Gaussian Processes (BO-GP)\n* Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)  \n\nIf you are interested in hyperparameter tuning of machine learning algorithms, please see the code in the following link:  \nhttps:\u002F\u002Fgithub.com\u002FLiYangHart\u002FHyperparameter-Optimization-of-Machine-Learning-Algorithms\n\n### Requirements & Libraries  \n* Python 3.6+ \n* [scikit-learn](https:\u002F\u002Fscikit-learn.org\u002Fstable\u002F)  \n* [Xgboost](https:\u002F\u002Fxgboost.readthedocs.io\u002Fen\u002Flatest\u002Fpython\u002Fpython_intro.html)\n* [lightgbm](https:\u002F\u002Flightgbm.readthedocs.io\u002Fen\u002Fv3.3.2\u002FPython-Intro.html)\n* [catboost](https:\u002F\u002Fxgboost.readthedocs.io\u002Fen\u002Flatest\u002Fpython\u002Fpython_intro.html)\n* [FCBF](https:\u002F\u002Fgithub.com\u002FSantiagoEG\u002FFCBF_module)\n* [scikit-optimize](https:\u002F\u002Fgithub.com\u002Fscikit-optimize\u002Fscikit-optimize)  \n* [hyperopt](https:\u002F\u002Fgithub.com\u002Fhyperopt\u002Fhyperopt)   \n* [River](https:\u002F\u002Friverml.xyz\u002Fdev\u002F)  \n\n## Contact-Info\nPlease feel free to contact us for any questions or cooperation opportunities. We will be happy to help.\n* Email: [liyanghart@gmail.com](mailto:liyanghart@gmail.com)\n* GitHub: [LiYangHart](https:\u002F\u002Fgithub.com\u002FLiYangHart) and [Western OC2 Lab](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002F)\n* LinkedIn: [Li Yang](https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fli-yang-phd-65a190176\u002F)  \n* Google Scholar: [Li Yang](https:\u002F\u002Fscholar.google.com.eg\u002Fcitations?user=XEfM7bIAAAAJ&hl=en) and [OC2 Lab](https:\u002F\u002Fscholar.google.com.eg\u002Fcitations?user=oiebNboAAAAJ&hl=en)\n\n## Citation\nIf you find this repository useful in your research, please cite one of the following two articles as:  \n\nL. Yang, A. Moubayed, I. Hamieh and A. Shami, \"Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles,\" 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1-6, doi: 10.1109\u002FGLOBECOM38437.2019.9013892.  \n```\n@INPROCEEDINGS{9013892,\n  author={Yang, Li and Moubayed, Abdallah and Hamieh, Ismail and Shami, Abdallah},\n  booktitle={2019 IEEE Global Communications Conference (GLOBECOM)}, \n  title={Tree-Based Intelligent Intrusion Detection System in Internet of Vehicles}, \n  year={2019},\n  pages={1-6},\n  doi={10.1109\u002FGLOBECOM38437.2019.9013892}\n  }\n```\n\nL. Yang, A. Moubayed, and A. Shami, “MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles,” IEEE Internet of Things Journal, vol. 9, no. 1, pp. 616-632, Jan.1, 2022, doi: 10.1109\u002FJIOT.2021.3084796.\n```\n@ARTICLE{9443234,\n  author={Yang, Li and Moubayed, Abdallah and Shami, Abdallah},\n  journal={IEEE Internet of Things Journal}, \n  title={MTH-IDS: A Multitiered Hybrid Intrusion Detection System for Internet of Vehicles}, \n  year={2022},\n  volume={9},\n  number={1},\n  pages={616-632},\n  doi={10.1109\u002FJIOT.2021.3084796}}\n```\n\nL. Yang, A. Shami, G. Stevens, and S. DeRusett, “LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles,\" in 2022 IEEE Global Communications Conference (GLOBECOM), 2022, pp. 1-6, doi: 10.1109\u002FGLOBECOM48099.2022.10001280.\n```\n@INPROCEEDINGS{10001280,\n  author={Yang, Li and Shami, Abdallah and Stevens, Gary and de Rusett, Stephen},\n  booktitle={GLOBECOM 2022 - 2022 IEEE Global Communications Conference}, \n  title={LCCDE: A Decision-Based Ensemble Framework for Intrusion Detection in The Internet of Vehicles}, \n  year={2022},\n  pages={3545-3550},\n  doi={10.1109\u002FGLOBECOM48099.2022.10001280}}\n```\n","# 基于机器学习的入侵检测系统\n\n本仓库包含项目“IDS-ML：基于机器学习的入侵检测系统开发”的代码。该代码及所提出的入侵检测系统（IDS）是通用模型，可用于任何IDS和异常检测应用。在本项目中，已发表三篇论文：\n\n* L. Yang, A. Moubayed, I. Hamieh 和 A. Shami，《基于树的车联网智能入侵检测系统》（[arXiv预印本](https:\u002F\u002Farxiv.org\u002Fpdf\u002F1910.08635.pdf)），2019年IEEE全球通信大会（GLOBECOM），2019年，第1–6页，doi: 10.1109\u002FGLOBECOM38437.2019.9013892。\n* L. Yang, A. Moubayed 和 A. Shami，《MTH-IDS：面向车联网的多层混合入侵检测系统》（[arXiv预印本](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2105.13289.pdf)），IEEE物联网期刊，第9卷，第1期，第616–632页，2022年1月1日，doi: 10.1109\u002FJIOT.2021.3084796。\n* L. Yang, A. Shami、G. Stevens 和 S. DeRusett，《LCCDE：基于决策的集成框架用于车联网入侵检测》（[arXiv预印本](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2208.03399.pdf)），2022年IEEE全球通信大会（GLOBECOM），2022年，第1–6页，doi: 10.1109\u002FGLOBECOM48099.2022.10001280。\n\n本仓库的代码介绍已在公开文献中发布：\n* L. Yang 和 A. Shami，《IDS-ML：基于机器学习的入侵检测系统开源代码》（[ScienceDirect文章](https:\u002F\u002Fwww.sciencedirect.com\u002Fscience\u002Farticle\u002Fpii\u002FS2665963822001300)），Software Impacts，第14卷，第1–4页，2022年，doi: 10.1016\u002Fj.simpa.2022.100446。\n\n本仓库通过实现多种机器学习算法，提出了三种入侵检测系统，包括基于树的算法（决策树、随机森林、XGBoost、LightGBM、CatBoost等）、无监督学习算法（k-means）、集成学习算法（堆叠、提出的LCCDE）以及超参数优化技术（贝叶斯优化）。\n\n- 另一种使用卷积神经网络（CNN）和迁移学习技术的入侵检测系统开发代码可在以下链接找到：[Intrusion-Detection-System-Using-CNN-and-Transfer-Learning](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-CNN-and-Transfer-Learning)\n\n- 一份全面的超参数优化教程代码可在以下链接找到：[Hyperparameter-Optimization-of-Machine-Learning-Algorithms](https:\u002F\u002Fgithub.com\u002FLiYangHart\u002FHyperparameter-Optimization-of-Machine-Learning-Algorithms)\n\n## 论文摘要\n### 论文1：基于树的车联网智能入侵检测系统\n  自动驾驶汽车（AVs）作为智能交通系统（ITSs）中的前沿技术，有望提升交通安全与驾驶效率。车辆到万物（V2X）通信技术使车辆与其他基础设施之间能够相互通信。然而，自动驾驶汽车和车联网（IoV）容易遭受拒绝服务攻击、欺骗攻击和嗅探攻击等多种网络攻击。本文提出了一种智能IDS，用于检测网络攻击，不仅可应用于自动驾驶汽车的控制器局域网（CAN）总线，也可推广至一般车联网环境。该IDS采用基于树的机器学习算法，包括决策树（DT）、随机森林（RF）、Extra Trees（ET）和极端梯度提升（XGBoost）。在标准数据集上实施该入侵检测系统的结果表明，该系统能够识别自动驾驶汽车网络中的各类网络攻击。此外，所提出的集成学习和特征选择方法使系统能够在高检测率的同时保持较低的计算开销。\n\n**\u003Cp align=\"center\">图1：基于树的IDS模型概览。\u003C\u002Fp>**\n\u003Cp align=\"center\">\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_readme_ccd787f96fed.jpg\" width=\"280\" \u002F>\n\u003C\u002Fp>\n\n### 论文2：MTH-IDS：面向车联网的多层混合入侵检测系统\n  现代车辆，包括联网汽车和自动驾驶汽车，通常配备多个电子控制单元，并通过车载网络连接以实现各种功能和操作。同时，现代车辆还通过V2X技术接入外部网络，从而与其他车辆、基础设施和智能设备进行通信。然而，随着车辆功能和连通性的不断增强，其针对车载网络和外部网络的攻击面也在扩大，使其更容易受到网络攻击。为保障车载网络的安全，许多研究人员致力于开发基于机器学习的入侵检测系统（IDS），以识别恶意网络攻击。本文探讨了车载网络和外部网络的脆弱性，并提出了一种多层混合IDS，结合签名检测和异常检测两种机制，以同时检测已知和未知的网络攻击。实验结果表明，该系统能够准确检测代表车载网络数据的CAN-Intrusion数据集以及反映外部车联网数据的CICIDS2017数据集中存在的各类已知攻击。\n  所提出的MTH-IDS框架由两个传统的机器学习阶段（数据预处理和特征工程）以及四个层次的学习模型组成：\n1. 四个基于树的监督学习器——决策树（DT）、随机森林（RF）、Extra Trees（ET）和极端梯度提升（XGBoost）——用作多分类器来检测已知攻击；\n2. 一个堆叠集成模型和基于树型帕尔森估计器的贝叶斯优化（BO-TPE）方法，用于优化监督学习器；\n3. 一个聚类标签（CL）k-means，用作无监督学习器来检测零日攻击；\n4. 两个偏置分类器和基于高斯过程的贝叶斯优化（BO-GP）方法，用于优化无监督学习器。\n\n**\u003Cp align=\"center\">图2：MTH-IDS模型概览。\u003C\u002Fp>**\n\u003Cp align=\"center\">\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_readme_deca41da326e.png\" width=\"700\" \u002F>\n\u003C\u002Fp>\n\n### 论文3：LCCDE——一种基于决策的集成框架，用于车联网入侵检测\n  现代车辆，包括自动驾驶汽车和联网汽车，通过与其他车辆、智能设备及基础设施的连接与通信，不断引入越来越多的功能。然而，车联网（IoV）日益增强的连通性也使其更容易受到网络攻击的威胁。为了保护车联网系统免受网络威胁，研究人员利用机器学习（ML）方法开发了能够识别恶意网络攻击的入侵检测系统（IDS）。为准确检测车联网网络中的各类攻击，我们提出了一种名为“领导者分类与置信度决策集成”（LCCDE）的新型集成IDS框架。该框架通过在三种先进的机器学习算法（XGBoost、LightGBM和CatBoost）中，为每类或每种类型的攻击确定表现最佳的模型来构建。随后，利用这些类别领导者模型及其预测置信度值，对各类网络攻击的检测做出准确决策。我们在两个公开的车联网安全数据集（Car-Hacking和CICIDS2017数据集）上进行的实验表明，所提出的LCCDE框架在车载内部网络和外部网络的入侵检测方面均具有显著效果。\n\n**\u003Cp align=\"center\">图3：LCCDE IDS模型的总体架构。\u003C\u002Fp>**\n\u003Cp align=\"center\">\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_readme_43a10d8127c9.jpg\" width=\"800\" \u002F>\n\u003C\u002Fp>\n\n\n## 实现 \n### 数据集 \nCICIDS2017数据集，一个广泛用于入侵检测问题的网络流量数据集\n* 公开获取地址：https:\u002F\u002Fwww.unb.ca\u002Fcic\u002Fdatasets\u002Fids-2017.html  \n* 为便于在Jupyter Notebook中展示实验结果，示例代码中使用了CICIDS2017数据集的采样子集，这些子集位于“data”文件夹中。\n\nCAN-intrusion数据集，一个用于车载内部入侵检测的基准网络安全数据集\n* 公开获取地址：https:\u002F\u002Focslab.hksecurity.net\u002FDatasets\u002FCAN-intrusion-dataset  \n* 可以使用相同的代码进行处理\n\n### 代码  \n* [Tree-based_IDS_GlobeCom19.ipynb](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning\u002Fblob\u002Fmain\u002FTree-based_IDS_GlobeCom19.ipynb)：论文《基于树结构的车联网智能入侵检测系统》的代码  \n* [MTH_IDS_IoTJ.ipynb](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning\u002Fblob\u002Fmain\u002FMTH_IDS_IoTJ.ipynb)：论文《MTH-IDS：一种面向车联网的多层混合入侵检测系统》的代码  \n* [LCCDE_IDS_GlobeCom22.ipynb](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning\u002Fblob\u002Fmain\u002FLCCDE_IDS_GlobeCom22.ipynb)：论文《LCCDE：一种基于决策的集成框架，用于车联网入侵检测》的代码  \n\n### 机器学习算法  \n* 决策树（DT）\n* 随机森林（RF）\n* 极端随机树（ET）\n* XGBoost  \n* LightGBM  \n* CatBoost  \n* 堆叠集成\n* K均值聚类\n\n### 超参数优化方法  \n* 基于高斯过程的贝叶斯优化（BO-GP）\n* 基于树结构Parzen估计器的贝叶斯优化（BO-TPE）  \n\n如果您对机器学习算法的超参数调优感兴趣，请参阅以下链接中的代码：  \nhttps:\u002F\u002Fgithub.com\u002FLiYangHart\u002FHyperparameter-Optimization-of-Machine-Learning-Algorithms\n\n### 环境要求与库  \n* Python 3.6及以上版本  \n* [scikit-learn](https:\u002F\u002Fscikit-learn.org\u002Fstable\u002F)  \n* [Xgboost](https:\u002F\u002Fxgboost.readthedocs.io\u002Fen\u002Flatest\u002Fpython\u002Fpython_intro.html)\n* [lightgbm](https:\u002F\u002Flightgbm.readthedocs.io\u002Fen\u002Fv3.3.2\u002FPython-Intro.html)\n* [catboost](https:\u002F\u002Fxgboost.readthedocs.io\u002Fen\u002Flatest\u002Fpython\u002Fpython_intro.html)\n* [FCBF](https:\u002F\u002Fgithub.com\u002FSantiagoEG\u002FFCBF_module)\n* [scikit-optimize](https:\u002F\u002Fgithub.com\u002Fscikit-optimize\u002Fscikit-optimize)  \n* [hyperopt](https:\u002F\u002Fgithub.com\u002Fhyperopt\u002Fhyperopt)   \n* [River](https:\u002F\u002Friverml.xyz\u002Fdev\u002F)  \n\n## 联系方式\n如您有任何问题或合作意向，欢迎随时与我们联系。我们将竭诚为您提供帮助。\n* 邮箱：[liyanghart@gmail.com](mailto:liyanghart@gmail.com)\n* GitHub：[LiYangHart](https:\u002F\u002Fgithub.com\u002FLiYangHart) 和 [Western OC2 Lab](https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002F)\n* LinkedIn：[Li Yang](https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fli-yang-phd-65a190176\u002F)  \n* Google学术：[Li Yang](https:\u002F\u002Fscholar.google.com.eg\u002Fcitations?user=XEfM7bIAAAAJ&hl=en) 和 [OC2 Lab](https:\u002F\u002Fscholar.google.com.eg\u002Fcitations?user=oiebNboAAAAJ&hl=en)\n\n## 引用\n如果您在研究中使用了本仓库中的内容，请引用以下两篇文章之一：\n\nL. Yang, A. Moubayed, I. Hamieh 和 A. Shami，“基于树结构的车联网智能入侵检测系统”，2019 IEEE全球通信大会（GLOBECOM），2019年，第1–6页，doi：10.1109\u002FGLOBECOM38437.2019.9013892。  \n```\n@INPROCEEDINGS{9013892,\n  author={Yang, Li and Moubayed, Abdallah and Hamieh, Ismail and Shami, Abdallah},\n  booktitle={2019 IEEE全球通信大会（GLOBECOM）}, \n  title={基于树结构的车联网智能入侵检测系统}, \n  year={2019},\n  pages={1-6},\n  doi={10.1109\u002FGLOBECOM38437.2019.9013892}\n  }\n```\n\nL. Yang, A. Moubayed 和 A. Shami，“MTH-IDS：一种面向车联网的多层混合入侵检测系统”，IEEE物联网期刊，第9卷第1期，第616–632页，2022年1月1日，doi：10.1109\u002FJIOT.2021.3084796。  \n```\n@ARTICLE{9443234,\n  author={Yang, Li and Moubayed, Abdallah and Shami, Abdallah},\n  journal={IEEE物联网期刊}, \n  title={MTH-IDS：一种多层混合车联网入侵检测系统}, \n  year={2022},\n  volume={9},\n  number={1},\n  pages={616-632},\n  doi={10.1109\u002FJIOT.2021.3084796}}\n```\n\nL. Yang, A. Shami、G. Stevens 和 S. DeRusett，“LCCDE：一种基于决策的集成框架，用于车联网入侵检测”，2022 IEEE全球通信大会（GLOBECOM），2022年，第1–6页，doi：10.1109\u002FGLOBECOM48099.2022.10001280。  \n```\n@INPROCEEDINGS{10001280,\n  author={Yang, Li and Shami, Abdallah and Stevens, Gary and de Rusett, Stephen},\n  booktitle={GLOBECOM 2022 - 2022 IEEE全球通信大会}, \n  title={LCCDE：一种基于决策的集成框架，用于车联网入侵检测}, \n  year={2022},\n  pages={3545-3550},\n  doi={10.1109\u002FGLOBECOM48099.2022.10001280}}\n```","# Intrusion-Detection-System-Using-Machine-Learning 快速上手指南\n\n本项目提供了一套基于机器学习的入侵检测系统（IDS）开源代码，适用于车联网（IoV）及通用网络异常检测场景。项目实现了多种树模型、无监督学习及集成学习算法，并包含超参数优化方案。\n\n## 环境准备\n\n### 系统要求\n- **操作系统**：Linux, macOS, 或 Windows\n- **Python 版本**：3.6 及以上\n\n### 前置依赖\n本项目依赖以下核心 Python 库：\n- `scikit-learn`: 基础机器学习算法\n- `xgboost`, `lightgbm`, `catboost`: 高级树模型\n- `scikit-optimize`, `hyperopt`: 贝叶斯超参数优化\n- `FCBF`: 特征选择\n- `River`: 在线机器学习\n\n> **提示**：国内开发者建议使用清华或阿里镜像源加速安装。\n\n## 安装步骤\n\n1. **克隆仓库**\n   ```bash\n   git clone https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab\u002FIntrusion-Detection-System-Using-Machine-Learning.git\n   cd Intrusion-Detection-System-Using-Machine-Learning\n   ```\n\n2. **安装依赖包**\n   推荐使用国内镜像源安装所需库：\n   ```bash\n   pip install -r requirements.txt -i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple\n   ```\n   \n   *若项目中未提供 `requirements.txt`，请手动执行以下命令安装核心依赖：*\n   ```bash\n   pip install scikit-learn xgboost lightgbm catboost scikit-optimize hyperopt river -i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple\n   ```\n   \n   *注：`FCBF` 模块可能需要单独从 GitHub 安装：*\n   ```bash\n   pip install git+https:\u002F\u002Fgithub.com\u002FSantiagoEG\u002FFCBF_module.git\n   ```\n\n3. **准备数据集**\n   项目主要使用以下两个数据集，代码中已包含采样子集用于演示：\n   - **CICIDS2017** (外部网络攻击): 数据位于 `data` 文件夹（采样版）。完整数据集可从 [UNB 官网](https:\u002F\u002Fwww.unb.ca\u002Fcic\u002Fdatasets\u002Fids-2017.html) 下载。\n   - **CAN-intrusion** (车内网络攻击): 可从 [OCSLab](https:\u002F\u002Focslab.hksecurity.net\u002FDatasets\u002FCAN-intrusion-dataset) 获取，处理逻辑与 CICIDS2017 类似。\n\n## 基本使用\n\n项目提供了三个主要的 Jupyter Notebook 文件，分别对应三篇学术论文的实现。您可以根据需求选择运行。\n\n### 示例：运行基于树的智能 IDS (Paper 1)\n此脚本演示了如何使用决策树、随机森林、XGBoost 等算法进行攻击检测。\n\n1. 启动 Jupyter Notebook：\n   ```bash\n   jupyter notebook\n   ```\n\n2. 在浏览器中打开并运行以下文件：\n   - `Tree-based_IDS_GlobeCom19.ipynb`\n\n   **代码逻辑简述**：\n   该 Notebook 将自动加载 `data` 文件夹中的采样数据，执行数据预处理、特征工程，并训练多种树模型进行评估。\n\n### 其他模型入口\n- **多层混合 IDS (MTH-IDS)**: 运行 `MTH_IDS_IoTJ.ipynb`。包含签名检测与异常检测的多层架构，适用于已知和未知攻击。\n- **LCCDE 集成框架**: 运行 `LCCDE_IDS_GlobeCom22.ipynb`。基于 XGBoost、LightGBM 和 CatBoost 的领导者类别与置信度决策集成框架。\n\n### 自定义训练\n若要使用完整数据集或调整模型，请在对应的 `.ipynb` 文件中修改数据加载路径（指向您下载的完整 CSV 文件），并根据需要调整超参数优化部分（已集成 BO-TPE 和 BO-GP 方法）。","某智能网联汽车研发团队正在构建车载网络（IoV）的安全防御体系，急需从海量 CAN 总线数据中实时识别拒绝服务、欺骗等新型网络攻击。\n\n### 没有 Intrusion-Detection-System-Using-Machine-Learning 时\n- **检测盲区多**：依赖传统规则匹配，无法有效识别未知的变种攻击或复杂的异常流量模式，导致漏报率高。\n- **开发周期长**：团队需从零编写决策树、随机森林等算法代码，并手动调试超参数，耗时数周才能产出原型。\n- **资源消耗大**：现有模型计算冗余度高，难以在车载嵌入式设备上运行，高延迟影响了行车安全的实时响应。\n- **准确率瓶颈**：缺乏集成学习（如 Stacking）和特征选择优化，单一模型在面对不平衡攻击数据时表现不稳定。\n\n### 使用 Intrusion-Detection-System-Using-Machine-Learning 后\n- **覆盖更全面**：直接调用内置的 XGBoost、Extra Trees 及无监督 K-means 算法，精准捕获各类已知与未知攻击，显著提升检出率。\n- **落地更高效**：复用经过论文验证的成熟代码框架，结合贝叶斯优化自动调参，将系统开发部署时间从数周缩短至几天。\n- **运行更轻量**：利用优化的树模型和特征选择技术，在保持高精度的同时大幅降低计算成本，完美适配车载边缘设备。\n- **性能更稳健**：通过 LCCDE 等集成框架整合多模型优势，有效解决数据不平衡问题，确保在复杂路况下的判断稳定性。\n\nIntrusion-Detection-System-Using-Machine-Learning 将原本繁琐的算法研发转化为高效的标准化流程，为智能交通系统提供了低成本、高精度的实时安全屏障。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FWestern-OC2-Lab_Intrusion-Detection-System-Using-Machine-Learning_e20d40e1.png","Western-OC2-Lab","Western OC2 Lab","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002FWestern-OC2-Lab_2c0c06fc.png","The Optimized Computing and Communications (OC2) Laboratory within the Department of Electrical and Computer Engineering at Western University, London, Canada.",null,"https:\u002F\u002Fwww.eng.uwo.ca\u002Foc2\u002F","https:\u002F\u002Fgithub.com\u002FWestern-OC2-Lab",[22],{"name":23,"color":24,"percentage":25},"Jupyter Notebook","#DA5B0B",100,579,158,"2026-04-06T06:21:04","MIT",2,"未说明",{"notes":33,"python":34,"dependencies":35},"该项目主要基于传统机器学习算法（如决策树、随机森林、XGBoost 等）和集成学习，未提及需要 GPU 加速。代码以 Jupyter Notebook 形式提供，使用了 CICIDS2017 和 CAN-intrusion 数据集。其中 FCBF 模块需从特定 GitHub 仓库安装。若需进行超参数优化，可参考作者提供的额外教程仓库。","3.6+",[36,37,38,39,40,41,42,43],"scikit-learn","xgboost","lightgbm","catboost","FCBF","scikit-optimize","hyperopt","River",[45,46],"开发框架","Agent",[48,49,50,37,51,52,53,54,55,56,57,58,59,60,61,62,39,63,38],"machine-learning","random-forest","decision-tree","bayesian-optimization","hyperparameter-optimization","hpo","kmeans","python-examples","intrusion-detection","intrusion-detection-system","autonomous-vehicles","cyber-security","network-security","stacking","cicids2017","ensemble-learning","ready","2026-03-27T02:49:30.150509","2026-04-08T01:07:24.053870",[],[],[70,81,89,98,106,115],{"id":71,"name":72,"github_repo":73,"description_zh":74,"stars":75,"difficulty_score":76,"last_commit_at":77,"category_tags":78,"status":64},4358,"openclaw","openclaw\u002Fopenclaw","OpenClaw 是一款专为个人打造的本地化 AI 助手，旨在让你在自己的设备上拥有完全可控的智能伙伴。它打破了传统 AI 助手局限于特定网页或应用的束缚，能够直接接入你日常使用的各类通讯渠道，包括微信、WhatsApp、Telegram、Discord、iMessage 等数十种平台。无论你在哪个聊天软件中发送消息，OpenClaw 都能即时响应，甚至支持在 macOS、iOS 和 Android 设备上进行语音交互，并提供实时的画布渲染功能供你操控。\n\n这款工具主要解决了用户对数据隐私、响应速度以及“始终在线”体验的需求。通过将 AI 部署在本地，用户无需依赖云端服务即可享受快速、私密的智能辅助，真正实现了“你的数据，你做主”。其独特的技术亮点在于强大的网关架构，将控制平面与核心助手分离，确保跨平台通信的流畅性与扩展性。\n\nOpenClaw 非常适合希望构建个性化工作流的技术爱好者、开发者，以及注重隐私保护且不愿被单一生态绑定的普通用户。只要具备基础的终端操作能力（支持 macOS、Linux 及 Windows WSL2），即可通过简单的命令行引导完成部署。如果你渴望拥有一个懂你",349277,3,"2026-04-06T06:32:30",[46,45,79,80],"图像","数据工具",{"id":82,"name":83,"github_repo":84,"description_zh":85,"stars":86,"difficulty_score":76,"last_commit_at":87,"category_tags":88,"status":64},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,"2026-04-05T11:01:52",[45,79,46],{"id":90,"name":91,"github_repo":92,"description_zh":93,"stars":94,"difficulty_score":30,"last_commit_at":95,"category_tags":96,"status":64},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",143909,"2026-04-07T11:33:18",[45,46,97],"语言模型",{"id":99,"name":100,"github_repo":101,"description_zh":102,"stars":103,"difficulty_score":30,"last_commit_at":104,"category_tags":105,"status":64},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",107888,"2026-04-06T11:32:50",[45,79,46],{"id":107,"name":108,"github_repo":109,"description_zh":110,"stars":111,"difficulty_score":30,"last_commit_at":112,"category_tags":113,"status":64},4721,"markitdown","microsoft\u002Fmarkitdown","MarkItDown 是一款由微软 AutoGen 团队打造的轻量级 Python 工具，专为将各类文件高效转换为 Markdown 格式而设计。它支持 PDF、Word、Excel、PPT、图片（含 OCR）、音频（含语音转录）、HTML 乃至 YouTube 链接等多种格式的解析，能够精准提取文档中的标题、列表、表格和链接等关键结构信息。\n\n在人工智能应用日益普及的今天，大语言模型（LLM）虽擅长处理文本，却难以直接读取复杂的二进制办公文档。MarkItDown 恰好解决了这一痛点，它将非结构化或半结构化的文件转化为模型“原生理解”且 Token 效率极高的 Markdown 格式，成为连接本地文件与 AI 分析 pipeline 的理想桥梁。此外，它还提供了 MCP（模型上下文协议）服务器，可无缝集成到 Claude Desktop 等 LLM 应用中。\n\n这款工具特别适合开发者、数据科学家及 AI 研究人员使用，尤其是那些需要构建文档检索增强生成（RAG）系统、进行批量文本分析或希望让 AI 助手直接“阅读”本地文件的用户。虽然生成的内容也具备一定可读性，但其核心优势在于为机器",93400,"2026-04-06T19:52:38",[114,45],"插件",{"id":116,"name":117,"github_repo":118,"description_zh":119,"stars":120,"difficulty_score":76,"last_commit_at":121,"category_tags":122,"status":64},4487,"LLMs-from-scratch","rasbt\u002FLLMs-from-scratch","LLMs-from-scratch 是一个基于 PyTorch 的开源教育项目，旨在引导用户从零开始一步步构建一个类似 ChatGPT 的大型语言模型（LLM）。它不仅是同名技术著作的官方代码库，更提供了一套完整的实践方案，涵盖模型开发、预训练及微调的全过程。\n\n该项目主要解决了大模型领域“黑盒化”的学习痛点。许多开发者虽能调用现成模型，却难以深入理解其内部架构与训练机制。通过亲手编写每一行核心代码，用户能够透彻掌握 Transformer 架构、注意力机制等关键原理，从而真正理解大模型是如何“思考”的。此外，项目还包含了加载大型预训练权重进行微调的代码，帮助用户将理论知识延伸至实际应用。\n\nLLMs-from-scratch 特别适合希望深入底层原理的 AI 开发者、研究人员以及计算机专业的学生。对于不满足于仅使用 API，而是渴望探究模型构建细节的技术人员而言，这是极佳的学习资源。其独特的技术亮点在于“循序渐进”的教学设计：将复杂的系统工程拆解为清晰的步骤，配合详细的图表与示例，让构建一个虽小但功能完备的大模型变得触手可及。无论你是想夯实理论基础，还是为未来研发更大规模的模型做准备",90106,"2026-04-06T11:19:32",[97,79,46,45]]