安全强化学习基准

该仓库用于安全强化学习（RL）研究，我们在此探索各种安全RL基准和安全RL评测环境，涵盖单智能体RL和多智能体RL。若任何作者不希望其论文在此列出，请随时联系 <gshangd[AT]foxmail.com>。（本仓库仍在积极开发中，我们欢迎任何建设性的意见和建议）

欢迎您更新此列表！如果您发现一篇未在此列出的安全强化学习相关论文，请：

• 分支本仓库，添加相关信息并合并回主仓库；
• 或在此处提交问题；
• 或发送邮件至 <gshangd[AT]foxmail.com>。

---

README的组织结构如下：

• 安全强化学习基准
  • 1. 支持的环境
    • 1.1. 安全单智能体RL基准
    • 1.2. 安全多智能体RL基准
  • 2. 安全RL基准方法
    • 2.1. 安全单智能体RL基准方法
    • 2.2. 安全多智能体RL基准方法
  • 3. 综述
  • 4. 学位论文
  • 5. 书籍
  • 6. 教程
  • 7. 练习题
• 出版物

---

1. 支持的环境

1.1. 安全单智能体RL基准

• AI Safety Gridworlds
• Safety-Gym
• Safety-Gymnasium

1.2. 安全多智能体RL基准

• Safe Multi-Agent Mujoco
• Safe Multi-Agent Isaac Gym
• Safe Multi-Agent Robosuite

2. 安全RL基准方法

2.1. 安全单智能体RL基准方法

• Consideration of risk in reinforcement learning, Paper, Not Find Code, (Accepted by ICML 1994)
• Multi-criteria Reinforcement Learning, Paper, Not Find Code, (Accepted by ICML 1998)
• Lyapunov design for safe reinforcement learning, Paper, Not Find Code, (Accepted by ICML 2002)
• Risk-sensitive reinforcement learning, Paper, Not Find Code, (Accepted by Machine Learning, 2002)
• Risk-Sensitive Reinforcement Learning Applied to Control under Constraints, Paper, Not Find Code, (Accepted by Journal of Artificial Intelligence Research, 2005)
• An actor-critic algorithm for constrained markov decision processes, Paper, Not Find Code, (Accepted by Systems & Control Letters, 2005)
• Reinforcement learning for MDPs with constraints, Paper, Not Find Code, (Accepted by European Conference on Machine Learning 2006)
• Discounted Markov decision processes with utility constraints, Paper, Not Find Code, (Accepted by Computers & Mathematics with Applications, 2006)
• Constrained reinforcement learning from intrinsic and extrinsic rewards, Paper, Not Find Code, (Accepted by International Conference on Development and Learning 2007)
• Safe exploration for reinforcement learning, Paper, Not Find Code, (Accepted by ESANN 2008)
• Percentile optimization for Markov decision processes with parameter uncertainty, Paper, Not Find Code, (Accepted by Operations research, 2010)
• Probabilistic goal Markov decision processes, Paper, Not Find Code, (Accepted by IJCAI 2011)
• Safe reinforcement learning in high-risk tasks through policy improvement, Paper, Not Find Code, (Accepted by IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) 2011)
• Safe Exploration in Markov Decision Processes, Paper, Not Find Code, (Accepted by ICML 2012)
• Policy gradients with variance related risk criteria, Paper, Not Find Code, (Accepted by ICML 2012)
• Risk aversion in Markov decision processes via near optimal Chernoff bounds, Paper, Not Find Code, (Accepted by NeurIPS 2012)
• Safe Exploration of State and Action Spaces in Reinforcement Learning, Paper, Not Find Code, (Accepted by Journal of Artificial Intelligence Research, 2012)
• An Online Actor–Critic Algorithm with Function Approximation for Constrained Markov Decision Processes, Paper, Not Find Code, (Accepted by Journal of Optimization Theory and Applications, 2012)
• Safe policy iteration, Paper, Not Find Code, (Accepted by ICML 2013)
• Reachability-based safe learning with Gaussian processes, Paper, Not Find Code (Accepted by IEEE CDC 2014)
• Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret, Paper, Not Find Code, (Accepted by ICML 2015)
• High-Confidence Off-Policy Evaluation, Paper, Code (Accepted by AAAI 2015)
• Safe Exploration for Optimization with Gaussian Processes, Paper, Not Find Code (Accepted by ICML 2015)
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes, Paper, Not Find Code (Accepted by NeurIPS 2016)
• Safe and efficient off-policy reinforcement learning, Paper, Code (Accepted by NeurIPS 2016)
• Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving, Paper, Not Find Code (only Arxiv, 2016, citation 530+)
• Safe Learning of Regions of Attraction in Uncertain, Nonlinear Systems with Gaussian Processes, Paper, Code (Accepetd by CDC 2016)
• Safety-constrained reinforcement learning for MDPs, Paper, Not Find Code (Accepted by InInternational Conference on Tools and Algorithms for the Construction and Analysis of Systems 2016)
• Convex synthesis of randomized policies for controlled Markov chains with density safety upper bound constraints, Paper, Not Find Code (Accepted by American Control Conference 2016)
• Combating Deep Reinforcement Learning's Sisyphean Curse with Intrinsic Fear, Paper, Not Find Code (only Openreview, 2016)
• Combating reinforcement learning's sisyphean curse with intrinsic fear, Paper, Not Find Code (only Arxiv, 2016)
• Constrained Policy Optimization (CPO), Paper, Code (Accepted by ICML 2017)
• Risk-constrained reinforcement learning with percentile risk criteria, Paper, , Not Find Code (Accepted by The Journal of Machine Learning Research, 2017)
• Probabilistically Safe Policy Transfer, Paper, Not Find Code (Accepted by ICRA 2017)
• Accelerated primal-dual policy optimization for safe reinforcement learning, Paper, Not Find Code (Arxiv, 2017)
• Stagewise safe bayesian optimization with gaussian processes, Paper, Not Find Code (Accepted by ICML 2018)
• Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning, Paper, Code (Accepted by ICLR 2018)
• Safe Model-based Reinforcement Learning with Stability Guarantees, Paper, Code (Accepted by NeurIPS 2018)
• A Lyapunov-based Approach to Safe Reinforcement Learning, Paper, Not Find Code (Accepted by NeurIPS 2018)
• Constrained Cross-Entropy Method for Safe Reinforcement Learning, Paper, Not Find Code (Accepted by NeurIPS 2018)
• Safe Reinforcement Learning via Formal Methods, Paper, Not Find Code (Accepted by AAAI 2018)
• Safe exploration and optimization of constrained mdps using gaussian processes, Paper, Not Find Code (Accepted by AAAI 2018)
• Safe reinforcement learning via shielding, Paper, Code (Accepted by AAAI 2018)
• Trial without Error: Towards Safe Reinforcement Learning via Human Intervention, Paper, Not Find Code (Accepted by AAMAS 2018)
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning, Paper, Not Find Code (Accepted by CDC 2018)
• The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems, Paper, Code (Accepted by CoRL 2018)
• OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World, Paper, Not Find Code (Accepted by ICRA 2018)
• Safe learning of quadrotor dynamics using barrier certificates, Paper, Not Find Code (Accepted by ICRA 2018)
• Safe reinforcement learning on autonomous vehicles, Paper, Not Find Code (Accepted by IROS 2018)
• Trial without error: Towards safe reinforcement learning via human intervention, Paper, Code (Accepted by AAMAS 2018)
• Safe reinforcement learning: Learning with supervision using a constraint-admissible set, Paper, Not Find Code (Accepted by Annual American Control Conference (ACC) 2018)
• A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems, Paper, Not Find Code (Accepted by IEEE Transactions on Automatic Control 2018)
• Safe exploration algorithms for reinforcement learning controllers, Paper, Not Find Code (Accepted by IEEE transactions on neural networks and learning systems 2018)
• Verification and repair of control policies for safe reinforcement learning, Paper, Not Find Code (Accepted by Applied Intelligence, 2018)
• Safe Exploration in Continuous Action Spaces, Paper, Code, (only Arxiv, 2018, citation 200+)
• Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning, Paper, Not Find Code (Arxiv, 2018, citation 40+)
• Batch policy learning under constraints, Paper, Code (Accepted by ICML 2019)
• Safe Policy Improvement with Baseline Bootstrapping, Paper, Not Find Code (Accepted by ICML 2019)
• Convergent Policy Optimization for Safe Reinforcement Learning, Paper, Code (Accepted by NeurIPS 2019)
• Constrained reinforcement learning has zero duality gap, Paper, Not Find Code (Accepted by NeurIPS 2019)
• Reinforcement learning with convex constraints, Paper, Code (Accepted by NeurIPS 2019)
• Reward constrained policy optimization, Paper, Not Find Code (Accepted by ICLR 2019)
• Supervised policy update for deep reinforcement learning, Paper, Code, (Accepted by ICLR 2019)
• End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks, Paper, Code (Accepted by AAAI 2019)
• Lyapunov-based safe policy optimization for continuous control, Paper, Not Find Code (Accepted by ICML Workshop RL4RealLife 2019)
• Safe reinforcement learning with model uncertainty estimates, Paper, Not Find Code (Accepted by ICRA 2019)
• Safe reinforcement learning with scene decomposition for navigating complex urban environments, Paper, Code, (Accepted by IV 2019)
• Verifiably safe off-model reinforcement learning, Paper, Code (Accepted by InInternational Conference on Tools and Algorithms for the Construction and Analysis of Systems 2019)
• Probabilistic policy reuse for safe reinforcement learning, Paper, Not Find Code, (Accepted by ACM Transactions on Autonomous and Adaptive Systems (TAAS), 2019)
• Projected stochastic primal-dual method for constrained online learning with kernels, Paper, Not Find Code, (Accepted by IEEE Transactions on Signal Processing, 2019)
• Resource constrained deep reinforcement learning, Paper, Not Find Code, (Accepted by 29th International Conference on Automated Planning and Scheduling 2019)
• Temporal logic guided safe reinforcement learning using control barrier functions, Paper, Not Find Code (Arxiv, Citation 25+, 2019)
• Safe policies for reinforcement learning via primal-dual methods, Paper, Not Find Code (Arxiv, Citation 25+, 2019)
• Value constrained model-free continuous control, Paper, Not Find Code (Arxiv, Citation 35+, 2019)
• Safe Reinforcement Learning in Constrained Markov Decision Processes (SNO-MDP), Paper, Code (Accepted by ICML 2020)
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, Paper, Code (Accepted by ICML 2020)
• Constrained markov decision processes via backward value functions, Paper, Code (Accepted by ICML 2020)
• Projection-Based Constrained Policy Optimization (PCPO), Paper, Code (Accepted by ICLR 2020)
• First order constrained optimization in policy space (FOCOPS),Paper, Code (Accepted by NeurIPS 2020)
• Safe reinforcement learning via curriculum induction, Paper, Code (Accepted by NeurIPS 2020)
• Constrained episodic reinforcement learning in concave-convex and knapsack settings, Paper, Code (Accepted by NeurIPS 2020)
• Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret, Paper, Not Find Code (Accepted by NeurIPS 2020)
• Upper confidence primal-dual reinforcement learning for CMDP with adversarial loss, Paper, Not Find Code (Accepted by NeurIPS 2020)
• IPO: Interior-point Policy Optimization under Constraints, Paper, Not Find Code (Accepted by AAAI 2020)
• Safe reinforcement learning using robust mpc, Paper, Not Find Code (IEEE Transactions on Automatic Control, 2020)
• Safe reinforcement learning via projection on a safe set: How to achieve optimality? Paper, Not Find Code (Accepted by IFAC 2020)
• Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions, Paper, Not Find Code (Accepted by RSS 2020)
• Learning Transferable Domain Priors for Safe Exploration in Reinforcement Learning, Paper, Code, (Accepted by International Joint Conference on Neural Networks (IJCNN) 2020)
• Safe reinforcement learning through meta-learned instincts, Paper, Not Find Code (Accepted by The Conference on Artificial Life 2020)
• Learning safe policies with cost-sensitive advantage estimation, Paper, Not Find Code (Openreview 2020)
• Safe reinforcement learning using probabilistic shields, Paper, Not Find Code (2020)
• A constrained reinforcement learning based approach for network slicing, Paper, Not Find Code (Accepted by IEEE 28th International Conference on Network Protocols (ICNP) 2020)
• Safe reinforcement learning: A control barrier function optimization approach, Paper, Not Find Code (Accepted by the International Journal of Robust and Nonlinear Control)
• Exploration-exploitation in constrained mdps, Paper, Not Find Code (Arxiv, 2020)
• Safe reinforcement learning using advantage-based intervention, Paper, Code (Accepted by ICML 2021)
• Shortest-path constrained reinforcement learning for sparse reward tasks, Paper, Code, (Accepted by ICML 2021)
• Density constrained reinforcement learning, Paper, Not Find Code (Accepted by ICML 2021)
• CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee, Paper, Not Find Code (Accepted by ICML 2021)
• Safe reinforcement learning with linear function approximation, Paper, Not Find Code (Accepted by ICML 2021)
• Safe Reinforcement Learning by Imagining the Near Future (SMBPO), Paper, Code (Accepted by NeurIPS 2021)
• Towards safe reinforcement learning with a safety editor policy, Paper, Code (Accepted by NeurIPS 2021)
• Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning, Paper, Not Find Code (Accepted by NeurIPS 2021)
• Risk-Sensitive Reinforcement Learning: Symmetry, Asymmetry, and Risk-Sample Tradeoff, Paper, Not Find Code (Accepted by NeurIPS 2021)
• Safe reinforcement learning with natural language constraints, Paper, Code, (Accepted by NeurIPS 2021)
• Learning policies with zero or bounded constraint violation for constrained mdps, Paper, Not Find Code (Accepted by NeurIPS 2021)
• Conservative safety critics for exploration, Paper, Not Find Code (Accepted by ICLR 2021)
• Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning, Paper, Not Find Code (Accepted by AAAI 2021)
• Risk-averse trust region optimization for reward-volatility reduction, Paper, Not Find Code (Accepted by IJCAI 2021)
• AlwaysSafe: Reinforcement Learning Without Safety Constraint Violations During Training, Paper, Code (Accepted by AAMAS 2021)
• Safe Continuous Control with Constrained Model-Based Policy Optimization (CMBPO), Paper, Code (Accepted by IROS 2021)
• Context-aware safe reinforcement learning for non-stationary environments, Paper, Code (Accepted by ICRA 2021)
• Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function, Paper, Code (Accepted by IROS 2021)
• Robot Reinforcement Learning on the Constraint Manifold, Paper, Code (Accepted by CoRL 2021)
• Provably efficient safe exploration via primal-dual policy optimization, Paper, Not Find Code (Accepted by the International Conference on Artificial Intelligence and Statistics 2021)
• Safe model-based reinforcement learning with robust cross-entropy method, Paper, Code (Accepted by ICLR 2021 Workshop on Security and Safety in Machine Learning Systems)
• MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance, Paper, Code (Accepted by Workshop on Safe and Robust Control of Uncertain Systems at NeurIPS 2021)
• Safe Reinforcement Learning of Control-Affine Systems with Vertex Networks, Paper, Code (Accepted by Conference on Learning for Dynamics and Control 2021)
• Can You Trust Your Autonomous Car? Interpretable and Verifiably Safe Reinforcement Learning, Paper, Not Find Code (Accepted by IV 2021)
• Provably safe model-based meta reinforcement learning: An abstraction-based approach, Paper, Not Find Code (Accepted by CDC 2021)
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones, Paper, Code, (Accepted by IEEE RAL, 2021)
• Reinforcement learning control of constrained dynamic systems with uniformly ultimate boundedness stability guarantee, Paper, Not Find Code (Accepted by Automatica, 2021)
• A predictive safety filter for learning-based control of constrained nonlinear dynamical systems, Paper, Not Find Code (Accepted by Automatica, 2021)
• A simple reward-free approach to constrained reinforcement learning, Paper, Not Find Code (Arxiv, 2021)
• State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards, Paper, Not Find Code (Arxiv, 2021)
• DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Intervention, Paper, Not Find Code (Arxiv, 2021)
• Safe Exploration in Model-based Reinforcement Learning using Control Barrier Functions, Paper, Not Find Code (Arxiv, 2021)
• Constrained Variational Policy Optimization for Safe Reinforcement Learning, Paper, Code (ICML 2022)
• Provably efficient model-free constrained rl with linear function approximation, Paper, Not Find Code (NeurIPS 2022)
• Constrained Policy Optimization via Bayesian World Models, Paper, Code (ICLR 2022)
• Stability-Constrained Markov Decision Processes Using MPC, Paper, Not Find Code (Accepted by Automatica, 2022)
• Constrained Reinforcement Learning for Vehicle Motion Planning with Topological Reachability Analysis, Paper, Not Find Code (Accepted by Robotics, 2022)
• Triple-Q: A Model-Free Algorithm for Constrained Reinforcement Learning with Sublinear Regret and Zero Constraint Violation, Paper, Code (Accepted by AISTATS 2022)
• Safe reinforcement learning using robust action governor, Paper, Not Find Code (Accepted by In Learning for Dynamics and Control, 2022)
• A primal-dual approach to constrained markov decision processes, Paper, Not Find Code (Arxiv, 2022)
• SAUTE RL: Almost Surely Safe Reinforcement Learning Using State Augmentation, Paper, Not Find Code (Arxiv, 2022)
• Finding Safe Zones of policies Markov Decision Processes, Paper, Not Find Code (Arxiv, 2022)
• CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning, Paper, Code (Arxiv, 2022)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition, Paper, Not Find Code (Arxiv, 2022)
• Penalized Proximal Policy Optimization for Safe Reinforcement Learning, Paper, Not Find Code (Arxiv, 2022)
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning, Paper, Not Find Code (Arxiv, 2022)
• Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs, Paper, Not Find Code (Arxiv, 2022)
• Guided Safe Shooting: model based reinforcement learning with safety constraints, Paper, Not Find Code (Arxiv, 2022)
• Safe Reinforcement Learning via Confidence-Based Filters, Paper, Not Find Code (Arxiv, 2022)
• TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning, Paper, Code (Accepted by IEEE RAL, 2022)
• Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Risk, Paper, Not Find Code (Accepted by IEEE RAL, 2022)
• Enhancing Safe Exploration Using Safety State Augmentation, Paper, Not Find Code (Arxiv, 2022)
• Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk, Paper, Not Find Code (Accepted by IJCAI 2022)
• Safe reinforcement learning of dynamic high-dimensional robotic tasks: navigation, manipulation, interaction, Paper, Not Find Code (Arxiv, 2022)
• Safe Exploration Method for Reinforcement Learning under Existence of Disturbance, Paper, Not Find Code (Arxiv, 2022)
• Guiding Safe Exploration with Weakest Preconditions, Paper, Code (Arxiv, 2022)
• Temporal logic guided safe model-based reinforcement learning: A hybrid systems approach, Paper, Not Find Code (Accepted by Nonlinear Analysis: Hybrid Systems, 2022)
• Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes, Paper, Not Find Code (Arxiv, 2022)
• Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm, Paper, Code (Arxiv, 2022)
• Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate, Paper, Not Find Code (Arxiv, 2022)
• UNIFY: a Unified Policy Designing Framework for Solving Constrained Optimization Problems with Machine Learning, Paper, Not Find Code (Arxiv, 2022)
• Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments, Paper, Not Find Code (Arxiv, 2022)
• Safe Reinforcement Learning Using Robust Control Barrier Functions, Paper, Not Find Code (Accepted by IEEE RAL, 2022)
• Model-free Neural Lyapunov Control for Safe Robot Navigation, Paper, Code, Demo (Accepted by IROS 2022)
• Safe Reinforcement Learning via Probabilistic Logic Shields, Paper, Code (Accepted by IJCAI 2023, Distinguished Paper Award)
• Towards robust and safe reinforcement learning with benign off-policy data, Paper, Not Find Code (Accepted by ICML 2023)
• Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments, Paper, Not Find Code (Accepted by ICML 2023)
• Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RL, Paper, Not Find Code (Accepted by ICLR 2023)
• A CMDP-within-online framework for Meta-Safe Reinforcement Learning, Paper, Not Find Code (Accepted by ICLR 2023)
• Datasets and Benchmarks for Offline Safe Reinforcement Learning, Paper, Code, (Accepted by Journal of Data-centric Machine Learning Research)
• SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization, Paper, Code (Arxiv, 2023)
• Shielded Reinforcement Learning for Hybrid Systems, Paper (Arxiv), Code (AISOLA, 2023)
• Adaptive primal-dual method for safe reinforcement learning, Paper, Not Find Code (Accepted by AAMAS 2024)
• Probabilistic constraint for safety-critical reinforcement learning, Paper, Not Find Code (Accepted by TAC)
• Generalized constraint for probabilistic safe reinforcement learning, Paper, Not Find Code (Accepted by DCC 2024)
• Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learning, Paper, Code (JMLR, 2024)
• Provably safe reinforcement learning with step-wise violation constraints, Paper, Not Find Code (Accepted by NeurIPS 2024)
• Feasibility Consistent Representation Learning for Safe Reinforcement Learning, Paper, Code, (Accepted by ICML 2024)
• Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation, Paper, Not Find Code (Accepted by AAAI 2024)
• Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models, Paper, Not Find Code (Accepted by AAMAS 2024)
• Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation, Paper, Not Find Code (Arxiv, 2024)
• Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning, Paper, Not Find Code (Arxiv, 2024)
• Confident Natural Policy Gradient for Local Planning in qπ-realizable Constrained MDPs, Paper, Not Find Code (Arxiv, 2024)
• Safe Exploration Using Bayesian World Models and Log-Barrier Optimization, Paper, Code (Arxiv, 2024)
• Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning, Paper, Code (Accepted by IEEE TPAMI 2025)
• Reward-Safety Balance in Offline Safe RL via Diffusion Regularization, Paper, Not Find Code (Accepted by NeurIPS 2025)

2.2. 安全多智能体强化学习基线

• 多智能体约束策略优化（MACPO），论文，代码（Arxiv，2021年）
• MAPPO-Lagrangian，论文，代码（Arxiv，2021年）
• 用于安全多智能体强化学习的去中心化策略梯度下降上升法，论文，未找到代码（被AAAI 2021接收）
• 基于屏蔽的安全多智能体强化学习，论文，未找到代码（被AAMAS 2021接收）
• CMIX：具有峰值和平均约束的深度多智能体强化学习，论文，未找到代码（被联合欧洲机器学习与数据库知识发现会议2021接收）
• 通过去中心化多重控制屏障函数实现的安全多智能体强化学习，论文，未找到代码（Arxiv 2021）
• CAMA：一种使用约束增强的安全多智能体强化学习新框架，论文，未找到代码（Openreview 2022）
• 用于安全多智能体强化学习的屏蔽去中心化，论文，未找到代码（NeurIPS 2022）
• 使用分布式上图形式多智能体强化学习求解多智能体安全最优控制，论文，代码（RSS 2025）

3. 综述

• 安全强化学习综述：方法、理论与应用，论文（IEEE TPAMI，2024年）
• 基于状态的安全强化学习：综述，论文（被IJCAI 2023接收）
• 无模型强化学习中的带约束策略学习：综述，论文（被IJCAI 2021接收）
• 机器人领域的安全学习：从基于学习的控制到安全强化学习，论文（被年度控制、机器人与自主系统评论接收，2021年）
• 安全学习与优化技术：迈向现状综述，论文（被国际可信人工智能基础研讨会——整合学习、优化与推理——接收，2020年）
• 安全强化学习综合综述，论文（被机器学习研究期刊接收，2015年）

4. 学位论文

• 用于机器人决策的安全强化学习，论文（Shangding Gu，慕尼黑工业大学博士论文，2024年）
• 强化学习中的安全探索：理论与机器人应用，论文（Felix Berkenkamp，苏黎世联邦理工学院博士论文，2019年）
• 安全强化学习，论文（Philip S. Thomas，马萨诸塞大学阿默斯特分校博士论文，2015年）

5. 书籍

• 约束马尔可夫决策过程：随机建模，书籍（Eitan Altman，Routledge，1999年）

6. 教程

• 安全强化学习：连接理论与实践，教程（Ming Jin & Shangding Gu，2024年）
• 用于智能电网控制与运营的安全强化学习，教程（Ming Jin & Shangding Gu，2024年）
• 安全强化学习，教程（Felix Berkenkamp，2023年）
• 原始-对偶方法，教程（Gergely Neu，2023年）

7. 练习

• 原始-对偶强化学习，练习代码和练习Colab（Germano Gabbianelli，2023年）

出版物

如果您觉得本仓库有用，请引用以下论文：

@article{gu2024review,
  title={A Review of Safe Reinforcement Learning: Methods, Theories and Applications},
  author={Gu, Shangding and Yang, Long and Du, Yali and Chen, Guang and Walter, Florian and Wang, Jun and Knoll, Alois},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2024},
  publisher={IEEE}
}

Safe-Reinforcement-Learning-Baselines 快速上手指南

本指南旨在帮助开发者快速了解并使用 Safe-Reinforcement-Learning-Baselines 项目。该项目是一个专注于安全强化学习（Safe RL）的研究资源库，汇集了单智能体与多智能体安全 RL 的基准环境、经典算法论文及代码实现。

注意：本项目主要作为一个文献与代码索引库存在，部分早期论文可能未提供官方代码。使用时请根据具体算法条目查看其对应的代码链接。

1. 环境准备

在开始之前，请确保您的开发环境满足以下基本要求：

• 操作系统：Linux (推荐 Ubuntu 18.04/20.04) 或 macOS。Windows 用户建议使用 WSL2。
• Python 版本：建议 Python 3.7 - 3.9（具体取决于子项目中使用的深度学习框架版本）。
• 前置依赖：
  • Git
  • PyTorch 或 TensorFlow (视具体算法实现而定，大多数现代实现基于 PyTorch)
  • MuJoCo (部分物理仿真环境需要授权和安装)
  • Gym / Gymnasium

国内加速建议：

• Git 克隆：如果直接克隆 GitHub 仓库速度慢，可使用国内镜像源（如 Gitee 镜像，若有）或配置代理。
• Python 包安装：推荐使用清华源或阿里源安装依赖。

  pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
  

2. 安装步骤

由于该仓库是多个独立算法和环境的集合，没有统一的 setup.py 来安装所有功能。请按照以下步骤获取资源并配置特定算法：

第一步：克隆仓库

git clone https://github.com/chauncygu/Safe-Reinforcement-Learning-Baseline.git
cd Safe-Reinforcement-Learning-Baseline

第二步：选择并配置具体算法

浏览仓库目录结构（通常在 Safe-RL/ 下），找到您需要的算法文件夹（例如 safety-starter-agents 对应 CPO 算法，safe_learning 对应基于高斯过程的方法等）。

以 Constrained Policy Optimization (CPO) 为例，进入对应目录并安装依赖：

# 进入具体算法目录 (示例路径，请以实际目录结构为准)
cd Safe-RL/safety-starter-agents

# 创建虚拟环境 (推荐)
python -m venv venv
source venv/bin/activate  # Windows 使用: venv\Scripts\activate

# 安装该算法特定的依赖
pip install -r requirements.txt

提示：如果某个条目标记为 "Not Find Code"，则表示该仓库仅提供了论文链接，您需要自行寻找第三方实现或复现。

第三步：安装仿真环境

根据您想运行的基准测试（Benchmark），安装相应的环境包。

• 安全单智能体环境：

  # 示例：安装 Safety-Gymnasium (PKU-Alignment 维护版，对国内更友好)
  pip install safety-gymnasium
  

• 安全多智能体环境： 需参考 Safe Multi-Agent Mujoco 等子项目的具体 README 进行安装，通常涉及 multi-agent-mujoco 等包。

3. 基本使用

以下以运行一个典型的安全强化学习算法（如 CPO）在安全环境中训练为例。

最简单的使用示例

假设您已进入包含 CPO 代码的目录，并且已安装 safety-gymnasium。

1. 导入环境与算法（Python 脚本示例）：

   import gymnasium as gym
   import safety_gymnasium
   from your_algo_module import CPOAgent  # 替换为实际的算法导入路径
   
   # 初始化安全环境 (例如：Safety-Gymnasium 中的 PointGoal1-v0)
   env = gym.make("SafetyPointGoal1-v0")
   
   # 初始化智能体
   agent = CPOAgent(env.observation_space, env.action_space)
   
   # 训练循环示例
   for episode in range(100):
       obs, _ = env.reset()
       done = False
       total_reward = 0
       total_cost = 0
       
       while not done:
           # 获取动作
           action = agent.select_action(obs)
           
           # 执行动作
           next_obs, reward, cost, terminated, truncated, info = env.step(action)
           done = terminated or truncated
           
           # 存储经验并更新策略 (具体方法名视实现而定)
           agent.store_transition(obs, action, reward, cost, next_obs, done)
           agent.update()
           
           obs = next_obs
           total_reward += reward
           total_cost += cost
           
       print(f"Episode {episode}: Reward: {total_reward:.2f}, Cost: {total_cost:.2f}")
   
   env.close()
   

2. 运行现有脚本： 大多数子项目都提供了直接的训练脚本。在项目目录下查找类似 run.py 或 main.py 的文件：

   # 示例命令，具体参数请参考各子项目的 README
   python run.py --algo CPO --env SafetyPointGoal1-v0 --epochs 50
   

核心资源索引

在使用过程中，您可以参考仓库 README 中的分类快速定位资源：

• 基准环境 (Environments): 查看 AI Safety Gridworlds, Safety-Gymnasium, Safe Multi-Agent Mujoco 等。
• 基线算法 (Baselines): 涵盖从早期的风险敏感 RL 到最新的 CPO, PPO-Lagrangian, TRPO-Lagrangian 等算法实现。
• 综述与教程: 适合入门安全强化学习理论。

---

注：本仓库处于活跃开发中，如遇问题或发现新的安全 RL 论文未被收录，欢迎通过 Issue 或邮件联系作者贡献。