[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-PacktPublishing--LLM-Engineers-Handbook":3,"tool-PacktPublishing--LLM-Engineers-Handbook":61},[4,18,26,36,44,53],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":17},4358,"openclaw","openclaw\u002Fopenclaw","OpenClaw 是一款专为个人打造的本地化 AI 助手，旨在让你在自己的设备上拥有完全可控的智能伙伴。它打破了传统 AI 助手局限于特定网页或应用的束缚，能够直接接入你日常使用的各类通讯渠道，包括微信、WhatsApp、Telegram、Discord、iMessage 等数十种平台。无论你在哪个聊天软件中发送消息，OpenClaw 都能即时响应，甚至支持在 macOS、iOS 和 Android 设备上进行语音交互，并提供实时的画布渲染功能供你操控。\n\n这款工具主要解决了用户对数据隐私、响应速度以及“始终在线”体验的需求。通过将 AI 部署在本地，用户无需依赖云端服务即可享受快速、私密的智能辅助，真正实现了“你的数据，你做主”。其独特的技术亮点在于强大的网关架构，将控制平面与核心助手分离，确保跨平台通信的流畅性与扩展性。\n\nOpenClaw 非常适合希望构建个性化工作流的技术爱好者、开发者，以及注重隐私保护且不愿被单一生态绑定的普通用户。只要具备基础的终端操作能力（支持 macOS、Linux 及 Windows WSL2），即可通过简单的命令行引导完成部署。如果你渴望拥有一个懂你",349277,3,"2026-04-06T06:32:30",[13,14,15,16],"Agent","开发框架","图像","数据工具","ready",{"id":19,"name":20,"github_repo":21,"description_zh":22,"stars":23,"difficulty_score":10,"last_commit_at":24,"category_tags":25,"status":17},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,"2026-04-05T11:01:52",[14,15,13],{"id":27,"name":28,"github_repo":29,"description_zh":30,"stars":31,"difficulty_score":32,"last_commit_at":33,"category_tags":34,"status":17},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",160411,2,"2026-04-18T23:33:24",[14,13,35],"语言模型",{"id":37,"name":38,"github_repo":39,"description_zh":40,"stars":41,"difficulty_score":32,"last_commit_at":42,"category_tags":43,"status":17},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",109154,"2026-04-18T11:18:24",[14,15,13],{"id":45,"name":46,"github_repo":47,"description_zh":48,"stars":49,"difficulty_score":32,"last_commit_at":50,"category_tags":51,"status":17},6121,"gemini-cli","google-gemini\u002Fgemini-cli","gemini-cli 是一款由谷歌推出的开源 AI 命令行工具，它将强大的 Gemini 大模型能力直接集成到用户的终端环境中。对于习惯在命令行工作的开发者而言，它提供了一条从输入提示词到获取模型响应的最短路径，无需切换窗口即可享受智能辅助。\n\n这款工具主要解决了开发过程中频繁上下文切换的痛点，让用户能在熟悉的终端界面内直接完成代码理解、生成、调试以及自动化运维任务。无论是查询大型代码库、根据草图生成应用，还是执行复杂的 Git 操作，gemini-cli 都能通过自然语言指令高效处理。\n\n它特别适合广大软件工程师、DevOps 人员及技术研究人员使用。其核心亮点包括支持高达 100 万 token 的超长上下文窗口，具备出色的逻辑推理能力；内置 Google 搜索、文件操作及 Shell 命令执行等实用工具；更独特的是，它支持 MCP（模型上下文协议），允许用户灵活扩展自定义集成，连接如图像生成等外部能力。此外，个人谷歌账号即可享受免费的额度支持，且项目基于 Apache 2.0 协议完全开源，是提升终端工作效率的理想助手。",100752,"2026-04-10T01:20:03",[52,13,15,14],"插件",{"id":54,"name":55,"github_repo":56,"description_zh":57,"stars":58,"difficulty_score":32,"last_commit_at":59,"category_tags":60,"status":17},4721,"markitdown","microsoft\u002Fmarkitdown","MarkItDown 是一款由微软 AutoGen 团队打造的轻量级 Python 工具，专为将各类文件高效转换为 Markdown 格式而设计。它支持 PDF、Word、Excel、PPT、图片（含 OCR）、音频（含语音转录）、HTML 乃至 YouTube 链接等多种格式的解析，能够精准提取文档中的标题、列表、表格和链接等关键结构信息。\n\n在人工智能应用日益普及的今天，大语言模型（LLM）虽擅长处理文本，却难以直接读取复杂的二进制办公文档。MarkItDown 恰好解决了这一痛点，它将非结构化或半结构化的文件转化为模型“原生理解”且 Token 效率极高的 Markdown 格式，成为连接本地文件与 AI 分析 pipeline 的理想桥梁。此外，它还提供了 MCP（模型上下文协议）服务器，可无缝集成到 Claude Desktop 等 LLM 应用中。\n\n这款工具特别适合开发者、数据科学家及 AI 研究人员使用，尤其是那些需要构建文档检索增强生成（RAG）系统、进行批量文本分析或希望让 AI 助手直接“阅读”本地文件的用户。虽然生成的内容也具备一定可读性，但其核心优势在于为机器",93400,"2026-04-06T19:52:38",[52,14],{"id":62,"github_repo":63,"name":64,"description_en":65,"description_zh":66,"ai_summary_zh":67,"readme_en":68,"readme_zh":69,"quickstart_zh":70,"use_case_zh":71,"hero_image_url":72,"owner_login":73,"owner_name":74,"owner_avatar_url":75,"owner_bio":76,"owner_company":77,"owner_location":77,"owner_email":77,"owner_twitter":73,"owner_website":78,"owner_url":79,"languages":80,"stars":89,"forks":90,"last_commit_at":91,"license":92,"difficulty_score":93,"env_os":94,"env_gpu":94,"env_ram":94,"env_deps":95,"category_tags":109,"github_topics":111,"view_count":32,"oss_zip_url":77,"oss_zip_packed_at":77,"status":17,"created_at":121,"updated_at":122,"faqs":123,"releases":153},9430,"PacktPublishing\u002FLLM-Engineers-Handbook","LLM-Engineers-Handbook","The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices","LLM-Engineers-Handbook 是一本专为大语言模型（LLM）工程师打造的实战指南，旨在帮助读者从零开始构建并部署生产级的 LLM 应用。它系统地解决了从数据收集、模型训练流水线搭建，到构建检索增强生成（RAG）系统，再到最终在 AWS 云端进行专业化部署与监控的全流程难题。\n\n这套资源特别适合希望深入掌握 LLMOps 最佳实践的开发者、算法工程师及技术研究人员。无论是想理解底层原理，还是需要将模型落地到真实业务场景，都能从中获得清晰的路径指引。其独特亮点在于提供了一个完整的端到端项目框架，不仅涵盖代码实现，还集成了 ZenML 编排、Opik 提示词监控、Qdrant 向量数据库以及 GitHub Actions 自动化流水线等现代工程化工具。\n\n通过跟随该手册，用户可以学习如何像专业团队一样，利用 Docker 容器化和 AWS 云服务打造高可用、可监控的 AI 系统。仓库中的代码保持活跃更新，甚至提供了可在 Hugging Face 直接下载的成品模型，确保学习者能接触到最新的技术方案。这不仅是一份理论教程，更是一套经过验证的工程化解决方案，帮助用户跨越从“实验D","LLM-Engineers-Handbook 是一本专为大语言模型（LLM）工程师打造的实战指南，旨在帮助读者从零开始构建并部署生产级的 LLM 应用。它系统地解决了从数据收集、模型训练流水线搭建，到构建检索增强生成（RAG）系统，再到最终在 AWS 云端进行专业化部署与监控的全流程难题。\n\n这套资源特别适合希望深入掌握 LLMOps 最佳实践的开发者、算法工程师及技术研究人员。无论是想理解底层原理，还是需要将模型落地到真实业务场景，都能从中获得清晰的路径指引。其独特亮点在于提供了一个完整的端到端项目框架，不仅涵盖代码实现，还集成了 ZenML 编排、Opik 提示词监控、Qdrant 向量数据库以及 GitHub Actions 自动化流水线等现代工程化工具。\n\n通过跟随该手册，用户可以学习如何像专业团队一样，利用 Docker 容器化和 AWS 云服务打造高可用、可监控的 AI 系统。仓库中的代码保持活跃更新，甚至提供了可在 Hugging Face 直接下载的成品模型，确保学习者能接触到最新的技术方案。这不仅是一份理论教程，更是一套经过验证的工程化解决方案，帮助用户跨越从“实验Demo\"到“生产系统”的鸿沟。","\u003Cp align='center'>\u003Ca href='https:\u002F\u002Fwww.eventbrite.com\u002Fe\u002Fship-production-pytorch-system-in-a-day-train-optimize-deploy-workshop-tickets-1983348934052?aff=GitHub'>\u003Cimg src='https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_readme_d11aced4566a.png'\u002F>\u003C\u002Fa>\u003C\u002Fp>\n\n\u003Cp align='center'>\u003Ca href='https:\u002F\u002Fwww.packtpub.com\u002Fen-us\u002Funlock?step=1'>\u003Cimg src='https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_readme_7416d3f6219e.png'\u002F>\u003C\u002Fa>\u003C\u002Fp>\n\n\u003Cdiv align=\"center\">\n  \u003Ch1>👷 LLM Engineer's Handbook\u003C\u002Fh1>\n  \u003Cp class=\"tagline\">Official repository of the \u003Ca href=\"https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F\">LLM Engineer's Handbook\u003C\u002Fa> by \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fiusztinpaul\">Paul Iusztin\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fmlabonne\">Maxime Labonne\u003C\u002Fa>\u003C\u002Fp>\n\u003C\u002Fdiv>\n\u003C\u002Fbr>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_readme_855fa73e767b.png\" alt=\"Book cover\">\n  \u003C\u002Fa>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  Find the book on \u003Ca href=\"https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F\">Amazon\u003C\u002Fa> or \u003Ca href=\"https:\u002F\u002Fwww.packtpub.com\u002Fen-us\u002Fproduct\u002Fllm-engineers-handbook-9781836200062\">Packt\u003C\u002Fa>\n\u003C\u002Fp>\n\n## 🌟 Features\n\nThe goal of this book is to create your own end-to-end LLM-based system using best practices:\n\n- 📝 Data collection & generation\n- 🔄 LLM training pipeline\n- 📊 Simple RAG system\n- 🚀 Production-ready AWS deployment\n- 🔍 Comprehensive monitoring\n- 🧪 Testing and evaluation framework\n\nYou can download and use the final trained model on [Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fmlabonne\u002FTwinLlama-3.1-8B-DPO).\n\n> [!IMPORTANT]\n> The code in this GitHub repository is actively maintained and may contain updates not reflected in the book. **Always refer to this repository for the latest version of the code.**\n\n## 🔗 Dependencies\n\n### Local dependencies\n\nTo install and run the project locally, you need the following dependencies.\n\n| Tool | Version | Purpose | Installation Link |\n|------|---------|---------|------------------|\n| pyenv | ≥2.3.36 | Multiple Python versions (optional) | [Install Guide](https:\u002F\u002Fgithub.com\u002Fpyenv\u002Fpyenv?tab=readme-ov-file#installation) |\n| Python | 3.11 | Runtime environment | [Download](https:\u002F\u002Fwww.python.org\u002Fdownloads\u002F) |\n| Poetry | >= 1.8.3 and \u003C 2.0 | Package management | [Install Guide](https:\u002F\u002Fpython-poetry.org\u002Fdocs\u002F#installation) |\n| Docker | ≥27.1.1 | Containerization | [Install Guide](https:\u002F\u002Fdocs.docker.com\u002Fengine\u002Finstall\u002F) |\n| AWS CLI | ≥2.15.42 | Cloud management | [Install Guide](https:\u002F\u002Fdocs.aws.amazon.com\u002Fcli\u002Flatest\u002Fuserguide\u002Fgetting-started-install.html) |\n| Git | ≥2.44.0 | Version control | [Download](https:\u002F\u002Fgit-scm.com\u002Fdownloads) |\n\n### Cloud services\n\nThe code also uses and depends on the following cloud services. For now, you don't have to do anything. We will guide you in the installation and deployment sections on how to use them:\n\n| Service | Purpose |\n|---------|---------|\n| [HuggingFace](https:\u002F\u002Fhuggingface.com\u002F) | Model registry |\n| [Comet ML](https:\u002F\u002Fwww.comet.com\u002Fsite\u002Fproducts\u002Fopik\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik) | Experiment tracker |\n| [Opik](https:\u002F\u002Fwww.comet.com\u002Fsite\u002Fproducts\u002Fopik\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik) | Prompt monitoring |\n| [ZenML](https:\u002F\u002Fwww.zenml.io\u002F) | Orchestrator and artifacts layer |\n| [AWS](https:\u002F\u002Faws.amazon.com\u002F) | Compute and storage |\n| [MongoDB](https:\u002F\u002Fwww.mongodb.com\u002F) | NoSQL database |\n| [Qdrant](https:\u002F\u002Fqdrant.tech\u002F) | Vector database |\n| [GitHub Actions](https:\u002F\u002Fgithub.com\u002Ffeatures\u002Factions) | CI\u002FCD pipeline |\n\nIn the [LLM Engineer's Handbook](https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F), Chapter 2 will walk you through each tool. Chapters 10 and 11 provide step-by-step guides on how to set up everything you need.\n\n## 🗂️ Project Structure\n\nHere is the directory overview:\n\n```bash\n.\n├── code_snippets\u002F       # Standalone example code\n├── configs\u002F             # Pipeline configuration files\n├── llm_engineering\u002F     # Core project package\n│   ├── application\u002F    \n│   ├── domain\u002F         \n│   ├── infrastructure\u002F \n│   ├── model\u002F         \n├── pipelines\u002F           # ML pipeline definitions\n├── steps\u002F               # Pipeline components\n├── tests\u002F               # Test examples\n├── tools\u002F               # Utility scripts\n│   ├── run.py\n│   ├── ml_service.py\n│   ├── rag.py\n│   ├── data_warehouse.py\n```\n\n`llm_engineering\u002F`  is the main Python package implementing LLM and RAG functionality. It follows Domain-Driven Design (DDD) principles:\n\n- `domain\u002F`: Core business entities and structures\n- `application\u002F`: Business logic, crawlers, and RAG implementation\n- `model\u002F`: LLM training and inference\n- `infrastructure\u002F`: External service integrations (AWS, Qdrant, MongoDB, FastAPI)\n\nThe code logic and imports flow as follows: `infrastructure` → `model` → `application` → `domain`\n\n`pipelines\u002F`: Contains the ZenML ML pipelines, which serve as the entry point for all the ML pipelines. Coordinates the data processing and model training stages of the ML lifecycle.\n\n`steps\u002F`: Contains individual ZenML steps, which are reusable components for building and customizing ZenML pipelines. Steps perform specific tasks (e.g., data loading, preprocessing) and can be combined within the ML pipelines.\n\n`tests\u002F`: Covers a few sample tests used as examples within the CI pipeline.\n\n`tools\u002F`: Utility scripts used to call the ZenML pipelines and inference code:\n- `run.py`: Entry point script to run ZenML pipelines.\n- `ml_service.py`: Starts the REST API inference server.\n- `rag.py`: Demonstrates usage of the RAG retrieval module.\n- `data_warehouse.py`: Used to export or import data from the MongoDB data warehouse through JSON files.\n\n`configs\u002F`: ZenML YAML configuration files to control the execution of pipelines and steps.\n\n`code_snippets\u002F`: Independent code examples that can be executed independently.\n\n## 💻 Installation\n\n> [!NOTE]\n> If you are experiencing issues while installing and running the repository, consider checking the [Issues](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues) GitHub section for other people who solved similar problems or directly asking us for help.\n\n### 1. Clone the Repository\n\nStart by cloning the repository and navigating to the project directory:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook.git\ncd LLM-Engineers-Handbook \n```\n\nNext, we have to prepare your Python environment and its adjacent dependencies. \n\n### 2. Set Up Python Environment\n\nThe project requires Python 3.11. You can either use your global Python installation or set up a project-specific version using pyenv.\n\n#### Option A: Using Global Python (if version 3.11 is installed)\n\nVerify your Python version:\n\n```bash\npython --version  # Should show Python 3.11.x\n```\n\n#### Option B: Using pyenv (recommended)\n\n1. Verify pyenv installation:\n\n```bash\npyenv --version   # Should show pyenv 2.3.36 or later\n```\n\n2. Install Python 3.11.8:\n\n```bash\npyenv install 3.11.8\n```\n\n3. Verify the installation:\n\n```bash\npython --version  # Should show Python 3.11.8\n```\n\n4. Confirm Python version in the project directory:\n\n```bash\npython --version\n# Output: Python 3.11.8\n```\n\n> [!NOTE]  \n> The project includes a `.python-version` file that automatically sets the correct Python version when you're in the project directory.\n\n### 3. Install Dependencies\n\nThe project uses Poetry for dependency management.\n\n1. Verify Poetry installation:\n\n```bash\npoetry --version  # Should show Poetry version 1.8.3 or later\n```\n\n2. Set up the project environment and install dependencies:\n\n```bash\npoetry env use 3.11\npoetry install --without aws\npoetry run pre-commit install\n```\n\nThis will:\n\n- Configure Poetry to use Python 3.11\n- Install project dependencies (excluding AWS-specific packages)\n- Set up pre-commit hooks for code verification\n\n### 4. Activate the Environment\n\nAs our task manager, we run all the scripts using [Poe the Poet](https:\u002F\u002Fpoethepoet.natn.io\u002Findex.html).\n\n1. Start a Poetry shell:\n\n```bash\npoetry shell\n```\n\n2. Run project commands using Poe the Poet:\n\n```bash\npoetry poe ...\n```\n\n\u003Cdetails>\n\u003Csummary>🔧 Troubleshooting Poe the Poet Installation\u003C\u002Fsummary>\n\n### Alternative Command Execution\n\nIf you're experiencing issues with `poethepoet`, you can still run the project commands directly through Poetry. Here's how:\n\n1. Look up the command definition in `pyproject.toml`\n2. Use `poetry run` with the underlying command\n\n#### Example:\nInstead of:\n```bash\npoetry poe local-infrastructure-up\n```\nUse the direct command from pyproject.toml:\n```bash\npoetry run \u003Cactual-command-from-pyproject-toml>\n```\nNote: All project commands are defined in the [tool.poe.tasks] section of pyproject.toml\n\u003C\u002Fdetails>\n\nNow, let's configure our local project with all the necessary credentials and tokens to run the code locally.\n\n### 5. Local Development Setup\n\nAfter you have installed all the dependencies, you must create and fill a `.env` file with your credentials to appropriately interact with other services and run the project. Setting your sensitive credentials in a `.env` file is a good security practice, as this file won't be committed to GitHub or shared with anyone else. \n\n1. First, copy our example by running the following:\n\n```bash\ncp .env.example .env # The file must be at your repository's root!\n```\n\n2. Now, let's understand how to fill in all the essential variables within the `.env` file to get you started. The following are the mandatory settings we must complete when working locally:\n\n#### OpenAI\n\nTo authenticate to OpenAI's API, you must fill out the `OPENAI_API_KEY` env var with an authentication token.\n\n```env\nOPENAI_API_KEY=your_api_key_here\n```\n\n→ Check out this [tutorial](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fquickstart) to learn how to provide one from OpenAI.\n\n#### Hugging Face\n\nTo authenticate to Hugging Face, you must fill out the `HUGGINGFACE_ACCESS_TOKEN` env var with an authentication token.\n\n```env\nHUGGINGFACE_ACCESS_TOKEN=your_token_here\n```\n\n→ Check out this [tutorial](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fhub\u002Fen\u002Fsecurity-tokens) to learn how to provide one from Hugging Face.\n\n#### Comet ML & Opik\n\nTo authenticate to Comet ML (required only during training) and Opik, you must fill out the `COMET_API_KEY` env var with your authentication token.\n\n```env\nCOMET_API_KEY=your_api_key_here\n```\n\n→ Check out this [tutorial](https:\u002F\u002Fwww.comet.com\u002Fdocs\u002Fopik\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik) to learn how to get started with Opik. You can also access Opik's dashboard using 🔗[this link](https:\u002F\u002Fwww.comet.com\u002Fopik?utm_source=llm_handbook&utm_medium=github&utm_content=opik).\n\n### 6. Deployment Setup\n\nWhen deploying the project to the cloud, we must set additional settings for Mongo, Qdrant, and AWS. If you are just working locally, the default values of these env vars will work out of the box. Detailed deployment instructions are available in Chapter 11 of the [LLM Engineer's Handbook](https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F).\n\n#### MongoDB\n\nWe must change the `DATABASE_HOST` env var with the URL pointing to your cloud MongoDB cluster.\n\n```env\nDATABASE_HOST=your_mongodb_url\n```\n\n→ Check out this [tutorial](https:\u002F\u002Fwww.mongodb.com\u002Fresources\u002Fproducts\u002Ffundamentals\u002Fmongodb-cluster-setup) to learn how to create and host a MongoDB cluster for free.\n\n#### Qdrant\n\nChange `USE_QDRANT_CLOUD` to `true`, `QDRANT_CLOUD_URL` with the URL point to your cloud Qdrant cluster, and `QDRANT_APIKEY` with its API key.\n\n```env\nUSE_QDRANT_CLOUD=true\nQDRANT_CLOUD_URL=your_qdrant_cloud_url\nQDRANT_APIKEY=your_qdrant_api_key\n```\n\n→ Check out this [tutorial](https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fcloud\u002Fcreate-cluster\u002F) to learn how to create a Qdrant cluster for free\n\n#### AWS\n\nFor your AWS set-up to work correctly, you need the AWS CLI installed on your local machine and properly configured with an admin user (or a user with enough permissions to create new SageMaker, ECR, and S3 resources; using an admin user will make everything more straightforward).\n\nChapter 2 provides step-by-step instructions on how to install the AWS CLI, create an admin user on AWS, and get an access key to set up the `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` environment variables. If you already have an AWS admin user in place, you have to configure the following env vars in your `.env` file:\n\n```bash\nAWS_REGION=eu-central-1 # Change it with your AWS region.\nAWS_ACCESS_KEY=your_aws_access_key\nAWS_SECRET_KEY=your_aws_secret_key\n```\n\nAWS credentials are typically stored in `~\u002F.aws\u002Fcredentials`. You can view this file directly using `cat` or similar commands:\n\n```bash\ncat ~\u002F.aws\u002Fcredentials\n```\n\n> [!IMPORTANT]\n> Additional configuration options are available in [settings.py](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fblob\u002Fmain\u002Fllm_engineering\u002Fsettings.py). Any variable in the `Settings` class can be configured through the `.env` file. \n\n## 🏗️ Infrastructure\n\n### Local infrastructure (for testing and development)\n\nWhen running the project locally, we host a MongoDB and Qdrant database using Docker. Also, a testing ZenML server is made available through their Python package.\n\n> [!WARNING]\n> You need Docker installed (>= v27.1.1)\n\nFor ease of use, you can start the whole local development infrastructure with the following command:\n```bash\npoetry poe local-infrastructure-up\n```\n\nAlso, you can stop the ZenML server and all the Docker containers using the following command:\n```bash\npoetry poe local-infrastructure-down\n```\n\n> [!WARNING]  \n> When running on MacOS, before starting the server, export the following environment variable:\n> `export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES`\n> Otherwise, the connection between the local server and pipeline will break. 🔗 More details in [this issue](https:\u002F\u002Fgithub.com\u002Fzenml-io\u002Fzenml\u002Fissues\u002F2369).\n> This is done by default when using Poe the Poet.\n\nStart the inference real-time RESTful API:\n```bash\npoetry poe run-inference-ml-service\n```\n\n> [!IMPORTANT]\n> The LLM microservice, called by the RESTful API, will work only after deploying the LLM to AWS SageMaker.\n\n#### ZenML\n\nDashboard URL: `localhost:8237`\n\nDefault credentials:\n  - `username`: default\n  - `password`: \n\n→ Find out more about using and setting up [ZenML](https:\u002F\u002Fdocs.zenml.io\u002F).\n\n#### Qdrant\n\nREST API URL: `localhost:6333`\n\nDashboard URL: `localhost:6333\u002Fdashboard`\n\n→ Find out more about using and setting up [Qdrant with Docker](https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fquick-start\u002F).\n\n#### MongoDB\n\nDatabase URI: `mongodb:\u002F\u002Fllm_engineering:llm_engineering@127.0.0.1:27017`\n\nDatabase name: `twin`\n\nDefault credentials:\n  - `username`: llm_engineering\n  - `password`: llm_engineering\n\n→ Find out more about using and setting up [MongoDB with Docker](https:\u002F\u002Fwww.mongodb.com\u002Fdocs\u002Fmanual\u002Ftutorial\u002Finstall-mongodb-community-with-docker).\n\nYou can search your MongoDB collections using your **IDEs MongoDB plugin** (which you have to install separately), where you have to use the database URI to connect to the MongoDB database hosted within the Docker container: `mongodb:\u002F\u002Fllm_engineering:llm_engineering@127.0.0.1:27017`\n\n> [!IMPORTANT]\n> Everything related to training or running the LLMs (e.g., training, evaluation, inference) can only be run if you set up AWS SageMaker, as explained in the next section on cloud infrastructure.\n\n### Cloud infrastructure (for production)\n\nHere we will quickly present how to deploy the project to AWS and other serverless services. We won't go into the details (as everything is presented in the book) but only point out the main steps you have to go through.\n\nFirst, reinstall your Python dependencies with the AWS group:\n```bash\npoetry install --with aws\n```\n\n#### AWS SageMaker\n\n> [!NOTE]\n> Chapter 10 provides step-by-step instructions in the section \"Implementing the LLM microservice using AWS SageMaker\".\n\nBy this point, we expect you to have AWS CLI installed and your AWS CLI and project's env vars (within the `.env` file) properly configured with an AWS admin user.\n\nTo ensure best practices, we must create a new AWS user restricted to creating and deleting only resources related to AWS SageMaker. Create it by running:\n```bash\npoetry poe create-sagemaker-role\n```\nIt will create a `sagemaker_user_credentials.json` file at the root of your repository with your new `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` values. **But before replacing your new AWS credentials, also run the following command to create the execution role (to create it using your admin credentials).**\n\nTo create the IAM execution role used by AWS SageMaker to access other AWS resources on our behalf, run the following:\n```bash\npoetry poe create-sagemaker-execution-role\n```\nIt will create a `sagemaker_execution_role.json` file at the root of your repository with your new `AWS_ARN_ROLE` value. Add it to your `.env` file. \n\nOnce you've updated the `AWS_ACCESS_KEY`, `AWS_SECRET_KEY`, and `AWS_ARN_ROLE` values in your `.env` file, you can use AWS SageMaker. **Note that this step is crucial to complete the AWS setup.**\n\n#### Training\n\nWe start the training pipeline through ZenML by running the following:\n```bash\npoetry poe run-training-pipeline\n```\nThis will start the training code using the configs from `configs\u002Ftraining.yaml` directly in SageMaker. You can visualize the results in Comet ML's dashboard.\n\nWe start the evaluation pipeline through ZenML by running the following:\n```bash\npoetry poe run-evaluation-pipeline\n```\nThis will start the evaluation code using the configs from `configs\u002Fevaluating.yaml` directly in SageMaker. You can visualize the results in `*-results` datasets saved to your Hugging Face profile.\n\n#### Inference\n\nTo create an AWS SageMaker Inference Endpoint, run:\n```bash\npoetry poe deploy-inference-endpoint\n```\nTo test it out, run:\n```bash\npoetry poe test-sagemaker-endpoint\n```\nTo delete it, run:\n```bash\npoetry poe delete-inference-endpoint\n```\n\n#### AWS: ML pipelines, artifacts, and containers\n\nThe ML pipelines, artifacts, and containers are deployed to AWS by leveraging ZenML's deployment features. Thus, you must create an account with ZenML Cloud and follow their guide on deploying a ZenML stack to AWS. Otherwise, we provide step-by-step instructions in **Chapter 11**, section **Deploying the LLM Twin's pipelines to the cloud** on what you must do.  \n\n#### Qdrant & MongoDB\n\nWe leverage Qdrant's and MongoDB's serverless options when deploying the project. Thus, you can either follow [Qdrant's](https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fcloud\u002Fcreate-cluster\u002F) and [MongoDB's](https:\u002F\u002Fwww.mongodb.com\u002Fresources\u002Fproducts\u002Ffundamentals\u002Fmongodb-cluster-setup) tutorials on how to create a freemium cluster for each or go through **Chapter 11**, section **Deploying the LLM Twin's pipelines to the cloud** and follow our step-by-step instructions.\n\n#### GitHub Actions\n\nWe use GitHub Actions to implement our CI\u002FCD pipelines. To implement your own, you have to fork our repository and set the following env vars as Actions secrets in your forked repository:\n- `AWS_ACCESS_KEY_ID`\n- `AWS_SECRET_ACCESS_KEY`\n- `AWS_ECR_NAME`\n- `AWS_REGION`\n\nAlso, we provide instructions on how to set everything up in **Chapter 11**, section **Adding LLMOps to the LLM Twin**.\n\n#### Comet ML & Opik\n\nYou can visualize the results on their self-hosted dashboards if you create a Comet account and correctly set the `COMET_API_KEY` env var. As Opik is powered by Comet, you don't have to set up anything else along Comet:\n- [Comet ML (for experiment tracking)](https:\u002F\u002Fwww.comet.com\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik)\n- [Opik (for prompt monitoring)](https:\u002F\u002Fwww.comet.com\u002Fopik?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik)\n\n### 💰 Running the Project Costs\n\nWe will mostly stick to free tiers for all the services except for AWS and OpenAI's API, which are both pay-as-you-go services. The cost of running the project once, with our default values, will be roughly ~$25 (most of it comes from using AWS SageMaker for training and inference).\n\n## ⚡ Pipelines\n\nAll the ML pipelines will be orchestrated behind the scenes by [ZenML](https:\u002F\u002Fwww.zenml.io\u002F). A few exceptions exist when running utility scrips, such as exporting or importing from the data warehouse.\n\nThe ZenML pipelines are the entry point for most processes throughout this project. They are under the `pipelines\u002F` folder. Thus, when you want to understand or debug a workflow, starting with the ZenML pipeline is the best approach.\n\nTo see the pipelines running and their results:\n- go to your ZenML dashboard\n- go to the `Pipelines` section\n- click on a specific pipeline (e.g., `feature_engineering`)\n- click on a specific run (e.g., `feature_engineering_run_2024_06_20_18_40_24`)\n- click on a specific step or artifact of the DAG to find more details about it\n\nNow, let's explore all the pipelines you can run. From data collection to training, we will present them in their natural order to go through the LLM project end-to-end.\n\n### Data pipelines\n\nRun the data collection ETL:\n```bash\npoetry poe run-digital-data-etl\n```\n\n> [!WARNING]\n> You must have Chrome (or another Chromium-based browser) installed on your system for LinkedIn and Medium crawlers to work (which use Selenium under the hood). Based on your Chrome version, the Chromedriver will be automatically installed to enable Selenium support. Another option is to run everything using our Docker image if you don't want to install Chrome. For example, to run all the pipelines combined you can run `poetry poe run-docker-end-to-end-data-pipeline`. Note that the command can be tweaked to support any other pipeline.\n>\n> If, for any other reason, you don't have a Chromium-based browser installed and don't want to use Docker, you have two other options to bypass this Selenium issue:\n> - Comment out all the code related to Selenium, Chrome and all the links that use Selenium to crawl them (e.g., Medium), such as the `chromedriver_autoinstaller.install()` command from [application.crawlers.base](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fblob\u002Fmain\u002Fllm_engineering\u002Fapplication\u002Fcrawlers\u002Fbase.py) and other static calls that check for Chrome drivers and Selenium.\n> - Install Google Chrome using your CLI in environments such as GitHub Codespaces or other cloud VMs using the same command as in our [Docker file](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fblob\u002Fmain\u002FDockerfile#L10).\n\nTo add additional links to collect from, go to `configs\u002Fdigital_data_etl_[author_name].yaml` and add them to the `links` field. Also, you can create a completely new file and specify it at run time, like this: `python -m llm_engineering.interfaces.orchestrator.run --run-etl --etl-config-filename configs\u002Fdigital_data_etl_[your_name].yaml`\n\nRun the feature engineering pipeline:\n```bash\npoetry poe run-feature-engineering-pipeline\n```\n\nGenerate the instruct dataset:\n```bash\npoetry poe run-generate-instruct-datasets-pipeline\n```\n\nGenerate the preference dataset:\n```bash\npoetry poe run-generate-preference-datasets-pipeline\n```\n\nRun all of the above compressed into a single pipeline:\n```bash\npoetry poe run-end-to-end-data-pipeline\n```\n\n### Utility pipelines\n\nExport the data from the data warehouse to JSON files:\n```bash\npoetry poe run-export-data-warehouse-to-json\n```\n\nImport data to the data warehouse from JSON files (by default, it imports the data from the `data\u002Fdata_warehouse_raw_data` directory):\n```bash\npoetry poe run-import-data-warehouse-from-json\n```\n\nExport ZenML artifacts to JSON:\n```bash\npoetry poe run-export-artifact-to-json-pipeline\n```\n\nThis will export the following ZenML artifacts to the `output` folder as JSON files (it will take their latest version):\n- cleaned_documents.json\n- instruct_datasets.json\n- preference_datasets.json\n- raw_documents.json\n\nYou can configure what artifacts to export by tweaking the `configs\u002Fexport_artifact_to_json.yaml` configuration file.\n\n### Training pipelines\n\nRun the training pipeline:\n```bash\npoetry poe run-training-pipeline\n```\n\nRun the evaluation pipeline:\n```bash\npoetry poe run-evaluation-pipeline\n```\n\n> [!WARNING]\n> For this to work, make sure you properly configured AWS SageMaker as described in [Set up cloud infrastructure (for production)](#set-up-cloud-infrastructure-for-production).\n\n### Inference pipelines\n\nCall the RAG retrieval module with a test query:\n```bash\npoetry poe call-rag-retrieval-module\n```\n\nStart the inference real-time RESTful API:\n```bash\npoetry poe run-inference-ml-service\n```\n\nCall the inference real-time RESTful API with a test query:\n```bash\npoetry poe call-inference-ml-service\n```\n\nRemember that you can monitor the prompt traces on [Opik](https:\u002F\u002Fwww.comet.com\u002Fopik).\n\n> [!WARNING]\n> For the inference service to work, you must have the LLM microservice deployed to AWS SageMaker, as explained in the setup cloud infrastructure section.\n\n### Linting & formatting (QA)\n\nCheck or fix your linting issues:\n```bash\npoetry poe lint-check\npoetry poe lint-fix\n```\n\nCheck or fix your formatting issues:\n```bash\npoetry poe format-check\npoetry poe format-fix\n```\n\nCheck the code for leaked credentials:\n```bash\npoetry poe gitleaks-check\n```\n\n### Tests\n\nRun all the tests using the following command:\n```bash\npoetry poe test\n```\n\n## 🏃 Run project\n\nBased on the setup and usage steps described above, assuming the local and cloud infrastructure works and the `.env` is filled as expected, follow the next steps to run the LLM system end-to-end:\n\n### Data\n\n1. Collect data: `poetry poe run-digital-data-etl`\n\n2. Compute features: `poetry poe run-feature-engineering-pipeline`\n\n3. Compute instruct dataset: `poetry poe run-generate-instruct-datasets-pipeline`\n\n4. Compute preference alignment dataset: `poetry poe run-generate-preference-datasets-pipeline`\n\n### Training\n\n> [!IMPORTANT]\n> From now on, for these steps to work, you need to properly set up AWS SageMaker, such as running `poetry install --with aws` and filling in the AWS-related environment variables and configs.\n\n5. SFT fine-tuning Llamma 3.1: `poetry poe run-training-pipeline`\n\n6. For DPO, go to `configs\u002Ftraining.yaml`, change `finetuning_type` to `dpo`, and run `poetry poe run-training-pipeline` again\n\n7. Evaluate fine-tuned models: `poetry poe run-evaluation-pipeline`\n\n### Inference\n\n> [!IMPORTANT]\n> From now on, for these steps to work, you need to properly set up AWS SageMaker, such as running `poetry install --with aws` and filling in the AWS-related environment variables and configs.\n\n8. Call only the RAG retrieval module: `poetry poe call-rag-retrieval-module`\n\n9. Deploy the LLM Twin microservice to SageMaker: `poetry poe deploy-inference-endpoint`\n\n10. Test the LLM Twin microservice: `poetry poe test-sagemaker-endpoint`\n\n11. Start end-to-end RAG server: `poetry poe run-inference-ml-service`\n\n12. Test RAG server: `poetry poe call-inference-ml-service`\n\n## 📄 License\n\nThis course is an open-source project released under the MIT license. Thus, as long you distribute our LICENSE and acknowledge our work, you can safely clone or fork this project and use it as a source of inspiration for whatever you want (e.g., university projects, college degree projects, personal projects, etc.).\n","\u003Cp align='center'>\u003Ca href='https:\u002F\u002Fwww.eventbrite.com\u002Fe\u002Fship-production-pytorch-system-in-a-day-train-optimize-deploy-workshop-tickets-1983348934052?aff=GitHub'>\u003Cimg src='https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_readme_d11aced4566a.png'\u002F>\u003C\u002Fa>\u003C\u002Fp>\n\n\u003Cp align='center'>\u003Ca href='https:\u002F\u002Fwww.packtpub.com\u002Fen-us\u002Funlock?step=1'>\u003Cimg src='https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_readme_7416d3f6219e.png'\u002F>\u003C\u002Fa>\u003C\u002Fp>\n\n\u003Cdiv align=\"center\">\n  \u003Ch1>👷 大语言模型工程师手册\u003C\u002Fh1>\n  \u003Cp class=\"tagline\">由 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fiusztinpaul\">Paul Iusztin\u003C\u002Fa> 和 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fmlabonne\">Maxime Labonne\u003C\u002Fa> 撰写的《大语言模型工程师手册》的官方仓库\u003C\u002Fp>\n\u003C\u002Fdiv>\n\u003C\u002Fbr>\n\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_readme_855fa73e767b.png\" alt=\"书籍封面\">\n  \u003C\u002Fa>\n\u003C\u002Fp>\n\n\u003Cp align=\"center\">\n  您可以在 \u003Ca href=\"https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F\">亚马逊\u003C\u002Fa> 或 \u003Ca href=\"https:\u002F\u002Fwww.packtpub.com\u002Fen-us\u002Fproduct\u002Fllm-engineers-handbook-9781836200062\">Packt\u003C\u002Fa> 上找到这本书。\n\u003C\u002Fp>\n\n## 🌟 特色功能\n\n本书的目标是帮助您使用最佳实践构建自己的端到端大语言模型系统：\n\n- 📝 数据收集与生成\n- 🔄 大语言模型训练流水线\n- 📊 简单的RAG系统\n- 🚀 生产就绪的AWS部署\n- 🔍 全面的监控\n- 🧪 测试与评估框架\n\n您可以从 [Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fmlabonne\u002FTwinLlama-3.1-8B-DPO) 下载并使用最终训练好的模型。\n\n> [!重要提示]\n> 本GitHub仓库中的代码会持续维护，可能会包含未反映在书中的更新。**请始终参考此仓库以获取最新版本的代码。**\n\n## 🔗 依赖项\n\n### 本地依赖\n\n要在本地安装并运行该项目，您需要以下依赖项。\n\n| 工具 | 版本 | 用途 | 安装链接 |\n|------|---------|---------|------------------|\n| pyenv | ≥2.3.36 | 多个Python版本（可选） | [安装指南](https:\u002F\u002Fgithub.com\u002Fpyenv\u002Fpyenv?tab=readme-ov-file#installation) |\n| Python | 3.11 | 运行时环境 | [下载](https:\u002F\u002Fwww.python.org\u002Fdownloads\u002F) |\n| Poetry | ≥ 1.8.3且\u003C 2.0 | 包管理 | [安装指南](https:\u002F\u002Fpython-poetry.org\u002Fdocs\u002F#installation) |\n| Docker | ≥27.1.1 | 容器化 | [安装指南](https:\u002F\u002Fdocs.docker.com\u002Fengine\u002Finstall\u002F) |\n| AWS CLI | ≥2.15.42 | 云管理 | [安装指南](https:\u002F\u002Fdocs.aws.amazon.com\u002Fcli\u002Flatest\u002Fuserguide\u002Fgetting-started-install.html) |\n| Git | ≥2.44.0 | 版本控制 | [下载](https:\u002F\u002Fgit-scm.com\u002Fdownloads) |\n\n### 云服务\n\n代码还使用并依赖于以下云服务。目前您无需做任何操作。我们将在安装和部署部分指导您如何使用它们：\n\n| 服务 | 用途 |\n|---------|---------|\n| [HuggingFace](https:\u002F\u002Fhuggingface.com\u002F) | 模型注册表 |\n| [Comet ML](https:\u002F\u002Fwww.comet.com\u002Fsite\u002Fproducts\u002Fopik\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik) | 实验跟踪器 |\n| [Opik](https:\u002F\u002Fwww.comet.com\u002Fsite\u002Fproducts\u002Fopik\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik) | 提示词监控 |\n| [ZenML](https:\u002F\u002Fwww.zenml.io\u002F) | 编排与工件层 |\n| [AWS](https:\u002F\u002Faws.amazon.com\u002F) | 计算与存储 |\n| [MongoDB](https:\u002F\u002Fwww.mongodb.com\u002F) | NoSQL数据库 |\n| [Qdrant](https:\u002F\u002Fqdrant.tech\u002F) | 向量数据库 |\n| [GitHub Actions](https:\u002F\u002Fgithub.com\u002Ffeatures\u002Factions) | CI\u002FCD流水线 |\n\n在《大语言模型工程师手册》（[亚马逊链接](https:\u002F\u002Fwww.amazon.com\u002FLLM-Engineers-Handbook-engineering-production\u002Fdp\u002F1836200072\u002F)）中，第2章将带您逐一了解每种工具。第10章和第11章则提供了设置所需一切的分步指南。\n\n## 🗂️ 项目结构\n\n以下是目录概览：\n\n```bash\n.\n├── code_snippets\u002F       # 独立示例代码\n├── configs\u002F             # 流水线配置文件\n├── llm_engineering\u002F     # 核心项目包\n│   ├── application\u002F    \n│   ├── domain\u002F         \n│   ├── infrastructure\u002F \n│   ├── model\u002F         \n├── pipelines\u002F           # ML流水线定义\n├── steps\u002F               # 流水线组件\n├── tests\u002F               # 测试示例\n├── tools\u002F               # 工具脚本\n│   ├── run.py\n│   ├── ml_service.py\n│   ├── rag.py\n│   ├── data_warehouse.py\n```\n\n`llm_engineering\u002F` 是实现大语言模型和RAG功能的主要Python包。它遵循领域驱动设计（DDD）原则：\n\n- `domain\u002F`：核心业务实体和结构\n- `application\u002F`：业务逻辑、爬虫及RAG实现\n- `model\u002F`：大语言模型的训练与推理\n- `infrastructure\u002F`：外部服务集成（AWS、Qdrant、MongoDB、FastAPI）\n\n代码逻辑和导入顺序如下：`infrastructure` → `model` → `application` → `domain`\n\n`pipelines\u002F`：包含ZenML ML流水线，作为所有ML流水线的入口点。协调ML生命周期中的数据处理和模型训练阶段。\n\n`steps\u002F`：包含独立的ZenML步骤，这些步骤是用于构建和定制ZenML流水线的可重用组件。步骤执行特定任务（如数据加载、预处理），并可在ML流水线中组合使用。\n\n`tests\u002F`：涵盖了一些作为CI流水线示例的测试用例。\n\n`tools\u002F`：用于调用ZenML流水线和推理代码的实用脚本：\n- `run.py`：运行ZenML流水线的入口脚本。\n- `ml_service.py`：启动REST API推理服务器。\n- `rag.py`：演示RAG检索模块的使用。\n- `data_warehouse.py`：用于通过JSON文件从MongoDB数据仓库导出或导入数据。\n\n`configs\u002F`：ZenML YAML配置文件，用于控制流水线和步骤的执行。\n\n`code_snippets\u002F`：可独立运行的独立代码示例。\n\n## 💻 安装\n\n> [!注意]\n> 如果您在安装和运行仓库时遇到问题，请查看 [Issues](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues) GitHub页面，看看是否有其他人解决了类似的问题，或者直接向我们寻求帮助。\n\n### 1. 克隆仓库\n\n首先克隆仓库并进入项目目录：\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook.git\ncd LLM-Engineers-Handbook \n```\n\n接下来，我们需要准备您的Python环境及其相关依赖。\n\n### 2. 配置 Python 环境\n\n该项目需要 Python 3.11。您可以使用全局安装的 Python，也可以通过 pyenv 设置一个项目专用版本。\n\n#### 选项 A：使用全局 Python（如果已安装 3.11 版本）\n\n验证您的 Python 版本：\n\n```bash\npython --version  # 应显示 Python 3.11.x\n```\n\n#### 选项 B：使用 pyenv（推荐）\n\n1. 验证 pyenv 是否已安装：\n\n```bash\npyenv --version   # 应显示 pyenv 2.3.36 或更高版本\n```\n\n2. 安装 Python 3.11.8：\n\n```bash\npyenv install 3.11.8\n```\n\n3. 验证安装是否成功：\n\n```bash\npython --version  # 应显示 Python 3.11.8\n```\n\n4. 在项目目录中确认 Python 版本：\n\n```bash\npython --version\n# 输出：Python 3.11.8\n```\n\n> [!注意]  \n> 项目包含一个 `.python-version` 文件，当您位于项目目录时，它会自动设置正确的 Python 版本。\n\n### 3. 安装依赖项\n\n该项目使用 Poetry 进行依赖管理。\n\n1. 验证 Poetry 是否已安装：\n\n```bash\npoetry --version  # 应显示 Poetry 1.8.3 或更高版本\n```\n\n2. 设置项目环境并安装依赖项：\n\n```bash\npoetry env use 3.11\npoetry install --without aws\npoetry run pre-commit install\n```\n\n这将：\n\n- 配置 Poetry 使用 Python 3.11\n- 安装项目依赖项（不包括 AWS 相关包）\n- 设置 pre-commit 钩子以进行代码验证\n\n### 4. 激活环境\n\n作为我们的任务管理工具，我们将使用 [Poe the Poet](https:\u002F\u002Fpoethepoet.natn.io\u002Findex.html) 来运行所有脚本。\n\n1. 启动 Poetry shell：\n\n```bash\npoetry shell\n```\n\n2. 使用 Poe the Poet 运行项目命令：\n\n```bash\npoetry poe ...\n```\n\n\u003Cdetails>\n\u003Csummary>🔧 Poe the Poet 安装故障排除\u003C\u002Fsummary>\n\n### 替代命令执行方式\n\n如果您在使用 `poethepoet` 时遇到问题，仍然可以通过 Poetry 直接运行项目命令。方法如下：\n\n1. 查看 `pyproject.toml` 中的命令定义\n2. 使用 `poetry run` 执行底层命令\n\n#### 示例：\n与其运行：\n```bash\npoetry poe local-infrastructure-up\n```\n不如直接使用 `pyproject.toml` 中的命令：\n```bash\npoetry run \u003C实际命令从 pyproject.toml 中获取>\n```\n注意：所有项目命令均定义在 `pyproject.toml` 的 [tool.poe.tasks] 部分。\n\u003C\u002Fdetails>\n\n现在，让我们配置本地项目所需的全部凭据和令牌，以便在本地运行代码。\n\n### 5. 本地开发设置\n\n在安装完所有依赖项后，您必须创建并填写 `.env` 文件，输入您的凭据，以便与其它服务正常交互并运行项目。将敏感凭据存储在 `.env` 文件中是一种良好的安全实践，因为该文件不会被提交到 GitHub 或与他人共享。\n\n1. 首先，复制示例文件：\n\n```bash\ncp .env.example .env # 文件必须位于仓库根目录！\n```\n\n2. 接下来，我们来了解如何填写 `.env` 文件中的必要变量，以便开始使用。以下是我们在本地工作时必须完成的必填设置：\n\n#### OpenAI\n\n要认证 OpenAI API，您需要将 `OPENAI_API_KEY` 环境变量设置为认证令牌。\n\n```env\nOPENAI_API_KEY=your_api_key_here\n```\n\n→ 请参阅此[教程](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fquickstart)，了解如何从 OpenAI 获取认证令牌。\n\n#### Hugging Face\n\n要认证 Hugging Face，您需要将 `HUGGINGFACE_ACCESS_TOKEN` 环境变量设置为认证令牌。\n\n```env\nHUGGINGFACE_ACCESS_TOKEN=your_token_here\n```\n\n→ 请参阅此[教程](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fhub\u002Fen\u002Fsecurity-tokens)，了解如何从 Hugging Face 获取认证令牌。\n\n#### Comet ML 和 Opik\n\n要认证 Comet ML（仅在训练期间需要）和 Opik，您需要将 `COMET_API_KEY` 环境变量设置为认证令牌。\n\n```env\nCOMET_API_KEY=your_api_key_here\n```\n\n→ 请参阅此[教程](https:\u002F\u002Fwww.comet.com\u002Fdocs\u002Fopik\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik)，了解如何开始使用 Opik。您还可以通过🔗[此链接](https:\u002F\u002Fwww.comet.com\u002Fopik?utm_source=llm_handbook&utm_medium=github&utm_content=opik)访问 Opik 的仪表板。\n\n### 6. 部署设置\n\n将项目部署到云端时，我们需要为 MongoDB、Qdrant 和 AWS 设置额外的参数。如果您只是在本地工作，这些环境变量的默认值即可直接使用。详细的部署说明可在《LLM 工程师手册》第 11 章中找到。\n\n#### MongoDB\n\n我们需要将 `DATABASE_HOST` 环境变量更改为指向您的云 MongoDB 集群的 URL。\n\n```env\nDATABASE_HOST=your_mongodb_url\n```\n\n→ 请参阅此[教程](https:\u002F\u002Fwww.mongodb.com\u002Fresources\u002Fproducts\u002Ffundamentals\u002Fmongodb-cluster-setup)，了解如何免费创建并托管 MongoDB 集群。\n\n#### Qdrant\n\n将 `USE_QDRANT_CLOUD` 设置为 `true`，并将 `QDRANT_CLOUD_URL` 更改为指向您的云 Qdrant 集群的 URL，同时设置 `QDRANT_APIKEY` 为该集群的 API 密钥。\n\n```env\nUSE_QDRANT_CLOUD=true\nQDRANT_CLOUD_URL=your_qdrant_cloud_url\nQDRANT_APIKEY=your_qdrant_api_key\n```\n\n→ 请参阅此[教程](https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fcloud\u002Fcreate-cluster\u002F)，了解如何免费创建 Qdrant 集群。\n\n#### AWS\n\n为了使您的 AWS 配置正常工作，您需要在本地机器上安装并正确配置 AWS CLI，并使用具有管理员权限的用户（或至少拥有足够权限以创建 SageMaker、ECR 和 S3 资源的用户；使用管理员用户会使流程更加简单）。\n\n第 2 章提供了逐步指导，介绍如何安装 AWS CLI、在 AWS 上创建管理员用户以及获取访问密钥，以便设置 `AWS_ACCESS_KEY` 和 `AWS_SECRET_KEY` 环境变量。如果您已经有一个 AWS 管理员用户，则需要在 `.env` 文件中配置以下环境变量：\n\n```bash\nAWS_REGION=eu-central-1 # 根据您的 AWS 区域进行更改。\nAWS_ACCESS_KEY=your_aws_access_key\nAWS_SECRET_KEY=your_aws_secret_key\n```\n\nAWS 凭据通常存储在 `~\u002F.aws\u002Fcredentials` 文件中。您可以通过 `cat` 等命令直接查看该文件：\n\n```bash\ncat ~\u002F.aws\u002Fcredentials\n```\n\n> [!重要提示]\n> 更多配置选项可在 [settings.py](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fblob\u002Fmain\u002Fllm_engineering\u002Fsettings.py) 中找到。`Settings` 类中的任何变量都可以通过 `.env` 文件进行配置。\n\n## 🏗️ 基础设施\n\n### 本地基础设施（用于测试和开发）\n\n在本地运行项目时，我们使用 Docker 托管 MongoDB 和 Qdrant 数据库。此外，还通过 ZenML 的 Python 包提供了一个用于测试的 ZenML 服务器。\n\n> [!WARNING]  \n> 您需要安装 Docker（>= v27.1.1）\n\n为了方便使用，您可以使用以下命令启动整个本地开发基础设施：\n```bash\npoetry poe local-infrastructure-up\n```\n\n您还可以使用以下命令停止 ZenML 服务器和所有 Docker 容器：\n```bash\npoetry poe local-infrastructure-down\n```\n\n> [!WARNING]  \n> 在 macOS 上运行时，在启动服务器之前，请导出以下环境变量：  \n> `export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES`  \n> 否则，本地服务器与管道之间的连接将会中断。🔗 更多详情请参见 [此问题](https:\u002F\u002Fgithub.com\u002Fzenml-io\u002Fzenml\u002Fissues\u002F2369)。  \n> 使用 Poe the Poet 时会默认执行此操作。\n\n启动实时推理 RESTful API：\n```bash\npoetry poe run-inference-ml-service\n```\n\n> [!IMPORTANT]  \n> 由 RESTful API 调用的 LLM 微服务，只有在将 LLM 部署到 AWS SageMaker 后才能正常工作。\n\n#### ZenML\n\n仪表板 URL：`localhost:8237`\n\n默认凭据：\n  - `username`: default\n  - `password`: （留空）\n\n→ 了解更多关于使用和设置 [ZenML](https:\u002F\u002Fdocs.zenml.io\u002F) 的信息。\n\n#### Qdrant\n\nREST API URL：`localhost:6333`\n\n仪表板 URL：`localhost:6333\u002Fdashboard`\n\n→ 了解更多关于使用和设置 [Qdrant with Docker](https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fquick-start\u002F) 的信息。\n\n#### MongoDB\n\n数据库 URI：`mongodb:\u002F\u002Fllm_engineering:llm_engineering@127.0.0.1:27017`\n\n数据库名称：`twin`\n\n默认凭据：\n  - `username`: llm_engineering\n  - `password`: llm_engineering\n\n→ 了解更多关于使用和设置 [MongoDB with Docker](https:\u002F\u002Fwww.mongodb.com\u002Fdocs\u002Fmanual\u002Ftutorial\u002Finstall-mongodb-community-with-docker) 的信息。\n\n您可以使用 IDE 的 MongoDB 插件（需单独安装）来查询您的 MongoDB 集合，连接时需使用 Docker 容器内托管的 MongoDB 数据库 URI：`mongodb:\u002F\u002Fllm_engineering:llm_engineering@127.0.0.1:27017`\n\n> [!IMPORTANT]  \n> 与训练或运行 LLM 相关的所有操作（例如训练、评估、推理）都必须在设置好 AWS SageMaker 的前提下才能执行，具体说明请参见下一节关于云基础设施的内容。\n\n### 云基础设施（用于生产）\n\n在这里，我们将简要介绍如何将项目部署到 AWS 及其他无服务器服务。我们不会深入细节（因为书中已详细说明），而只列出您需要完成的主要步骤。\n\n首先，重新安装包含 AWS 组的 Python 依赖项：\n```bash\npoetry install --with aws\n```\n\n#### AWS SageMaker\n\n> [!NOTE]  \n> 第 10 章“使用 AWS SageMaker 实现 LLM 微服务”部分提供了逐步指导。\n\n此时，我们假设您已安装 AWS CLI，并且已正确配置了 AWS CLI 以及项目 `.env` 文件中的环境变量，使用的是具有管理员权限的 AWS 用户。\n\n为确保最佳实践，我们需要创建一个仅限于创建和删除与 AWS SageMaker 相关资源的新 AWS 用户。通过运行以下命令创建该用户：\n```bash\npoetry poe create-sagemaker-role\n```\n这将在您的仓库根目录下生成一个 `sagemaker_user_credentials.json` 文件，其中包含新的 `AWS_ACCESS_KEY` 和 `AWS_SECRET_KEY` 值。**但在替换您的新 AWS 凭证之前，还需运行以下命令以创建执行角色（使用您的管理员凭证创建）。**\n\n要创建 AWS SageMaker 代表我们访问其他 AWS 资源所使用的 IAM 执行角色，请运行：\n```bash\npoetry poe create-sagemaker-execution-role\n```\n这将在您的仓库根目录下生成一个 `sagemaker_execution_role.json` 文件，其中包含新的 `AWS_ARN_ROLE` 值。请将其添加到您的 `.env` 文件中。\n\n一旦您更新了 `.env` 文件中的 `AWS_ACCESS_KEY`、`AWS_SECRET_KEY` 和 `AWS_ARN_ROLE` 值，就可以开始使用 AWS SageMaker。**请注意，此步骤是完成 AWS 设置的关键。**\n\n#### 训练\n\n我们通过 ZenML 运行以下命令来启动训练管道：\n```bash\npoetry poe run-training-pipeline\n```\n这将直接在 SageMaker 中使用 `configs\u002Ftraining.yaml` 中的配置启动训练代码。您可以在 Comet ML 的仪表板上查看结果。\n\n我们通过 ZenML 运行以下命令来启动评估管道：\n```bash\npoetry poe run-evaluation-pipeline\n```\n这将直接在 SageMaker 中使用 `configs\u002Fevaluating.yaml` 中的配置启动评估代码。您可以在保存到 Hugging Face 个人主页的 `*-results` 数据集中查看结果。\n\n#### 推理\n\n要创建 AWS SageMaker 推理端点，请运行：\n```bash\npoetry poe deploy-inference-endpoint\n```\n要测试它，请运行：\n```bash\npoetry poe test-sagemaker-endpoint\n```\n要删除它，请运行：\n```bash\npoetry poe delete-inference-endpoint\n```\n\n#### AWS：ML 管道、工件和容器\n\nML 管道、工件和容器都是通过利用 ZenML 的部署功能部署到 AWS 的。因此，您必须注册一个 ZenML Cloud 账户，并按照其指南将 ZenML 堆栈部署到 AWS。否则，我们在 **第 11 章**“将 LLM Twin 的管道部署到云端”部分提供了逐步指导，说明您需要做什么。\n\n#### Qdrant 和 MongoDB\n\n我们在部署项目时利用了 Qdrant 和 MongoDB 的无服务器选项。因此，您可以选择遵循 [Qdrant](https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fcloud\u002Fcreate-cluster\u002F) 和 [MongoDB](https:\u002F\u002Fwww.mongodb.com\u002Fresources\u002Fproducts\u002Ffundamentals\u002Fmongodb-cluster-setup) 提供的教程，分别为它们创建一个免费集群，或者参考 **第 11 章**“将 LLM Twin 的管道部署到云端”部分，按照我们的逐步指导进行操作。\n\n#### GitHub Actions\n\n我们使用 GitHub Actions 来实现 CI\u002FCD 管道。要实现您自己的 CI\u002FCD 管道，您需要 fork 我们的仓库，并在您的 forked 仓库中将以下环境变量设置为 Actions secrets：\n- `AWS_ACCESS_KEY_ID`\n- `AWS_SECRET_ACCESS_KEY`\n- `AWS_ECR_NAME`\n- `AWS_REGION`\n\n我们还在 **第 11 章**“将 LLMOps 添加到 LLM Twin”部分提供了关于如何设置这一切的说明。\n\n#### Comet ML 和 Opik\n\n如果您创建了一个 Comet 账户并正确设置了 `COMET_API_KEY` 环境变量，就可以在它们的自托管仪表板上查看结果。由于 Opik 是基于 Comet 构建的，因此您无需为 Opik 做任何额外的设置：\n- [Comet ML（用于实验跟踪）](https:\u002F\u002Fwww.comet.com\u002F?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik)\n- [Opik（用于提示监控）](https:\u002F\u002Fwww.comet.com\u002Fopik?utm_source=llm_handbook&utm_medium=github&utm_campaign=opik)\n\n### 💰 项目运行成本\n\n我们将主要使用各服务的免费层级，除了 AWS 和 OpenAI 的 API，它们都是按需付费的服务。按照我们的默认配置，项目一次性运行的成本大约为 ~$25（其中大部分来自使用 AWS SageMaker 进行训练和推理）。\n\n## ⚡ 管道\n\n所有机器学习管道都将由 [ZenML](https:\u002F\u002Fwww.zenml.io\u002F) 在后台进行编排。在运行一些实用脚本时会有少数例外，比如从数据仓库导出或导入数据。\n\nZenML 管道是本项目中大多数流程的入口点。它们位于 `pipelines\u002F` 文件夹下。因此，当你想要理解或调试某个工作流时，从 ZenML 管道入手是最好的方法。\n\n要查看正在运行的管道及其结果：\n- 前往你的 ZenML 控制台\n- 进入 `Pipelines` 部分\n- 点击特定的管道（例如 `feature_engineering`）\n- 点击特定的运行实例（例如 `feature_engineering_run_2024_06_20_18_40_24`）\n- 点击 DAG 中的特定步骤或工件以获取更多详细信息。\n\n现在，让我们来探索你可以运行的所有管道。从数据收集到训练，我们将按照自然顺序呈现它们，以便端到端地完成整个 LLM 项目。\n\n### 数据管道\n\n运行数据收集 ETL：\n```bash\npoetry poe run-digital-data-etl\n```\n\n> [!WARNING]\n> 为了使 LinkedIn 和 Medium 的爬虫正常工作（它们底层使用 Selenium），你的系统必须安装 Chrome 浏览器（或其他基于 Chromium 的浏览器）。根据你当前的 Chrome 版本，Chromedriver 将会自动安装以支持 Selenium。另一种选择是直接使用我们的 Docker 镜像来运行所有任务，这样你就无需安装 Chrome。例如，要同时运行所有管道，可以执行 `poetry poe run-docker-end-to-end-data-pipeline`。请注意，该命令也可以调整以支持其他任何管道。\n>\n> 如果由于其他原因你没有安装基于 Chromium 的浏览器，也不想使用 Docker，还有两种方法可以绕过这个 Selenium 问题：\n> - 注释掉所有与 Selenium、Chrome 相关的代码，以及所有依赖 Selenium 进行抓取的链接（如 Medium），例如从 [application.crawlers.base](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fblob\u002Fmain\u002Fllm_engineering\u002Fapplication\u002Fcrawlers\u002Fbase.py) 中的 `chromedriver_autoinstaller.install()` 命令，以及其他检查 Chrome 驱动和 Selenium 是否可用的静态调用。\n> - 在 GitHub Codespaces 或其他云虚拟机等环境中，使用 CLI 安装 Google Chrome，命令与我们 [Dockerfile](https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fblob\u002Fmain\u002FDockerfile#L10) 中的命令相同。\n\n要添加更多需要收集数据的链接，请前往 `configs\u002Fdigital_data_etl_[author_name].yaml` 文件，并将它们添加到 `links` 字段中。此外，你还可以创建一个全新的配置文件，并在运行时指定它，例如：`python -m llm_engineering.interfaces.orchestrator.run --run-etl --etl-config-filename configs\u002Fdigital_data_etl_[your_name].yaml`。\n\n运行特征工程管道：\n```bash\npoetry poe run-feature-engineering-pipeline\n```\n\n生成指令数据集：\n```bash\npoetry poe run-generate-instruct-datasets-pipeline\n```\n\n生成偏好数据集：\n```bash\npoetry poe run-generate-preference-datasets-pipeline\n```\n\n将上述所有步骤压缩成一个单一管道运行：\n```bash\npoetry poe run-end-to-end-data-pipeline\n``。\n\n### 实用管道\n\n将数据从数据仓库导出为 JSON 文件：\n```bash\npoetry poe run-export-data-warehouse-to-json\n```\n\n从 JSON 文件向数据仓库导入数据（默认情况下，它会从 `data\u002Fdata_warehouse_raw_data` 目录导入数据）：\n```bash\npoetry poe run-import-data-warehouse-from-json\n``。\n\n将 ZenML 工件导出为 JSON：\n```bash\npoetry poe run-export-artifact-to-json-pipeline\n``。\n\n这会将以下 ZenML 工件以 JSON 文件的形式导出到 `output` 文件夹中（采用最新版本）：\n- cleaned_documents.json\n- instruct_datasets.json\n- preference_datasets.json\n- raw_documents.json\n\n你可以通过调整 `configs\u002Fexport_artifact_to_json.yaml` 配置文件来指定要导出哪些工件。\n\n### 训练管道\n\n运行训练管道：\n```bash\npoetry poe run-training-pipeline\n``。\n\n运行评估管道：\n```bash\npoetry poe run-evaluation-pipeline\n``。\n\n> [!WARNING]\n> 为使此功能正常运行，务必按照 [设置云基础设施（用于生产环境）](#set-up-cloud-infrastructure-for-production) 中的说明正确配置 AWS SageMaker。\n\n### 推理管道\n\n使用测试查询调用 RAG 检索模块：\n```bash\npoetry poe call-rag-retrieval-module\n``。\n\n启动实时推理 RESTful API：\n```bash\npoetry poe run-inference-ml-service\n``。\n\n使用测试查询调用实时推理 RESTful API：\n```bash\npoetry poe call-inference-ml-service\n``。\n\n请记住，你可以在 [Opik](https:\u002F\u002Fwww.comet.com\u002Fopik) 上监控提示追踪信息。\n\n> [!WARNING]\n> 为使推理服务正常运行，你必须已将 LLM 微服务部署到 AWS SageMaker，具体操作请参考“设置云基础设施”部分。\n\n### 代码检查与格式化（QA）\n\n检查或修复代码中的 linting 问题：\n```bash\npoetry poe lint-check\npoetry poe lint-fix\n``。\n\n检查或修复代码格式问题：\n```bash\npoetry poe format-check\npoetry poe format-fix\n``。\n\n检查代码中是否存在泄露的凭据：\n```bash\npoetry poe gitleaks-check\n``。\n\n### 测试\n\n使用以下命令运行所有测试：\n```bash\npoetry poe test\n``。\n\n## 🏃 运行项目\n\n根据上述设置和使用步骤，假设本地和云基础设施均正常运行，并且 `.env` 文件已按预期填写完毕，按照以下步骤即可端到端运行 LLM 系统：\n\n### 数据\n\n1. 收集数据：`poetry poe run-digital-data-etl`\n\n2. 计算特征：`poetry poe run-feature-engineering-pipeline`\n\n3. 计算指令数据集：`poetry poe run-generate-instruct-datasets-pipeline`\n\n4. 计算偏好对齐数据集：`poetry poe run-generate-preference-datasets-pipeline`\n\n### 训练\n\n> [!IMPORTANT]\n> 从现在开始，要使这些步骤正常运行，你需要正确设置 AWS SageMaker，例如执行 `poetry install --with aws` 并填写与 AWS 相关的环境变量和配置。\n\n5. 使用 SFT 对 Llama 3.1 进行微调：`poetry poe run-training-pipeline`\n\n6. 如果要进行 DPO 训练，进入 `configs\u002Ftraining.yaml` 文件，将 `finetuning_type` 修改为 `dpo`，然后再次运行 `poetry poe run-training-pipeline`。\n\n7. 评估微调后的模型：`poetry poe run-evaluation-pipeline`\n\n### 推理\n\n> [!IMPORTANT]\n> 从现在开始，要使这些步骤正常运行，您需要正确设置 AWS SageMaker，例如执行 `poetry install --with aws` 命令，并填写与 AWS 相关的环境变量和配置。\n\n8. 仅调用 RAG 检索模块：`poetry poe call-rag-retrieval-module`\n\n9. 将 LLM Twin 微服务部署到 SageMaker：`poetry poe deploy-inference-endpoint`\n\n10. 测试 LLM Twin 微服务：`poetry poe test-sagemaker-endpoint`\n\n11. 启动端到端 RAG 服务器：`poetry poe run-inference-ml-service`\n\n12. 测试 RAG 服务器：`poetry poe call-inference-ml-service`\n\n## 📄 许可证\n\n本课程是一个基于 MIT 许可证发布的开源项目。因此，只要您分发我们的 LICENSE 文件并注明我们的工作出处，就可以放心地克隆或 fork 该项目，并将其作为您各种项目的灵感来源（例如，大学作业、毕业设计、个人项目等）。","# LLM-Engineers-Handbook 快速上手指南\n\n本指南基于《LLM Engineer's Handbook》官方仓库，帮助开发者快速构建端到端的 LLM 系统（包含数据收集、训练流水线、RAG 系统及 AWS 部署）。\n\n## 环境准备\n\n在开始之前，请确保您的开发环境满足以下要求：\n\n### 系统要求\n- **操作系统**: Linux, macOS 或 Windows (WSL2 推荐)\n- **Python 版本**: 3.11 (必须)\n- **磁盘空间**: 建议预留 20GB+ 用于模型和依赖安装\n\n### 前置依赖工具\n请预先安装以下工具（版本需满足最低要求）：\n\n| 工具 | 最低版本 | 用途 |\n|------|---------|------|\n| **Git** | 2.44.0 | 版本控制 |\n| **Docker** | 27.1.1 | 容器化运行 (本地数据库等) |\n| **Poetry** | 1.8.3 - 2.0 | Python 包管理 |\n| **pyenv** (可选) | 2.3.36 | 多 Python 版本管理 |\n| **AWS CLI** | 2.15.42 | 云资源管理 (部署阶段需要) |\n\n> **提示**: 国内用户若下载 Python 或 Docker 较慢，建议使用国内镜像源（如清华源、阿里源）进行安装。\n\n## 安装步骤\n\n### 1. 克隆项目\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook.git\ncd LLM-Engineers-Handbook\n```\n\n### 2. 配置 Python 环境\n项目强制要求 Python 3.11。推荐使用 `pyenv` 管理，或直接使用全局环境。\n\n**选项 A：使用 pyenv (推荐)**\n```bash\n# 安装 Python 3.11.8\npyenv install 3.11.8\n\n# 确认版本 (项目根目录包含 .python-version 文件会自动切换)\npython --version \n# 输出应为: Python 3.11.8\n```\n\n**选项 B：使用全局 Python**\n确保 `python --version` 显示为 3.11.x。\n\n### 3. 安装项目依赖\n本项目使用 Poetry 进行依赖管理。\n\n```bash\n# 指定使用 Python 3.11\npoetry env use 3.11\n\n# 安装依赖 (默认排除 AWS 特定包以加快安装速度)\npoetry install --without aws\n\n# 安装预提交钩子\npoetry run pre-commit install\n```\n\n### 4. 激活环境与任务运行器\n项目使用 `Poe the Poet` 作为任务管理器。\n\n```bash\n# 激活 Poetry 虚拟环境\npoetry shell\n\n# 验证命令是否可用 (后续所有操作均通过 poetry poe 执行)\npoetry poe --help\n```\n\n> **注意**: 如果 `poethepoet` 运行有问题，可直接查看 `pyproject.toml` 中的 `[tool.poe.tasks]` 部分，使用 `poetry run \u003C具体命令>` 替代。\n\n### 5. 配置环境变量\n创建 `.env` 文件并填入必要的 API 密钥。\n\n```bash\ncp .env.example .env\n```\n\n编辑 `.env` 文件，填入以下**必填**字段（本地运行最低要求）：\n\n```ini\n# OpenAI API Key (用于部分模型调用或评估)\nOPENAI_API_KEY=sk-...\n\n# Hugging Face Token (用于下载模型和推送)\nHUGGINGFACE_ACCESS_TOKEN=hf_...\n\n# Comet ML \u002F Opik Key (用于实验追踪和监控)\nCOMET_API_KEY=...\n```\n\n> **说明**: \n> - 本地开发时，MongoDB 和 Qdrant 将通过 Docker 自动启动，无需配置云端地址。\n> - 若需部署到云端，需在 `.env` 中额外配置 `DATABASE_HOST`, `QDRANT_CLOUD_URL` 及 AWS 凭证。\n\n## 基本使用\n\n项目核心功能通过 `tools\u002F` 目录下的脚本和 ZenML 流水线暴露。以下是几个最常用的入门操作：\n\n### 启动本地基础设施\n在运行任何流水线之前，先启动本地所需的数据库和服务（MongoDB, Qdrant 等）。\n\n```bash\npoetry poe local-infrastructure-up\n```\n\n### 运行数据处理与训练流水线\n执行完整的机器学习流水线（数据加载 -> 预处理 -> 训练）。\n\n```bash\npoetry poe run-pipeline\n```\n*注：具体流水线名称可在 `pipelines\u002F` 目录查看，上述命令通常指向默认的主流水线。*\n\n### 启动推理服务\n训练完成后，启动 REST API 服务进行模型推理。\n\n```bash\npoetry poe start-ml-service\n```\n服务启动后，可通过 `http:\u002F\u002Flocalhost:8000` 访问 API 文档。\n\n### 测试 RAG 检索模块\n运行简单的脚本来测试检索增强生成（RAG）功能。\n\n```bash\npoetry poe run-rag-demo\n```\n\n### 停止本地服务\n使用完毕后，关闭本地 Docker 容器以释放资源。\n\n```bash\npoetry poe local-infrastructure-down\n```","某初创团队急需在两周内为电商客服构建一个能处理复杂售后问题的智能问答系统，并部署到 AWS 云端。\n\n### 没有 LLM-Engineers-Handbook 时\n- **架构混乱**：团队成员各自为战，数据收集、模型微调与 RAG 检索模块缺乏统一标准，导致代码难以整合。\n- **部署困难**：缺乏成熟的 LLMOps 经验，手动配置 AWS 环境和容器化流程耗时耗力，多次因依赖冲突导致上线失败。\n- **监控缺失**：系统上线后无法有效追踪 Prompt 效果和模型表现，遇到回答错误时只能盲目猜测原因，缺乏评估框架。\n- **维护成本高**：由于未采用最佳实践，后续迭代需要重构大量代码，实验记录散乱，无法复现之前的训练结果。\n\n### 使用 LLM-Engineers-Handbook 后\n- **流程标准化**：直接复用书中端到端的最佳实践，快速搭建起从数据生成、训练流水线到简单 RAG 系统的完整架构。\n- **一键上云**：利用提供的 AWS 部署指南和 Docker 配置，将生产级应用顺利部署至云端，大幅缩短环境搭建时间。\n- **全链路可观测**：集成 Opik 和 Comet ML 实现了对 Prompt 的实时监控与实验追踪，能快速定位并优化回答质量。\n- **可持续迭代**：基于 ZenML 构建的编排层和 GitHub Actions CI\u002FCD 流水线，让模型更新和测试评估变得自动化且可复现。\n\nLLM-Engineers-Handbook 将原本需要数月摸索的工程化难题，转化为可按图索骥的标准作业流程，帮助团队高效交付生产级大模型应用。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FPacktPublishing_LLM-Engineers-Handbook_5e1edab6.png","PacktPublishing","Packt","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002FPacktPublishing_6d855195.jpg","Providing books, eBooks, video tutorials, and articles for IT developers, administrators, and users.",null,"https:\u002F\u002Fwww.packtpub.com","https:\u002F\u002Fgithub.com\u002FPacktPublishing",[81,85],{"name":82,"color":83,"percentage":84},"Python","#3572A5",99.2,{"name":86,"color":87,"percentage":88},"Dockerfile","#384d54",0.8,4917,1180,"2026-04-18T15:18:34","MIT",4,"未说明",{"notes":96,"python":97,"dependencies":98},"该项目主要依赖云端服务（AWS, MongoDB, Qdrant, HuggingFace 等）进行计算和存储，本地主要用于编排和开发。本地运行需配置 .env 文件填入 OpenAI、HuggingFace 和 Comet ML 的 API Key。建议使用 Poetry 管理依赖，并通过 'poetry install --without aws' 安装基础环境。若需部署到云端，需额外配置 MongoDB 集群、Qdrant 云服务和 AWS 凭证。代码逻辑遵循领域驱动设计 (DDD)，包含数据收集、LLM 训练、RAG 系统及生产级部署流程。","3.11",[99,100,101,102,103,104,105,106,107,108],"Poetry>=1.8.3","Docker>=27.1.1","AWS CLI>=2.15.42","Git>=2.44.0","pyenv>=2.3.36 (可选)","HuggingFace","Comet ML \u002F Opik","ZenML","MongoDB","Qdrant",[14,35,110],"其他",[112,113,114,115,116,117,118,119,120],"genai","llm","llmops","mlops","rag","aws","fine-tuning-llm","llm-evaluation","ml-system-design","2026-03-27T02:49:30.150509","2026-04-19T09:37:51.057365",[124,129,134,139,143,148],{"id":125,"question_zh":126,"answer_zh":127,"source_url":128},42320,"运行 `poetry poe` 命令时提示 'The requested command poe does not exist' 或 'Command not found' 怎么办？","这是因为未将 'Poe the Poet' 安装为 Poetry 的插件。请在全局安装 poe 是不够的，必须在当前 Poetry 环境中执行以下命令：\n\npoetry self add 'poethepoet[poetry_plugin]'\n\n执行后即可正常使用 `poetry poe` 相关命令。","https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues\u002F54",{"id":130,"question_zh":131,"answer_zh":132,"source_url":133},42321,"连接本地 MongoDB 失败或出现端口冲突错误如何解决？","该问题通常由以下两个原因导致：\n1. 本地已有另一个 MongoDB 实例在运行并占用了相同端口（默认 27017）。请先停止现有的 MongoDB 服务，再启动 Docker 容器。\n2. Docker 基础设施未正常启动。尝试重启计算机，然后重新运行启动本地基础设施的命令（如 `poetry poe local-infrastructure-up`）。","https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues\u002F15",{"id":135,"question_zh":136,"answer_zh":137,"source_url":138},42322,"在 macOS 上使用 pyenv 安装 Python 后，运行项目时报错 `ModuleNotFoundError: No module named '_lzma'` 怎么办？","这是由于 macOS 上通过 pyenv 编译 Python 时缺少 `xz` 库导致的。请按以下步骤修复：\n\n1. 使用 Homebrew 安装 xz：\n   brew install xz\n2. 卸载当前有问题的 Python 版本（替换 \u003Cversion> 为你的版本号）：\n   pyenv uninstall \u003Cversion>\n3. 重新安装该 Python 版本：\n   pyenv install \u003Cversion>\n\n完成后重新配置 Poetry 环境即可。","https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues\u002F14",{"id":140,"question_zh":141,"answer_zh":142,"source_url":138},42323,"运行 Pipeline 后，为什么 ZenML 仪表板中没有显示任何 Pipeline？","这是预期行为。Pipeline 只有在实际成功运行后才会出现在 ZenML 仪表板中。如果你只是启动了基础设施但未执行具体的 ETL 或训练命令，仪表板将是空的。请运行具体的任务命令（例如 `poetry poe run-digital-data-etl`），执行成功后刷新仪表板即可看到记录。",{"id":144,"question_zh":145,"answer_zh":146,"source_url":147},42324,"代码中的 `ChunkingDispatcher` 类属性命名 `cleaning_factory` 是否错误？","是的，这是一个拼写错误（Typo）。该属性本应命名为 `chunking_factory` 以匹配其功能（ChunkingHandlerFactory）。维护者已确认并修复了此问题。如果你在旧版本代码中看到此命名，可以手动将其重命名为 `chunking_factory` 以符合逻辑。","https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues\u002F21",{"id":149,"question_zh":150,"answer_zh":151,"source_url":152},42325,"Poetry 版本升级到 2.0 以上后，安装 `poethepoet` 插件失败怎么办？","Poetry 2.0+ 版本可能导致 `poethepoet` 插件安装兼容性问题。建议暂时将 Poetry 版本降级至 2.0 以下（推荐范围：>= 1.8.3 且 \u003C 2.0）。\n\n或者，你可以考虑使用替代方案 `pixi`（基于 uv），该项目有社区成员提供的 pixi 版本实现，但需注意这并非官方文档默认推荐的路径。","https:\u002F\u002Fgithub.com\u002FPacktPublishing\u002FLLM-Engineers-Handbook\u002Fissues\u002F32",[]]