[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-allenai--OLMo":3,"tool-allenai--OLMo":64},[4,17,27,35,43,56],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":16},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,3,"2026-04-05T11:01:52",[13,14,15],"开发框架","图像","Agent","ready",{"id":18,"name":19,"github_repo":20,"description_zh":21,"stars":22,"difficulty_score":23,"last_commit_at":24,"category_tags":25,"status":16},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",138956,2,"2026-04-05T11:33:21",[13,15,26],"语言模型",{"id":28,"name":29,"github_repo":30,"description_zh":31,"stars":32,"difficulty_score":23,"last_commit_at":33,"category_tags":34,"status":16},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",107662,"2026-04-03T11:11:01",[13,14,15],{"id":36,"name":37,"github_repo":38,"description_zh":39,"stars":40,"difficulty_score":23,"last_commit_at":41,"category_tags":42,"status":16},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[13,26],{"id":44,"name":45,"github_repo":46,"description_zh":47,"stars":48,"difficulty_score":23,"last_commit_at":49,"category_tags":50,"status":16},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",84991,"2026-04-05T10:45:23",[14,51,52,53,15,54,26,13,55],"数据工具","视频","插件","其他","音频",{"id":57,"name":58,"github_repo":59,"description_zh":60,"stars":61,"difficulty_score":10,"last_commit_at":62,"category_tags":63,"status":16},3128,"ragflow","infiniflow\u002Fragflow","RAGFlow 是一款领先的开源检索增强生成（RAG）引擎，旨在为大语言模型构建更精准、可靠的上下文层。它巧妙地将前沿的 RAG 技术与智能体（Agent）能力相结合，不仅支持从各类文档中高效提取知识，还能让模型基于这些知识进行逻辑推理和任务执行。\n\n在大模型应用中，幻觉问题和知识滞后是常见痛点。RAGFlow 通过深度解析复杂文档结构（如表格、图表及混合排版），显著提升了信息检索的准确度，从而有效减少模型“胡编乱造”的现象，确保回答既有据可依又具备时效性。其内置的智能体机制更进一步，使系统不仅能回答问题，还能自主规划步骤解决复杂问题。\n\n这款工具特别适合开发者、企业技术团队以及 AI 研究人员使用。无论是希望快速搭建私有知识库问答系统，还是致力于探索大模型在垂直领域落地的创新者，都能从中受益。RAGFlow 提供了可视化的工作流编排界面和灵活的 API 接口，既降低了非算法背景用户的上手门槛，也满足了专业开发者对系统深度定制的需求。作为基于 Apache 2.0 协议开源的项目，它正成为连接通用大模型与行业专有知识之间的重要桥梁。",77062,"2026-04-04T04:44:48",[15,14,13,26,54],{"id":65,"github_repo":66,"name":67,"description_en":68,"description_zh":69,"ai_summary_zh":69,"readme_en":70,"readme_zh":71,"quickstart_zh":72,"use_case_zh":73,"hero_image_url":74,"owner_login":75,"owner_name":76,"owner_avatar_url":77,"owner_bio":78,"owner_company":79,"owner_location":79,"owner_email":80,"owner_twitter":79,"owner_website":81,"owner_url":82,"languages":83,"stars":119,"forks":120,"last_commit_at":121,"license":122,"difficulty_score":10,"env_os":123,"env_gpu":124,"env_ram":125,"env_deps":126,"category_tags":132,"github_topics":79,"view_count":133,"oss_zip_url":79,"oss_zip_packed_at":79,"status":16,"created_at":134,"updated_at":135,"faqs":136,"releases":165},482,"allenai\u002FOLMo","OLMo","Modeling, training, eval, and inference code for OLMo","OLMo 是由艾伦人工智能研究所（AI2）开发的开源语言模型工具包，专为研究人员和开发者设计。它提供了一套完整的代码框架，支持从模型训练、评估到推理的全流程操作，尤其适合需要定制化大语言模型的研究场景。通过分阶段训练策略——先基于海量网络数据进行预训练，再结合高质量数据微调——OLMo 能够高效构建性能优异的语言模型。\n\n这一工具解决了传统大模型开发中数据利用效率低、训练成本高和部署复杂等问题。其核心亮点包括：模块化代码结构便于科研复现与改进、提供1B至32B参数量级的多种模型变体（如OLMo-2系列）、支持Hugging Face格式转换以适配主流生态。对于需要探索模型架构、优化训练策略或部署垂直领域应用的研究者而言，OLMo 提供了灵活的技术基座。\n\n用户群体主要面向自然语言处理领域的研究人员、机器学习工程师及高校实验室成员。开发者可通过PyPI或源码安装快速上手，而预训练模型和阶段性检查点的开放则降低了实验门槛。需要注意的是，当前GitHub仓库已停止更新，最新版本和文档请访问 OLMo-core 项目。","\u003Cdiv align=\"center\">\n  \u003C!-- \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fallenai_OLMo_readme_7d4cc5b19f85.png\" width=\"300\"\u002F> -->\n  \u003Cbr>\n  \u003Cbr>\n  \u003Ch1>OLMo: Open Language Model\u003C\u002Fh1>\n\u003C\u002Fdiv>\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002FLICENSE\">\n    \u003Cimg alt=\"GitHub License\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002Fallenai\u002FOLMo\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Freleases\">\n    \u003Cimg alt=\"GitHub release\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Frelease\u002Fallenai\u002FOLMo.svg\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fpdf\u002F2501.00656.pdf\">\n    \u003Cimg alt=\"Paper URL\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Farxiv-2402.00838-blue\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fplayground.allenai.org\">\n    \u003Cimg alt=\"Playground\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FAi2-Playground-F0529C\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FsZq3jTNVNG\">\n    \u003Cimg alt=\"Discord\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDiscord%20-%20blue?style=flat&logo=discord&label=Ai2&color=%235B65E9\">\n  \u003C\u002Fa>\n\u003C\u002Fp>\n\n# ⚠️ NOTICE ⚠️ This repository is out of date with our more recent releases and is no longer active. For the latest Olmo releases and updates, please visit: https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002F \n\nOLMo is a repository for training and using AI2's state-of-the-art open language models. It is designed by scientists, for scientists.\n\n## Installation\n\nFirst, install [PyTorch](https:\u002F\u002Fpytorch.org) following the instructions specific to your operating system.\n\nFor training and fine-tuning, we recommend installing from source:\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo.git\ncd OLMo\npip install -e .[all]\n```\nYou can also install from PyPI with:\n```bash\npip install ai2-olmo\n```\n\n## Pretraining\n\nOLMo pretraining follows a two-stage training procedure.\nIn the first stage, we train on large amounts of mostly web-based data: [OLMo-mix-1124](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Folmo-mix-1124)\nIn the second stage, we train on a smaller amount of high-quality, targeted data: [Dolmino-mix-1124](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Fdolmino-mix-1124)\n\nYou can find *all* the checkpoints, at minimum every 1000 training steps in OLMo core and Hugging Face format:\n\n\n| Variant         | OLMo Format (Stage 1)                                                                                         | OLMo Format (Stage 2) | Hugging Face Format                                                               |\n|----------------|-----------------------------------------------------------------------------------------------------|--------|----------------------------------------------------------------------------------|\n| **OLMo-2 1B**  | [OLMo-2 1B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-0425\u002FOLMo-2-0425-1B.csv)     | [OLMo-2 1B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-0425\u002FOLMo-2-0425-1B-stage2.csv)      | [Hugging Face for the 1B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B)  |\n| **OLMo-2 7B**  | [OLMo-2 7B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-7B.csv)     | [OLMo-2 7B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-7B-stage2.csv)      | [Hugging Face for the 7B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B)  |\n| **OLMo-2 13B** | [OLMo-2 13B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-13B.csv)   | [OLMo-2 13B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-13B-stage2.csv)       | [Hugging Face for the 13B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B) |\n| **OLMo-2 32B** | [OLMo-2 32B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002Fblob\u002Fmain\u002Fsrc\u002Fscripts\u002Fofficial\u002FOLMo2-0325-32B.csv)   | [OLMo-2 32B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002Fblob\u002Fmain\u002Fsrc\u002Fscripts\u002Fofficial\u002FOLMo-2-0325-32B-stage2.csv) | [Hugging Face for the 32B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0325-32B) |\n\n> Note: The 32B variant was trained on our new trainer. To train or fine-tune OLMo-2 32B, visit [OLMo-core](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core).\n\n### Steps to reproduce\n\nTo reproduce any of the training processes described below, run this:\n\n```bash\ntorchrun --nproc_per_node=8 scripts\u002Ftrain.py {path_to_train_config}\n```\n\nFor the training config, use any of the configs listed below.\n\nIf you want to override any of the settings in the training config without having to write a new config every time,\nyou can do this:\n\n```bash\ntorchrun --nproc_per_node=8 scripts\u002Ftrain.py {path_to_train_config} \\\n  --setting1=value \\\n  --setting2=value \\\n  --setting3.subsetting1=value\n```\n\nThe training configs below refer to training data that gets streamed in live over HTTP.\nTo reproduce at large scale, we recommend downloading the files locally and changing the paths to point to your\nlocal file system.\n\n#### To run on Mac silicon devices:\n```bash\npython scripts\u002Ftrain.py {path_to_train_config}\n```\nExample:\n```bash\npython scripts\u002Ftrain.py configs\u002Ftiny\u002FOLMo-20M.yaml --save_overwrite\n```\n> Note: You need to upgrade PyTorch to 2.5.x to run.\n\n### Stage 1\n\nStage 1 is the biggest stage, where we train on 4T or 5T tokens on largely web-based data. \n\n|                 | OLMo2 1B                                                                                                          | OLMo2 7B                                                                                                          | OLMo2 13B                                                                                                          |\n|-----------------|-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| Number of tokens| 4 Trillion | 4 Trillion                                                                                                        | 5 Trillion                                                                                                         |\n| Checkpoint      |[stage1-step1907359-tokens4001B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage1-step1907359-tokens4001B)| [stage1-step928646-tokens3896B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage1-step928646-tokens3896B) | [stage1-step596057-tokens5001B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage1-step596057-tokens5001B) |\n| Training config | [OLMo2-1B-stage1.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage1.yaml) |[OLMo2-7B-stage1.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage1.yaml)                                                | [OLMo2-13B-stage1.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage1.yaml)                                               |                                              |\n| WandB           | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0)|[wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002FOLMo-2-7B-Nov-2024--VmlldzoxMDUzMzE1OA)       | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n\nYou can find the .csv.gz files containing the training data [here](configs\u002Fofficial-1124\u002Fprovenance.csv).\n\n### Stage 2 for the 1B\n\nFor the 1B model, we have trained three times with different data order on 50B high quality tokens, used last checkpoint of seed 42 as final checkpoint.\n\n|                        | Checkpoint                                                                                                                          | Training config                                                                        | WandB       |\n|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-------------|\n| random seed 42069         | [stage2-ingredient1-step23852-tokens51B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage2-ingredient1-step23852-tokens51B) | [OLMo2-1B-stage2-seed42069.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage2-seed42069.yaml)       | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0) |\n| random seed 666      | [stage2-ingredient2-step23852-tokens51B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage2-ingredient2-step23852-tokens51B) | [OLMo2-1B-stage2-seed666.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage2-seed666.yaml) | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0) |\n| random seed 42  (main)      | [stage2-ingredient3-step23852-tokens51B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage2-ingredient3-step23852-tokens51B) | [OLMo2-1B-stage2-seed42.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage2-seed42.yaml)     | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0) |\n\n\n### Stage 2 for the 7B\n\nFor the 7B model, we train three times with different data order on 50B high quality tokens, and then average (\"soup\") the models.\n\n|                        | Checkpoint                                                                                                                          | Training config                                                                        | WandB       |\n|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-------------|\n| random seed 42         | [stage2-ingredient1-step11931-tokens50B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage2-ingredient1-step11931-tokens50B) | [OLMo2-7B-stage2-seed42.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage2-seed42.yaml)       | [wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002F) |\n| random seed 42069      | [stage2-ingredient2-step11931-tokens50B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage2-ingredient2-step11931-tokens50B) | [OLMo2-7B-stage2-seed42069.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage2-seed42069.yaml) | [wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002F) |\n| random seed 666        | [stage2-ingredient3-step11931-tokens50B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage2-ingredient3-step11931-tokens50B) | [OLMo2-7B-stage2-seed666.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage2-seed666.yaml)     | [wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002F) |\n| **final souped model** | [main](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fmain) | no config, we just averaged the weights in Python                                      | |\n\nThe training configs linked here are set up to download the latest checkpoint after stage 1, and start training from there.\n\n### Stage 2 for the 13B\n\nFor the 13B model, we train three times with different data order on 100B high quality tokens, and one more time\non 300B high quality tokens. Then we average (\"soup\") the models.\n\n|                        | Checkpoint                                                                                                                             | Training config                                                                                  | WandB       |\n|------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|-------------|\n| random seed 1110, 100B | [stage2-ingredient1-step11931-tokens100B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient1-step11931-tokens100B) | [OLMo2-13B-stage2-seed1110-100B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed1110-100B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| random seed 2662, 100B | [stage2-ingredient2-step11931-tokens100B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient2-step11931-tokens100B) | [OLMo2-13B-stage2-seed2662-100B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed2662-100B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| random seed 6209, 100B | [stage2-ingredient3-step11931-tokens100B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient3-step11931-tokens100B) | [OLMo2-13B-stage2-seed6209-100B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed6209-100B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| random seed 2662, 300B | [stage2-ingredient4-step11931-tokens300B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient4-step35773-tokens300B) | [OLMo2-13B-stage2-seed2662-300B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed2662-300B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| **final souped model** | [main](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fmain)                                                                       | no config, we just averaged the weights in Python                                                | |\n\nThe training configs linked here are set up to download the latest checkpoints after stage 1, and start training from there.\n\n> Note: You can find all the information about the 32B in the [OLMo-core](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core) repository.\n\n## Instruction tuned variants\n\nFor instruction tuned variants of these models, go to\n * [OLMo2 1B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B-Instruct)\n * [OLMo2 7B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B-Instruct)\n * [OLMo2 13B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B-Instruct)\n * [OLMo2 32B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0325-32B-Instruct)\n\n## Inference\n\nYou can use our Hugging Face integration to run inference on the OLMo Transformers checkpoints:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai\u002FOLMo-2-0425-1B\")\ntokenizer = AutoTokenizer.from_pretrained(\"allenai\u002FOLMo-2-0425-1B\")\nmessage = [\"Language modeling is \"]\ninputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)\n# optional verifying cuda\n# inputs = {k: v.to('cuda') for k,v in inputs.items()}\n# olmo = olmo.to('cuda')\nresponse = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\nprint(tokenizer.batch_decode(response, skip_special_tokens=True)[0])\n```\n\nAlternatively, with the Hugging Face pipeline abstraction:\n\n```python\nfrom transformers import pipeline\nolmo_pipe = pipeline(\"text-generation\", model=\"allenai\u002FOLMo-2-0425-1B\")\nprint(olmo_pipe(\"Language modeling is\"))\n```\n\n### Quantization\n\n```python\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai\u002FOLMo-2-0425-1B\", torch_dtype=torch.float16, load_in_8bit=True)  # requires bitsandbytes\n```\n\nThe quantized model is sensitive to input types and CUDA handling. To avoid potential issues, we recommend explicitly converting input IDs to CUDA using: `inputs.input_ids.to('cuda')`\n\n## Evaluation\n\nAdditional tools for evaluating OLMo models are available at the [OLMo Eval](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-eval) and [olmes](https:\u002F\u002Fgithub.com\u002Fallenai\u002Folmes) repositories.\n\n## Modal.com Hosting\n\nAn example script is provided for hosting an OLMo 2 model on Modal.com using the OpenAI API in `.\u002Fscripts\u002Folmo2_modal_openai.py`.\nTo run that:\n\n1. Follow the instructions under Getting Started in [the Modal.com Guide](https:\u002F\u002Fmodal.com\u002Fdocs\u002Fguide) to install\nthe Modal library and command line tools.\u003C\u002Fli>\n2. Follow the instructions under [Secrets](https:\u002F\u002Fmodal.com\u002Fdocs\u002Fguide\u002Fsecrets) in the Modal.com Guide to create a Modal secret named \"example-secret-token\"\nthat defines a value for the variable MODAL_TOKEN for your server.\u003C\u002Fli>\n3. Then run\n```bash\nmodal deploy .\u002Fscripts\u002Folmo2_modal_openai.py\n```\n\nYou can check your endpoint using curl similar to the following:\n```bash\ncurl -X POST \\\n  -H \"Authorization: Bearer [the secret token from above]\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d @body.json \\\n  https:\u002F\u002F[the web endpoint modal creates above]\u002Fv1\u002Fchat\u002Fcompletions\n```\n\nwhere `body.json` is of the form:\n```\n{\n    \"model\": \"OLMo-2-1124-13B-Instruct\",\n    \"messages\": [\n        {\n            \"role\": \"user\",\n            \"content\": \"Who was Alan Turing?\"\n        }\n      ],\n    \"max_tokens\": 100,\n    \"temperature\": 0.9,\n    \"stream\": true\n}\n```\n\n\n## Citing\n\n```bibtex\n@misc{olmo20242olmo2furious,\n      title={2 OLMo 2 Furious}, \n      author={Team OLMo and Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and Dustin Schwenk and Oyvind Tafjord and Taira Anderson and David Atkinson and Faeze Brahman and Christopher Clark and Pradeep Dasigi and Nouha Dziri and Michal Guerquin and Hamish Ivison and Pang Wei Koh and Jiacheng Liu and Saumya Malik and William Merrill and Lester James V. Miranda and Jacob Morrison and Tyler Murray and Crystal Nam and Valentina Pyatkin and Aman Rangapur and Michael Schmitz and Sam Skjonsberg and David Wadden and Christopher Wilhelm and Michael Wilson and Luke Zettlemoyer and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi},\n      year={2024},\n      eprint={2501.00656},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.00656}, \n}\n```\n","\u003Cdiv align=\"center\">\n  \u003C!-- \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fallenai_OLMo_readme_7d4cc5b19f85.png\" width=\"300\"\u002F> -->\n  \u003Cbr>\n  \u003Cbr>\n  \u003Ch1>OLMo: 开放语言模型\u003C\u002Fh1>\n\u003C\u002Fdiv>\n\u003Cp align=\"center\">\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002FLICENSE\">\n    \u003Cimg alt=\"GitHub 许可证\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002Fallenai\u002FOLMo\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Freleases\">\n    \u003Cimg alt=\"GitHub 发布版本\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Frelease\u002Fallenai\u002FOLMo.svg\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Farxiv.org\u002Fpdf\u002F2501.00656.pdf\">\n    \u003Cimg alt=\"论文地址\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002Farxiv-2402.00838-blue\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fplayground.allenai.org\">\n    \u003Cimg alt=\"体验平台\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FAi2-Playground-F0529C\">\n  \u003C\u002Fa>\n  \u003Ca href=\"https:\u002F\u002Fdiscord.gg\u002FsZq3jTNVNG\">\n    \u003Cimg alt=\"Discord\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDiscord%20-%20blue?style=flat&logo=discord&label=Ai2&color=%235B65E9\">\n  \u003C\u002Fa>\n\u003C\u002Fp>\n\n# ⚠️ 通知 ⚠️ 此仓库已落后于我们最近的发布版本，且不再活跃。获取最新的 Olmo 发布和更新，请访问：https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002F \n\nOLMo 是用于训练和使用 AI2 最先进的开源**语言模型 (Language Model)** 的仓库。它由科学家设计，专为科学家打造。\n\n## 安装\n\n首先，根据您的操作系统特定说明安装 [PyTorch (PyTorch 深度学习框架)](https:\u002F\u002Fpytorch.org)。\n\n对于训练和**微调 (Fine-tuning)**，我们推荐从源代码安装：\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo.git\ncd OLMo\npip install -e .[all]\n```\n您也可以通过以下方式从 PyPI 安装：\n```bash\npip install ai2-olmo\n```\n\n## 预训练 (Pretraining)\n\nOLMo 预训练遵循两阶段训练流程。\n在第一阶段，我们在大量主要基于网络的数据上进行训练：[OLMo-mix-1124](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Folmo-mix-1124)\n在第二阶段，我们在少量高质量、定向的数据上进行训练：[Dolmino-mix-1124](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Fdolmino-mix-1124)\n\n您可以在 OLMo core 和 Hugging Face 格式中找到所有**检查点 (Checkpoints)**，至少每 1000 个训练步骤一次：\n\n\n| Variant         | OLMo 格式（阶段 1）                                                                                         | OLMo 格式（阶段 2） | Hugging Face 格式                                                               |\n|----------------|-----------------------------------------------------------------------------------------------------|--------|----------------------------------------------------------------------------------|\n| **OLMo-2 1B**  | [OLMo-2 1B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-0425\u002FOLMo-2-0425-1B.csv)     | [OLMo-2 1B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-0425\u002FOLMo-2-0425-1B-stage2.csv)      | [Hugging Face for the 1B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B)  |\n| **OLMo-2 7B**  | [OLMo-2 7B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-7B.csv)     | [OLMo-2 7B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-7B-stage2.csv)      | [Hugging Face for the 7B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B)  |\n| **OLMo-2 13B** | [OLMo-2 13B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-13B.csv)   | [OLMo-2 13B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fblob\u002Fmain\u002Fconfigs\u002Fofficial-1124\u002FOLMo-2-1124-13B-stage2.csv)       | [Hugging Face for the 13B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B) |\n| **OLMo-2 32B** | [OLMo-2 32B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002Fblob\u002Fmain\u002Fsrc\u002Fscripts\u002Fofficial\u002FOLMo2-0325-32B.csv)   | [OLMo-2 32B](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002Fblob\u002Fmain\u002Fsrc\u002Fscripts\u002Fofficial\u002FOLMo-2-0325-32B-stage2.csv) | [Hugging Face for the 32B variant](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0325-32B) |\n\n> 注意：32B 变体是在我们的新**训练器 (Trainer)** 上训练的。要训练或微调 (Fine-tuning) OLMo-2 32B，请访问 [OLMo-core](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core)。\n\n### 复现步骤\n\n要复现以下描述的任何训练过程，请运行：\n\n```bash\ntorchrun --nproc_per_node=8 scripts\u002Ftrain.py {path_to_train_config}\n```\n\n对于训练**配置 (Config)**，请使用下列列出的任意配置。\n\n如果您想在不每次编写新配置的情况下覆盖训练配置中的任何设置，可以这样做：\n\n```bash\ntorchrun --nproc_per_node=8 scripts\u002Ftrain.py {path_to_train_config} \\\n  --setting1=value \\\n  --setting2=value \\\n  --setting3.subsetting1=value\n```\n\n下面的训练配置引用的是通过 HTTP 实时流式传输的训练数据。\n为了大规模复现，我们建议将文件下载到本地并将路径更改为指向您的本地文件系统。\n\n#### 在 Mac 硅基设备 (Silicon Devices) 上运行：\n```bash\npython scripts\u002Ftrain.py {path_to_train_config}\n```\n示例：\n```bash\npython scripts\u002Ftrain.py configs\u002Ftiny\u002FOLMo-20M.yaml --save_overwrite\n```\n> 注意：您需要升级 PyTorch 到 2.5.x 才能运行。\n\n### 第一阶段\n\n第一阶段是规模最大的阶段，我们在大量基于网络的数据上进行了 4T 或 5T token（词元）的训练。\n\n|                 | OLMo2 1B                                                                                                          | OLMo2 7B                                                                                                          | OLMo2 13B                                                                                                          |\n|-----------------|-----------------|-------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| Token 数量      | 4 Trillion | 4 Trillion                                                                                                        | 5 Trillion                                                                                                         |\n| 检查点          |[stage1-step1907359-tokens4001B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage1-step1907359-tokens4001B)| [stage1-step928646-tokens3896B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage1-step928646-tokens3896B) | [stage1-step596057-tokens5001B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage1-step596057-tokens5001B) |\n| 训练配置        | [OLMo2-1B-stage1.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage1.yaml) |[OLMo2-7B-stage1.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage1.yaml)                                                | [OLMo2-13B-stage1.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage1.yaml)                                               |                                              |\n| WandB           | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0)|[wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002FOLMo-2-7B-Nov-2024--VmlldzoxMDUzMzE1OA)       | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n\n你可以在 [这里](configs\u002Fofficial-1124\u002Fprovenance.csv) 找到包含训练数据的 .csv.gz 文件。\n\n### 1B 模型的第二阶段\n\n针对 1B 模型，我们在 50B 高质量 token 上以不同的数据顺序训练了三次，并使用 seed 42（随机种子）的最终检查点作为最终模型。\n\n|                        | 检查点                                                                                                                          | 训练配置                                                                        | WandB       |\n|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-------------|\n| 随机种子 42069         | [stage2-ingredient1-step23852-tokens51B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage2-ingredient1-step23852-tokens51B) | [OLMo2-1B-stage2-seed42069.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage2-seed42069.yaml)       | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0) |\n| 随机种子 666      | [stage2-ingredient2-step23852-tokens51B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage2-ingredient2-step23852-tokens51B) | [OLMo2-1B-stage2-seed666.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage2-seed666.yaml) | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0) |\n| 随机种子 42（主）      | [stage2-ingredient3-step23852-tokens51B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B\u002Ftree\u002Fstage2-ingredient3-step23852-tokens51B) | [OLMo2-1B-stage2-seed42.yaml](configs\u002Fofficial-0425\u002FOLMo2-1B-stage2-seed42.yaml)     | [wandb.ai\u002FOLMo2-1B](https:\u002F\u002Fapi.wandb.ai\u002Flinks\u002Fai2-llm\u002Fizdtrtu0) |\n\n\n### 7B 模型的第二阶段\n\n针对 7B 模型，我们在 50B 高质量 token 上以不同的数据顺序训练了三次，随后对这些模型进行平均（即\"Model Soup\"）。\n\n|                        | 检查点                                                                                                                          | 训练配置                                                                        | WandB       |\n|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|-------------|\n| 随机种子 42         | [stage2-ingredient1-step11931-tokens50B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage2-ingredient1-step11931-tokens50B) | [OLMo2-7B-stage2-seed42.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage2-seed42.yaml)       | [wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002F) |\n| 随机种子 42069      | [stage2-ingredient2-step11931-tokens50B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage2-ingredient2-step11931-tokens50B) | [OLMo2-7B-stage2-seed42069.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage2-seed42069.yaml) | [wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002F) |\n| 随机种子 666        | [stage2-ingredient3-step11931-tokens50B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fstage2-ingredient3-step11931-tokens50B) | [OLMo2-7B-stage2-seed666.yaml](configs\u002Fofficial-1124\u002FOLMo2-7B-stage2-seed666.yaml)     | [wandb.ai\u002FOLMo2-7B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-7B\u002Freports\u002F) |\n| **最终融合模型** | [main](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Ftree\u002Fmain) | 无配置文件，我们仅在 Python 中平均了权重                                      | |\n\n此处链接的训练配置已设置好，用于下载 Stage 1 后的最新检查点，并从中继续训练。\n\n### 13B 模型的 Stage 2\n\n对于 13B 模型，我们在 100B 高质量 tokens 上以不同的数据顺序训练了三次，并在 300B 高质量 tokens 上再训练一次。然后我们对模型进行平均（即“模型汤”\u002Fmodel soup）。\n\n|                        | 检查点 (Checkpoint)                                                                                                                             | 训练配置 (Training config)                                                                                  | WandB       |\n|------------------------|----------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|-------------|\n| 随机种子 1110, 100B | [stage2-ingredient1-step11931-tokens100B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient1-step11931-tokens100B) | [OLMo2-13B-stage2-seed1110-100B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed1110-100B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| 随机种子 2662, 100B | [stage2-ingredient2-step11931-tokens100B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient2-step11931-tokens100B) | [OLMo2-13B-stage2-seed2662-100B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed2662-100B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| 随机种子 6209, 100B | [stage2-ingredient3-step11931-tokens100B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient3-step11931-tokens100B) | [OLMo2-13B-stage2-seed6209-100B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed6209-100B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| 随机种子 2662, 300B | [stage2-ingredient4-step11931-tokens300B](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fstage2-ingredient4-step35773-tokens300B) | [OLMo2-13B-stage2-seed2662-300B.yaml](configs\u002Fofficial-1124\u002FOLMo2-13B-stage2-seed2662-300B.yaml) | [wandb.ai\u002FOLMo2-13B](https:\u002F\u002Fwandb.ai\u002Fai2-llm\u002FOLMo-2-1124-13B\u002Freports\u002FOLMo-2-13B-Nov-2024--VmlldzoxMDUzMjQxNg) |\n| **最终融合模型** | [main](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B\u002Ftree\u002Fmain)                                                                       | 无配置，我们仅在 Python 中平均了权重                                                | |\n\n此处链接的训练配置已设置好，用于下载 Stage 1 之后的最新检查点 (checkpoint)，并从此处开始训练。\n\n> 注意：您可以在 [OLMo-core](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core) 仓库中找到关于 32B 的所有信息。\n\n## 指令微调变体\n\n对于这些模型的指令微调 (instruction tuned) 变体，请访问：\n * [OLMo2 1B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0425-1B-Instruct)\n * [OLMo2 7B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B-Instruct)\n * [OLMo2 13B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-13B-Instruct)\n * [OLMo2 32B Instruct](https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-0325-32B-Instruct)\n\n## 推理\n\n您可以使用我们的 Hugging Face 集成来在 OLMo Transformers 检查点上运行推理：\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai\u002FOLMo-2-0425-1B\")\ntokenizer = AutoTokenizer.from_pretrained(\"allenai\u002FOLMo-2-0425-1B\")\nmessage = [\"Language modeling is \"]\ninputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)\n# optional verifying cuda\n# inputs = {k: v.to('cuda') for k,v in inputs.items()}\n# olmo = olmo.to('cuda')\nresponse = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\nprint(tokenizer.batch_decode(response, skip_special_tokens=True)[0])\n```\n\n或者，使用 Hugging Face 的 pipeline 抽象：\n\n```python\nfrom transformers import pipeline\nolmo_pipe = pipeline(\"text-generation\", model=\"allenai\u002FOLMo-2-0425-1B\")\nprint(olmo_pipe(\"Language modeling is\"))\n```\n\n### 量化\n\n```python\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai\u002FOLMo-2-0425-1B\", torch_dtype=torch.float16, load_in_8bit=True)  # requires bitsandbytes\n```\n\n量化模型对输入类型和 CUDA 处理比较敏感。为了避免潜在问题，我们建议显式地将输入 ID 转换为 CUDA，使用：`inputs.input_ids.to('cuda')`\n\n## 评估\n\n评估 OLMo 模型的额外工具可在 [OLMo Eval](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-eval) 和 [olmes](https:\u002F\u002Fgithub.com\u002Fallenai\u002Folmes) 仓库中找到。\n\n## Modal.com 托管\n\n在 `.\u002Fscripts\u002Folmo2_modal_openai.py` 中提供了一个示例脚本，用于在 Modal.com 上使用 OpenAI API 托管 OLMo 2 模型。\n要运行该脚本：\n\n1. 按照 [Modal.com 指南](https:\u002F\u002Fmodal.com\u002Fdocs\u002Fguide) 中“入门”部分的说明安装 Modal 库和命令行工具。\n2. 按照 [Modal.com 指南](https:\u002F\u002Fmodal.com\u002Fdocs\u002Fguide\u002Fsecrets) 中“密钥”部分的说明创建一个名为 \"example-secret-token\" 的 Modal 密钥，该密钥为服务器变量 MODAL_TOKEN 定义一个值。\n3. 然后运行\n```bash\nmodal deploy .\u002Fscripts\u002Folmo2_modal_openai.py\n```\n\n您可以使用 curl 检查您的端点 (endpoint)，类似于以下内容：\n```bash\ncurl -X POST \\\n  -H \"Authorization: Bearer [the secret token from above]\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d @body.json \\\n  https:\u002F\u002F[the web endpoint modal creates above]\u002Fv1\u002Fchat\u002Fcompletions\n```\n\n其中 `body.json` 的格式如下：\n```\n{\n    \"model\": \"OLMo-2-1124-13B-Instruct\",\n    \"messages\": [\n        {\n            \"role\": \"user\",\n            \"content\": \"Who was Alan Turing?\"\n        }\n      ],\n    \"max_tokens\": 100,\n    \"temperature\": 0.9,\n    \"stream\": true\n}\n```\n\n\n## 引用\n\n```bibtex\n@misc{olmo20242olmo2furious,\n      title={2 OLMo 2 Furious}, \n      author={Team OLMo and Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and Dustin Schwenk and Oyvind Tafjord and Taira Anderson and David Atkinson and Faeze Brahman and Christopher Clark and Pradeep Dasigi and Nouha Dziri and Michal Guerquin and Hamish Ivison and Pang Wei Koh and Jiacheng Liu and Saumya Malik and William Merrill and Lester James V. Miranda and Jacob Morrison and Tyler Murray and Crystal Nam and Valentina Pyatkin and Aman Rangapur and Michael Schmitz and Sam Skjonsberg and David Wadden and Christopher Wilhelm and Michael Wilson and Luke Zettlemoyer and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi},\n      year={2024},\n      eprint={2501.00656},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https:\u002F\u002Farxiv.org\u002Fabs\u002F2501.00656}, \n}\n```","# OLMo 快速上手指南\n\n> ⚠️ **重要提示**：当前仓库（`allenai\u002FOLMo`）已不再活跃且版本过时。如需获取最新的 OLMo 发布版本和更新，请访问官方核心仓库：[https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002F](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core\u002F)。本指南基于提供的文档整理，适用于复现训练流程。\n\n## 一、环境准备\n\nOLMo 专为科研人员设计，支持在多种操作系统上运行。\n\n*   **操作系统**：Linux \u002F macOS \u002F Windows\n*   **Python 版本**：建议使用较新的稳定版\n*   **深度学习框架**：需先安装 [PyTorch](https:\u002F\u002Fpytorch.org)。\n    *   **Mac Silicon 设备**：需升级 PyTorch 至 **2.5.x** 版本才能运行。\n*   **网络要求**：训练数据通过 HTTP 流式传输，建议在国内环境下配置稳定的网络连接或使用代理。\n\n## 二、安装步骤\n\n### 方式 1：从源码安装（推荐用于训练和微调）\n\n克隆仓库并安装依赖：\n\n```bash\ngit clone https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo.git\ncd OLMo\npip install -e .[all]\n```\n\n### 方式 2：通过 PyPI 安装\n\n直接安装包：\n\n```bash\npip install ai2-olmo\n```\n\n## 三、基本使用\n\nOLMo 采用两阶段训练流程。以下是启动训练脚本的基本命令。\n\n### 1. 启动训练\n\n使用 `torchrun` 启动多卡训练（以 8 张卡为例）：\n\n```bash\ntorchrun --nproc_per_node=8 scripts\u002Ftrain.py {path_to_train_config}\n```\n\n*   `{path_to_train_config}` 替换为具体的配置文件路径（见下文）。\n*   **覆盖设置**：无需新建配置文件，可直接在命令行覆盖参数：\n    ```bash\n    torchrun --nproc_per_node=8 scripts\u002Ftrain.py {path_to_train_config} \\\n      --setting1=value \\\n      --setting2=value \\\n      --setting3.subsetting1=value\n    ```\n\n### 2. Mac Silicon 设备运行\n\n在 Apple Silicon 设备上运行单进程训练：\n\n```bash\npython scripts\u002Ftrain.py {path_to_train_config}\n```\n\n**示例**：\n```bash\npython scripts\u002Ftrain.py configs\u002Ftiny\u002FOLMo-20M.yaml --save_overwrite\n```\n\n### 3. 模型变体与配置\n\n不同规模的模型（1B, 7B, 13B, 32B）对应不同的训练配置和数据集。主要配置位于 `configs\u002Fofficial-*` 目录下。\n\n| 模型变体 | 说明 | 配置参考 |\n| :--- | :--- | :--- |\n| **OLMo-2 1B** | 第一阶段 4T tokens，第二阶段 50B tokens | `configs\u002Fofficial-0425\u002FOLMo2-1B-stage1.yaml` |\n| **OLMo-2 7B** | 第一阶段 4T tokens，第二阶段 50B tokens (多模型融合) | `configs\u002Fofficial-1124\u002FOLMo2-7B-stage1.yaml` |\n| **OLMo-2 13B** | 第一阶段 5T tokens，第二阶段 100B+300B tokens | `configs\u002Fofficial-1124\u002FOLMo2-13B-stage1.yaml` |\n| **OLMo-2 32B** | 需使用新 Trainer，请前往 [OLMo-core](https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo-core) | N\u002FA |\n\n> **注意**：32B 变体使用了新的 Trainer，若需训练或微调该版本，请务必访问 `OLMo-core` 仓库。\n\n### 4. 数据与检查点\n\n*   **预训练数据**：\n    *   第一阶段：[OLMo-mix-1124](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Folmo-mix-1124)\n    *   第二阶段：[Dolmino-mix-1124](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fallenai\u002Fdolmino-mix-1124)\n*   **模型检查点**：所有检查点均提供 OLMo 格式和 Hugging Face 格式，可在 GitHub 配置表或 Hugging Face 页面下载。","某高校科研团队计划开发一个专注于医学文献分析的私有化大模型，需要确保数据不出内网且具备高度的可解释性。\n\n### 没有 OLMo 时\n- 依赖闭源商业 API，无法针对特定医学术语进行深度微调，效果受限\n- 从零搭建训练环境耗时漫长，缺乏经过验证的预训练代码库和配置\n- 模型决策过程是黑盒，难以排查医疗建议中潜在的逻辑偏见或事实错误\n- 高昂的商用授权费用限制了大规模实验迭代，导致研究进度受阻\n\n### 使用 OLMo 后\n- 直接加载官方提供的 7B 或 13B 预训练权重，快速启动高性能基座模型\n- 利用内置的两阶段训练脚本，在自有高质量医疗数据集上进行高效微调\n- 完全掌控推理与评估流程，确保敏感患者数据无需上传至第三方服务器\n- 开源架构支持深入检查模型内部机制，满足科研对代码可复现性的严格要求\n\nOLMo 让科研团队能够以低成本构建透明、可控且符合伦理的垂直领域大模型。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fallenai_OLMo_7d4cc5b1.png","allenai","Ai2","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002Fallenai_65c450d5.png","",null,"ai2-info@allenai.org","http:\u002F\u002Fwww.allenai.org","https:\u002F\u002Fgithub.com\u002Fallenai",[84,88,92,96,99,103,107,111,115],{"name":85,"color":86,"percentage":87},"Python","#3572A5",89.5,{"name":89,"color":90,"percentage":91},"Cuda","#3A4E3A",3.5,{"name":93,"color":94,"percentage":95},"Jupyter Notebook","#DA5B0B",3.2,{"name":97,"color":98,"percentage":23},"Shell","#89e051",{"name":100,"color":101,"percentage":102},"C++","#f34b7d",0.9,{"name":104,"color":105,"percentage":106},"Jsonnet","#0064bd",0.6,{"name":108,"color":109,"percentage":110},"Dockerfile","#384d54",0.2,{"name":112,"color":113,"percentage":114},"Makefile","#427819",0.1,{"name":116,"color":117,"percentage":118},"C","#555555",0,6457,735,"2026-04-05T10:59:56","Apache-2.0","Linux, macOS","需要多 GPU 环境（示例配置 8 卡），具体显存大小未说明","未说明",{"notes":127,"python":125,"dependencies":128},"1. ⚠️ 警告：此仓库已过期，请使用 OLMo-core 获取最新版本；2. Mac Silicon 设备需升级 PyTorch 至 2.5.x；3. 预训练涉及 TB 级数据（如 4T tokens），需充足磁盘空间；4. 使用 torchrun 命令进行多卡分布式训练",[129,130,131],"torch","ai2-olmo","wandb",[26,13],7,"2026-03-27T02:49:30.150509","2026-04-06T05:35:28.501310",[137,142,146,151,155,160],{"id":138,"question_zh":139,"answer_zh":140,"source_url":141},1902,"运行 `unshard.py` 时遇到 `ImportError: cannot import name 'unshard_model_state'` 错误怎么办？","这是一个已知问题。请切换到分支 `amanr\u002Fcheckpoint_bug_newtrainer`。然后按以下步骤操作：1. 创建 `model_and_optim` 子目录并将所有 `.distcp` 和 `.metadata` 文件移入其中；2. 使用脚本 `scripts\u002Fconvert_fsdp_to_hf.sh` 对检查点进行解分片；3. 将 `model.pt` 从子目录移动到检查点根目录；4. 下载对应 tokenizer (`wget https:\u002F\u002Fhuggingface.co\u002Fallenai\u002FOLMo-2-1124-7B\u002Fresolve\u002Fmain\u002Ftokenizer.json`)；5. 运行转换脚本 `python scripts\u002Fconvert_olmo2_to_hf.py`。","https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fissues\u002F862",{"id":143,"question_zh":144,"answer_zh":145,"source_url":141},1903,"如何将 OLMo 的训练检查点转换为 HuggingFace 格式？","在成功解分片并确保 `model.pt` 位于检查点根目录后，首先需要下载训练时使用的 tokenizer。然后运行转换命令：`python scripts\u002Fconvert_olmo2_to_hf.py --input_dir \u003C检查点路径>`。注意根据实际路径调整参数，确保输入目录结构正确。",{"id":147,"question_zh":148,"answer_zh":149,"source_url":150},1904,"训练过程中出现 `AssertionError: expected X bytes, got Y` 或数据读取不完整错误如何解决？","这通常是因为服务器返回了截断的内容。维护者已合并修复代码，建议更新到最新版本。如果是临时方案，可以在代码中将数据读取函数包裹在 `while true` 循环中，并使用 `try-except` 捕获异常进行重试，直到读取成功为止。","https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fissues\u002F766",{"id":152,"question_zh":153,"answer_zh":154,"source_url":150},1905,"训练时 DataLoader 报错 `IncompleteRead` 网络错误如何处理？","这是由于网络连接不稳定导致的数据流中断。除了等待官方代码修复外，用户可以在本地实现重试机制。例如在 `_http_get_bytes_range` 等函数中添加重试逻辑，或者在外部循环中处理该异常，确保在网络波动时能自动恢复连接并重试读取。",{"id":156,"question_zh":157,"answer_zh":158,"source_url":159},1906,"OLMo 预训练数据的来源、规模和分布情况是什么？","数据上传至 `s3:\u002F\u002Fai2-llm\u002Fpretraining-data\u002Fsources\u002Fcommon-crawl\u002Fv1\u002Fdocuments`。包含 25 个 URL 去重后的 Common Crawl dump，总压缩大小 11 TB，约 30 亿文档，总计 4.8T tokens。高\u002F中\u002F低流利度比例分别为 20%\u002F25%\u002F55%。未seen URL 的比例在流式处理后稳定在 30-40%。","https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fissues\u002F1",{"id":161,"question_zh":162,"answer_zh":163,"source_url":164},1907,"如何确保训练数据的顺序可复现以便进行实验对比？","建议在 dataloaders 中记录每个 batch 包含的文档 ID 及 token span。可以使用 Dolma v1.5 的工具将 NPY 文件映射回具体的 JSONL 行和文档 ID，构建从检查点索引 -> NPY 文件 -> 文档 ID -> token span 的完整追踪链路，从而实现精确的数据顺序复现。","https:\u002F\u002Fgithub.com\u002Fallenai\u002FOLMo\u002Fissues\u002F95",[166,171,176,181,186,191,196,201,206,211,216,221,226],{"id":167,"version":168,"summary_zh":169,"released_at":170},101399,"v0.6.0","## What's new\n\n### Added 🎉\n\n- A bunch of annealing configs\n- `constant_with_warmup` learning rate schedule\n- `one_in_eight` configuration for activation checkpointing\n- New tokenizer in the source instead of from huggingface\n- Improved support for GCS\n- `torch.compile()` now only compiles each block, not the whole model.\n- Support for `torch.compile()` with `dynamic=True`\n- Resetting the `torch.compile()` after every evaluation, because evaluation messes with the compiled versions\n- Added more in-loop evaluation tasks to pick from, mostly for scaling law.\n\n## Commits\n\nb41634f4 One more hint for what's going on.\nd74e8351 A little more help for getting started\n24ce0cae Merge pull request #756 from allenai\u002FMoreCheckpoints2\n69d1e4ef Note about and link to Huggingface\n3c6d5150 Merge pull request #754 from allenai\u002FMoreCheckpoints\n645587e9 Merge branch 'main' of https:\u002F\u002Fgithub.com\u002Fallenai\u002FLLM\na3466749 Fix link\n4f0d7d1d We use safetensors now.\na6e6e2b6 Remove links that don't work\ne6f6b455 Remove obsolete docs\n0d141580 Merge pull request #745 from allenai\u002Fimprove-documentation\n1048c16c Merge pull request #750 from allenai\u002Fdave\u002Fannealing_peteish_v2\n767047c3 Merge pull request #749 from allenai\u002Fmattj\u002Flegalwhammy2-augusta\n9c677c90 Merge pull request #748 from allenai\u002Foeeval-ladder-testtrain\n7e81a6c1 Merge pull request #739 from allenai\u002Fpeteish13-augusta\n31c385f3 Merge pull request #742 from allenai\u002FGoogleStorage\nafd728f2 Merge pull request #738 from allenai\u002Fannealing_peteish_v2_neweval\n837a4ff8 Merge pull request #687 from allenai\u002Fkylel\u002Fconfig-diff\n\n","2024-12-19T17:03:31",{"id":172,"version":173,"summary_zh":174,"released_at":175},101400,"v0.5.1","## What's new\n\n### Added 🎉\n\n- Added ability to try loading latest checkpoint from save folder using `--try_load_latest_save`.\n- Added support for flash attention and gradient checkpointing to `hf_olmo`.\n- Added `effective_n_kv_heads` to OLMoConfig for hacky VLLM support.\n\n## Commits\n\n68899916 Merge pull request #735 from allenai\u002Ftrue-version-0.5.1\n4d81b1bf Merge pull request #733 from allenai\u002Fversion-0.5.1\n76ad7587 Merge pull request #724 from allenai\u002Fshanea\u002Flumi-24.03-2\n885bc22c Merge pull request #721 from allenai\u002Fhey-my-first-pr-to-olmo\naa1863e7 Merge pull request #725 from allenai\u002Fshanea\u002Ffix-build-errors\n59360be2 add missing function\nd2b655a3 Merge pull request #720 from allenai\u002Fshanea\u002Fset-device-early\n47f8f5ab Merge pull request #719 from allenai\u002Fshanea\u002Fhf-olmo-gradient-checkpointing\n0b920774 Merge pull request #718 from allenai\u002Fot-fix-mmlu-bpb\nca81901e Merge pull request #717 from allenai\u002Fshanea\u002Ftry-load-latest-save-2\n46f06cbc Merge pull request #712 from allenai\u002Fot-fix-oe-eval-bpb\n\n","2024-10-17T21:32:09",{"id":177,"version":178,"summary_zh":179,"released_at":180},101401,"v0.5.0","## What's new\n\n- Fixed conversion to HuggingFace model for DDP-trained models.\n- Added support for remote source and destination for HuggingFace model conversion.\n\n### Added 🎉\n\n- Added support for document masking via flash-attn during training with `--data.generate_doc_lengths`.\n- Added config options for `model.norm_after`, `model.scale_emb_init`, and `auxiliary_loss_multiplier` (used with zloss).\n- Added scripts for running experiments on qk_norm, norm reordering, and zloss.\n- Added `model.rope_theta` configuration option.\n- Added `model.embedding_layer_norm` configuration option for adding a LN to the embeddings.\n- Added `model.emb_init_std` configuration option to override the standard deviation used to initialize the embeddings.\n- Added downstream eval task for requests dumped from oe-eval tasks\n- Added `CosLinearEnvelope` scheduler, which is a pointwise product of a cosine schedule and a linear decay.\n- Added ability to save outputs of submodules for debugging purposes.\n- Version dolma flan change in named_data_mix.py\n\n### Changed ⚠️\n\n- Changed default distributed training strategy from single-GPU to FSDP\n- Fixed behavior of `effective_memmap_dtype` to prevent unrecognized dtypes to be parsed as `uint16`.\n\n### Fixed ✅\n\n- Fixed restarting a training run in later epochs so that we no longer need to set the flag `--epoch=INT`.\n- Swapped in correct flan data mix.\n- Fix bug where the attention norm, when applied before the attention block, was modifying the residual stream.\n- Fixed `OLMo.from_checkpoint()` so that it correctly loads `olmo_core` and `torch_new` style checkpoints.\n- Fixed `preserve_rng_state` being incorrectly set to False when doing gradient checkpointing with dropout\n\n## Commits\n\ncee1a5df Merge pull request #710 from allenai\u002Fversion-dolma-flan-change\n213a6395 Merge pull request #711 from allenai\u002Fepwalsh\u002Ffix-unbound-qkv\n4575d405 Fix Conversion Issues + add support for remote upload.  (#694)\n78d79a51 Merge pull request #709 from allenai\u002Fshanea\u002Fdebugging-docs\n91478898 Merge pull request #685 from allenai\u002Fot-oe-eval-requests\n6cdc4cc0 Merge pull request #698 from allenai\u002Fshanea\u002Fcompare-model-state\ne5217cfa Merge pull request #705 from allenai\u002Fdave\u002Fcheckpoint_style_naming\nf4b386e6 Merge pull request #704 from allenai\u002Fshanea\u002Ffix-olmo-1.7-batch-size\n1e71ce34 Merge pull request #547 from allenai\u002Fshanea\u002Fadd-olmo-1.7-7b-to-readme\n6c4d53fe Merge pull request #702 from chrisc36\u002Fmain\n0bc7f6c7 Merge pull request #690 from allenai\u002Fshanea\u002Ftrace-model-outputs-2\n4332c322 Merge pull request #691 from allenai\u002Fdave\u002Fcosine_linear_envelope\n6587ddb9 Merge pull request #674 from allenai\u002Fdave\u002Fflan_data_mix\n7d63fe09 Merge pull request #671 from allenai\u002Fs3_unshard_to_hf\nc322b9a3 Merge pull request #686 from allenai\u002Ffix-from-checkpoint\nc482df74 Merge pull request #680 from allenai\u002Fshanea\u002Ffix-incorrect-attn-norm\n3e307106 Merge pull request #629 from allenai\u002Fepwalsh\u002Famberish\n4e004602 Add support for document masking during training (#661)\nb45002e8 make epoch logging less confusing\n1b7d2756 Fix restarts in later epochs (#670)\n345edc6f Merge branch 'main' of https:\u002F\u002Fgithub.com\u002Fallenai\u002FLLM\n66d2be71 Revert \"Update Beaker image\"\n07572231 Merge pull request #649 from allenai\u002FModelLadder\n90b3889b Merge pull request #660 from allenai\u002Ffix_convert_olmo_to_hf\ndfb7212f Merge pull request #616 from allenai\u002Fchameleon\nd627c94e Merge pull request #665 from allenai\u002Fddp-ckpt-fix\nab63296a Improving memmap type parser (#663)\nb55fb5f7 Merge pull request #662 from allenai\u002Ftiny-olmo-config-fix\n56d1fe07 Merge pull request #657 from allenai\u002Fshanea\u002Flumi-torch2.3-3\n26c2d536 Merge pull request #648 from allenai\u002Fshanea\u002Fdefault-fsdp-strategy\n65f1fff6 Merge pull request #656 from jeqcho\u002Fpatch-1\n20b82f86 Merge pull request #653 from allenai\u002Fshanea\u002Folmo-v0.4.0\n\n","2024-08-27T02:00:22",{"id":182,"version":183,"summary_zh":184,"released_at":185},101402,"v0.4.0","## What's new\n\n### Added 🎉\n\n- Added clipping fix to `Optimizer` class to make it work with FSDP `no_shard` and DDP.\n- Added tests to compare grad norm differences between torch optimizer and clipping and OLMo optimizer and clipping on both CPU and GPU.\n- Expose memmap dtype in data config \n- Added support for DDP training.\n- Added caching to disk of HF datasets used in downstream evals\n- Added FLOPs logging\n- Added configs for OLMo tiny set of models\n- Added configuration field `optimizer.record_update_metrics`, which defaults to `False`, but when set to `True` will trigger AdamW to collect the step size norm and absolute max for each parameter.\n- Added configuration field `optimizer.selective_updates`, which defaults to `False`, but when set to `True` will tell the optimizer to skip updating the parameter and state when the corresponding gradient is 0.\n- Added configuration field `optimizer.record_update_metrics`, which defaults to `False`, but when set to True will trigger AdamW to collect the step size norm and absolute max for each parameter.\n- Added `olmo_data`, a package holding data files like tokenizers.\n- Added ability to load tokenizers from `olmo_data` package data.\n\n### Changed ⚠️\n\n- Added original legacy unsharding implementation back, as the default. The new\nshared memory implementation can be used by passing `use_legacy_shared_mem_impl` to `unshard.py`.\n- Refactor weight initialization. IMPORTANT: this does not maintain backwards-compatibility with older configs; the jobs will still run, but may produce different outputs.\n- Changed the behavior of the Lion optimizer to only record the update cosine similarity when `optimizer.record_update_metrics` is `True` in order to be consistent with the API.\n- Added HF datasets into `olmo_data`, and changed downstream eval to load from the package.\n\n### Fixed ✅\n\n- Changed from `ignored_index` to `ignore_index` for `cross_entropy_loss` when `flash-attn>=2.5.8`.\n- Make `hf_olmo` support `AutoModelForCasualLM` and similar HF methods again.\n\n## Commits\n\nd423c11a Merge pull request #652 from allenai\u002Fshanea\u002Fupdate-to-torch2.3\nb10ab4b2 Merge pull request #651 from allenai\u002Fshanea\u002Flumi-torch2.3-2\na101b31b Merge pull request #646 from allenai\u002Fshanea\u002Fhf-datasets-from-package\n429a7525 Merge pull request #647 from allenai\u002Fshanea\u002Ffix-tokenizer-break\nbc60b8ae Add option to skip optim steps for 0 grad params (#636)\ncbc7c25b Merge pull request #645 from allenai\u002Fshanea\u002Ftokenizer-package-data\n1b2658bf Add option to record step size metrics from AdamW (#605)\na3e2ea7b multiple epoch fix\na1f118aa Merge pull request #628 from allenai\u002Folmo-tiny\nd7994c86 Fix Z-loss calculation (#634)\na5539f42 Merge pull request #631 from allenai\u002Fshanea\u002Fhf-olmo-auto-model\nd72a2626 Merge pull request #626 from allenai\u002Fshanea\u002Finspect-train-data-improvements\n2417b117 Make olmo-core checkpointer more robust on weka (#624)\nddc88471 Merge pull request #612 from allenai\u002Fddp\n41ed20a6 Merge pull request #623 from allenai\u002Fshanea\u002Fhf-save-to-disk-2\na33caa99 Merge pull request #604 from allenai\u002FWandbDiff\ne5d63a37 Merge pull request #619 from allenai\u002Fshanea\u002Fadd-olmo-1.7-7b-checkpoints\ne207df77 Officially add OLMo-core as a dependency (#615)\n72159aec Merge pull request #614 from allenai\u002Fshanea\u002Fpass-include-instance-metadata\nc2cedbc3 Merge pull request #607 from allenai\u002Frewrite-init\n578234d8 Merge pull request #611 from allenai\u002Fshanea\u002Fhf-get-tokenizer-from-config-2\nde43ee8a Merge pull request #610 from allenai\u002Fshanea\u002Fhf-get-tokenizer-from-config\n26392798 Merge pull request #594 from NeuralFabricAI\u002Flx\u002Fexpose-data-dtype\n9e894081 Create sensible filenames\n02a8a586 Merge pull request #603 from allenai\u002Fshanea\u002Funshard-without-passing-type\nae84d479 Merge pull request #602 from allenai\u002Fno_shard_ddp_clip\n40210bb1 Merge pull request #599 from allenai\u002Ftrain-olmo-large\n55c1e2f9 Merge pull request #601 from allenai\u002Fno_shard_ddp_clip\n5789cfe3 Merge pull request #593 from allenai\u002Fshanea\u002Finspect-train-data-no-indices\neafd154d Merge pull request #579 from MLgdg\u002Fmain\n652c7456 Merge pull request #590 from allenai\u002Fshanea\u002Fupdate-readme-to-olmo-1.7\n8ec28097 Merge pull request #589 from allenai\u002Fshanea\u002Fupdate-main-readme-hf\n6e714b89 Merge pull request #588 from allenai\u002Fshanea\u002Fhf-olmo-docs-auto-methods\n65d55755 Merge pull request #587 from allenai\u002Fshanea\u002Fstorage-cleaner-improvemnts\n0bddfe00 Merge pull request #585 from allenai\u002Fshanea\u002Fadd-hf-docs\ne6430a07 Merge pull request #582 from allenai\u002Fshanea\u002Fhybrid-shard-as-no-shard\nc29787a8 Merge pull request #569 from allenai\u002FMuennighoff\u002Ffix-torchv\n7a462c57 Merge pull request #580 from allenai\u002Fshanea\u002Fupdate-ignore-index-kwarg\n4f917fb7 Merge pull request #575 from allenai\u002Fshanea\u002Fadd-weka\n5c721cc8 Fix GPU tests CI (#574)\n467adcc9 Merge remote-tracking branch 'origin\u002Ftrain-olmo-large'\n4b2d12ea Merge pull request #565 from allenai\u002Freadme\nccc49fde Merge pull request #564 from allenai\u002Fshanea\u002Fadd-new-hf-converter\nb17abd05 Merge pull request #512 from liaoleo\u002Fmain\n295d3096 Merge pull request #561 from a","2024-07-11T21:52:25",{"id":187,"version":188,"summary_zh":189,"released_at":190},101403,"v0.3.0","## What's new\n\n### Added 🎉\n\n- Added support for Grouped Query Attention.\n- Added commonsense_qa and social_iqa downstream evaluation tasks\n- Makes it possible to read from http\u002Fhttps the same way we read from s3\u002Fr2.\n- Added MMLU multiple choice (A\u002FB\u002FC\u002FD) 5-shot variant downstream tasks\n- Tokenizer patch\n- Added option to specify number of model replicas when using hybrid sharding.\n\n### Changed ⚠️\n\n- Rename `Olmo` to `OLMo` everywhere in the codebase\n- Disabled automatic garbage collection during training, instead we run manually at regular intervals to avoid ranks getting out-of-sync with their own gc.\n\n### Removed 👋\n\n- Removed `AMDLayerNorm`, since the original layer norm bug has been fixed and we don't need this workaround anymore.\n- Removed `OLMoParallelBlock`.\n\n### Fixed ✅\n\n- Don't log garbage on nodes that aren't rank 0\n- Don't crash in the HF code when we are referring to a tokenizer in a local file\n- Point official training scripts to publicly available URLs\n- Corrected the `resize_token_embeddings` method in the `OLMoForCausalLM` class to properly update the token embeddings when resizing the vocabulary.\n- Changed `tie_weights` method to a no-op as weight tying is handled in olmo\u002Fmodel.py\n- Fixed the size calculation for qk layer norm\n- Fixed pipeline test failure that occurs due to a bug in transformers version 4.39.1\n- Make `hf_olmo` compatible with transformers versions >=4.40.0\n\n## Commits\n\n3b16e218 Merge pull request #556 from allenai\u002Fshanea\u002Fmake-hf-olmo-support-new-transformers\nccf7bf0a Merge pull request #555 from allenai\u002Fshanea\u002Fwandb-cancel-failure-bypass\n7be71cd7 use correct PG when collecting metrics with HYBRID shard (#551)\n06786a7b Merge pull request #548 from allenai\u002Fshanea\u002Ffix-olmo-name-hf\n4ed135e2 Merge pull request #540 from allenai\u002Fshanea\u002Fhybrid-sharding-num-groups-2\n2eae9888 Merge pull request #546 from allenai\u002Fshanea\u002Fadd-olmo-1.7-7b-checkpoints\nd2afcaaf Add cfg option `--scheduler.warmup_min_lr` (#542)\n9d408986 Merge pull request #537 from allenai\u002FAkshitaB-tokenizer-patch\n62c7954e Merge pull request #536 from allenai\u002Fshanea\u002Fstorage-cleaner-wandb-path-from-checkpoint\n657a55e8 Merge pull request #494 from allenai\u002Fshanea\u002Fstorage-cleaner-move-entry\n9a0a84a1 Merge pull request #527 from allenai\u002FPublicTrainingData\n0de5fdc8 Merge pull request #501 from djliden\u002Fdl\u002Ffix-embedding-resize\n4792f94c Adds a new experimental sharded checkpointer from OLMo-core (#532)\n1c129802 make garbage collection interval configurable (#533)\ndb2dee2e Merge pull request #503 from djliden\u002Fdl\u002Fhf-weight-tying\n8fad6498 Merge pull request #534 from allenai\u002Fshanea\u002Ffix-transformer-cache-position-regression\n71f7014e Merge pull request #528 from allenai\u002Fadd-mmlu-mc-5shot\n8472d0b4 Merge pull request #521 from allenai\u002Fdavidbrandfonbrener-patch-1\n194012a0 Merge pull request #523 from allenai\u002Fdavidbrandfonbrener-patch-2\n8949bd85 Added deprecation for memmap (#517)\n83cc8b10 Merge pull request #464 from allenai\u002Folmo7-ablations\nf8aef844 Merge pull request #509 from allenai\u002Fepwalsh\u002Fmanual-gc\n0ac82a93 Merge pull request #508 from allenai\u002FRunDataloader\n74de51d3 Merge pull request #414 from allenai\u002Fmitchish65-2\n417af0ed Merge pull request #504 from allenai\u002Fadd-csqa-siqa\n666da70f Patch other S3 methods with 404 detection fix\n0b6e28c0 Fix checking HTTP status code for boto3 responses\n0b835a8d Merge pull request #500 from allenai\u002Fshanea\u002Fexpose-official-checkpoints\n50da7a49 Add work-arounds for new-style checkpointing issues\n6d42d7ab Fix hang when training is canceled\n7eb7f3d6 Merge pull request #455 from gahdritz\u002Fmain\ned47c298 Merge pull request #453 from hxdtest\u002Fonly_rank0_log_metrics\nad8198e4 Merge pull request #495 from allenai\u002Fadd-basic-math\n1511fed2 Merge pull request #487 from allenai\u002Ffix-mmlu-prompt-bug\nc2840e4f Merge pull request #493 from allenai\u002Fshanea\u002Fstorage-cleaner-move-improvements\n658f7cc1 Merge pull request #466 from allenai\u002Frename\neb5b2dad Merge pull request #490 from allenai\u002FRemoveAMDLN\n752353bf Merge pull request #488 from allenai\u002Fshanea\u002Foptimize-unsharding-2\n\n","2024-04-25T19:23:44",{"id":192,"version":193,"summary_zh":194,"released_at":195},101404,"v0.2.5","## What's new\n\n### Fixed ✅\n\n- Fixed default value of `--tokenizer` argument to `scripts\u002Fprepare_tulu_data.py` to be an absolute path, not relative path, the script can be run from other directories.\n- Added the option to directly pass input embeddings to `OLMo` and `OLMoForCausalLM`.\n- Added support for Python 3.8.\n- Added code to throw an error if `output_attentions` is set to `True` in forward call to `OLMoForCausalLM`. This functionality hasn't been implemented yet.\n- Fixed running with data loading workers on LUMI\n\n### Added 🎉\n- Added `output_hidden_states` argument and associated functionality to `OLMo` and `OLMoForCausalLM` to return model intermediate hidden states.\n- Added MMLU downstream evaluation tasks, with prompt variations.\n- Added support for PyTorch v2.2.\n- Added ability to show logs from all ranks\n- Added option for QKV clipping.\n\n### Changed ⚠️\n\n- Refactor torch.load monkey patching for legacy checkpoint unsharding in anticipation of unsharding implementation change.\n\n## Commits\n\nc4996325 Add option for QKV clipping (#489)\n31d85287 Pull checkpoint patch from `mitchish-gqa-2`\n03d7643c Merge pull request #486 from allenai\u002Fshanea\u002Fmonkey-patch-ctx-manager\nfd3a57ba Merge pull request #483 from allenai\u002Fshanea\u002Fstorage-cleaner-unshard-improvements\n1d264e44 Merge pull request #481 from allenai\u002FWorkersOnLumi\n70ad30c6 Merge pull request #480 from allenai\u002FFirehose\n493c0b83 Add MMLU prompt variants (#484)\ncb711e24 Add support for PyTorch v2.2 (#476)\n67d24f5e Merge pull request #468 from allenai\u002Fmmlu-downstream\n0c58beed Fix bug when clipping is disabled\n922db6aa Only run the profiler through a single cycle (#463)\n37ca7893 Merge pull request #462 from allenai\u002Fepwalsh\u002Ffsdp-wrap-patch\ncc367095 Add attn bias arg to HF wrapper (#458)\n7f7abbb6 Merge pull request #451 from sarahwie\u002Fmain\n9fd9130d Add support for Python 3.8 (#448)\nd9c09937 Require Python>=3.9 for now\n97296e61 Merge pull request #442 from allenai\u002Fshanea\u002Fadd-input-embedding-arg\n3be4c1ec add link to W&B logs for 1B run\nd7d4de4c Add link to OLMo-7B-Twin-2T W&B logs\ncf121084 Update README.md (#429)\n15af6688 freeze official configs for reproductions (#421)\n7739fe17 Add link to W&B logs for OLMo-7B\n80db5e3d Fix default value of `--tokenizer`\n6765317e Add link to paper in README badge\n\n","2024-03-07T00:31:06",{"id":197,"version":198,"summary_zh":199,"released_at":200},101405,"v0.2.4","## What's new\n\n### Fixed ✅\n\n- Fixed an issue with the HuggingFace integration where we were inadvertently using a feature that was introduced in Python 3.10, causing an error for older Python versions.\n\n## Commits\n\n8a3f2d86 Fix HF integration for Python \u003C 3.10 (#426)\n49c8647f Use temp branding GIF for logo (for now) (#419)\n\n","2024-02-02T18:40:15",{"id":202,"version":203,"summary_zh":204,"released_at":205},101406,"v0.2.3","## What's new\n\n\n\n## Commits\n\n98c115cf Bump version to v0.2.3 for release\n0e53b338 specify dependencies in pyproject.toml (#418)\n18e5dada update PyPI release process\n141cc945 Merge pull request #415 from allenai\u002Freadme-inf\n25872405 Merge pull request #417 from allenai\u002FMuennighoff\u002Fckpt\na5a01a2f Merge pull request #416 from allenai\u002Fnol_rdme\n98425a55 Merge pull request #413 from allenai\u002Fshanea\u002Fstorage-cleaner-s3-upload-cleanup\n3053bfae Update install instructions in README\nf36ac42e Merge pull request #410 from allenai\u002Fepwalsh\u002Ffine-tune-with-label-masking\ndcae8e82 Merge pull request #411 from allenai\u002Fepwalsh\u002Flr-schedule-tokens\n45ed078b Add more mcli configs\n905359e6 fix bug with saving unsharded checkpoint\n3e3df714 Merge pull request #409 from allenai\u002Fepwalsh\u002Ftulu-fine-tune\na2e1d13b Merge pull request #368 from allenai\u002Fmitchish-lumi\n5a735dd2 Merge pull request #350 from allenai\u002Fmitchish\ndf195549 Merge pull request #388 from allenai\u002Fmitchish65\n23eb949f Train a few steps after time limit reached (#362)\nac1aee12 Merge pull request #408 from allenai\u002FNixLogz\n6da42cf5 ensure we save checkpoint at end of loop\n568a3d89 Merge pull request #406 from allenai\u002Fhf-olmo-loading\n3c514027 Merge pull request #407 from allenai\u002Fshanea\u002Fstorage-cleaner-avoid-redundant-copy\n53217d21 Merge pull request #405 from allenai\u002Fshanea\u002Fstorage-cleaner-fix-upload-path\n5eb26aaa Merge pull request #404 from allenai\u002Fshanea\u002Fstorage-cleaner-minor-fixes\n87ed747d backwards compat fix\n1c13e5fd Merge pull request #403 from allenai\u002Fshanea\u002Fstorage-cleaner-fix-max-archive-size\n685d11ba Merge pull request #400 from allenai\u002Fshanea\u002Fstorage-cleaner-wandb\n5bdccc32 Merge pull request #402 from allenai\u002Fshanea\u002Fstorage-cleaner-is-run-improvement\n75d67383 Merge pull request #401 from allenai\u002Fshanea\u002Fstorage-cleaner-is-file-no-key\n0475f3ad Make logo a little smaller\n1184050c Add logo to README\ne2d77c47 Ephemeral checkpoints (#397)\n6f2abfbe Merge pull request #399 from allenai\u002Fshane\u002Fstorage-cleaner-fix-s3-upload\nf8beb5be Merge pull request #398 from allenai\u002Fshanea\u002Fstorage-cleaner-move-run\n185d7e26 Move remaining top-level mkd docs into docs folder (#395)\n5d03d385 Merge pull request #396 from allenai\u002Fshanea\u002Fstorage-cleaner-delete-temp-files\nfe496935 Merge pull request #382 from allenai\u002Fshanea\u002Fstorage-cleaner-unsharding-legacy\n1ede949c Merge pull request #381 from allenai\u002Fshanea\u002Fstorage-cleaner-unsharding-2\n9cc71544 update some links to new repo (#394)\n\n","2024-01-31T18:36:09",{"id":207,"version":208,"summary_zh":209,"released_at":210},101407,"v0.2.2","## What's new\n\n\n\n## Commits\n\n364e21ec Merge pull request #393 from allenai\u002Fhf-olmo-auto-map\n\n","2023-12-11T05:58:51",{"id":212,"version":213,"summary_zh":214,"released_at":215},101408,"v0.2.1","## What's new\n\n\n\n## Commits\n\nad3e6763 missing readme\n9fa23b4d Merge pull request #392 from allenai\u002Fhf-bug-fix\n\n","2023-12-11T00:11:14",{"id":217,"version":218,"summary_zh":219,"released_at":220},101409,"v0.2.0","## What's new\n\n### Added 🎉\n\n- GPT-based model.\n- Tokenizer and data pre-processing pipeline.\n- training script.\n- Triton-based FlashAttention.\n\n## Commits\n\ne801af8e add release proc\ne643f5ed update pyproject\ndbc81777 Bump version to v0.2.0 for release\ne99dbe5d Merge pull request #391 from allenai\u002Fhf-olmo-new\na120ab24 Merge pull request #380 from allenai\u002Fshanea\u002Fstorage-cleaner-download-upload\n4e849e4b Merge pull request #390 from allenai\u002Fshanea\u002Fstorage-cleaner-archive-fix-2\n1dbc3461 Merge pull request #378 from allenai\u002Fshanea\u002Fstorage-cleaner-cached-path\n22cefa28 Merge pull request #389 from allenai\u002Fshanea\u002Fadd-r2-scheme\nac01778e fix\n6c79c634 add option to only unshard model\nd1c185b6 Merge pull request #387 from allenai\u002Fepwalsh\u002Fdist-init\ne30d29f1 Merge pull request #364 from allenai\u002Fshanea\u002Fstorage-cleaner\nff883e5d Merge pull request #385 from allenai\u002Fepwalsh\u002Fmax-duration-tokens\ne16e606c Merge pull request #383 from allenai\u002Fepwalsh\u002Fstart-new-epoch\n\n","2023-12-10T06:43:49",{"id":222,"version":223,"summary_zh":224,"released_at":225},101410,"v0.1.1","## What's new\n\n\n\n## Commits\n\n\n","2023-11-27T01:04:15",{"id":227,"version":228,"summary_zh":229,"released_at":230},101411,"v0.1.0","## What's new\n\n### Added 🎉\n\n- GPT-based model.\n- Tokenizer and data pre-processing pipeline.\n- training script.\n- Triton-based FlashAttention.\n\n## Commits\n\nf1ba78e2 moving readme to notes\n6c94994c Bump version to v0.1.0 for release\nf09a500d Add a \"constant\" LR scheduler (#376)\ndcdadc5e Merge pull request #377 from allenai\u002FMuennighoff\u002Fsplit-model-comps\n80b081b4 Merge pull request #374 from allenai\u002Fepwalsh\u002Fthreaded-data-loading\n9c8e67ed Merge pull request #373 from allenai\u002Fchore\u002Fpaths\n1f51fece Merge pull request #375 from allenai\u002FMuennighoff\u002Fmove-torch-utils\n9d5aa119 Merge pull request #370 from allenai\u002FCheckpointLoading\n38be6a7d Merge pull request #372 from allenai\u002Fepwalsh\u002Foptim-state-fix\nc205912b Fix how we update grad_norm_exp_avg (#371)\n9320f9b9 Fix unsharding local checkpoints w\u002F torch 2.1 (#369)\nb8a174fe Merge pull request #367 from allenai\u002FFacePalm\n13548fd3 Merge pull request #365 from allenai\u002Fwrap_and_shard\n6c0e4196 Add gradient clipping warmup (#363)\n0afafd6d Fix stale links in README, scripts cleanup  (#359)\n42dba3cd remove data team's stuff (#357)\n4bb69666 consolidate Python configs into pyproject.toml, other clean up (#353)\na952f448 minor fixes to kempner docs (#354)\n026793ea Merge pull request #347 from allenai\u002Fepwalsh\u002Fblock-groups-load-fix\n62fc2fe4 Add two more FSDP wrapping strategies (#355)\n4ccf2bdc Merge pull request #346 from allenai\u002Fshanea\u002Fllama-block\nda91f341 Merge pull request #317 from allenai\u002FLlama\nfd2425f2 Adds a YAML validator to automatically find the last checkpoint (#348)\n10999429 Upload profiler data to remote save folder (#338)\ndb0756ff Merge pull request #335 from allenai\u002FKempner\n5c64338b Merge pull request #343 from allenai\u002FActivationCheckpointing\n558102ef Merge pull request #342 from allenai\u002FS3Client\ncd73387a Add option to FSDP wrap by groups of blocks (#340)\nc1a45198 Fix dtype casting on CPU (#339)\n104d1cef Move remaining checkpointing logic to Checkpointer class (#331)\na465caa9 Merge pull request #337 from allenai\u002FUnshardSkipKeys\n4980badf set mcli time limit to null\nf974a1dd update mitch ish configs\n07404f81 Merge pull request #308 from allenai\u002Ffine-grained-metrics\n4644ff53 Lazily init s3 client (#333)\n809fe9d0 Load state dicts to CPU (#328)\n1bff3085 ensure bias is created in fp32 (#327)\nd4744d08 Bring back global gradient clipping and improve speed of collecting metrics (#326)\n54572d35 Add stop_at config option\ne63b3896 Fix SDP NaN bug (#323)\nfddded56 Features to match OpenLM (#302)\nd2e84fea Refactor checkpointing, bring back legacy sharded checkpointing as the default (#316)\nfed4cf39 Merge pull request #311 from allenai\u002Fpass-thru-model-kwargs\na5cd0e6c Merge pull request #304 from allenai\u002Fppl-suite-v3\n536d029e Merge pull request #306 from allenai\u002Fkeep-instance-info\n0b5f68dd Merge pull request #314 from allenai\u002FResetOptimizerState\ne8bd1227 Merge pull request #315 from allenai\u002FMemoryEnvVar\nda1f0b80 Merge pull request #313 from allenai\u002FNanCheck\n94133da0 Fixes pyspy script\n602968ae New-style checkpointing (again) (#307)\n973090f4 implement bytes range for GS\n18e061dc Merge pull request #303 from allenai\u002Fshanea\u002Ffix-leftover-data-partitioning\n0a1455b0 Merge pull request #301 from allenai\u002Fshanea\u002Ffix-s3-keyerror-failures\ne7b92a69 comment\n6ebd5d3a Add configs for v1.5 mix\n8e2b8be8 Merge pull request #297 from allenai\u002FPerfTests\n62dde55e Make `resource_path()` more robust\n900544ed Prepare 7B config for MCLI (#295)\n309bf84d Merge pull request #294 from allenai\u002Fpetew\u002Flinear-schedule\n91f499b3 Ignore warnings from urllib3, don't print config when it's huge\n012e97fb Merge pull request #290 from allenai\u002Ftorch2.1init\naec449c4 update mcli config\n27dd5127 MCLI configs (#286)\na2b369aa Merge pull request #279 from allenai\u002Fpetew\u002Ftrain-metrics\n5ad0d8ce Merge pull request #282 from allenai\u002Frsqrt\ncc787ed9 Merge pull request #277 from allenai\u002Fshanea\u002Fadd-truncated-normal-init\nfabda71f Merge pull request #274 from allenai\u002Fpetew\u002Flayer-norm\n70a3f4c1 Merge pull request #280 from allenai\u002Fpetew\u002Freduce-dtype\n2a7f694f Merge pull request #278 from allenai\u002Fupdate-hf-olmo-config\nef85d5cf Merge pull request #265 from allenai\u002FLayerNormAffine-ManualLayerNorm-Profiling\n2df922b0 Merge pull request #276 from allenai\u002Fpetew\u002Fsys-metrics\n921c2545 Merge pull request #275 from allenai\u002Fsimplify-eff-benchmark\n400a1d28 Minor cleanup of grad clipping (#273)\n18f34595 fix updating grad_norm_exp_avg (#272)\n54dbd489 Merge pull request #238 from allenai\u002Finference-efficiency-pentathlon\n95555f4e Refactor how we clip gradients and collect optimizer metrics (#261)\n6cc09fe2 Merge pull request #271 from allenai\u002FPythonProfiling2-UnwindingChanges\n41b06631 Merge pull request #269 from allenai\u002FPythonProfiling\n2eedf07b Fix speed issue on LUMI with 7B model (#270)\nd2abecdd Merge pull request #267 from allenai\u002Fv2-pii-tagging\n5b4c68ed fix isort config\nc8a27007 Merge pull request #253 from allenai\u002FSavedTokenizer\n26e17c38 Merge pull request #264 from allenai\u002FLayerNormAffine-ManualLayerNorm-TurnedOffForSafety\na49f4ece Make Dropout a no-op when p=0.0 ","2023-11-27T00:49:00"]