[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-chenfei-wu--TaskMatrix":3,"tool-chenfei-wu--TaskMatrix":64},[4,17,27,35,43,56],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":16},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,3,"2026-04-05T11:01:52",[13,14,15],"开发框架","图像","Agent","ready",{"id":18,"name":19,"github_repo":20,"description_zh":21,"stars":22,"difficulty_score":23,"last_commit_at":24,"category_tags":25,"status":16},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",138956,2,"2026-04-05T11:33:21",[13,15,26],"语言模型",{"id":28,"name":29,"github_repo":30,"description_zh":31,"stars":32,"difficulty_score":23,"last_commit_at":33,"category_tags":34,"status":16},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",107662,"2026-04-03T11:11:01",[13,14,15],{"id":36,"name":37,"github_repo":38,"description_zh":39,"stars":40,"difficulty_score":23,"last_commit_at":41,"category_tags":42,"status":16},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[13,26],{"id":44,"name":45,"github_repo":46,"description_zh":47,"stars":48,"difficulty_score":23,"last_commit_at":49,"category_tags":50,"status":16},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",84991,"2026-04-05T10:45:23",[14,51,52,53,15,54,26,13,55],"数据工具","视频","插件","其他","音频",{"id":57,"name":58,"github_repo":59,"description_zh":60,"stars":61,"difficulty_score":10,"last_commit_at":62,"category_tags":63,"status":16},3128,"ragflow","infiniflow\u002Fragflow","RAGFlow 是一款领先的开源检索增强生成（RAG）引擎，旨在为大语言模型构建更精准、可靠的上下文层。它巧妙地将前沿的 RAG 技术与智能体（Agent）能力相结合，不仅支持从各类文档中高效提取知识，还能让模型基于这些知识进行逻辑推理和任务执行。\n\n在大模型应用中，幻觉问题和知识滞后是常见痛点。RAGFlow 通过深度解析复杂文档结构（如表格、图表及混合排版），显著提升了信息检索的准确度，从而有效减少模型“胡编乱造”的现象，确保回答既有据可依又具备时效性。其内置的智能体机制更进一步，使系统不仅能回答问题，还能自主规划步骤解决复杂问题。\n\n这款工具特别适合开发者、企业技术团队以及 AI 研究人员使用。无论是希望快速搭建私有知识库问答系统，还是致力于探索大模型在垂直领域落地的创新者，都能从中受益。RAGFlow 提供了可视化的工作流编排界面和灵活的 API 接口，既降低了非算法背景用户的上手门槛，也满足了专业开发者对系统深度定制的需求。作为基于 Apache 2.0 协议开源的项目，它正成为连接通用大模型与行业专有知识之间的重要桥梁。",77062,"2026-04-04T04:44:48",[15,14,13,26,54],{"id":65,"github_repo":66,"name":67,"description_en":68,"description_zh":69,"ai_summary_zh":69,"readme_en":70,"readme_zh":71,"quickstart_zh":72,"use_case_zh":73,"hero_image_url":74,"owner_login":75,"owner_name":76,"owner_avatar_url":77,"owner_bio":78,"owner_company":79,"owner_location":80,"owner_email":81,"owner_twitter":68,"owner_website":82,"owner_url":83,"languages":84,"stars":97,"forks":98,"last_commit_at":99,"license":100,"difficulty_score":10,"env_os":101,"env_gpu":102,"env_ram":103,"env_deps":104,"category_tags":116,"github_topics":68,"view_count":117,"oss_zip_url":68,"oss_zip_packed_at":68,"status":16,"created_at":118,"updated_at":119,"faqs":120,"releases":148},375,"chenfei-wu\u002FTaskMatrix","TaskMatrix",null,"TaskMatrix 是一个开源项目，旨在将 ChatGPT 与多种视觉基础模型无缝连接，赋予大语言模型“看图说话”的能力。在传统对话中，AI 无法直接处理图像，TaskMatrix 解决了这一痛点，让用户在聊天时既能发送也能接收图片，实现真正的全能多模态交互。\n\nTaskMatrix 非常适合开发者、研究人员以及对 AI 创意工作流感兴趣的设计师。其核心亮点在于提出了“模板”机制，这是一种预定义的执行流程，能够协助 ChatGPT 自动组装涉及多个专业模型的复杂任务。例如，结合 GroundingDINO 和 Segment Anything，TaskMatrix 可以直接根据文本指令完成图像定位、分割或无损扩展画面，无需额外训练。此外，项目已支持中文输入，并允许社区贡献新模板以拓展功能。通过将 ChatGPT 的通用理解力与视觉模型的领域专业知识相结合，TaskMatrix 为构建更智能的 AI 助手提供了灵活且高效的解决方案。","# TaskMatrix\n\n**TaskMatrix** connects ChatGPT and a series of Visual Foundation Models to enable **sending** and **receiving** images during chatting.\n\nSee our paper: [\u003Cfont size=5>Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models\u003C\u002Ffont>](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.04671)\n\n\u003Ca src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97-Open%20in%20Spaces-blue\" href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmicrosoft\u002Fvisual_chatgpt\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97-Open%20in%20Spaces-blue\" alt=\"Open in Spaces\">\n\u003C\u002Fa>\n\n\u003Ca src=\"https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg\" href=\"https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1P3jJqKEWEaeNcZg8fODbbWeQ3gxOHk2-?usp=sharing\">\n    \u003Cimg src=\"https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg\" alt=\"Open in Colab\">\n\u003C\u002Fa>\n\n## Updates:\n- Now TaskMatrix supports [GroundingDINO](https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGroundingDINO) and [segment-anything](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything)! Thanks **@jordddan** for his efforts. For the image editing case, `GroundingDINO` is first used to locate bounding boxes guided by given text, then `segment-anything` is used to generate the related mask, and finally stable diffusion inpainting is used to edit image based on the mask. \n    - Firstly, run `python visual_chatgpt.py --load \"Text2Box_cuda:0,Segmenting_cuda:0,Inpainting_cuda:0,ImageCaptioning_cuda:0\"`\n    - Then, say `find xxx in the image` or `segment xxx in the image`. `xxx` is an object. TaskMatrix will return the detection or segmentation result!\n\n\n- Now TaskMatrix can support Chinese! Thanks to **@Wang-Xiaodong1899** for his efforts.\n- We propose the **template** idea in TaskMatrix!\n    - A template is a **pre-defined execution flow** that assists ChatGPT in assembling complex tasks involving multiple foundation models. \n    - A template contains the **experiential solution** to complex tasks as determined by humans. \n    - A template can **invoke multiple foundation models** or even **establish a new ChatGPT session**\n    - To define a **template**, simply adding a class with attributes `template_model = True`\n- Thanks to **@ShengmingYin** and **@thebestannie** for providing a template example in `InfinityOutPainting` class (see the following gif)\n    - Firstly, run `python visual_chatgpt.py --load \"Inpainting_cuda:0,ImageCaptioning_cuda:0,VisualQuestionAnswering_cuda:0\"`\n    - Secondly, say `extend the image to 2048x1024` to TaskMatrix!\n    - By simply creating an `InfinityOutPainting` template, TaskMatrix can seamlessly extend images to any size through collaboration with existing `ImageCaptioning`, `Inpainting`, and `VisualQuestionAnswering` foundation models, **without the need for additional training**.\n- **TaskMatrix needs the effort of the community! We crave your contribution to add new and interesting features!**\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_readme_01af52a8bd27.gif\" width=\"750\">\n\n\n## Insight & Goal:\nOn the one hand, **ChatGPT (or LLMs)** serves as a **general interface** that provides a broad and diverse understanding of a\nwide range of topics. On the other hand, **Foundation Models** serve as **domain experts** by providing deep knowledge in specific domains.\nBy leveraging **both general and deep knowledge**, we aim at building an AI that is capable of handling various tasks.\n\n\n## Demo \n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_readme_b02dd9c36177.gif\" width=\"750\">\n\n##  System Architecture \n\n \n\u003Cp align=\"center\">\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_readme_240ed7dd178c.jpg\" alt=\"Logo\">\u003C\u002Fp>\n\n\n## Quick Start\n\n```\n# clone the repo\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FTaskMatrix.git\n\n# Go to directory\ncd visual-chatgpt\n\n# create a new environment\nconda create -n visgpt python=3.8\n\n# activate the new environment\nconda activate visgpt\n\n#  prepare the basic environments\npip install -r requirements.txt\npip install  git+https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGroundingDINO.git\npip install  git+https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything.git\n\n# prepare your private OpenAI key (for Linux)\nexport OPENAI_API_KEY={Your_Private_Openai_Key}\n\n# prepare your private OpenAI key (for Windows)\nset OPENAI_API_KEY={Your_Private_Openai_Key}\n\n# Start TaskMatrix !\n# You can specify the GPU\u002FCPU assignment by \"--load\", the parameter indicates which \n# Visual Foundation Model to use and where it will be loaded to\n# The model and device are separated by underline '_', the different models are separated by comma ','\n# The available Visual Foundation Models can be found in the following table\n# For example, if you want to load ImageCaptioning to cpu and Text2Image to cuda:0\n# You can use: \"ImageCaptioning_cpu,Text2Image_cuda:0\"\n\n# Advice for CPU Users\npython visual_chatgpt.py --load ImageCaptioning_cpu,Text2Image_cpu\n\n# Advice for 1 Tesla T4 15GB  (Google Colab)                       \npython visual_chatgpt.py --load \"ImageCaptioning_cuda:0,Text2Image_cuda:0\"\n                                \n# Advice for 4 Tesla V100 32GB                            \npython visual_chatgpt.py --load \"Text2Box_cuda:0,Segmenting_cuda:0,\n    Inpainting_cuda:0,ImageCaptioning_cuda:0,\n    Text2Image_cuda:1,Image2Canny_cpu,CannyText2Image_cuda:1,\n    Image2Depth_cpu,DepthText2Image_cuda:1,VisualQuestionAnswering_cuda:2,\n    InstructPix2Pix_cuda:2,Image2Scribble_cpu,ScribbleText2Image_cuda:2,\n    SegText2Image_cuda:2,Image2Pose_cpu,PoseText2Image_cuda:2,\n    Image2Hed_cpu,HedText2Image_cuda:3,Image2Normal_cpu,\n    NormalText2Image_cuda:3,Image2Line_cpu,LineText2Image_cuda:3\"\n\n```\n\n## GPU memory usage\nHere we list the GPU memory usage of each visual foundation model, you can specify which one you like:\n\n| Foundation Model        | GPU Memory (MB) |\n|------------------------|-----------------|\n| ImageEditing           | 3981            |\n| InstructPix2Pix        | 2827            |\n| Text2Image             | 3385            |\n| ImageCaptioning        | 1209            |\n| Image2Canny            | 0               |\n| CannyText2Image        | 3531            |\n| Image2Line             | 0               |\n| LineText2Image         | 3529            |\n| Image2Hed              | 0               |\n| HedText2Image          | 3529            |\n| Image2Scribble         | 0               |\n| ScribbleText2Image     | 3531            |\n| Image2Pose             | 0               |\n| PoseText2Image         | 3529            |\n| Image2Seg              | 919             |\n| SegText2Image          | 3529            |\n| Image2Depth            | 0               |\n| DepthText2Image        | 3531            |\n| Image2Normal           | 0               |\n| NormalText2Image       | 3529            |\n| VisualQuestionAnswering| 1495            |\n\n## Acknowledgement\nWe appreciate the open source of the following projects:\n\n[Hugging Face](https:\u002F\u002Fgithub.com\u002Fhuggingface)  \n[LangChain](https:\u002F\u002Fgithub.com\u002Fhwchase17\u002Flangchain)  \n[Stable Diffusion](https:\u002F\u002Fgithub.com\u002FCompVis\u002Fstable-diffusion)   \n[ControlNet](https:\u002F\u002Fgithub.com\u002Flllyasviel\u002FControlNet)   \n[InstructPix2Pix](https:\u002F\u002Fgithub.com\u002Ftimothybrooks\u002Finstruct-pix2pix)   \n[CLIPSeg](https:\u002F\u002Fgithub.com\u002Ftimojl\u002Fclipseg)  \n[BLIP](https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FBLIP)  \n\n## Contact Information\nFor help or issues using the TaskMatrix, please submit a GitHub issue.\n\nFor other communications, please contact Chenfei WU (chewu@microsoft.com) or Nan DUAN (nanduan@microsoft.com).\n\n## Trademark Notice\n\nTrademarks This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft’s Trademark & Brand Guidelines](https:\u002F\u002Fwww.microsoft.com\u002Fen-us\u002Flegal\u002Fintellectualproperty\u002Ftrademarks). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.\n\n## Disclaimer\nThe recommended models in this Repo are just examples, used for scientific research exploring the concept of task automation and benchmarking with the paper published at [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.04671). Users can replace the models in this Repo according to their research needs. When using the recommended models in this Repo, you need to comply with the licenses of these models respectively. Microsoft shall not be held liable for any infringement of third-party rights resulting from your usage of this repo. Users agree to defend, indemnify and hold Microsoft harmless from and against all damages, costs, and attorneys' fees in connection with any claims arising from this Repo. If anyone believes that this Repo infringes on your rights, please notify the project owner [email](chewu@microsoft.com).\n","# TaskMatrix\n\n**TaskMatrix** 连接 ChatGPT 和一系列**视觉基础模型 (Visual Foundation Models)**，实现在聊天过程中**发送**和**接收**图像。\n\n查看我们的论文：[\u003Cfont size=5>Visual ChatGPT: 使用视觉基础模型进行对话、绘图和编辑\u003C\u002Ffont>](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.04671)\n\n\u003Ca src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97-Open%20in%20Spaces-blue\" href=\"https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmicrosoft\u002Fvisual_chatgpt\">\n    \u003Cimg src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F%F0%9F%A4%97-Open%20in%20Spaces-blue\" alt=\"Open in Spaces\">\n\u003C\u002Fa>\n\n\u003Ca src=\"https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg\" href=\"https:\u002F\u002Fcolab.research.google.com\u002Fdrive\u002F1P3jJqKEWEaeNcZg8fODbbWeQ3gxOHk2-?usp=sharing\">\n    \u003Cimg src=\"https:\u002F\u002Fcolab.research.google.com\u002Fassets\u002Fcolab-badge.svg\" alt=\"Open in Colab\">\n\u003C\u002Fa>\n\n## 更新：\n- 现在 TaskMatrix 支持 [GroundingDINO](https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGroundingDINO) 和 [segment-anything](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything)! 感谢 **@jordddan** 的努力。对于图像编辑案例，首先使用 `GroundingDINO` 根据给定文本定位边界框，然后使用 `segment-anything` 生成相关掩码，最后使用 stable diffusion inpainting 基于掩码编辑图像。 \n    - 首先，运行 `python visual_chatgpt.py --load \"Text2Box_cuda:0,Segmenting_cuda:0,Inpainting_cuda:0,ImageCaptioning_cuda:0\"`\n    - 然后，说 `find xxx in the image` 或 `segment xxx in the image`。`xxx` 是一个对象。TaskMatrix 将返回检测或分割结果！\n\n\n- 现在 TaskMatrix 可以支持中文！感谢 **@Wang-Xiaodong1899** 的努力。\n- 我们在 TaskMatrix 中提出了**模板 (Template)** 的概念！\n    - 模板是一个**预定义的执行流程**，协助 ChatGPT 组装涉及多个基础模型的复杂任务。 \n    - 模板包含由人类确定的复杂任务的**经验性解决方案**。 \n    - 模板可以**调用多个基础模型**，甚至可以**建立新的 ChatGPT 会话**。\n    - 要定义一个**模板**，只需添加一个具有属性 `template_model = True` 的类。\n- 感谢 **@ShengmingYin** 和 **@thebestannie** 在 `InfinityOutPainting` 类中提供的模板示例（见以下 gif）\n    - 首先，运行 `python visual_chatgpt.py --load \"Inpainting_cuda:0,ImageCaptioning_cuda:0,VisualQuestionAnswering_cuda:0\"`\n    - 其次，向 TaskMatrix 说 `extend the image to 2048x1024`！\n    - 通过简单地创建 `InfinityOutPainting` 模板，TaskMatrix 可以与现有的 `ImageCaptioning`、`Inpainting` 和 `VisualQuestionAnswering` 基础模型协作，无缝地将图像扩展到任何大小，**无需额外的训练**。\n- **TaskMatrix 需要社区的努力！我们渴望你的贡献来添加新的有趣功能！**\n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_readme_01af52a8bd27.gif\" width=\"750\">\n\n\n## 洞察与目标：\n一方面，**ChatGPT（或大语言模型 LLMs）** 作为一个**通用接口**，提供对广泛主题的广泛而多样的理解。另一方面，**基础模型 (Foundation Models)** 作为**领域专家**，提供特定领域的深入知识。\n通过利用**通用知识和深度知识**，我们旨在构建能够处理各种任务的 AI。\n\n\n## 演示 \n\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_readme_b02dd9c36177.gif\" width=\"750\">\n\n## 系统架构 \n\n \n\u003Cp align=\"center\">\u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_readme_240ed7dd178c.jpg\" alt=\"Logo\">\u003C\u002Fp>\n\n\n## 快速开始\n\n```\n# clone the repo\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FTaskMatrix.git\n\n# Go to directory\ncd visual-chatgpt\n\n# create a new environment\nconda create -n visgpt python=3.8\n\n# activate the new environment\nconda activate visgpt\n\n#  prepare the basic environments\npip install -r requirements.txt\npip install  git+https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGroundingDINO.git\npip install  git+https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything.git\n\n# prepare your private OpenAI key (for Linux)\nexport OPENAI_API_KEY={Your_Private_Openai_Key}\n\n# prepare your private OpenAI key (for Windows)\nset OPENAI_API_KEY={Your_Private_Openai_Key}\n\n# Start TaskMatrix !\n# You can specify the GPU\u002FCPU assignment by \"--load\", the parameter indicates which \n# Visual Foundation Model to use and where it will be loaded to\n# The model and device are separated by underline '_', the different models are separated by comma ','\n# The available Visual Foundation Models can be found in the following table\n# For example, if you want to load ImageCaptioning to cpu and Text2Image to cuda:0\n# You can use: \"ImageCaptioning_cpu,Text2Image_cuda:0\"\n\n# Advice for CPU Users\npython visual_chatgpt.py --load ImageCaptioning_cpu,Text2Image_cpu\n\n# Advice for 1 Tesla T4 15GB  (Google Colab)                       \npython visual_chatgpt.py --load \"ImageCaptioning_cuda:0,Text2Image_cuda:0\"\n                                \n# Advice for 4 Tesla V100 32GB                            \npython visual_chatgpt.py --load \"Text2Box_cuda:0,Segmenting_cuda:0,\n    Inpainting_cuda:0,ImageCaptioning_cuda:0,\n    Text2Image_cuda:1,Image2Canny_cpu,CannyText2Image_cuda:1,\n    Image2Depth_cpu,DepthText2Image_cuda:1,VisualQuestionAnswering_cuda:2,\n    InstructPix2Pix_cuda:2,Image2Scribble_cpu,ScribbleText2Image_cuda:2,\n    SegText2Image_cuda:2,Image2Pose_cpu,PoseText2Image_cuda:2,\n    Image2Hed_cpu,HedText2Image_cuda:3,Image2Normal_cpu,\n    NormalText2Image_cuda:3,Image2Line_cpu,LineText2Image_cuda:3\"\n\n```\n\n## GPU 显存使用情况\n在此我们列出每个视觉基础模型的 GPU 显存使用情况，您可以指定您喜欢的模型：\n\n| 基础模型        | GPU 显存 (MB) |\n|------------------------|-----------------|\n| ImageEditing           | 3981            |\n| InstructPix2Pix        | 2827            |\n| Text2Image             | 3385            |\n| ImageCaptioning        | 1209            |\n| Image2Canny            | 0               |\n| CannyText2Image        | 3531            |\n| Image2Line             | 0               |\n| LineText2Image         | 3529            |\n| Image2Hed              | 0               |\n| HedText2Image          | 3529            |\n| Image2Scribble         | 0               |\n| ScribbleText2Image     | 3531            |\n| Image2Pose             | 0               |\n| PoseText2Image         | 3529            |\n| Image2Seg              | 919             |\n| SegText2Image          | 3529            |\n| Image2Depth            | 0               |\n| DepthText2Image        | 3531            |\n| Image2Normal           | 0               |\n| NormalText2Image       | 3529            |\n| VisualQuestionAnswering| 1495            |\n\n## 致谢\n我们感谢以下项目的开源贡献：\n\n[Hugging Face](https:\u002F\u002Fgithub.com\u002Fhuggingface)  \n[LangChain](https:\u002F\u002Fgithub.com\u002Fhwchase17\u002Flangchain)  \n[Stable Diffusion](https:\u002F\u002Fgithub.com\u002FCompVis\u002Fstable-diffusion)   \n[ControlNet](https:\u002F\u002Fgithub.com\u002Flllyasviel\u002FControlNet)   \n[InstructPix2Pix](https:\u002F\u002Fgithub.com\u002Ftimothybrooks\u002Finstruct-pix2pix)   \n[CLIPSeg](https:\u002F\u002Fgithub.com\u002Ftimojl\u002Fclipseg)  \n[BLIP](https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FBLIP)  \n\n## 联系信息\n如需使用 TaskMatrix 的帮助或遇到问题，请提交 GitHub Issue。\n\n其他沟通事宜，请联系 Chenfei WU (chewu@microsoft.com) 或 Nan DUAN (nanduan@microsoft.com)。\n\n## 商标声明\n\n商标 本项目可能包含项目、产品或服务的商标或徽标。微软商标或徽标的授权使用须遵守并遵循 [Microsoft 的商标和品牌指南](https:\u002F\u002Fwww.microsoft.com\u002Fen-us\u002Flegal\u002Fintellectualproperty\u002Ftrademarks)。在本项目的修改版本中使用微软商标或徽标不得引起混淆或暗示微软赞助。任何第三方商标或徽标的使用均受该第三方政策的约束。\n\n## 免责声明\n本仓库中推荐的模型仅为示例，用于科学研究，探索任务自动化的概念，并使用发表于 [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.04671) 的论文进行基准测试。用户可以根据研究需求替换本仓库中的模型。使用本仓库中推荐的模型时，您需要分别遵守这些模型的许可协议。对于您使用此仓库导致的任何第三方权利侵权，微软不承担任何责任。用户同意就由此仓库引起的任何索赔，为微软辩护、赔偿并使微软免受所有损害、费用和律师费的侵害。如果有人认为此仓库侵犯了您的权利，请通知项目所有者 [email](chewu@microsoft.com)。","# TaskMatrix 快速上手指南\n\n## 简介\nTaskMatrix 将 ChatGPT 与一系列视觉基础模型连接起来，实现在聊天过程中**发送**和**接收**图像。它支持中文交互、模板定义以及 GroundingDINO 和 Segment Anything 等高级功能。\n\n## 环境准备\n*   **系统要求**: Python 3.8+，推荐使用 NVIDIA GPU (CUDA)。\n*   **前置依赖**: Git, Conda, 有效的 OpenAI API Key。\n*   **网络建议**: 国内用户建议使用清华或阿里镜像源加速 pip 安装。\n\n## 安装步骤\n1.  **克隆仓库**\n    ```bash\n    git clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FTaskMatrix.git\n    cd visual-chatgpt\n    ```\n\n2.  **创建并激活虚拟环境**\n    ```bash\n    conda create -n visgpt python=3.8\n    conda activate visgpt\n    ```\n\n3.  **安装依赖包**\n    ```bash\n    pip install -r requirements.txt\n    pip install  git+https:\u002F\u002Fgithub.com\u002FIDEA-Research\u002FGroundingDINO.git\n    pip install  git+https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything.git\n    ```\n    *(注：国内网络较慢时，可在 pip 命令后添加 `-i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple`)*\n\n4.  **配置 OpenAI API Key**\n    *   **Linux \u002F macOS**:\n        ```bash\n        export OPENAI_API_KEY={Your_Private_Openai_Key}\n        ```\n    *   **Windows**:\n        ```cmd\n        set OPENAI_API_KEY={Your_Private_Openai_Key}\n        ```\n\n## 基本使用\n运行 `visual_chatgpt.py` 脚本，通过 `--load` 参数指定要加载的视觉模型及其设备（CPU\u002FGPU）。\n\n**参数说明**：\n*   模型和设备用下划线 `_` 分隔（如 `ImageCaptioning_cpu`）。\n*   多个模型用逗号 `,` 分隔。\n*   可用模型及显存占用可参考 README 中的表格。\n\n**运行示例**：\n\n1.  **CPU 模式**（适合无显卡环境）：\n    ```bash\n    python visual_chatgpt.py --load ImageCaptioning_cpu,Text2Image_cpu\n    ```\n\n2.  **单卡 GPU 模式**（如 Google Colab T4 15GB）：\n    ```bash\n    python visual_chatgpt.py --load \"ImageCaptioning_cuda:0,Text2Image_cuda:0\"\n    ```\n\n3.  **多卡 GPU 模式**（如 4 张 V100 32GB）：\n    ```bash\n    python visual_chatgpt.py --load \"Text2Box_cuda:0,Segmenting_cuda:0,Inpainting_cuda:0,ImageCaptioning_cuda:0,Text2Image_cuda:1,Image2Canny_cpu,CannyText2Image_cuda:1,Image2Depth_cpu,DepthText2Image_cuda:1,VisualQuestionAnswering_cuda:2,InstructPix2Pix_cuda:2,Image2Scribble_cpu,ScribbleText2Image_cuda:2,SegText2Image_cuda:2,Image2Pose_cpu,PoseText2Image_cuda:2,Image2Hed_cpu,HedText2Image_cuda:3,Image2Normal_cpu,NormalText2Image_cuda:3,Image2Line_cpu,LineText2Image_cuda:3\"\n    ```\n\n启动后，即可在命令行界面与 ChatGPT 进行图文交互。如需使用 GroundingDINO 等功能，请确保加载了相应的模型（如 `Text2Box`, `Segmenting`）。","电商运营人员小张需要在一张杂乱的仓库实拍图中快速提取指定商品，并修改其背景颜色以适配新的促销海报。\n\n### 没有 TaskMatrix 时\n- 必须打开 Photoshop 等专业软件，手动用套索工具逐像素精细抠图，单个商品耗时极长。\n- 难以精准定位图中某类商品的具体坐标，往往需要放大缩小反复确认，容易遗漏细节。\n- 若需去除商品上的旧标签或瑕疵，需结合多种滤镜和修复工具，操作流程极其繁琐。\n- 每次更换海报尺寸都要重新设计布局，无法通过简单指令批量调整，重复劳动多。\n\n### 使用 TaskMatrix 后\n- 直接在对话框输入“找到图中的蓝色箱子”，TaskMatrix 利用 GroundingDINO 自动返回检测框。\n- 配合 Segment Anything 模型，一键生成精确分割掩码，无需手动描边即可获得干净主体。\n- 通过自然语言指令如“去掉箱子上的贴纸”，Stable Diffusion 即可基于掩码智能重绘修复。\n- 利用 InfinityOutPainting 模板，只需说“扩展图片至 2048x1024”，自动无缝延展画面适应新尺寸。\n\nTaskMatrix 通过串联视觉大模型与聊天机器人，让非技术人员也能通过对话完成复杂的图像处理工作。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fchenfei-wu_TaskMatrix_7fba23d8.png","chenfei-wu","Chenfei Wu","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002Fchenfei-wu_d805f75c.jpg","Qwen's Painter now.\r\nMy research interests lie in visual generation, with a focus on joint modeling of perception and generation.","Tongyi Lab, Alibaba","Beijing","cqwuchenfei@163.com","https:\u002F\u002Fchenfei-wu.github.io\u002F","https:\u002F\u002Fgithub.com\u002Fchenfei-wu",[85,89,93],{"name":86,"color":87,"percentage":88},"Python","#3572A5",80.6,{"name":90,"color":91,"percentage":92},"HTML","#e34c26",19,{"name":94,"color":95,"percentage":96},"Dockerfile","#384d54",0.3,34187,3241,"2026-04-05T10:04:55","NOASSERTION","Linux, Windows","需要 NVIDIA GPU，建议显存 15GB+ (参考 T4 15GB 建议)，支持多卡分配","未说明",{"notes":105,"python":106,"dependencies":107},"需配置 OpenAI API Key；支持通过 --load 参数灵活分配模型至 GPU\u002FCPU；提供模板功能简化复杂任务流程；部分轻量模型支持 CPU 运行。","3.8",[108,109,110,111,112,113,114,115],"GroundingDINO","segment-anything","LangChain","Stable Diffusion","ControlNet","InstructPix2Pix","CLIPSeg","BLIP",[54,15,14,26],81,"2026-03-27T02:49:30.150509","2026-04-06T06:56:49.115818",[121,126,131,135,139,144],{"id":122,"question_zh":123,"answer_zh":124,"source_url":125},1359,"运行时报错 `AttributeError: module 'openai' has no attribute 'error'` 如何解决？","这通常是由于 openai 库版本更新导致的 API 不兼容。根据社区测试，建议使用 openai==0.24.0 版本以保持兼容性。如果已升级，请尝试降级该包或检查代码是否适配了新版本的 API 结构。","https:\u002F\u002Fgithub.com\u002Fchenfei-wu\u002FTaskMatrix\u002Fissues\u002F121",{"id":127,"question_zh":128,"answer_zh":129,"source_url":130},1360,"M1 Mac 用户如何配置正确的依赖环境？","M1 Mac 需要特定的依赖版本组合。建议参考以下经过验证的 requirements.txt 配置：\naccelerate==0.17.0\naddict==2.4.0\nalbumentations==1.3.0\nbasicsr==1.4.2\ndiffusers==0.14.0\neinops==0.3.2\ngradio==3.20.1\nimageio==2.26.0\nimageio-ffmpeg==0.4.8\nkornia==0.6\nlangchain==0.0.101\nnumpy==1.23.1\nomegaconf==2.1.1\nopencv-contrib-python==4.4.0.46\nopen_clip_torch==2.0.2\npytorch-lightning==1.5.0\nprettytable==3.6.0\nsafetensors==0.2.7\nstreamlit==1.12.1\nstreamlit-drawable-canvas==0.8.0\ntest-tube>=0.7.5\ntimmm==0.6.12\ntorch==1.12.1\ntorchmetrics==0.6.0\ntorchvision==0.13.1\ntransformers==4.26.1\nwebdataset==0.2.5\nyapf==0.32.0","https:\u002F\u002Fgithub.com\u002Fchenfei-wu\u002FTaskMatrix\u002Fissues\u002F37",{"id":132,"question_zh":133,"answer_zh":134,"source_url":130},1361,"M1 Mac 上运行时报 `NotImplementedError: The operator 'aten::cumsum.out' is not currently implemented for the MPS device` 怎么办？","这是 PyTorch MPS 设备支持的问题。请在运行命令前设置环境变量以强制回退到 CPU 模式：\nexport PYTORCH_ENABLE_MPS_FALLBACK=1",{"id":136,"question_zh":137,"answer_zh":138,"source_url":130},1362,"安装依赖前是否需要先运行其他脚本？","是的，必须在应用补丁之前先运行 README 中提到的下载脚本。请执行：\nbash download.sh",{"id":140,"question_zh":141,"answer_zh":142,"source_url":143},1363,"为什么免费 OpenAI 账号在使用 API 时会报 RateLimitError？","免费账号通常没有可用的 API 额度，或者额度非常有限，一旦开始调用就会立即触发配额限制。这不是代码错误，而是账户权限问题。","https:\u002F\u002Fgithub.com\u002Fchenfei-wu\u002FTaskMatrix\u002Fissues\u002F53",{"id":145,"question_zh":146,"answer_zh":147,"source_url":143},1364,"遇到免费账号 API Key 报错（RateLimitError）有什么临时解决方法？","可以尝试以下步骤绕过部分限制：\n1. 在 .env 文件中注释掉你的 API Key。\n2. 启动应用，此时会报错提示未找到 Key。\n3. 停止服务器（Ctrl+C）。\n4. 取消注释 .env 中的 OpenAI Key。\n5. 重新启动服务器。如果计划有效，这应该能正常工作。",[]]