[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-UMass-Embodied-AGI--3D-LLM":3,"tool-UMass-Embodied-AGI--3D-LLM":65},[4,17,27,35,48,57],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":16},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",149489,2,"2026-04-10T11:32:46",[13,14,15],"开发框架","Agent","语言模型","ready",{"id":18,"name":19,"github_repo":20,"description_zh":21,"stars":22,"difficulty_score":23,"last_commit_at":24,"category_tags":25,"status":16},4487,"LLMs-from-scratch","rasbt\u002FLLMs-from-scratch","LLMs-from-scratch 是一个基于 PyTorch 的开源教育项目，旨在引导用户从零开始一步步构建一个类似 ChatGPT 的大型语言模型（LLM）。它不仅是同名技术著作的官方代码库，更提供了一套完整的实践方案，涵盖模型开发、预训练及微调的全过程。\n\n该项目主要解决了大模型领域“黑盒化”的学习痛点。许多开发者虽能调用现成模型，却难以深入理解其内部架构与训练机制。通过亲手编写每一行核心代码，用户能够透彻掌握 Transformer 架构、注意力机制等关键原理，从而真正理解大模型是如何“思考”的。此外，项目还包含了加载大型预训练权重进行微调的代码，帮助用户将理论知识延伸至实际应用。\n\nLLMs-from-scratch 特别适合希望深入底层原理的 AI 开发者、研究人员以及计算机专业的学生。对于不满足于仅使用 API，而是渴望探究模型构建细节的技术人员而言，这是极佳的学习资源。其独特的技术亮点在于“循序渐进”的教学设计：将复杂的系统工程拆解为清晰的步骤，配合详细的图表与示例，让构建一个虽小但功能完备的大模型变得触手可及。无论你是想夯实理论基础，还是为未来研发更大规模的模型做准备",90106,3,"2026-04-06T11:19:32",[15,26,14,13],"图像",{"id":28,"name":29,"github_repo":30,"description_zh":31,"stars":32,"difficulty_score":10,"last_commit_at":33,"category_tags":34,"status":16},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[13,15],{"id":36,"name":37,"github_repo":38,"description_zh":39,"stars":40,"difficulty_score":10,"last_commit_at":41,"category_tags":42,"status":16},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",85092,"2026-04-10T11:13:16",[26,43,44,45,14,46,15,13,47],"数据工具","视频","插件","其他","音频",{"id":49,"name":50,"github_repo":51,"description_zh":52,"stars":53,"difficulty_score":54,"last_commit_at":55,"category_tags":56,"status":16},5784,"funNLP","fighting41love\u002FfunNLP","funNLP 是一个专为中文自然语言处理（NLP）打造的超级资源库，被誉为\"NLP 民工的乐园”。它并非单一的软件工具，而是一个汇集了海量开源项目、数据集、预训练模型和实用代码的综合性平台。\n\n面对中文 NLP 领域资源分散、入门门槛高以及特定场景数据匮乏的痛点，funNLP 提供了“一站式”解决方案。这里不仅涵盖了分词、命名实体识别、情感分析、文本摘要等基础任务的标准工具，还独特地收录了丰富的垂直领域资源，如法律、医疗、金融行业的专用词库与数据集，甚至包含古诗词生成、歌词创作等趣味应用。其核心亮点在于极高的全面性与实用性，从基础的字典词典到前沿的 BERT、GPT-2 模型代码，再到高质量的标注数据和竞赛方案，应有尽有。\n\n无论是刚刚踏入 NLP 领域的学生、需要快速验证想法的算法工程师，还是从事人工智能研究的学者，都能在这里找到急需的“武器弹药”。对于开发者而言，它能大幅减少寻找数据和复现模型的时间；对于研究者，它提供了丰富的基准测试资源和前沿技术参考。funNLP 以开放共享的精神，极大地降低了中文自然语言处理的开发与研究成本，是中文 AI 社区不可或缺的宝藏仓库。",79857,1,"2026-04-08T20:11:31",[15,43,46],{"id":58,"name":59,"github_repo":60,"description_zh":61,"stars":62,"difficulty_score":54,"last_commit_at":63,"category_tags":64,"status":16},5773,"cs-video-courses","Developer-Y\u002Fcs-video-courses","cs-video-courses 是一个精心整理的计算机科学视频课程清单，旨在为自学者提供系统化的学习路径。它汇集了全球知名高校（如加州大学伯克利分校、新南威尔士大学等）的完整课程录像，涵盖从编程基础、数据结构与算法，到操作系统、分布式系统、数据库等核心领域，并深入延伸至人工智能、机器学习、量子计算及区块链等前沿方向。\n\n面对网络上零散且质量参差不齐的教学资源，cs-video-courses 解决了学习者难以找到成体系、高难度大学级别课程的痛点。该项目严格筛选内容，仅收录真正的大学层级课程，排除了碎片化的简短教程或商业广告，确保用户能接触到严谨的学术内容。\n\n这份清单特别适合希望夯实计算机基础的开发者、需要补充特定领域知识的研究人员，以及渴望像在校生一样系统学习计算机科学的自学者。其独特的技术亮点在于分类极其详尽，不仅包含传统的软件工程与网络安全，还细分了生成式 AI、大语言模型、计算生物学等新兴学科，并直接链接至官方视频播放列表，让用户能一站式获取高质量的教育资源，免费享受世界顶尖大学的课堂体验。",79792,"2026-04-08T22:03:59",[46,26,43,13],{"id":66,"github_repo":67,"name":68,"description_en":69,"description_zh":70,"ai_summary_zh":70,"readme_en":71,"readme_zh":72,"quickstart_zh":73,"use_case_zh":74,"hero_image_url":75,"owner_login":76,"owner_name":77,"owner_avatar_url":78,"owner_bio":79,"owner_company":80,"owner_location":80,"owner_email":80,"owner_twitter":80,"owner_website":81,"owner_url":82,"languages":83,"stars":100,"forks":101,"last_commit_at":102,"license":103,"difficulty_score":104,"env_os":105,"env_gpu":106,"env_ram":107,"env_deps":108,"category_tags":119,"github_topics":80,"view_count":10,"oss_zip_url":80,"oss_zip_packed_at":80,"status":16,"created_at":120,"updated_at":121,"faqs":122,"releases":158},6369,"UMass-Embodied-AGI\u002F3D-LLM","3D-LLM","Code for  3D-LLM: Injecting the 3D World into Large Language Models","3D-LLM 是首个能够直接理解三维空间信息的大型语言模型，旨在将真实的 3D 世界注入 AI 的认知体系。传统大模型通常仅处理文本或二维图像，难以应对复杂的立体空间数据，而 3D-LLM 突破了这一局限，既能识别单个 3D 物体（如 Objaverse 数据集），也能理解完整的室内场景（如 ScanNet 和 HM3D）。\n\n该工具主要解决了 AI 在三维几何感知与空间推理方面的短板，使其能够基于 3D 点云或多视图特征回答关于物体属性、空间关系及场景布局的复杂问题。其核心技术亮点在于创新性地融合了多视角图像编码器与大型语言模型，通过独特的投影机制将三维几何特征转化为语言模型可理解的序列，实现了从“看平面图”到“懂立体空间”的跨越。\n\n3D-LLM 非常适合人工智能研究人员、计算机视觉开发者以及从事具身智能、机器人导航和虚拟现实领域的专业人士使用。由于项目提供了详细的预训练权重、微调配置及推理代码，具备一定深度学习基础的用户可以快速复现论文成果，或在此基础上开发需要空间理解能力的下游应用，推动三维智能交互技术的发展。","\u003Cbr \u002F>\n\u003Cp align=\"center\">\n  \u003Ch1 align=\"center\">3D-LLM: Injecting the 3D World into Large Language Models (NeurIPS 2023 Spotlight) \u003C\u002Fh1>\n  \u003Cp align=\"center\">\n    \u003Ca href=\"https:\u002F\u002Fevelinehong.github.io\">Yining Hong\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fhaoyuzhen.com\">Haoyu Zhen\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fpeihaochen.github.io\">Peihao Chen\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fzsh2000.github.io\">Shuhong Zheng\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fyilundu.github.io\">Yilun Du\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fzfchenunique.github.io\">Zhenfang Chen\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fpeople.csail.mit.edu\u002Fganchuang\">Chuang Gan\u003C\u002Fa>\n  \u003C\u002Fp>\n  \u003Cp align=\"center\">\n    \u003Ca href='https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.12981'>\n      \u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-PDF-red?style=flat&logo=arXiv&logoColor=red' alt='Paper PDF'>\n    \u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Fvis-www.cs.umass.edu\u002F3dllm\u002F' style='padding-left: 0.5rem;'>\n      \u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-blue?style=flat&logo=Google%20chrome&logoColor=blue' alt='Project Page'>\n    \u003C\u002Fa>\n  \u003C\u002Fp>\n  \u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FUMass-Embodied-AGI_3D-LLM_readme_08cffee1809d.png\" alt=\"Logo\" width=\"80%\">\n  \u003C\u002Fp>\n\u003C\u002Fp>\n\n3D-LLM is the first Large Language Model that could take 3D representations as inputs. It is able to handle both object (e.g., objaverse) and scene data (e.g., scannet & hm3d). \n\n## Installation\nInstall [salesforce-lavis](https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS)\n\n```shell\n$ conda create -n lavis python=3.8\n$ conda activate lavis\n\n$ git clone https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS.git SalesForce-LAVIS\n$ cd SalesForce-LAVIS\n$ pip install -e .\n\n$ pip install positional_encodings\n```\n\n## Checkpoints\n### Pretraining Checkpoints\n[Pretrained checkpoints](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr\u002Fview?usp=drive_link) are released (Please use v2!)\n\n### Finetuning Checkpoints\n[Finetuning checkpoints](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link) for [ScanQA](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1sPynAO8pI_RPR4pwWTrx8weDTdMPsqtW\u002Fview?usp=drive_link), [SQA3d](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1Ka9TWv6cs6h-pPaaQG1auIiQma2xbNFk\u002Fview?usp=drive_link), and [3DMV_VQA](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1_h2wPPGO64HY5LUcA1bD8DlZx3WCsY8b\u002Fview?usp=drive_link) are released.\nThe results are better than preprint-version paper. We will update the camera-ready paper to the arxiv soon.\n\n## Quick Start: Inference\nDownload the objaverse subset features [here](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1mJZONfWREfIUAPYXP65D65uS2EoplAfR\u002Fview?usp=drive_link). Download the [pretrained checkpoints](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1urI2I3S8SgLD8L9brl4ae1Mul_yhCxJe?usp=drive_link). For more details, please refer to `3DLLM_BLIP2-base\u002FDEMO.md`.\n```\n$ cd 3DLLM_BLIP2-base\n$ conda activate lavis\n\npython inference.py # for objects\npython inference.py --mode room # for scenes\n```\nTODO: huggingface auto load checkpoint.\n\n## Finetuning\nFinetuning config yaml files that need to be changed are in [this directory](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Ftree\u002Fmain\u002F3DLLM_BLIP2-base\u002Flavis\u002Fprojects\u002Fblip2\u002Ftrain)\n1. Download the [pretrained checkpoints](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr\u002Fview?usp=drive_link). Modify the \"resume_checkpoint_path\" path in the yaml files\n2. Download the [questions](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F14MDiDl6Cch_B27Q0aZgdElhAEOBBpn2o?usp=drive_link), modify the \"annotations\" path in the yaml files\n3. Download the [scannet features](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1H1SKZsK_XpMkXntwhY4BqSxsGCRP8cgp?usp=drive_link)  or [3dmv-vqa features](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1NdFKKn_IZxGezi6fXA60rF1uxTOmhOet?usp=drive_link). Modify the path (both train and val) in lavis\u002Fdatasets\u002Fdatasets\u002Fthreedvqa_datasets.py\n4.\n```\n$ cd 3DLLM_BLIP2-base\n\n$ conda activate lavis\n\npython -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis\u002Fprojects\u002Fblip2\u002Ftrain\u002F\u003Cfinetune_yaml_file>\n```\nYou can also load the finetuning checkpoints [here](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link).\n\n5.**Calculating scores**\n```\ncd calculate_scores\npython calculate_score_\u003Ctask>.py --folder \u003Cyour result dir> --epoch \u003Cyour epoch>\n```\nplease also modify the feature and question path in the scripts\n\nTODO: huggingface auto load checkpoint.\n\n\n## Data\nAll data will be gradually released in [Google Drive](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F188Yd7tmiUfyct-dVMpkQ8q_tnqkb-4bo?usp=sharing) and [Huggingface](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FShuhongZheng\u002F3D-LLM) (All files are released in Google Drive first and then Huggingface. Please refer to the Google Drive for file structure)\n\n### Pretraining Data\nWe are still cleaning the grounding & navigation part. All other pre-training data are released.\n#### Object Data\n   Language annotations of object data released [here](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F17K1ZGb1HpvPj36a84GefGQK7DOkRp_32\u002Fview?usp=sharing).\n\n   For downloading Objaverse data, please refer to [Objaverse website](https:\u002F\u002Fobjaverse.allenai.org\u002F).\n  \n   To get 3D features and point clouds of the Objaverse data, please refer to [Step1](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Fblob\u002Fmain\u002FREADME.md#step1-render-images-from-different-views-of-a-scene) and [Step3](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Fblob\u002Fmain\u002FREADME.md#step3-3d-feature-construction-from-rendered-images) of 3DLanguage Data generation - ChatCaptioner based\n\n   A small set of objaverse features is released [here](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1mJZONfWREfIUAPYXP65D65uS2EoplAfR\u002Fview?usp=drive_link).\n  \n   TODO: We will probably release the whole set of Objaverse 3D features \n#### Scene Data\n   [Language data released here](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F18AVyAWwMwqP8nyZQfTpEMw1Wfhghjw0b\u002Fview?usp=drive_link). \n \n   3D features and point clouds (~250G) are released [here](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1bJpK00UFRZGH7tdx3PaqDrrTfHNP6B2K?usp=drive_link). However, if you want to explore generating the features yourself, please refer to the Three-step 3D Feature Extraction part [here](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM#three-step-3d-feature-extraction). Please use v2 to be consistent with the checkpoints (and also result in better performances).\n\n  chat: 73103. task: 84531\n\n### Finetuning Data\n3D features and point clouds of Scannet (used for finetuning ScanQA and SQA3D) are released in [here](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1CsEt48jj5uCyelGcXXJBkGH86QYeCE8D?usp=drive_link). 3D features and point clouds of 3DMV-VQA are released [here](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1NdFKKn_IZxGezi6fXA60rF1uxTOmhOet?usp=drive_link) (3DMV-VQA data will be further updated for a clearer structure).\n\nAll questions can be found [here](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F14MDiDl6Cch_B27Q0aZgdElhAEOBBpn2o?usp=drive_link).\n\n\n\n## 3DLanguage Data Generation\n\n### ChatCaptioner based \u002F Three-step 3D Feature Extraction (Objaverse)\n#### Step1: render images from different views of a scene\nFollow the instruction in ``3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\u002FREADME.md`` for installation. \n\nThe following code will render images of a objaverse scene (e.g. f6e9ec5953854dff94176c36b877c519). The rendered images will be saved at ``3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\u002Foutput``.\n(Please refer to ``3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\u002FREADME.md`` for more details about the command)\n\n```shell\n$ cd .\u002F3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\n\n$ {path\u002Fto\u002Fblender} -b -P render.py -noaudio --disable-crash-handler -- --uid f6e9ec5953854dff94176c36b877c519\n\n```\n\n#### Step2: generate caption for this objaverse scene\nInstallation: \n\nPlease follow [ChatCaptioner](https:\u002F\u002Fgithub.com\u002FVision-CAIR\u002FChatCaptioner) to install the environment\u002F\n\n\nThe following code will read the rended images of an objaverse scene (e.g., f6e9ec5953854dff94176c36b877c519) and generate scene caption at ``3DLanguage_data\u002FChatCaptioner_based\u002Foutput``\n\n```shell\n$ cd .\u002F3DLanguage_data\u002FChatCaptioner_based\n\n$ python chatcaption.py --specific_scene f6e9ec5953854dff94176c36b877c519\n```\n\n#### Step3: 3D feature construction from rendered images\n\nFollow the instruction in ``3DLanguage_data\u002FChatCaptioner_based\u002Fgen_features\u002FREADME.md`` for extracting 3D features from rendered images.\n```shell\n$ cd .\u002F3DLanguage_data\u002FChatCaptioner_based\u002Fgen_features\n```\n\n\n### Box-Demonstration-Instruction based\nTODO\n\n### Revision based\nTODO\n\n## Three-step 3D Feature Extraction (Scene)\nThis section is for constructing 3D features for scene data. If you already downloaded our released scene data, please skip this section.\n### First step\nInstallation: \n\nPlease follow [Mask2Former](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002FMask2Former) to install the environment and download the [pretrained weight](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fmaskformer\u002Fmask2former\u002Fcoco\u002Finstance\u002Fmaskformer2_swin_large_IN21k_384_bs16_100ep\u002Fmodel_final_e5f453.pkl) to the current directory\nif extracting the masks with [Mask2Former](https:\u002F\u002Fopenaccess.thecvf.com\u002Fcontent\u002FCVPR2022\u002Fpapers\u002FCheng_Masked-Attention_Mask_Transformer_for_Universal_Image_Segmentation_CVPR_2022_paper.pdf).\n\nPlease follow [Segment Anything](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything) to install the environment and download the [pretrained weight](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fsegment_anything\u002Fsam_vit_h_4b8939.pth) to the current directory if extracting the masks with [SAM](https:\u002F\u002Farxiv.org\u002Fabs\u002F2304.02643).\n\nExtract masks with Mask2Former:\n\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Ffirst_step\n\n$ python maskformer_mask.py --scene_dir_path DATA_DIR_WITH_RGB_IMAGES --save_dir_path DIR_YOU_WANT_TO_SAVE_THE_MASKS\n```\n\nExtract masks with Segment Anything:\n\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Ffirst_step\n\n$ python sam_mask.py --scene_dir_path DATA_DIR_WITH_RGB_IMAGES --save_dir_path DIR_YOU_WANT_TO_SAVE_THE_MASKS\n```\n\nAfter the first step, we are expected to obtain a directory of masks (specified by ``--save_dir_path``) that contains extracted masks for\nmulti-view images of the scenes.\n\n### Second step\nNote: BLIP features are for LAVIS(BLIP2), CLIP features are for open-flamingo.\n\nInstallation: The same as the following ``3D-LLM_BLIP2-based`` section to install [salesforce-lavis](https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS).\n\nThere are four options: (1) Extract CLIP feature with Mask2Former masks; (2) Extract CLIP feature with SAM masks;\n(3) Extract BLIP feature with Mask2Former masks; (4) Extract BLIP feature with SAM masks.\n\nExtract 2D CLIP features with Mask2Former masks:\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Fsecond_step\u002F\n\n$ python clip_maskformer.py --scene_dir_path DATA_DIR_WITH_RGB_IMAGES --mask_dir_path MASK_DIR_FROM_1ST_STEP --save_dir_path DIR_YOU_WANT_TO_SAVE_THE_FEAT\n```\n\nFor the other options, the scripts are in similar format.\n\nAfter the second step, we are expected to obtain a directory of features (specified by ``--save_dir_path``) that contains 2D features for\nmulti-view images of the scenes.\n\n### Third step\n#### Direct Reconstruction\nInstallation:\n\nPlease install the [Habitat environment](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fhabitat-lab\u002Ftree\u002Fchallenge-2022).\n\nReconstruct 3D feature from multi-view 2D features:\n\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Fthird_step\u002F\n\n$ python sam_mask.py --data_dir_path DATA_DIR_WITH_RGB_IMAGES --depth_dir_path DATA_DIR_WITH_DEPTH_IMAGES --feat_dir_path FEATURE_DIR_FROM_2ND_STEP\n```\n\nAfter the third step, we are expected to obtain two files (``pcd_pos.pt`` and ``pcd_feat.pt``) for each room inside the corresponding RGB directory.\n``pcd_pos.pt`` contains the point positions of the 3D point cloud (shape: ``N * 3``). ``pcd_feat.pt`` contains the point features of the 3D point cloud (shape: ``N * n_dim``).\n``N`` is the number of sampled points in the point cloud (default: 300000) and ``n_dim`` is the feature dimension (1024 for CLIP feature, 1408 for BLIP feature).\n\n#### GradSLAM (Feature Fusion)\nRefer to [Concept Fusion](https:\u002F\u002Fgithub.com\u002Fconcept-fusion\u002Fconcept-fusion).\n\nWe will also release our reproduced version of Concept Fusion for our feature generation (we reproduced the paper before their official release).\n\n#### Neural Field\nPlease refer to [3D-CLR](https:\u002F\u002Fgithub.com\u002Fevelinehong\u002F3D-CLR-Official) repository.\n\n## Pre-training\n\n```shell\n$ cd 3DLLM_BLIP2-base\n\n$ conda activate lavis\n# use facebook\u002Fopt-2.7b:\n$ TODO\n# use flant5\n$ python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis\u002Fprojects\u002Fblip2\u002Ftrain\u002Fpretrain.yaml\n```\n## 3D-LLM_flamingo-based\nTODO.\n\n## Citation\n\nIf you find our work useful, please consider citing:\n\n```\n@article{3dllm,\n author = {Hong, Yining and Zhen, Haoyu and Chen, Peihao and Zheng, Shuhong and Du, Yilun and Chen, Zhenfang and Gan, Chuang},\n title = {3D-LLM: Injecting the 3D World into Large Language Models},\n journal = {NeurIPS},\n year = {2023},\n} \n```\n\n### Acknowledgements\n\nhttps:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS\n\nhttps:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002FMask2Former\n\nhttps:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything\n\nhttps:\u002F\u002Fgithub.com\u002Fmlfoundations\u002Fopen_flamingo\n\nhttps:\u002F\u002Fgithub.com\u002Fconcept-fusion\u002Fconcept-fusion\n\nhttps:\u002F\u002Fgithub.com\u002Fevelinehong\u002F3D-CLR-Official\n","\u003Cbr \u002F>\n\u003Cp align=\"center\">\n  \u003Ch1 align=\"center\">3D-LLM：将3D世界注入大型语言模型（NeurIPS 2023 Spotlight）\u003C\u002Fh1>\n  \u003Cp align=\"center\">\n    \u003Ca href=\"https:\u002F\u002Fevelinehong.github.io\">洪怡宁\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fhaoyuzhen.com\">甄浩宇\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fpeihaochen.github.io\">陈培豪\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fzsh2000.github.io\">郑书鸿\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fyilundu.github.io\">杜一伦\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fzfchenunique.github.io\">陈振芳\u003C\u002Fa>,\n    \u003Ca href=\"https:\u002F\u002Fpeople.csail.mit.edu\u002Fganchuang\">甘闯\u003C\u002Fa>\n  \u003C\u002Fp>\n  \u003Cp align=\"center\">\n    \u003Ca href='https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.12981'>\n      \u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FPaper-PDF-red?style=flat&logo=arXiv&logoColor=red' alt='Paper PDF'>\n    \u003C\u002Fa>\n    \u003Ca href='https:\u002F\u002Fvis-www.cs.umass.edu\u002F3dllm\u002F' style='padding-left: 0.5rem;'>\n      \u003Cimg src='https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FProject-Page-blue?style=flat&logo=Google%20chrome&logoColor=blue' alt='Project Page'>\n    \u003C\u002Fa>\n  \u003C\u002Fp>\n  \u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FUMass-Embodied-AGI_3D-LLM_readme_08cffee1809d.png\" alt=\"Logo\" width=\"80%\">\n  \u003C\u002Fp>\n\u003C\u002Fp>\n\n3D-LLM是首个能够以3D表示作为输入的大型语言模型。它既可以处理对象数据（如objaverse），也可以处理场景数据（如scannet和hm3d）。\n\n## 安装\n安装 [salesforce-lavis](https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS)\n\n```shell\n$ conda create -n lavis python=3.8\n$ conda activate lavis\n\n$ git clone https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS.git SalesForce-LAVIS\n$ cd SalesForce-LAVIS\n$ pip install -e .\n\n$ pip install positional_encodings\n```\n\n## 检查点\n### 预训练检查点\n[预训练检查点](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr\u002Fview?usp=drive_link)已发布（请使用v2版本！）\n\n### 微调检查点\n针对[ScanQA](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1sPynAO8pI_RPR4pwWTrx8weDTdMPsqtW\u002Fview?usp=drive_link)、[SQA3d](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1Ka9TWv6cs6h-pPaaQG1auIiQma2xbNFk\u002Fview?usp=drive_link)和[3DMV_VQA](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1_h2wPPGO64HY5LUcA1bD8DlZx3WCsY8b\u002Fview?usp=drive_link)的[微调检查点](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link)已发布。\n这些结果比预印本论文中的效果更好。我们很快会将最终版论文更新到arxiv上。\n\n## 快速入门：推理\n从[这里](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1mJZONfWREfIUAPYXP65D65uS2EoplAfR\u002Fview?usp=drive_link)下载objaverse子集特征。同时下载[预训练检查点](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1urI2I3S8SgLD8L9brl4ae1Mul_yhCxJe?usp=drive_link)。更多详细信息，请参阅`3DLLM_BLIP2-base\u002FDEMO.md`。\n```\n$ cd 3DLLM_BLIP2-base\n$ conda activate lavis\n\npython inference.py # 用于对象\npython inference.py --mode room # 用于场景\n```\n待办事项：实现Hugging Face自动加载检查点。\n\n## 微调\n需要修改的微调配置yaml文件位于[此目录](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Ftree\u002Fmain\u002F3DLLM_BLIP2-base\u002Flavis\u002Fprojects\u002Fblip2\u002Ftrain)。\n1. 下载[预训练检查点](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr\u002Fview?usp=drive_link)。在yaml文件中修改“resume_checkpoint_path”路径。\n2. 下载[问题](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F14MDiDl6Cch_B27Q0aZgdElhAEOBBpn2o?usp=drive_link)，并在yaml文件中修改“annotations”路径。\n3. 下载[scannet特征](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1H1SKZsK_XpMkXntwhY4BqSxsGCRP8cgp?usp=drive_link)或[3dmv-vqa特征](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1NdFKKn_IZxGezi6fXA60rF1uxTOmhOet?usp=drive_link)。在lavis\u002Fdatasets\u002Fdatasets\u002Fthreedvqa_datasets.py中修改训练和验证数据的路径。\n4.\n```\n$ cd 3DLLM_BLIP2-base\n\n$ conda activate lavis\n\npython -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis\u002Fprojects\u002Fblip2\u002Ftrain\u002F\u003Cfinetune_yaml_file>\n```\n您也可以从[这里](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link)加载微调检查点。\n\n5.**计算分数**\n```\ncd calculate_scores\npython calculate_score_\u003Ctask>.py --folder \u003Cyour result dir> --epoch \u003Cyour epoch>\n```\n请同时在脚本中修改特征和问题路径。\n\n待办事项：实现Hugging Face自动加载检查点。\n\n\n## 数据\n所有数据将逐步在[Google Drive](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F188Yd7tmiUfyct-dVMpkQ8q_tnqkb-4bo?usp=sharing)和[Hugging Face](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FShuhongZheng\u002F3D-LLM)上发布（所有文件先在Google Drive上发布，随后再上传至Hugging Face。请参考Google Drive了解文件结构）。\n\n### 预训练数据\n我们仍在整理接地与导航部分的数据。其他所有预训练数据均已发布。\n#### 对象数据\n   对象数据的语言标注已[此处](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F17K1ZGb1HpvPj36a84GefGQK7DOkRp_32\u002Fview?usp=sharing)发布。\n\n   如需下载Objaverse数据，请访问[Objaverse官网](https:\u002F\u002Fobjaverse.allenai.org\u002F)。\n\n   要获取Objaverse数据的3D特征和点云，请参考基于ChatCaptioner的3D语言数据生成中的[步骤1](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Fblob\u002Fmain\u002FREADME.md#step1-render-images-from-different-views-of-a-scene)和[步骤3](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Fblob\u002Fmain\u002FREADME.md#step3-3d-feature-construction-from-rendered-images)。\n\n   一小部分objaverse特征已[此处](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1mJZONfWREfIUAPYXP65D65uS2EoplAfR\u002Fview?usp=drive_link)发布。\n  \n   待办事项：我们可能会发布完整的Objaverse 3D特征集。\n#### 场景数据\n   [语言数据在此处发布](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F18AVyAWwMwqP8nyZQfTpEMw1Wfhghjw0b\u002Fview?usp=drive_link)。 \n \n   3D特征和点云（约250G）已[此处](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1bJpK00UFRZGH7tdx3PaqDrrTfHNP6B2K?usp=drive_link)发布。然而，如果您想自行探索生成这些特征的方法，请参考[三步3D特征提取](https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM#three-step-3d-feature-extraction)部分。请使用v2版本，以确保与检查点一致，并获得更好的性能。\n\n  聊天：73103条，任务：84531个\n\n### 微调数据\n用于微调ScanQA和SQA3D的Scannet的3D特征和点云已在[此处](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1CsEt48jj5uCyelGcXXJBkGH86QYeCE8D?usp=drive_link)发布。3DMV-VQA的3D特征和点云则在[此处](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1NdFKKn_IZxGezi6fXA60rF1uxTOmhOet?usp=drive_link)发布（3DMV-VQA数据将进一步更新，以使结构更加清晰）。\n\n所有问题均可在[此处](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F14MDiDl6Cch_B27Q0aZgdElhAEOBBpn2o?usp=drive_link)找到。\n\n\n\n## 3D语言数据生成\n\n### 基于 ChatCaptioner 的三步 3D 特征提取（Objaverse）\n#### 第一步：从场景的不同视角渲染图像\n请按照 ``3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\u002FREADME.md`` 中的说明进行安装。\n\n以下代码将渲染一个 Objaverse 场景的图像（例如 f6e9ec5953854dff94176c36b877c519）。渲染后的图像将保存在 ``3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\u002Foutput`` 目录下。\n（有关该命令的更多详细信息，请参阅 ``3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\u002FREADME.md``）\n\n```shell\n$ cd .\u002F3DLanguage_data\u002FChatCaptioner_based\u002Fobjaverse_render\n\n$ {blender路径} -b -P render.py -noaudio --disable-crash-handler -- --uid f6e9ec5953854dff94176c36b877c519\n\n```\n\n#### 第二步：为该 Objaverse 场景生成描述\n安装：\n\n请按照 [ChatCaptioner](https:\u002F\u002Fgithub.com\u002FVision-CAIR\u002FChatCaptioner) 的说明安装环境。\n\n\n以下代码将读取一个 Objaverse 场景的渲染图像（例如 f6e9ec5953854dff94176c36b877c519），并在 ``3DLanguage_data\u002FChatCaptioner_based\u002Foutput`` 目录下生成场景描述。\n\n```shell\n$ cd .\u002F3DLanguage_data\u002FChatCaptioner_based\n\n$ python chatcaption.py --specific_scene f6e9ec5953854dff94176c36b877c519\n```\n\n#### 第三步：从渲染图像中构建 3D 特征\n请按照 ``3DLanguage_data\u002FChatCaptioner_based\u002Fgen_features\u002FREADME.md`` 中的说明，从渲染图像中提取 3D 特征。\n```shell\n$ cd .\u002F3DLanguage_data\u002FChatCaptioner_based\u002Fgen_features\n```\n\n\n### 基于 Box-Demonstration-Instruction 的方法\n待完成\n\n### 基于 Revision 的方法\n待完成\n\n## 三步 3D 特征提取（场景）\n本节用于为场景数据构建 3D 特征。如果您已经下载了我们发布的场景数据，请跳过本节。\n### 第一步\n安装：\n\n如果使用 [Mask2Former](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002FMask2Former) 提取掩码，请按照其说明安装环境，并将 [预训练权重](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fmaskformer\u002Fmask2former\u002Fcoco\u002Finstance\u002Fmaskformer2_swin_large_IN21k_384_bs16_100ep\u002Fmodel_final_e5f453.pkl) 下载到当前目录。\n\n如果使用 [SAM](https:\u002F\u002Farxiv.org\u002Fabs\u002F2304.02643) 提取掩码，请按照 [Segment Anything](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything) 的说明安装环境，并将 [预训练权重](https:\u002F\u002Fdl.fbaipublicfiles.com\u002Fsegment_anything\u002Fsam_vit_h_4b8939.pth) 下载到当前目录。\n\n使用 Mask2Former 提取掩码：\n\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Ffirst_step\n\n$ python maskformer_mask.py --scene_dir_path 包含RGB图像的数据目录 --save_dir_path 您希望保存掩码的目录\n```\n\n使用 Segment Anything 提取掩码：\n\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Ffirst_step\n\n$ python sam_mask.py --scene_dir_path 包含RGB图像的数据目录 --save_dir_path 您希望保存掩码的目录\n```\n\n完成第一步后，您应该会得到一个包含场景多视角图像掩码的目录（由 ``--save_dir_path`` 指定）。\n\n### 第二步\n注意：BLIP 特征适用于 LAVIS（BLIP2），CLIP 特征适用于 open-flamingo。\n\n安装：与下面的 ``3D-LLM_BLIP2-based`` 部分相同，安装 [salesforce-lavis](https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS)。\n\n有四种选择：(1) 使用 Mask2Former 掩码提取 CLIP 特征；(2) 使用 SAM 掩码提取 CLIP 特征；(3) 使用 Mask2Former 掩码提取 BLIP 特征；(4) 使用 SAM 掩码提取 BLIP 特征。\n\n使用 Mask2Former 掩码提取 2D CLIP 特征：\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Fsecond_step\u002F\n\n$ python clip_maskformer.py --scene_dir_path 包含RGB图像的数据目录 --mask_dir_path 第一步生成的掩码目录 --save_dir_path 您希望保存特征的目录\n```\n\n其他选项的脚本格式类似。\n\n完成第二步后，您应该会得到一个包含场景多视角图像 2D 特征的目录（由 ``--save_dir_path`` 指定）。\n\n### 第三步\n#### 直接重建\n安装：\n\n请安装 [Habitat 环境](https:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fhabitat-lab\u002Ftree\u002Fchallenge-2022)。\n\n从多视角 2D 特征重建 3D 特征：\n\n```shell\n$ cd .\u002Fthree_steps_3d_feature\u002Fthird_step\u002F\n\n$ python sam_mask.py --data_dir_path 包含RGB图像的数据目录 --depth_dir_path 包含深度图像的数据目录 --feat_dir_path 第二步生成的特征目录\n```\n\n完成第三步后，您应该会为每个房间在对应的 RGB 数据目录中获得两个文件（``pcd_pos.pt`` 和 ``pcd_feat.pt``）。``pcd_pos.pt`` 包含 3D 点云的点位置（形状：``N * 3``）。``pcd_feat.pt`` 包含 3D 点云的点特征（形状：``N * n_dim``）。``N`` 是点云中采样的点数（默认为 300,000），``n_dim`` 是特征维度（CLIP 特征为 1024，BLIP 特征为 1408）。\n\n#### GradSLAM（特征融合）\n请参考 [Concept Fusion](https:\u002F\u002Fgithub.com\u002Fconcept-fusion\u002Fconcept-fusion)。\n\n我们还将发布我们复现的 Concept Fusion 版本，用于我们的特征生成（我们在他们正式发布之前就已复现了该论文）。\n\n#### 神经场\n请参考 [3D-CLR](https:\u002F\u002Fgithub.com\u002Fevelinehong\u002F3D-CLR-Official) 仓库。\n\n## 预训练\n\n```shell\n$ cd 3DLLM_BLIP2-base\n\n$ conda activate lavis\n# 使用 facebook\u002Fopt-2.7b:\n$ 待完成\n# 使用 flant5\n$ python -m torch.distributed.run --nproc_per_node=8 train.py --cfg-path lavis\u002Fprojects\u002Fblip2\u002Ftrain\u002Fpretrain.yaml\n```\n## 3D-LLM_flamingo-based\n待完成。\n\n## 引用\n如果您觉得我们的工作有用，请考虑引用：\n\n```\n@article{3dllm,\n author = {Hong, Yining and Zhen, Haoyu and Chen, Peihao and Zheng, Shuhong and Du, Yilun and Chen, Zhenfang and Gan, Chuang},\n title = {3D-LLM: 将 3D 世界注入大型语言模型},\n journal = {NeurIPS},\n year = {2023},\n} \n```\n\n### 致谢\n\nhttps:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS\n\nhttps:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002FMask2Former\n\nhttps:\u002F\u002Fgithub.com\u002Ffacebookresearch\u002Fsegment-anything\n\nhttps:\u002F\u002Fgithub.com\u002Fmlfoundations\u002Fopen_flamingo\n\nhttps:\u002F\u002Fgithub.com\u002Fconcept-fusion\u002Fconcept-fusion\n\nhttps:\u002F\u002Fgithub.com\u002Fevelinehong\u002F3D-CLR-Official","# 3D-LLM 快速上手指南\n\n3D-LLM 是首个能够直接接受 3D 表示作为输入的大型语言模型，支持处理物体（如 Objaverse）和场景（如 ScanNet, HM3D）数据。\n\n## 1. 环境准备\n\n*   **操作系统**: Linux (推荐 Ubuntu)\n*   **Python 版本**: 3.8\n*   **硬件要求**: 需要 NVIDIA GPU 以支持模型推理和训练\n*   **前置依赖**:\n    *   Conda (用于环境管理)\n    *   Git\n    *   CUDA Toolkit (需与 PyTorch 版本匹配)\n\n> **注意**：本项目基于 `salesforce-lavis` 构建，请确保网络通畅以下载相关依赖。国内用户若遇到下载缓慢，可配置 pip 使用清华或阿里镜像源。\n\n## 2. 安装步骤\n\n### 2.1 创建并激活 Conda 环境\n```shell\nconda create -n lavis python=3.8\nconda activate lavis\n```\n\n### 2.2 安装 LAVIS 框架\n克隆 Salesforce LAVIS 仓库并进行可编辑安装：\n```shell\ngit clone https:\u002F\u002Fgithub.com\u002Fsalesforce\u002FLAVIS.git SalesForce-LAVIS\ncd SalesForce-LAVIS\npip install -e .\n```\n\n### 2.3 安装额外依赖\n安装位置编码库：\n```shell\npip install positional_encodings\n```\n\n### 2.4 获取预训练权重\n下载预训练检查点（请务必使用 **v2** 版本以获得最佳效果）：\n*   **预训练权重**: [Google Drive 下载链接](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1tiis8mOdZGBzmR7vgZtRE4Ni_2FE4nTr\u002Fview?usp=drive_link)\n*   **微调权重** (ScanQA, SQA3d, 3DMV_VQA): [Google Drive 下载链接](https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1RKP1cz6R6H8YziEc4f3MHW9dCCXQChbA?usp=drive_link)\n\n下载完成后，请将权重文件放置在项目相应目录下（具体路径参考后续推理脚本或配置文件）。\n\n### 2.5 准备测试数据 (可选)\n如需快速测试物体推理，请下载 Objaverse 子集特征：\n*   **Objaverse 特征子集**: [Google Drive 下载链接](https:\u002F\u002Fdrive.google.com\u002Ffile\u002Fd\u002F1mJZONfWREfIUAPYXP65D65uS2EoplAfR\u002Fview?usp=drive_link)\n\n## 3. 基本使用 (推理)\n\n以下演示如何进行最简单的推理操作。请确保已激活 `lavis` 环境并进入项目基础目录。\n\n### 3.1 启动推理\n进入主目录并激活环境：\n```shell\ncd 3DLLM_BLIP2-base\nconda activate lavis\n```\n\n#### 场景 A：物体推理 (Object Inference)\n针对单个 3D 物体进行问答或描述生成：\n```shell\npython inference.py\n```\n\n#### 场景 B：场景推理 (Scene Inference)\n针对 3D 室内场景（如房间）进行推理：\n```shell\npython inference.py --mode room\n```\n\n> **提示**: 更多详细的参数配置和 Demo 说明，请参考项目内的 `3DLLM_BLIP2-base\u002FDEMO.md` 文件。\n\n---\n*注：如需进行微调（Finetuning），需修改对应的 yaml 配置文件中的检查点路径、标注文件路径及数据集特征路径，并使用多卡分布式训练命令启动。详细步骤请参阅原始文档的 \"Finetuning\" 章节。*","某智能家居公司正在开发一款能理解家庭空间布局的语音管家，需要让 AI 不仅听懂指令，还能“看懂”房间的三维结构。\n\n### 没有 3D-LLM 时\n- **空间感知缺失**：传统大模型只能处理文本或 2D 图片，无法直接解析 ScanNet 等场景的 3D 点云数据，导致 AI 不知道沙发具体在电视的左侧还是后方。\n- **交互逻辑僵硬**：用户询问“把灯调到沙发旁边那盏”，系统因缺乏三维位置关系推理能力，只能机械地报错或随机操作。\n- **开发成本高昂**：工程师需手动编写大量规则代码来映射 3D 坐标与语义标签，且难以覆盖复杂的物体遮挡和多变的空间布局。\n- **数据利用率低**：海量的 Objaverse 3D 物体库和真实场景扫描数据无法被大模型直接学习，只能作为静态素材闲置。\n\n### 使用 3D-LLM 后\n- **原生 3D 理解**：3D-LLM 直接将 3D 表征注入大语言模型，AI 能像人类一样“脑补”出房间立体结构，精准定位物体间的相对位置。\n- **自然语义交互**：面对“去厨房拿冰箱顶层的牛奶”这类复杂指令，3D-LLM 能结合三维空间信息进行多步推理，准确执行任务。\n- **端到端智能训练**：利用 3D-LLM 的微调能力，团队可直接用 3D 问答数据集（如 SQA3d）训练模型，大幅减少人工规则编写，提升泛化性。\n- **全域数据激活**：无论是单一物体还是完整室内场景，3D-LLM 均能统一处理，让原本沉睡的 3D 资产转化为模型的空間常识。\n\n3D-LLM 通过打破二维文本与三维世界的壁垒，让 AI 真正具备了在物理空间中思考与交互的核心能力。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FUMass-Embodied-AGI_3D-LLM_0c7f109a.png","UMass-Embodied-AGI","UMass Embodied AGI Group","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002FUMass-Embodied-AGI_bd6a3d2e.png","Embodied AGI Group at University of Massachusetts Amherst",null,"https:\u002F\u002Fembodied-agi.cs.umass.edu\u002F","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI",[84,88,92,96],{"name":85,"color":86,"percentage":87},"Python","#3572A5",96.9,{"name":89,"color":90,"percentage":91},"Cuda","#3A4E3A",2.7,{"name":93,"color":94,"percentage":95},"C++","#f34b7d",0.3,{"name":97,"color":98,"percentage":99},"Shell","#89e051",0,1191,74,"2026-04-10T08:22:20","MIT",4,"Linux","需要 NVIDIA GPU（训练命令使用 --nproc_per_node=8，暗示多卡环境），需支持 CUDA（具体版本未说明，依赖 PyTorch 环境），显存需求未说明（但处理 3D 点云和大模型通常建议 24GB+）","未说明（场景数据特征文件约 250GB，建议大内存）",{"notes":109,"python":110,"dependencies":111},"1. 必须使用 conda 创建名为 'lavis' 的虚拟环境。2. 核心依赖为 Salesforce-LAVIS 库。3. 若需自行生成 3D 特征数据，需额外安装 Blender（用于渲染）、Mask2Former 或 SAM（用于分割）、以及 Habitat（用于 3D 重建）。4. 预训练和微调检查点及大规模数据集（如 ScanNet 特征约 250GB）托管在 Google Drive，需手动下载并配置路径。5. 训练脚本默认配置为 8 卡分布式训练。","3.8",[112,113,114,115,116,117,118],"salesforce-lavis","positional_encodings","torch","blender","mask2former","segment-anything (SAM)","habitat-lab",[15,46],"2026-03-27T02:49:30.150509","2026-04-11T03:26:18.440353",[123,128,133,138,143,148,153],{"id":124,"question_zh":125,"answer_zh":126,"source_url":127},28928,"如何获取因下载人数过多而无法访问的 Google Drive 数据集？","由于 Google Drive 限制，团队正在将数据集迁移至 Huggingface。您可以访问 https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FShuhongZheng\u002F3D-LLM 下载数据。迁移可能需要几天时间，请耐心等待。","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F22",{"id":129,"question_zh":130,"answer_zh":131,"source_url":132},28929,"运行代码时遇到 'Input type (unsigned char) and bias type (c10::Half) should be the same' 错误如何解决？","该问题已在代码提交中修复。请拉取最新代码或参考修复提交：https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Fcommit\u002Fbeff99e110a86d2f8b93ee2941cd9212450a9ce0。此外，生成 3D 特征时应使用 8 张渲染图像而非 4 张。","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F41",{"id":134,"question_zh":135,"answer_zh":136,"source_url":137},28930,"微调时加载预训练权重报错 'loaded state dict contains a parameter group that doesn't match the size of optimizer's group' 怎么办？","此问题是由于预训练权重与代码定义不匹配导致的，现已修复。请执行以下操作：\n1. 更新代码至最新版本。\n2. 重新下载最新的检查点文件。\n修复提交地址：https:\u002F\u002Fgithub.com\u002FUMass-Foundation-Model\u002F3D-LLM\u002Fcommit\u002Ff5cd275fa573eb428e09f49ff29e8dec6746faf4","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F34",{"id":139,"question_zh":140,"answer_zh":141,"source_url":142},28931,"如何将 data_part2_scene.json 中的 scene_ids 与 ScanNet 数据集中的 ID 对齐？","可以使用映射文件进行对齐。参考代码如下：\n```python\nimport json\nmapping_file = json.load(open('final_scene_map_dict_scan_v2.json', 'r'))\ndata = json.load(open('data_part2_scene.json', 'r'))\nscannet_annotations = []\nfor anno in data:\n    scannet_scene_id = mapping_file.get(str(anno['scene_id']), None)\n    if scannet_scene_id is None:\n        continue\n    scannet_annotations.append(anno)\nprint(len(scannet_annotations))\n```\n注意：请确保使用正确的映射文件版本（如 v2）。","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F23",{"id":144,"question_zh":145,"answer_zh":146,"source_url":147},28932,"在哪里可以找到 3D Captioning、3D-assisted Dialog 和 Task Decomposition 任务的验证集以进行评估？","验证集已发布在 Google Drive 文件夹中，访问链接为：https:\u002F\u002Fdrive.google.com\u002Fdrive\u002Ffolders\u002F1tdvEW5iB-ZEPo1tCqEAqKKhgHiDd4Dx0?usp=drive_link。请在该目录下查找对应的评估数据。","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F47",{"id":149,"question_zh":150,"answer_zh":151,"source_url":152},28933,"Objaverse 对象数据集的总大小是多少？预训练模型使用了多少数据？","Objaverse 对象数据集在压缩前总大小约为 10TB。预训练模型（如 'pretrain_blip2_sam_flant5xl_v2.pth'）是基于这 10TB 的对象文件以及其他场景数据进行训练的。","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F32",{"id":154,"question_zh":155,"answer_zh":156,"source_url":157},28934,"ScanNet 用于微调的特征（voxelized_features_sam_nonzero_preprocess）和用于预训练的特征有什么区别？","这两组特征主要区别在于覆盖的场景不同。虽然部分场景 ID 可能对应同一个物理场景（通过映射文件关联），但由于生成时间或处理流程不同，文件内容和大小可能存在差异。官方表示将会重新整理 Drive 上的数据结构以消除混淆。","https:\u002F\u002Fgithub.com\u002FUMass-Embodied-AGI\u002F3D-LLM\u002Fissues\u002F58",[]]