[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-JunnYu--RoFormer_pytorch":3,"tool-JunnYu--RoFormer_pytorch":64},[4,17,27,35,43,56],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":16},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,3,"2026-04-05T11:01:52",[13,14,15],"开发框架","图像","Agent","ready",{"id":18,"name":19,"github_repo":20,"description_zh":21,"stars":22,"difficulty_score":23,"last_commit_at":24,"category_tags":25,"status":16},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",138956,2,"2026-04-05T11:33:21",[13,15,26],"语言模型",{"id":28,"name":29,"github_repo":30,"description_zh":31,"stars":32,"difficulty_score":23,"last_commit_at":33,"category_tags":34,"status":16},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",107662,"2026-04-03T11:11:01",[13,14,15],{"id":36,"name":37,"github_repo":38,"description_zh":39,"stars":40,"difficulty_score":23,"last_commit_at":41,"category_tags":42,"status":16},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[13,26],{"id":44,"name":45,"github_repo":46,"description_zh":47,"stars":48,"difficulty_score":23,"last_commit_at":49,"category_tags":50,"status":16},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",84991,"2026-04-05T10:45:23",[14,51,52,53,15,54,26,13,55],"数据工具","视频","插件","其他","音频",{"id":57,"name":58,"github_repo":59,"description_zh":60,"stars":61,"difficulty_score":10,"last_commit_at":62,"category_tags":63,"status":16},3128,"ragflow","infiniflow\u002Fragflow","RAGFlow 是一款领先的开源检索增强生成（RAG）引擎，旨在为大语言模型构建更精准、可靠的上下文层。它巧妙地将前沿的 RAG 技术与智能体（Agent）能力相结合，不仅支持从各类文档中高效提取知识，还能让模型基于这些知识进行逻辑推理和任务执行。\n\n在大模型应用中，幻觉问题和知识滞后是常见痛点。RAGFlow 通过深度解析复杂文档结构（如表格、图表及混合排版），显著提升了信息检索的准确度，从而有效减少模型“胡编乱造”的现象，确保回答既有据可依又具备时效性。其内置的智能体机制更进一步，使系统不仅能回答问题，还能自主规划步骤解决复杂问题。\n\n这款工具特别适合开发者、企业技术团队以及 AI 研究人员使用。无论是希望快速搭建私有知识库问答系统，还是致力于探索大模型在垂直领域落地的创新者，都能从中受益。RAGFlow 提供了可视化的工作流编排界面和灵活的 API 接口，既降低了非算法背景用户的上手门槛，也满足了专业开发者对系统深度定制的需求。作为基于 Apache 2.0 协议开源的项目，它正成为连接通用大模型与行业专有知识之间的重要桥梁。",77062,"2026-04-04T04:44:48",[15,14,13,26,54],{"id":65,"github_repo":66,"name":67,"description_en":68,"description_zh":69,"ai_summary_zh":69,"readme_en":70,"readme_zh":71,"quickstart_zh":72,"use_case_zh":73,"hero_image_url":74,"owner_login":75,"owner_name":76,"owner_avatar_url":77,"owner_bio":78,"owner_company":79,"owner_location":80,"owner_email":81,"owner_twitter":81,"owner_website":81,"owner_url":82,"languages":83,"stars":88,"forks":89,"last_commit_at":90,"license":91,"difficulty_score":10,"env_os":92,"env_gpu":93,"env_ram":92,"env_deps":94,"category_tags":102,"github_topics":103,"view_count":10,"oss_zip_url":81,"oss_zip_packed_at":81,"status":16,"created_at":108,"updated_at":109,"faqs":110,"releases":140},773,"JunnYu\u002FRoFormer_pytorch","RoFormer_pytorch","RoFormer V1 & V2 pytorch","RoFormer_pytorch 是基于 PyTorch 框架实现的 RoFormer 及 RoFormer-V2 模型开源库。它成功将苏神提出的先进中文预训练模型迁移至 PyTorch 生态，解决了用户无法直接在 PyTorch 环境中使用原版 PaddlePaddle 实现的问题。\n\n在技术层面，RoFormer_pytorch 采用了旋转位置编码（RoPE）机制，显著增强了模型对长文本的理解能力。根据 CLUE 评测数据，其在多个中文分类任务中的表现优于传统 BERT 和 RoBERTa 模型，尤其在 WSC 和 CSL 等推理任务上优势明显。此外，项目还集成了 RoFormerForCausalLM 以支持生成式任务，并修复了微调时 token_type_id 处理的细节 bug，确保与官方行为一致。\n\n无论是需要复现论文效果的研究人员，还是希望在生产环境中部署高性能中文模型的开发者，都能从中受益。代码结构清晰，支持一键安装，为中文 NLP 领域的模型探索与落地提供了便捷且强大的基础设施。","# PyTorch RoFormer & RoFormer-V2\nRoFormer模型和RoFormer-V2模型\n\n## 更新\n- **2022\u002F05\u002F18**\n\n添加paddle版本[RoFormerV2](https:\u002F\u002Fgithub.com\u002FPaddlePaddle\u002FPaddleNLP\u002Ftree\u002Fdevelop\u002Fpaddlenlp\u002Ftransformers\u002Froformerv2)在分类任务上的训练结果。\n\n- **2022\u002F05\u002F11** \n\n感谢苏神提醒，添加了一个注释，其中RoFormerV2*表示未经多任务学习的RoFormerV2模型。\n\n- **2022\u002F05\u002F01**\n\n添加`clue分类任务`的代码和dev集结果，代码在`examples\u002Fclue`文件夹，缺少啥依赖安装啥，比如需要这个`pip install -U accelerate`。\n- **2022\u002F04\u002F30** \n\n有个细节需要注意一下，苏神在微调时无论输入是`text`还是`text pair`类型时，`token_type_id`都置为了0。\n\n如果想要使用与苏神保持一致，那么可以在`tokenizer`时候设置`return_token_type_ids=False`，这样模型会在内部处理。\n\n否则对于`text pair`类型时，会返回与`0，1`两种类型的`token_type_id`\n- **2022\u002F04\u002F02**\n\n（1）修改RoFormerForCausalLM，支持`roformer-sim`并提供相关的例子，请见`examples\u002Ftest_sim.py`。\n\n（2）修改`apply_rotary`实现方式，看起来更简单。\n```python\ndef apply_rotary(x, sinusoidal_pos=None):\n    if sinusoidal_pos is None:\n        return x\n    sin, cos = sinusoidal_pos\n    # x.shape [batch, seq_len, 2]\n    x1, x2 = x[..., 0::2], x[..., 1::2]\n    # [cos_nθ, -sin_nθ] [x1]\n    # [sin_nθ,  cos_nθ] [x2]\n    # => [x1 * cos_nθ - x2 * sin_nθ, x1 * sin_nθ + x2 * cos_nθ]\n    # 苏神的rotary，使用了下面的计算方法。\n    # return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1).flatten(-2, -1)\n    # 考虑到矩阵乘法torch.einsum(\"bhmd,bhnd->bhmn\", q, k)，因此可以直接在最后一个维度拼接（无需奇偶交错）\n    return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)\n```\n- **2022\u002F03\u002F21** 添加`roformer-v2`的权重, 注：必须使用本仓库的代码，不能使用transformers仓库的代码!!!\n\n\n\n## 安装\n\n```bash\n# v2版本\npip install roformer>=0.4.3\n# v1版本(代码已经加入到huggingface仓库，请使用新版本的transformers)\npip install -U transformers\n```\n\n\n\n## 评测对比\n### CLUE-dev榜单分类任务结果，base+large版本。\n\n|         | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl | avg |\n| :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: | ----- |\n| BERT | 60.06 | 56.80 | 72.41 | 79.56 | 73.93 | 78.62 | 83.93 | 72.19 |\n| RoBERTa | 60.64 | 58.06 | 74.05 | 81.24 | 76.00 | 87.50 | 84.50 | 74.57 |\n| RoFormer | 60.91 | 57.54 | 73.52 | 80.92 | 76.07 | 86.84 | 84.63 | 74.35 |\n| RoFormerV2\u003Csup>*\u003C\u002Fsup> | 60.87 | 56.54 | 72.75 | 80.34 | 75.36 | 80.92 | 84.67 | 73.06 |\n| GAU-α | 61.41 | 57.76 | 74.17 | 81.82 | 75.86 | 79.93 | 85.67 | 73.8 |\n| RoFormer-pytorch(本仓库代码) | 60.60 | 57.51 | 74.44 | 80.79 | 75.67 | 86.84 | 84.77 | 74.37 |\n| RoFormerV2-pytorch(本仓库代码) | 62.87 | 59.03 | 76.20 | 80.85 | 79.73 |   87.82   | **91.87** | 76.91 |\n| GAU-α-pytorch（Adafactor） | 61.18 | 57.52 | 73.42 | 80.91 | 75.69 | 80.59 | 85.5 | 73.54 |\n| GAU-α-pytorch（AdamW wd0.01 warmup0.1） | 60.68 | 57.95 | 73.08 | 81.02 | 75.36 | 81.25 | 83.93 | 73.32 |\n| RoFormerV2-large-pytorch(本仓库代码) | 61.75 | 59.21 | 76.14 | 82.35 | 81.73 | 91.45 | 91.5 | 77.73 |\n| Chinesebert-large-pytorch | 61.25 | 58.67 | 74.70 | 82.65 | 79.63 | 87.83 | 84.97 | 75.67 |\n| RoFormerV2-base-paddle |   63.76   |   59.53   |   77.06   |   81.58   |  81.56   |   87.83   |   86.73   | 76.87 |\n| RoFormerV2-large-paddle | **64.02** | **60.08** | **77.92** | **82.87** | **83.9** | **92.43** | 86.87 | **78.30** |\n\n### CLUE-1.0-test榜单分类任务结果，base+large版本。\n\n|         | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl | avg |\n| :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: | ----- |\n| RoFormer-pytorch(本仓库代码) | 59.54 | 57.34 | 74.46 | 80.23 | 73.67 |   80.69   | 84.57 | 72.93 |\n| RoFormerV2-pytorch(本仓库代码) | 63.15 | 58.24 | 75.42 | 80.59 | 74.17 |   83.79   | 83.73 | 74.16 |\n| GAU-α-pytorch（Adafactor） | 61.38 | 57.08 | 74.05 | 80.37 | 73.53 | 74.83 | **85.6** | 72.41 |\n| GAU-α-pytorch（AdamW wd0.01 warmup0.1） | 60.54 | 57.67 | 72.44 | 80.32 | 72.97 | 76.55 | 84.13 | 72.09 |\n| RoFormerV2-large-pytorch(本仓库代码) | 61.85 | 59.13 | 76.38 | 80.97 | 76.23 | **85.86** | 84.33 | 74.96 |\n| Chinesebert-large-pytorch | 61.54 | 58.57 | 74.8 | 81.94 | **76.93** | 79.66 | 85.1 | 74.08 |\n| RoFormerV2-large-paddle | **64.23** | **59.99** | **76.85** | **81.97** |   76.57   | 84.48 | 83.37 | **75.35** |\n\n### 注：\n- 其中RoFormerV2\u003Csup>*\u003C\u002Fsup>表示的是未进行多任务学习的RoFormerV2模型，该模型苏神并未开源，感谢苏神的提醒。\n- 其中不带有pytorch后缀结果都是从[GAU-alpha](https:\u002F\u002Fgithub.com\u002FZhuiyiTechnology\u002FGAU-alpha)仓库复制过来的。\n- 其中带有pytorch后缀的结果都是自己训练得出的。\n- 苏神代码中拿了cls标签后直接进行了分类，而本仓库使用了如下的分类头，多了2个dropout，1个dense，1个relu激活。\n- paddle版本的代码进行了grid search！\n\n```python\nclass RoFormerClassificationHead(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.dense = nn.Linear(config.hidden_size, config.hidden_size)\n        self.dropout = nn.Dropout(config.hidden_dropout_prob)\n        self.out_proj = nn.Linear(config.hidden_size, config.num_labels)\n\n        self.config = config\n\n    def forward(self, features, **kwargs):\n        x = features[:, 0, :]  # take \u003Cs> token (equiv. to [CLS])\n        x = self.dropout(x)\n        x = self.dense(x)\n        x = ACT2FN[self.config.hidden_act](x) # 这里是relu\n        x = self.dropout(x)\n        x = self.out_proj(x)\n        return x\n```\n\n### Tips:\n\n- 实验环境**RTX 3090**\n\n### Leadborad截图\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_a4d0b13a6a34.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_232ac2fefdd9.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_0cdb38a29076.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_b0c5960dc47d.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\n## Roformer-sim测试例子\n\n```python\nimport torch\nimport numpy as np\nfrom roformer import RoFormerForCausalLM, RoFormerConfig\nfrom transformers import BertTokenizer\n\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n# 可选以下几个。\n# junnyu\u002Froformer_chinese_sim_char_small, junnyu\u002Froformer_chinese_sim_char_base\n# junnyu\u002Froformer_chinese_sim_char_ft_small, roformer_chinese_sim_char_ft_base\npretrained_model = \"junnyu\u002Froformer_chinese_sim_char_base\"\ntokenizer = BertTokenizer.from_pretrained(pretrained_model)\nconfig = RoFormerConfig.from_pretrained(pretrained_model)\nconfig.is_decoder = True\nconfig.eos_token_id = tokenizer.sep_token_id\nconfig.pooler_activation = \"linear\"\nmodel = RoFormerForCausalLM.from_pretrained(pretrained_model, config=config)\nmodel.to(device)\nmodel.eval()\n\ndef gen_synonyms(text, n=100, k=20):\n    ''''含义： 产生sent的n个相似句，然后返回最相似的k个。\n    做法：用seq2seq生成，并用encoder算相似度并排序。\n    '''\n    # 寻找所有相似的句子\n    r = []\n    inputs1 = tokenizer(text, return_tensors=\"pt\")\n    for _ in range(n):\n        inputs1.to(device)\n        output = tokenizer.batch_decode(model.generate(**inputs1, top_p=0.95, do_sample=True, max_length=128), skip_special_tokens=True)[0].replace(\" \",\"\").replace(text, \"\") # 去除空格，去除原始text文本。\n        r.append(output)\n    \n    # 对相似的句子进行排序\n    r = [i for i in set(r) if i != text and len(i) > 0]\n    r = [text] + r\n    inputs2 = tokenizer(r, padding=True, return_tensors=\"pt\")\n    with torch.no_grad():\n        inputs2.to(device)\n        outputs = model(**inputs2)\n        Z = outputs.pooler_output.cpu().numpy()\n    Z \u002F= (Z**2).sum(axis=1, keepdims=True)**0.5\n    argsort = np.dot(Z[1:], -Z[0]).argsort()\n    \n    return [r[i + 1] for i in argsort[:k]]\n\nout = gen_synonyms(\"广州和深圳哪个好？\")\nprint(out)\n# ['深圳和广州哪个好？',\n#  '广州和深圳哪个好',\n#  '深圳和广州哪个好',\n#  '深圳和广州哪个比较好。',\n#  '深圳和广州哪个最好？',\n#  '深圳和广州哪个比较好',\n#  '广州和深圳那个比较好',\n#  '深圳和广州哪个更好？',\n#  '深圳与广州哪个好',\n#  '深圳和广州，哪个比较好',\n#  '广州与深圳比较哪个好',\n#  '深圳和广州哪里比较好',\n#  '深圳还是广州比较好？',\n#  '广州和深圳哪个地方好一些？',\n#  '广州好还是深圳好？',\n#  '广州好还是深圳好呢？',\n#  '广州与深圳哪个地方好点？',\n#  '深圳好还是广州好',\n#  '广州好还是深圳好',\n#  '广州和深圳哪个城市好？']\n```\n\n\n\n## 模型权重对照表\n\n### 中文模型 roformer-v2\n| huggingface.co                     | bert4keras                                       |\n| ---------------------------------- | ------------------------------------------------ |\n| [roformer_v2_chinese_char_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_small)              | [chinese_roformer-v2-char_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1huUrC9P60Afggo8AfiUcmA) (download code：ttn4)             |\n| [roformer_v2_chinese_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_base)              | [chinese_roformer-v2-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1qcnN4LVKVe0-mnHlkN3-6Q) (download code：pfoh)               |\n| [roformer_v2_chinese_char_large](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_large)          | [chinese_roformer-v2-char_L-24_H-1024_A-16.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1QiJWSZrGxn8vek-8myvL6w) (download code：npfv)        |\n\n\n### 中文模型 roformer-v1\n| huggingface.co                     | bert4keras                                       |\n| ---------------------------------- | ------------------------------------------------ |\n| [roformer_chinese_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_base)              | [chinese_roformer_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1fiss862YsGCwf2HvU_Jm-g) (download code：xy9x)             |\n| [roformer_chinese_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_small)             | [chinese_roformer_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1iIXgZHHCgrYGXVRRSSCVPg) (download code：gy97)               |\n| [roformer_chinese_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_char_base)         | [chinese_roformer-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1Q1pq8F4Fsl6bTipUAkqeDQ) (download code：bt94)        |\n| [roformer_chinese_char_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_char_small)        | [chinese_roformer-char_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1cc281-M0Rsjlwws5phqzbQ) (download code：a44c)          |\n| [roformer_chinese_sim_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_base)     | [chinese_roformer-sim-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1f1FB288nv1a6jYjsNCordg) (download code：2cgz)    |\n| [roformer_chinese_sim_char_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_small)    | [chinese_roformer-sim-char_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1r0eJ7shGwQ0RzV9BTFFW4g) (download code：h68q)      |\n| [roformer_chinese_sim_char_ft_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_ft_base)  | [chinese_roformer-sim-char-ft_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1Igh3tSvSu_ahDZmGaOlVoA) (download code：w15n) |\n| [roformer_chinese_sim_char_ft_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_ft_small) | [chinese_roformer-sim-char-ft_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1G36x7YQF1b6nzW0OzyJS_Q) (download code：gty5)   |\n\n\n\n\n### 英文模型（使用electra的训练方法在openwebtext上训练的small模型（rotary value = True）） \n| huggingface.co                     |\n| ---------------------------------- |\n|[roformer_small_generator](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_small_generator)|\n|[roformer_small_discriminator](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_small_discriminator)|\n\n\n\n## Roformer-v2 MLM测试\n\n```python\nimport torch\nimport tensorflow as tf\nfrom transformers import BertTokenizer\nfrom roformer import RoFormerForMaskedLM, TFRoFormerForMaskedLM\n\ntext = \"今天[MASK]很好，我[MASK]去公园玩。\"\ntokenizer = BertTokenizer.from_pretrained(\"junnyu\u002Froformer_v2_chinese_char_base\")\npt_model = RoFormerForMaskedLM.from_pretrained(\"junnyu\u002Froformer_v2_chinese_char_base\")\ntf_model = TFRoFormerForMaskedLM.from_pretrained(\n    \"junnyu\u002Froformer_v2_chinese_char_base\", from_pt=True\n)\npt_inputs = tokenizer(text, return_tensors=\"pt\")\ntf_inputs = tokenizer(text, return_tensors=\"tf\")\n# pytorch\nwith torch.no_grad():\n    pt_outputs = pt_model(**pt_inputs).logits[0]\npt_outputs_sentence = \"pytorch: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])\n        pt_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        pt_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(pt_outputs_sentence)\n# tf\ntf_outputs = tf_model(**tf_inputs, training=False).logits[0]\ntf_outputs_sentence = \"tf: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(tf.math.top_k(tf_outputs[i], k=5)[1])\n        tf_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        tf_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(tf_outputs_sentence)\n# small\n# pytorch: 今天[的||，||是||很||也]很好，我[要||会||是||想||在]去公园玩。\n# tf: 今天[的||，||是||很||也]很好，我[要||会||是||想||在]去公园玩。\n# base\n# pytorch: 今天[我||天||晴||园||玩]很好，我[想||要||会||就||带]去公园玩。\n# tf: 今天[我||天||晴||园||玩]很好，我[想||要||会||就||带]去公园玩。\n# large\n# pytorch: 今天[天||气||我||空||阳]很好，我[又||想||会||就||爱]去公园玩。\n# tf: 今天[天||气||我||空||阳]很好，我[又||想||会||就||爱]去公园玩。\n```\n\n\n\n## Roformer-v1 MLM测试\n\n```python\nimport torch\nimport tensorflow as tf\nfrom transformers import RoFormerForMaskedLM, RoFormerTokenizer, TFRoFormerForMaskedLM\n\ntext = \"今天[MASK]很好，我[MASK]去公园玩。\"\ntokenizer = RoFormerTokenizer.from_pretrained(\"junnyu\u002Froformer_chinese_base\")\npt_model = RoFormerForMaskedLM.from_pretrained(\"junnyu\u002Froformer_chinese_base\")\ntf_model = TFRoFormerForMaskedLM.from_pretrained(\n    \"junnyu\u002Froformer_chinese_base\", from_pt=True\n)\npt_inputs = tokenizer(text, return_tensors=\"pt\")\ntf_inputs = tokenizer(text, return_tensors=\"tf\")\n# pytorch\nwith torch.no_grad():\n    pt_outputs = pt_model(**pt_inputs).logits[0]\npt_outputs_sentence = \"pytorch: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])\n        pt_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        pt_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(pt_outputs_sentence)\n# tf\ntf_outputs = tf_model(**tf_inputs, training=False).logits[0]\ntf_outputs_sentence = \"tf: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(tf.math.top_k(tf_outputs[i], k=5)[1])\n        tf_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        tf_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(tf_outputs_sentence)\n# pytorch: 今天[天气||天||心情||阳光||空气]很好，我[想||要||打算||准备||喜欢]去公园玩。\n# tf:      今天[天气||天||心情||阳光||空气]很好，我[想||要||打算||准备||喜欢]去公园玩。\n\n```\n\n\n\n## 手动权重转换\n\n```bash\npython convert_roformer_original_tf_checkpoint_to_pytorch.py \\\n    --tf_checkpoint_path=xxxxxx\u002Fchinese_roformer_L-12_H-768_A-12\u002Fbert_model.ckpt \\\n    --bert_config_file=pretrained_models\u002Fchinese_roformer_base\u002Fconfig.json \\\n    --pytorch_dump_path=pretrained_models\u002Fchinese_roformer_base\u002Fpytorch_model.bin\n```\n\n\n\n## tf与pytorch精度对齐\n\n```python\nsmall版本\nbert4keras vs pytorch\nmean diff : tensor(5.9108e-07)\nmax diff : tensor(5.7220e-06)\nbert4keras vs tf2.0\nmean diff : tensor(4.5976e-07)\nmax diff : tensor(3.5763e-06)\n\nbase版本\npython compare_model.py\nbert4keras vs pytorch\nmean diff : tensor(4.3340e-07)\nmax diff : tensor(5.7220e-06)\nbert4keras vs tf2.0\nmean diff : tensor(3.4319e-07)\nmax diff : tensor(5.2452e-06)\n```\n\n\n\n## 参考\n\nhttps:\u002F\u002Fgithub.com\u002Fpengming617\u002Fbert_classification\n\nhttps:\u002F\u002Fgithub.com\u002Fbojone\u002Fbert4keras\n\nhttps:\u002F\u002Fgithub.com\u002FZhuiyiTechnology\u002Froformer \n\nhttps:\u002F\u002Fgithub.com\u002FlonePatient\u002FNeZha_Chinese_PyTorch \n\nhttps:\u002F\u002Fgithub.com\u002FlonePatient\u002FTorchBlocks\n\nhttps:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\n\n\n\n##  Citation\n\nBibtex:\n\n```tex\n\n@misc{su2021roformer,\n      title={RoFormer: Enhanced Transformer with Rotary Position Embedding}, \n      author={Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu},\n      year={2021},\n      eprint={2104.09864},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n\n```\n\n```tex\n@techreport{roformerv2,\n  title={RoFormerV2: A Faster and Better RoFormer - ZhuiyiAI},\n  author={Jianlin Su, Shengfeng Pan, Bo Wen, Yunfeng Liu},\n  year={2022},\n  url=\"https:\u002F\u002Fgithub.com\u002FZhuiyiTechnology\u002Froformer-v2\",\n}\n```\n","# PyTorch RoFormer & RoFormer-V2\nRoFormer 模型和 RoFormer-V2 模型\n\n## 更新\n- **2022\u002F05\u002F18**\n\n添加 paddle 版本 [RoFormerV2](https:\u002F\u002Fgithub.com\u002FPaddlePaddle\u002FPaddleNLP\u002Ftree\u002Fdevelop\u002Fpaddlenlp\u002Ftransformers\u002Froformerv2) 在分类任务上的训练结果。\n\n- **2022\u002F05\u002F11** \n\n感谢苏神提醒，添加了一个注释，其中 RoFormerV2*表示未经多任务学习的 RoFormerV2 模型。\n\n- **2022\u002F05\u002F01**\n\n添加 `clue 分类任务` 的代码和 dev 集结果，代码在 `examples\u002Fclue` 文件夹，缺少啥依赖安装啥，比如需要这个 `pip install -U accelerate`。\n- **2022\u002F04\u002F30** \n\n有个细节需要注意一下，苏神在微调时无论输入是 `text` 还是 `text pair` 类型时，`token_type_id` 都置为了 0。\n\n如果想要使用与苏神保持一致，那么可以在 `tokenizer` 时候设置 `return_token_type_ids=False`，这样模型会在内部处理。\n\n否则对于 `text pair` 类型时，会返回与 `0，1` 两种类型的 `token_type_id`\n- **2022\u002F04\u002F02**\n\n（1）修改 RoFormerForCausalLM，支持 `roformer-sim` 并提供相关的例子，请见 `examples\u002Ftest_sim.py`。\n\n（2）修改 `apply_rotary` 实现方式，看起来更简单。\n```python\ndef apply_rotary(x, sinusoidal_pos=None):\n    if sinusoidal_pos is None:\n        return x\n    sin, cos = sinusoidal_pos\n    # x.shape [batch, seq_len, 2]\n    x1, x2 = x[..., 0::2], x[..., 1::2]\n    # [cos_nθ, -sin_nθ] [x1]\n    # [sin_nθ,  cos_nθ] [x2]\n    # => [x1 * cos_nθ - x2 * sin_nθ, x1 * sin_nθ + x2 * cos_nθ]\n    # 苏神的 rotary，使用了下面的计算方法。\n    # return torch.stack([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1).flatten(-2, -1)\n    # 考虑到矩阵乘法 torch.einsum(\"bhmd,bhnd->bhmn\", q, k)，因此可以直接在最后一个维度拼接（无需奇偶交错）\n    return torch.cat([x1 * cos - x2 * sin, x1 * sin + x2 * cos], dim=-1)\n```\n- **2022\u002F03\u002F21** 添加 `roformer-v2` 的权重，注：必须使用本仓库的代码，不能使用 transformers 仓库的代码!!!\n\n\n\n## 安装\n\n```bash\n# v2 版本\npip install roformer>=0.4.3\n# v1 版本 (代码已经加入到 huggingface 仓库，请使用新版本的 transformers)\npip install -U transformers\n```\n\n\n\n## 评测对比\n### CLUE-dev 榜单分类任务结果，base+large 版本。\n\n|         | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl | avg |\n| :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: | ----- |\n| BERT | 60.06 | 56.80 | 72.41 | 79.56 | 73.93 | 78.62 | 83.93 | 72.19 |\n| RoBERTa | 60.64 | 58.06 | 74.05 | 81.24 | 76.00 | 87.50 | 84.50 | 74.57 |\n| RoFormer | 60.91 | 57.54 | 73.52 | 80.92 | 76.07 | 86.84 | 84.63 | 74.35 |\n| RoFormerV2\u003Csup>*\u003C\u002Fsup> | 60.87 | 56.54 | 72.75 | 80.34 | 75.36 | 80.92 | 84.67 | 73.06 |\n| GAU-α | 61.41 | 57.76 | 74.17 | 81.82 | 75.86 | 79.93 | 85.67 | 73.8 |\n| RoFormer-pytorch(本仓库代码) | 60.60 | 57.51 | 74.44 | 80.79 | 75.67 | 86.84 | 84.77 | 74.37 |\n| RoFormerV2-pytorch(本仓库代码) | 62.87 | 59.03 | 76.20 | 80.85 | 79.73 |   87.82   | **91.87** | 76.91 |\n| GAU-α-pytorch（Adafactor） | 61.18 | 57.52 | 73.42 | 80.91 | 75.69 | 80.59 | 85.5 | 73.54 |\n| GAU-α-pytorch（AdamW wd0.01 warmup0.1） | 60.68 | 57.95 | 73.08 | 81.02 | 75.36 | 81.25 | 83.93 | 73.32 |\n| RoFormerV2-large-pytorch(本仓库代码) | 61.75 | 59.21 | 76.14 | 82.35 | 81.73 | 91.45 | 91.5 | 77.73 |\n| Chinesebert-large-pytorch | 61.25 | 58.67 | 74.70 | 82.65 | 79.63 | 87.83 | 84.97 | 75.67 |\n| RoFormerV2-base-paddle |   63.76   |   59.53   |   77.06   |   81.58   |  81.56   |   87.83   |   86.73   | 76.87 |\n| RoFormerV2-large-paddle | **64.02** | **60.08** | **77.92** | **82.87** | **83.9** | **92.43** | 86.87 | **78.30** |\n\n### CLUE-1.0-test 榜单分类任务结果，base+large 版本。\n\n|         | iflytek | tnews | afqmc | cmnli | ocnli | wsc | csl | avg |\n| :-----: | :-----: | :---: | :---: | :---: | :---: | :---: | :---: | ----- |\n| RoFormer-pytorch(本仓库代码) | 59.54 | 57.34 | 74.46 | 80.23 | 73.67 |   80.69   | 84.57 | 72.93 |\n| RoFormerV2-pytorch(本仓库代码) | 63.15 | 58.24 | 75.42 | 80.59 | 74.17 |   83.79   | 83.73 | 74.16 |\n| GAU-α-pytorch（Adafactor） | 61.38 | 57.08 | 74.05 | 80.37 | 73.53 | 74.83 | **85.6** | 72.41 |\n| GAU-α-pytorch（AdamW wd0.01 warmup0.1） | 60.54 | 57.67 | 72.44 | 80.32 | 72.97 | 76.55 | 84.13 | 72.09 |\n| RoFormerV2-large-pytorch(本仓库代码) | 61.85 | 59.13 | 76.38 | 80.97 | 76.23 | **85.86** | 84.33 | 74.96 |\n| Chinesebert-large-pytorch | 61.54 | 58.57 | 74.8 | 81.94 | **76.93** | 79.66 | 85.1 | 74.08 |\n| RoFormerV2-large-paddle | **64.23** | **59.99** | **76.85** | **81.97** |   76.57   | 84.48 | 83.37 | **75.35** |\n\n### 注：\n- 其中 RoFormerV2\u003Csup>*\u003C\u002Fsup>表示的是未进行多任务学习的 RoFormerV2 模型，该模型苏神并未开源，感谢苏神的提醒。\n- 其中不带有 pytorch 后缀结果都是从 [GAU-alpha](https:\u002F\u002Fgithub.com\u002FZhuiyiTechnology\u002FGAU-alpha) 仓库复制过来的。\n- 其中带有 pytorch 后缀的结果都是自己训练得出的。\n- 苏神代码中拿了 cls 标签后直接进行了分类，而本仓库使用了如下的分类头，多了 2 个 dropout，1 个 dense，1 个 relu 激活。\n- paddle 版本的代码进行了 grid search！\n\n```python\nclass RoFormerClassificationHead(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.dense = nn.Linear(config.hidden_size, config.hidden_size)\n        self.dropout = nn.Dropout(config.hidden_dropout_prob)\n        self.out_proj = nn.Linear(config.hidden_size, config.num_labels)\n\n        self.config = config\n\n    def forward(self, features, **kwargs):\n        x = features[:, 0, :]  # take \u003Cs> token (equiv. to [CLS])\n        x = self.dropout(x)\n        x = self.dense(x)\n        x = ACT2FN[self.config.hidden_act](x) # 这里是 relu\n        x = self.dropout(x)\n        x = self.out_proj(x)\n        return x\n```\n\n### 提示：\n\n- 实验环境**RTX 3090**\n\n### 排行榜截图\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_a4d0b13a6a34.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_232ac2fefdd9.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_0cdb38a29076.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\u003Cp align=\"center\">\n    \u003Cimg src=\"https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_readme_b0c5960dc47d.jpg\" width=\"100%\" \u002F>\n\u003C\u002Fp>\n\n## Roformer-sim 测试例子\n\n```python\nimport torch\nimport numpy as np\nfrom roformer import RoFormerForCausalLM, RoFormerConfig\nfrom transformers import BertTokenizer\n\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n# 可选以下几个。\n# junnyu\u002Froformer_chinese_sim_char_small, junnyu\u002Froformer_chinese_sim_char_base\n# junnyu\u002Froformer_chinese_sim_char_ft_small, roformer_chinese_sim_char_ft_base\npretrained_model = \"junnyu\u002Froformer_chinese_sim_char_base\"\ntokenizer = BertTokenizer.from_pretrained(pretrained_model)\nconfig = RoFormerConfig.from_pretrained(pretrained_model)\nconfig.is_decoder = True\nconfig.eos_token_id = tokenizer.sep_token_id\nconfig.pooler_activation = \"linear\"\nmodel = RoFormerForCausalLM.from_pretrained(pretrained_model, config=config)\nmodel.to(device)\nmodel.eval()\n\ndef gen_synonyms(text, n=100, k=20):\n    ''''含义：产生 sent 的 n 个相似句，然后返回最相似的 k 个。\n    做法：用 seq2seq 生成，并用 encoder 算相似度并排序。\n    '''\n    # 寻找所有相似的句子\n    r = []\n    inputs1 = tokenizer(text, return_tensors=\"pt\")\n    for _ in range(n):\n        inputs1.to(device)\n        output = tokenizer.batch_decode(model.generate(**inputs1, top_p=0.95, do_sample=True, max_length=128), skip_special_tokens=True)[0].replace(\" \",\"\").replace(text, \"\") # 去除空格，去除原始 text 文本。\n        r.append(output)\n    \n    # 对相似的句子进行排序\n    r = [i for i in set(r) if i != text and len(i) > 0]\n    r = [text] + r\n    inputs2 = tokenizer(r, padding=True, return_tensors=\"pt\")\n    with torch.no_grad():\n        inputs2.to(device)\n        outputs = model(**inputs2)\n        Z = outputs.pooler_output.cpu().numpy()\n    Z \u002F= (Z**2).sum(axis=1, keepdims=True)**0.5\n    argsort = np.dot(Z[1:], -Z[0]).argsort()\n    \n    return [r[i + 1] for i in argsort[:k]]\n\nout = gen_synonyms(\"广州和深圳哪个好？\")\nprint(out)\n```\n\n# ['深圳和广州哪个好？',\n#  '广州和深圳哪个好',\n#  '深圳和广州哪个好',\n#  '深圳和广州哪个比较好。',\n#  '深圳和广州哪个最好？',\n#  '深圳和广州哪个比较好',\n#  '广州和深圳那个比较好',\n#  '深圳和广州哪个更好？',\n#  '深圳与广州哪个好',\n#  '深圳和广州，哪个比较好',\n#  '广州与深圳比较哪个好',\n#  '深圳和广州哪里比较好',\n#  '深圳还是广州比较好？',\n#  '广州和深圳哪个地方好一些？',\n#  '广州好还是深圳好？',\n#  '广州好还是深圳好呢？',\n#  '广州与深圳哪个地方好点？',\n#  '深圳好还是广州好',\n#  '广州好还是深圳好',\n#  '广州和深圳哪个城市好？']\n```\n\n\n\n## 模型权重对照表\n\n### 中文模型 roformer-v2\n| huggingface.co                     | bert4keras                                       |\n| ---------------------------------- | ------------------------------------------------ |\n| [roformer_v2_chinese_char_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_small)              | [chinese_roformer-v2-char_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1huUrC9P60Afggo8AfiUcmA) (下载码：ttn4)             |\n| [roformer_v2_chinese_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_base)              | [chinese_roformer-v2-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1qcnN4LVKVe0-mnHlkN3-6Q) (下载码：pfoh)               |\n| [roformer_v2_chinese_char_large](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_large)          | [chinese_roformer-v2-char_L-24_H-1024_A-16.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1QiJWSZrGxn8vek-8myvL6w) (下载码：npfv)        |\n\n\n### 中文模型 roformer-v1\n| huggingface.co                     | bert4keras                                       |\n| ---------------------------------- | ------------------------------------------------ |\n| [roformer_chinese_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_base)              | [chinese_roformer_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1fiss862YsGCwf2HvU_Jm-g) (下载码：xy9x)             |\n| [roformer_chinese_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_small)             | [chinese_roformer_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1iIXgZHHCgrYGXVRRSSCVPg) (下载码：gy97)               |\n| [roformer_chinese_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_char_base)         | [chinese_roformer-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1Q1pq8F4Fsl6bTipUAkqeDQ) (下载码：bt94)        |\n| [roformer_chinese_char_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_char_small)        | [chinese_roformer-char_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1cc281-M0Rsjlwws5phqzbQ) (下载码：a44c)          |\n| [roformer_chinese_sim_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_base)     | [chinese_roformer-sim-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1f1FB288nv1a6jYjsNCordg) (下载码：2cgz)    |\n| [roformer_chinese_sim_char_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_small)    | [chinese_roformer-sim-char_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1r0eJ7shGwQ0RzV9BTFFW4g) (下载码：h68q)      |\n| [roformer_chinese_sim_char_ft_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_ft_base)  | [chinese_roformer-sim-char-ft_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1Igh3tSvSu_ahDZmGaOlVoA) (下载码：w15n) |\n| [roformer_chinese_sim_char_ft_small](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_sim_char_ft_small) | [chinese_roformer-sim-char-ft_L-6_H-384_A-6.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1G36x7YQF1b6nzW0OzyJS_Q) (下载码：gty5)   |\n\n\n\n\n### 英文模型（使用 ELECTRA (一种预训练模型架构) 的训练方法在 OpenWebText (开源网络文本数据集) 上训练的 Small (小型) 模型（Rotary Value (旋转位置编码值) = True）） \n| huggingface.co                     |\n| ---------------------------------- |\n|[roformer_small_generator](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_small_generator)|\n|[roformer_small_discriminator](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_small_discriminator)|\n\n\n\n## RoFormer-v2 掩码语言模型 (Masked Language Model, MLM) 测试\n\n```python\nimport torch\nimport tensorflow as tf\nfrom transformers import BertTokenizer\nfrom roformer import RoFormerForMaskedLM, TFRoFormerForMaskedLM\n\ntext = \"今天[MASK]很好，我[MASK]去公园玩。\"\ntokenizer = BertTokenizer.from_pretrained(\"junnyu\u002Froformer_v2_chinese_char_base\")\npt_model = RoFormerForMaskedLM.from_pretrained(\"junnyu\u002Froformer_v2_chinese_char_base\")\ntf_model = TFRoFormerForMaskedLM.from_pretrained(\n    \"junnyu\u002Froformer_v2_chinese_char_base\", from_pt=True\n)\npt_inputs = tokenizer(text, return_tensors=\"pt\")\ntf_inputs = tokenizer(text, return_tensors=\"tf\")\n# pytorch\nwith torch.no_grad():\n    pt_outputs = pt_model(**pt_inputs).logits[0]\npt_outputs_sentence = \"pytorch: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])\n        pt_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        pt_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(pt_outputs_sentence)\n# tf\ntf_outputs = tf_model(**tf_inputs, training=False).logits[0]\ntf_outputs_sentence = \"tf: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(tf.math.top_k(tf_outputs[i], k=5)[1])\n        tf_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        tf_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(tf_outputs_sentence)\n# small\n# pytorch: 今天 [的 ||，||是||很||也] 很好，我 [要 ||会||是||想||在] 去公园玩。\n# tf: 今天 [的 ||，||是||很||也] 很好，我 [要 ||会||是||想||在] 去公园玩。\n# base\n# pytorch: 今天 [我 ||天||晴||园||玩] 很好，我 [想 ||要||会||就||带] 去公园玩。\n# tf: 今天 [我 ||天||晴||园||玩] 很好，我 [想 ||要||会||就||带] 去公园玩。\n# large\n# pytorch: 今天 [天 ||气||我||空||阳] 很好，我 [又 ||想||会||就||爱] 去公园玩。\n# tf: 今天 [天 ||气||我||空||阳] 很好，我 [又 ||想||会||就||爱] 去公园玩。\n```\n\n\n\n## RoFormer-v1 掩码语言模型 (Masked Language Model, MLM) 测试\n\n```python\nimport torch\nimport tensorflow as tf\nfrom transformers import RoFormerForMaskedLM, RoFormerTokenizer, TFRoFormerForMaskedLM\n\ntext = \"今天[MASK]很好，我[MASK]去公园玩。\"\ntokenizer = RoFormerTokenizer.from_pretrained(\"junnyu\u002Froformer_chinese_base\")\npt_model = RoFormerForMaskedLM.from_pretrained(\"junnyu\u002Froformer_chinese_base\")\ntf_model = TFRoFormerForMaskedLM.from_pretrained(\n    \"junnyu\u002Froformer_chinese_base\", from_pt=True\n)\npt_inputs = tokenizer(text, return_tensors=\"pt\")\ntf_inputs = tokenizer(text, return_tensors=\"tf\")\n# pytorch\nwith torch.no_grad():\n    pt_outputs = pt_model(**pt_inputs).logits[0]\npt_outputs_sentence = \"pytorch: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k=5)[1])\n        pt_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        pt_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(pt_outputs_sentence)\n```\n\n# tf\ntf_outputs = tf_model(**tf_inputs, training=False).logits[0]\ntf_outputs_sentence = \"tf: \"\nfor i, id in enumerate(tokenizer.encode(text)):\n    if id == tokenizer.mask_token_id:\n        tokens = tokenizer.convert_ids_to_tokens(tf.math.top_k(tf_outputs[i], k=5)[1])\n        tf_outputs_sentence += \"[\" + \"||\".join(tokens) + \"]\"\n    else:\n        tf_outputs_sentence += \"\".join(\n            tokenizer.convert_ids_to_tokens([id], skip_special_tokens=True)\n        )\nprint(tf_outputs_sentence)\n# pytorch: 今天 [天气||天||心情||阳光||空气] 很好，我 [想||要||打算||准备||喜欢] 去公园玩。\n# tf:      今天 [天气||天||心情||阳光||空气] 很好，我 [想||要||打算||准备||喜欢] 去公园玩。\n\n```\n\n\n\n## 手动权重转换\n\n```bash\npython convert_roformer_original_tf_checkpoint_to_pytorch.py \\\n    --tf_checkpoint_path=xxxxxx\u002Fchinese_roformer_L-12_H-768_A-12\u002Fbert_model.ckpt \\\n    --bert_config_file=pretrained_models\u002Fchinese_roformer_base\u002Fconfig.json \\\n    --pytorch_dump_path=pretrained_models\u002Fchinese_roformer_base\u002Fpytorch_model.bin\n```\n\n\n\n## tf 与 pytorch 精度对齐\n\n```python\nsmall 版本\nbert4keras vs pytorch\nmean diff : tensor(5.9108e-07)\nmax diff : tensor(5.7220e-06)\nbert4keras vs tf2.0\nmean diff : tensor(4.5976e-07)\nmax diff : tensor(3.5763e-06)\n\nbase 版本\npython compare_model.py\nbert4keras vs pytorch\nmean diff : tensor(4.3340e-07)\nmax diff : tensor(5.7220e-06)\nbert4keras vs tf2.0\nmean diff : tensor(3.4319e-07)\nmax diff : tensor(5.2452e-06)\n```\n\n\n\n## 参考\n\nhttps:\u002F\u002Fgithub.com\u002Fpengming617\u002Fbert_classification\n\nhttps:\u002F\u002Fgithub.com\u002Fbojone\u002Fbert4keras\n\nhttps:\u002F\u002Fgithub.com\u002FZhuiyiTechnology\u002Froformer \n\nhttps:\u002F\u002Fgithub.com\u002FlonePatient\u002FNeZha_Chinese_PyTorch \n\nhttps:\u002F\u002Fgithub.com\u002FlonePatient\u002FTorchBlocks\n\nhttps:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\n\n\n\n## 引用\n\nBibtex:\n\n```tex\n\n@misc{su2021roformer,\n      title={RoFormer: Enhanced Transformer with Rotary Position Embedding}, \n      author={Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu},\n      year={2021},\n      eprint={2104.09864},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n\n```\n\n```tex\n@techreport{roformerv2,\n  title={RoFormerV2: A Faster and Better RoFormer - ZhuiyiAI},\n  author={Jianlin Su, Shengfeng Pan, Bo Wen, Yunfeng Liu},\n  year={2022},\n  url=\"https:\u002F\u002Fgithub.com\u002FZhuiyiTechnology\u002Froformer-v2\",\n}\n```","# RoFormer_pytorch 快速上手指南\n\n## 简介\n本项目是 RoFormer 及 RoFormer-V2 模型的 PyTorch 实现，针对中文自然语言处理任务进行了优化，支持分类、生成及语义相似度等任务。\n\n## 环境准备\n- **系统要求**: Linux \u002F Windows \u002F macOS\n- **Python**: >= 3.7\n- **深度学习框架**: PyTorch (推荐 CUDA 版本以利用 GPU 加速)\n- **其他依赖**: `transformers`, `numpy`, `torchvision` (视具体任务而定)\n\n## 安装步骤\n### 1. 安装核心库\n根据需求选择版本进行安装。**RoFormer-V2 必须使用本仓库代码**，无法直接兼容标准 transformers 库。\n\n```bash\n# 安装 v2 版本\npip install roformer>=0.4.3\n\n# 如需使用 v1 版本（代码已并入 transformers，但本仓库提供特定功能）\npip install -U transformers\n```\n\n### 2. 可选依赖\n若需运行 CLUE 分类任务示例，请安装加速库：\n```bash\npip install -U accelerate\n```\n\n## 基本使用\n以下示例演示如何加载 **RoFormer-Sim** 模型并生成相似句（语义相似度任务）。\n\n```python\nimport torch\nimport numpy as np\nfrom roformer import RoFormerForCausalLM, RoFormerConfig\nfrom transformers import BertTokenizer\n\ndevice = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n# 可选以下几个模型\n# junnyu\u002Froformer_chinese_sim_char_small, junnyu\u002Froformer_chinese_sim_char_base\n# junnyu\u002Froformer_chinese_sim_char_ft_small, roformer_chinese_sim_char_ft_base\npretrained_model = \"junnyu\u002Froformer_chinese_sim_char_base\"\ntokenizer = BertTokenizer.from_pretrained(pretrained_model)\nconfig = RoFormerConfig.from_pretrained(pretrained_model)\nconfig.is_decoder = True\nconfig.eos_token_id = tokenizer.sep_token_id\nconfig.pooler_activation = \"linear\"\nmodel = RoFormerForCausalLM.from_pretrained(pretrained_model, config=config)\nmodel.to(device)\nmodel.eval()\n\ndef gen_synonyms(text, n=100, k=20):\n    ''''含义：产生 sent 的 n 个相似句，然后返回最相似的 k 个。\n    做法：用 seq2seq 生成，并用 encoder 算相似度并排序。\n    '''\n    r = []\n    inputs1 = tokenizer(text, return_tensors=\"pt\")\n    for _ in range(n):\n        inputs1.to(device)\n        output = tokenizer.batch_decode(model.generate(**inputs1, top_p=0.95, do_sample=True, max_length=128), skip_special_tokens=True)[0].replace(\" \",\"\").replace(text, \"\") \n        r.append(output)\n    \n    r = [i for i in set(r) if i != text and len(i) > 0]\n    r = [text] + r\n    inputs2 = tokenizer(r, padding=True, return_tensors=\"pt\")\n    with torch.no_grad():\n        inputs2.to(device)\n        outputs = model(**inputs2)\n        Z = outputs.pooler_output.cpu().numpy()\n    Z \u002F= (Z**2).sum(axis=1, keepdims=True)**0.5\n    argsort = np.dot(Z[1:], -Z[0]).argsort()\n    \n    return [r[i + 1] for i in argsort[:k]]\n\nout = gen_synonyms(\"广州和深圳哪个好？\")\nprint(out)\n```\n\n## 模型权重下载\n模型主要托管于 HuggingFace，国内用户如遇网络问题可使用百度网盘镜像。\n\n| 模型类型 | HuggingFace 链接 | 百度网盘 (提取码) |\n| :--- | :--- | :--- |\n| **RoFormer-V2 Base** | [roformer_v2_chinese_char_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_base) | [chinese_roformer-v2-char_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1qcnN4LVKVe0-mnHlkN3-6Q) (pfoh) |\n| **RoFormer-V2 Large** | [roformer_v2_chinese_char_large](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_v2_chinese_char_large) | [chinese_roformer-v2-char_L-24_H-1024_A-16.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1QiJWSZrGxn8vek-8myvL6w) (npfv) |\n| **RoFormer-V1 Base** | [roformer_chinese_base](https:\u002F\u002Fhuggingface.co\u002Fjunnyu\u002Froformer_chinese_base) | [chinese_roformer_L-12_H-768_A-12.zip](https:\u002F\u002Fpan.baidu.com\u002Fs\u002F1fiss862YsGCwf2HvU_Jm-g) (xy9x) |\n\n## 注意事项\n1. **Token Type ID**: 微调时无论输入是 `text` 还是 `text pair`，建议设置 `return_token_type_ids=False`，使模型内部自动处理（置为 0），以保持与原作者苏神代码一致。\n2. **分类头差异**: 本仓库分类头包含额外的 dropout、dense 层和 ReLU 激活，与原始代码直接取 cls 标签略有不同。\n3. **实验环境**: 评测结果基于 RTX 3090 显卡环境。","某金融科技公司的算法团队正在开发智能工单分类系统，核心需求是解决中文语境下复杂的语义歧义与指代关系识别问题。\n\n### 没有 RoFormer_pytorch 时\n- 采用传统 BERT 模型处理中文长文本时，固定位置编码难以捕捉深层语法结构，导致关键上下文信息严重遗漏。\n- 在涉及指代消解等复杂逻辑任务上，模型表现平平，误判率较高，直接影响客服工单的自动派单效率与用户体验。\n- 环境依赖繁琐，若需使用苏神优化的 V2 版本，往往面临代码兼容性差、权重加载困难以及缺乏官方支持的问题。\n\n### 使用 RoFormer_pytorch 后\n- 利用旋转位置嵌入机制，显著增强了对长序列上下文的动态理解能力，有效解决了因位置信息丢失导致的语义偏差。\n- 基于 CLUE 榜单验证的高性能数据，在 WSC 和 CSL 等中文特有任务上准确率大幅提升，大幅降低了业务场景中的误分类风险。\n- 原生 PyTorch 实现开箱即用，支持 V2 版本一键安装与微调，无需反复调试底层代码即可快速完成模型部署与迭代。\n\n核心价值：RoFormer_pytorch 凭借更优的中文语义建模能力，帮助团队以更低算力成本实现了高精度的业务落地。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002FJunnYu_RoFormer_pytorch_92649050.png","JunnYu","yujun","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002FJunnYu_6315126c.jpg","Please show me your code.","@ECUST","Shanghai, China",null,"https:\u002F\u002Fgithub.com\u002FJunnYu",[84],{"name":85,"color":86,"percentage":87},"Python","#3572A5",100,522,45,"2026-03-11T13:23:47","Apache-2.0","未说明","非必需（支持 CPU），实验环境 RTX 3090",{"notes":95,"python":92,"dependencies":96},"RoFormerV2 版本必须使用本仓库代码，不可直接使用 transformers 官方仓库代码；微调时 text pair 类型建议设置 return_token_type_ids=False 以保持与苏神代码一致；支持 CPU 和 GPU 运行。",[97,98,99,100,101],"roformer>=0.4.3","transformers","accelerate","torch","numpy",[26,13],[104,105,106,107],"pytorch","roformer-v1","roformer-v2","language-model","2026-03-27T02:49:30.150509","2026-04-06T06:46:00.651649",[111,116,120,125,130,135],{"id":112,"question_zh":113,"answer_zh":114,"source_url":115},3322,"使用官方 transformers 库加载 RoFormer-Sim 模型训练时 Loss 无法下降，原因是什么？","transformers 库的代码与本仓库基本一致，但缺少了 `pooler` 部分。最好的办法是使用本仓库的代码（已添加 pooler 部分），然后调用。如果必须使用 transformers，需注意配置差异可能导致训练效果不佳。","https:\u002F\u002Fgithub.com\u002FJunnYu\u002FRoFormer_pytorch\u002Fissues\u002F14",{"id":117,"question_zh":118,"answer_zh":119,"source_url":115},3323,"如何使用 RoFormer-Sim 模型生成相似句子？","需使用 `RoFormerForCausalLM` 并配置解码参数。关键配置如下：\n```python\nconfig.pooler_activation = \"linear\"\nconfig.is_decoder = True\nconfig.eos_token_id = tokenizer.sep_token_id\nmodel = RoFormerForCausalLM.from_pretrained(\"junnyu\u002Froformer_chinese_sim_char_base\", config=config)\n```\n这样可以正确生成结果并进行相似度计算。",{"id":121,"question_zh":122,"answer_zh":123,"source_url":124},3324,"使用 transformers 加载 `roformer_chinese_sim_char_small` 时报错 `size mismatch for roformer.encoder.embed_positions.weight` 如何解决？","这是因为位置编码权重在模型 `__init__` 初始化时会自动生成（SinusoidalPositionalEmbedding），无需从 checkpoint 加载匹配。该模块会在初始化时自动创建 sinusoidal 位置嵌入，且 `requires_grad = False`，因此报错中的 size mismatch 不影响使用，可直接忽略。","https:\u002F\u002Fgithub.com\u002FJunnYu\u002FRoFormer_pytorch\u002Fissues\u002F12",{"id":126,"question_zh":127,"answer_zh":128,"source_url":129},3325,"能否将其他模型（如 MacBERT）的权重转换为 RoFormer 格式？","可以。本仓库代码支持 `embedding_size` 与 `hidden_size` 不同的情况（类似于 ALBERT、ELECTRA），通过 `embeddings_project` 层处理。可以使用 `convert_roformer_original_tf_checkpoint_to_pytorch.py` 脚本将 TF 版本的权重转换为 PyTorch 格式。","https:\u002F\u002Fgithub.com\u002FJunnYu\u002FRoFormer_pytorch\u002Fissues\u002F2",{"id":131,"question_zh":132,"answer_zh":133,"source_url":134},3326,"RoFormer 的 Char 版本和 Word 版本在模型调用上有什么区别？","转换权重的代码是一样的，主要区别在于使用时使用的 tokenizer 不同。\n\n**Char 版本模型调用：**\n```python\nfrom transformers import BertTokenizer\ntokenizer = BertTokenizer.from_pretrained(\"模型权重文件夹\")\n```\n\n**Word 版本模型调用：**\n```python\nfrom roformer import RoFormerTokenizer\ntokenizer = RoFormerTokenizer.from_pretrained(\"模型权重文件夹\")\n```","https:\u002F\u002Fgithub.com\u002FJunnYu\u002FRoFormer_pytorch\u002Fissues\u002F23",{"id":136,"question_zh":137,"answer_zh":138,"source_url":139},3327,"加载 PyTorch 权重时遇到 `ValueError: tensorflow.__spec__ is None` 或版本兼容性问题怎么办？","这通常与 PyTorch 版本及保存格式有关。PyTorch 1.6 切换到了新的 zipfile 格式。解决方案有二：一是更新 PyTorch 至 1.6+；二是在旧环境下下载原版 TF 权重并使用转换脚本重新转换。若必须加载旧格式文件，可在 `torch.load` 时传入 `_use_new_zipfile_serialization=False`。","https:\u002F\u002Fgithub.com\u002FJunnYu\u002FRoFormer_pytorch\u002Fissues\u002F3",[141,146,151,156,160,164,169,173,177],{"id":142,"version":143,"summary_zh":144,"released_at":145},102891,"v0.4.1","支持 roformer-sim","2022-04-02T15:31:04",{"id":147,"version":148,"summary_zh":149,"released_at":150},102892,"v0.4.0","support roformer-v2!!!","2022-03-21T14:05:33",{"id":152,"version":153,"summary_zh":154,"released_at":155},102893,"v0.3.1","删除临时的transformers库，更新依赖 transformers>=4.13.0","2021-12-15T09:09:08",{"id":157,"version":158,"summary_zh":81,"released_at":159},102894,"v0.3.0","2021-12-03T03:57:39",{"id":161,"version":162,"summary_zh":81,"released_at":163},102895,"v0.2.2","2021-09-14T15:13:11",{"id":165,"version":166,"summary_zh":167,"released_at":168},102896,"v0.1.0","同步huggingface仓库roformer最新代码到本仓库","2021-07-05T03:35:30",{"id":170,"version":171,"summary_zh":81,"released_at":172},102897,"0.0.8","2021-05-27T04:24:50",{"id":174,"version":175,"summary_zh":81,"released_at":176},102898,"0.0.6","2021-05-18T01:49:42",{"id":178,"version":179,"summary_zh":180,"released_at":181},102899,"v0.0.4","pytorch\r\ntensorflow2.0","2021-05-09T07:21:50"]