[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-google-deepmind--synthid-text":3,"tool-google-deepmind--synthid-text":64},[4,17,25,39,48,56],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":16},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",138956,2,"2026-04-05T11:33:21",[13,14,15],"开发框架","Agent","语言模型","ready",{"id":18,"name":19,"github_repo":20,"description_zh":21,"stars":22,"difficulty_score":10,"last_commit_at":23,"category_tags":24,"status":16},3704,"NextChat","ChatGPTNextWeb\u002FNextChat","NextChat 是一款轻量且极速的 AI 助手，旨在为用户提供流畅、跨平台的大模型交互体验。它完美解决了用户在多设备间切换时难以保持对话连续性，以及面对众多 AI 模型不知如何统一管理的痛点。无论是日常办公、学习辅助还是创意激发，NextChat 都能让用户随时随地通过网页、iOS、Android、Windows、MacOS 或 Linux 端无缝接入智能服务。\n\n这款工具非常适合普通用户、学生、职场人士以及需要私有化部署的企业团队使用。对于开发者而言，它也提供了便捷的自托管方案，支持一键部署到 Vercel 或 Zeabur 等平台。\n\nNextChat 的核心亮点在于其广泛的模型兼容性，原生支持 Claude、DeepSeek、GPT-4 及 Gemini Pro 等主流大模型，让用户在一个界面即可自由切换不同 AI 能力。此外，它还率先支持 MCP（Model Context Protocol）协议，增强了上下文处理能力。针对企业用户，NextChat 提供专业版解决方案，具备品牌定制、细粒度权限控制、内部知识库整合及安全审计等功能，满足公司对数据隐私和个性化管理的高标准要求。",87618,"2026-04-05T07:20:52",[13,15],{"id":26,"name":27,"github_repo":28,"description_zh":29,"stars":30,"difficulty_score":10,"last_commit_at":31,"category_tags":32,"status":16},2268,"ML-For-Beginners","microsoft\u002FML-For-Beginners","ML-For-Beginners 是由微软推出的一套系统化机器学习入门课程，旨在帮助零基础用户轻松掌握经典机器学习知识。这套课程将学习路径规划为 12 周，包含 26 节精炼课程和 52 道配套测验，内容涵盖从基础概念到实际应用的完整流程，有效解决了初学者面对庞大知识体系时无从下手、缺乏结构化指导的痛点。\n\n无论是希望转型的开发者、需要补充算法背景的研究人员，还是对人工智能充满好奇的普通爱好者，都能从中受益。课程不仅提供了清晰的理论讲解，还强调动手实践，让用户在循序渐进中建立扎实的技能基础。其独特的亮点在于强大的多语言支持，通过自动化机制提供了包括简体中文在内的 50 多种语言版本，极大地降低了全球不同背景用户的学习门槛。此外，项目采用开源协作模式，社区活跃且内容持续更新，确保学习者能获取前沿且准确的技术资讯。如果你正寻找一条清晰、友好且专业的机器学习入门之路，ML-For-Beginners 将是理想的起点。",84991,"2026-04-05T10:45:23",[33,34,35,36,14,37,15,13,38],"图像","数据工具","视频","插件","其他","音频",{"id":40,"name":41,"github_repo":42,"description_zh":43,"stars":44,"difficulty_score":45,"last_commit_at":46,"category_tags":47,"status":16},3128,"ragflow","infiniflow\u002Fragflow","RAGFlow 是一款领先的开源检索增强生成（RAG）引擎，旨在为大语言模型构建更精准、可靠的上下文层。它巧妙地将前沿的 RAG 技术与智能体（Agent）能力相结合，不仅支持从各类文档中高效提取知识，还能让模型基于这些知识进行逻辑推理和任务执行。\n\n在大模型应用中，幻觉问题和知识滞后是常见痛点。RAGFlow 通过深度解析复杂文档结构（如表格、图表及混合排版），显著提升了信息检索的准确度，从而有效减少模型“胡编乱造”的现象，确保回答既有据可依又具备时效性。其内置的智能体机制更进一步，使系统不仅能回答问题，还能自主规划步骤解决复杂问题。\n\n这款工具特别适合开发者、企业技术团队以及 AI 研究人员使用。无论是希望快速搭建私有知识库问答系统，还是致力于探索大模型在垂直领域落地的创新者，都能从中受益。RAGFlow 提供了可视化的工作流编排界面和灵活的 API 接口，既降低了非算法背景用户的上手门槛，也满足了专业开发者对系统深度定制的需求。作为基于 Apache 2.0 协议开源的项目，它正成为连接通用大模型与行业专有知识之间的重要桥梁。",77062,3,"2026-04-04T04:44:48",[14,33,13,15,37],{"id":49,"name":50,"github_repo":51,"description_zh":52,"stars":53,"difficulty_score":45,"last_commit_at":54,"category_tags":55,"status":16},519,"PaddleOCR","PaddlePaddle\u002FPaddleOCR","PaddleOCR 是一款基于百度飞桨框架开发的高性能开源光学字符识别工具包。它的核心能力是将图片、PDF 等文档中的文字提取出来，转换成计算机可读取的结构化数据，让机器真正“看懂”图文内容。\n\n面对海量纸质或电子文档，PaddleOCR 解决了人工录入效率低、数字化成本高的问题。尤其在人工智能领域，它扮演着连接图像与大型语言模型（LLM）的桥梁角色，能将视觉信息直接转化为文本输入，助力智能问答、文档分析等应用场景落地。\n\nPaddleOCR 适合开发者、算法研究人员以及有文档自动化需求的普通用户。其技术优势十分明显：不仅支持全球 100 多种语言的识别，还能在 Windows、Linux、macOS 等多个系统上运行，并灵活适配 CPU、GPU、NPU 等各类硬件。作为一个轻量级且社区活跃的开源项目，PaddleOCR 既能满足快速集成的需求，也能支撑前沿的视觉语言研究，是处理文字识别任务的理想选择。",74913,"2026-04-05T10:44:17",[15,33,13,37],{"id":57,"name":58,"github_repo":59,"description_zh":60,"stars":61,"difficulty_score":45,"last_commit_at":62,"category_tags":63,"status":16},2181,"OpenHands","OpenHands\u002FOpenHands","OpenHands 是一个专注于 AI 驱动开发的开源平台，旨在让智能体（Agent）像人类开发者一样理解、编写和调试代码。它解决了传统编程中重复性劳动多、环境配置复杂以及人机协作效率低等痛点，通过自动化流程显著提升开发速度。\n\n无论是希望提升编码效率的软件工程师、探索智能体技术的研究人员，还是需要快速原型验证的技术团队，都能从中受益。OpenHands 提供了灵活多样的使用方式：既可以通过命令行（CLI）或本地图形界面在个人电脑上轻松上手，体验类似 Devin 的流畅交互；也能利用其强大的 Python SDK 自定义智能体逻辑，甚至在云端大规模部署上千个智能体并行工作。\n\n其核心技术亮点在于模块化的软件智能体 SDK，这不仅构成了平台的引擎，还支持高度可组合的开发模式。此外，OpenHands 在 SWE-bench 基准测试中取得了 77.6% 的优异成绩，证明了其解决真实世界软件工程问题的能力。平台还具备完善的企业级功能，支持与 Slack、Jira 等工具集成，并提供细粒度的权限管理，适合从个人开发者到大型企业的各类用户场景。",70612,"2026-04-05T11:12:22",[15,14,13,36],{"id":65,"github_repo":66,"name":67,"description_en":68,"description_zh":69,"ai_summary_zh":69,"readme_en":70,"readme_zh":71,"quickstart_zh":72,"use_case_zh":73,"hero_image_url":74,"owner_login":75,"owner_name":76,"owner_avatar_url":77,"owner_bio":78,"owner_company":68,"owner_location":68,"owner_email":68,"owner_twitter":68,"owner_website":79,"owner_url":80,"languages":81,"stars":90,"forks":91,"last_commit_at":92,"license":93,"difficulty_score":45,"env_os":94,"env_gpu":95,"env_ram":96,"env_deps":97,"category_tags":106,"github_topics":68,"view_count":10,"oss_zip_url":68,"oss_zip_packed_at":68,"status":16,"created_at":107,"updated_at":108,"faqs":109,"releases":130},952,"google-deepmind\u002Fsynthid-text","synthid-text",null,"synthid-text是Google DeepMind推出的AI文本水印研究工具，用于为AI生成内容嵌入隐形水印并验证来源。它解决了当前AI生成文本难以溯源的问题，帮助识别虚假信息，维护内容真实性。基于Nature论文实现，支持Gemma和GPT-2模型，提供两种检测方法：无需训练的加权平均检测器和需训练的贝叶斯检测器。开发者可通过Colab或本地环境快速部署，但需注意当前版本仅限学术研究，不适用于生产系统。技术亮点在于水印配置通过keys参数灵活控制各模型层的处理方式，以及Hugging Face生态无缝集成，不过哈希函数不具备密码学安全性。适合AI安全研究人员使用，普通用户无需直接操作。","# SynthID Text\n\nThis repository provides a reference implementation of the SynthID Text\nwatermarking and detection capabilities for the [research paper][nature-paper]\npublished in _Nature_. It is not intended for production use. The core library\nis [distributed on PyPI][synthid-pypi] for easy installation in the\n[Python Notebook example][synthid-colab], which demonstrates how to apply these\ntools with the [Gemma][gemma] and [GPT-2][gpt2] models.\n\n## Installation and usage\n\nThe [Colab Notebook][synthid-colab] is self-contained reference implementation\nthat:\n\n1.  Extends the [`GemmaForCausalLM`][transformers-gemma] and\n    [`GPT2LMHeadModel`][transformers-gpt2] classes from\n    [Hugging Face Transformers][transformers] with a [mix-in][synthid-mixin] to\n    enable watermarking text content generated by models running in\n    [PyTorch][pytorch]; and\n1.  Detects the watermark. This can be done either with the simple [Weighted Mean\n    detector][synthid-detector-mean] which requires no training, or with the\n    more powerful [Bayesian detector][synthid-detector-bayesian] that requires\n    [training][synthid-detector-trainer]. If using the [Weighted Mean\n    detector][synthid-detector-mean] approach across texts of varying token lengths,\n    we recommend empirically\u002Ftheoretically computing the thresholds at the desired\n    false positives rate at specific token lengths, or using a weighted\n    frequentist approach as described in Appendix A.3.1.\n\nThe notebook is designed to be run end-to-end with either a Gemma or GPT-2\nmodel, and runs best on the following runtime hardware, some of which may\nrequire a [Colab Subscription][colab-subscriptions].\n\n*   Gemma v1.0 2B IT: Use a GPU with 16GB of memory, such as a T4.\n*   Gemma v1.0 7B IT: Use a GPU with 32GB of memory, such as an A100.\n*   GPT-2: Any runtime will work, though a High-RAM CPU or any GPU will be\n    faster.\n\nNOTE: This implementation is for reference and research reproducibility purposes\nonly. Due to minor variations in Gemma and Mistral models across\nimplementations, we expect minor fluctuations in the detectability and\nperplexity results obtained from this repository versus those reported in the\npaper. The subclasses introduced herein are not designed to be used in\nproduction systems. Check out the official SynthID Text implementation in\n[Hugging Face Transformers][transformers-blog] for a production-ready\nimplementation.\n\nNOTE: The `synthid_text.hashing_function.accumulate_hash()` function, used while\ncomputing G values in this reference implementation, does not provide any\nguarantees of cryptographic security.\n\n### Local notebook use\n\nThe notebook can also be used locally if installed from source. Using a virtual\nenvironment is highly recommended for any local use.\n\n```shell\n# Create and activate the virtual environment\npython3 -m venv ~\u002F.venvs\u002Fsynthid\nsource ~\u002F.venvs\u002Fsynthid\u002Fbin\u002Factivate\n\n# Download and install SynthID Text and Jupyter\ngit clone https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text.git\ncd synthid-text\npip install '.[notebook-local]'\n\n# Start the Jupyter server\npython -m notebook\n```\n\nOnce your kernel is running navigate to .pynb file to execute.\n\n### Running the tests\n\nThe source installation also includes a small test suite to verify that the\nlibrary is working as expected.\n\n```shell\n# Create and activate the virtual environment\npython3 -m venv ~\u002F.venvs\u002Fsynthid\nsource ~\u002F.venvs\u002Fsynthid\u002Fbin\u002Factivate\n\n# Download and install SynthID Text with test dependencies from source\ngit clone https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text.git\ncd synthid-text\npip install '.[test]'\n\n# Run the test suite\npytest .\n```\n\n## How it works\n\n### Defining a watermark configuration\n\nSynthID Text produces unique watermarks given a configuration, with the most\nimportant piece of these configurations being the `keys`: a sequence of unique\nintegers where `len(keys)` corresponds to the number of layers in the\nwatermarking or detection models.\n\nThe structure of a configuration is described in the following `TypedDict`\nsubclass, though in practice, the [mixin][synthid-mixin] class in this library\nuses a static configuration.\n\n```python\nfrom collections.abc import Sequence\nfrom typing import TypedDict\n\nimport torch\n\n\nclass WatermarkingConfig(TypedDict):\n    ngram_len: int\n    keys: Sequence[int]\n    sampling_table_size: int\n    sampling_table_seed: int\n    context_history_size: int\n    device: torch.device\n```\n\n### Applying a watermark\n\nWatermarks are applied by a [mix-in][synthid-mixin] class that wraps the\n[`GemmaForCausalLM`][transformers-gemma] and\n[`GPT2LMHeadModel`][transformers-gpt2] classes from Transformers, which results\nin two subclasses with the same API that you are used to from Transformers.\nRemember that the mix-in provided by this library uses a static watermarking\nconfiguration, making it unsuitable for production use.\n\n```python\nfrom synthid_text import synthid_mixin\nimport transformers\nimport torch\n\n\nDEVICE = (\n    torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')\n)\nINPUTS = [\n    \"I enjoy walking with my cute dog\",\n    \"I am from New York\",\n    \"The test was not so very hard after all\",\n    \"I don't think they can score twice in so short a time\",\n]\nMODEL_NAME = 'google\u002Fgemma-2b-it'\nTEMPERATURE = 0.5\nTOP_K = 40\nTOP_P = 0.99\n\n# Initialize a standard tokenizer from Transformers.\ntokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_NAME)\n# Initialize a SynthID Text-enabled model.\nmodel = synthid_mixin.SynthIDGemmaForCausalLM.from_pretrained(\n    MODEL_NAME,\n    device_map='auto',\n    torch_dtype=torch.bfloat16,\n)\n# Prepare your inputs in the usual way.\ninputs = tokenizer(\n    INPUTS,\n    return_tensors='pt',\n    padding=True,\n).to(DEVICE)\n# Generate watermarked text.\noutputs = model.generate(\n    **inputs,\n    do_sample=True,\n    max_length=1024,\n    temperature=TEMPERATURE,\n    top_k=TOP_K,\n    top_p=TOP_P,\n)\n```\n\n### Detecting a watermark\n\nWatermark detection can be done using a variety of scoring functions (see\npaper). This repository contains code for the Mean, Weighted Mean, and Bayesian\nscoring functions described in the paper. The colab contains examples for how\nto use these scoring functions.\n\nThe Bayesian detector must be trained on watermarked and unwatermarked data\nbefore it can be used. The Bayesian detector must be trained for each unique\nwatermarking key, and the training data used for this detector model should be\nindependent from, but representative of the expected character and quality of\nthe text content the system will generate in production.\n\n```python\nimport jax.numpy as jnp\nfrom synthid_text import train_detector_bayesian\n\n\ndef load_data():\n  # Get your training and test data into the system.\n  pass\n\n\ndef process_training_data(split):\n  # Get the G values, masks, and labels for the provided split.\n  pass\n\n\ntrain_split, test_split = load_data()\ntrain_g_values, train_masks, train_labels = process_training_data(train_split)\ntest_g_values, test_masks, test_labels = process_training_data(test_split)\n\ndetector, loss = train_detector_bayesian.optimize_model(\n    jnp.squeeze(train_g_values),\n    jnp.squeeze(train_masks),\n    jnp.squeeze(train_labels),\n    jnp.squeeze(test_g_values),\n    jnp.squeeze(test_masks),\n    jnp.squeeze(test_labels),\n)\n```\n\nOnce the Bayesian detector is trained, use the `detector.score()` function to\ngenerate a per-example score indicating if the text was generated with the given\nwatermarking configuration. Score values will be between 0 and 1, with scores\ncloser to 1 indicating higher likelihood that the text was generated with the\ngiven watermark. You can adjust the acceptance threshold to your needs.\n\n```python\nfrom synthid_text import logits_processing\n\n\nCONFIG = synthid_mixin.DEFAULT_WATERMARKING_CONFIG\n\nlogits_processor = logits_processing.SynthIDLogitsProcessor(\n    **CONFIG, top_k=TOP_K, temperature=TEMPERATURE\n)\n\n# Get only the generated text from the models predictions.\noutputs = outputs[:, inputs_len:]\n\n# Compute the end-of-sequence mask, skipping first ngram_len - 1 tokens\n# \u003Cbool>[batch_size, output_len]\neos_token_mask = logits_processor.compute_eos_token_mask(\n    input_ids=outputs,\n    eos_token_id=tokenizer.eos_token_id,\n)[:, CONFIG['ngram_len'] - 1 :]\n# Compute the context repetition mask\n# \u003Cbool>[batch_size, output_len - (ngram_len - 1)]\ncontext_repetition_mask = logits_processor.compute_context_repetition_mask(\n    input_ids=outputs\n)\n\n# Compute the mask that isolates the generated text.\ncombined_mask = context_repetition_mask * eos_token_mask\n# Compute the G values for the generated text.\ng_values = logits_processor.compute_g_values(input_ids=outputs)\n\n# Score the G values, given the combined mask, and output a per-example score\n# indicating whether the\ndetector.score(g_values.cpu().numpy(), combined_mask.cpu().numpy())\n```\n\n\n## Human Data\n\nWe release the human evaluation data, where we compare watermarked text against unwatermarked text generated from the Gemma 7B model.\nThe data is located in `data\u002Fhuman_eval.jsonl`.\nTo get the prompts used for generating the responses, please use the following code.\n\n```python\nimport json\nimport tensorflow_datasets as tfds\n\nds = tfds.load('huggingface:eli5\u002FLFQA_reddit', split='test_eli5')\nid_to_prompt = {}\nfor x in ds.as_numpy_iterator():\n  id_to_prompt[x['q_id'].decode()] = x['title'].decode()\n\nfull_data = []\nwith open('.\u002Fdata\u002Fhuman_eval.jsonl') as f:\n  for json_str in f:\n    x = json.loads(json_str)\n    x['question'] = id_to_prompt[x['q_id']]\n    full_data.append(x)\n```\n\n## Citing this work\n\n```bibtex\n@article{Dathathri2024,\n    author={Dathathri, Sumanth and See, Abigail and Ghaisas, Sumedh and Huang, Po-Sen and McAdam, Rob and Welbl, Johannes and Bachani, Vandana and Kaskasoli, Alex and Stanforth, Robert and Matejovicova, Tatiana and Hayes, Jamie and Vyas, Nidhi and Merey, Majd Al and Brown-Cohen, Jonah and Bunel, Rudy and Balle, Borja and Cemgil, Taylan and Ahmed, Zahra and Stacpoole, Kitty and Shumailov, Ilia and Baetu, Ciprian and Gowal, Sven and Hassabis, Demis and Kohli, Pushmeet},\n    title={Scalable watermarking for identifying large language model outputs},\n    journal={Nature},\n    year={2024},\n    month={Oct},\n    day={01},\n    volume={634},\n    number={8035},\n    pages={818-823},\n    issn={1476-4687},\n    doi={10.1038\u002Fs41586-024-08025-4},\n    url={https:\u002F\u002Fdoi.org\u002F10.1038\u002Fs41586-024-08025-4}\n}\n```\n\n## License and disclaimer\n\nCopyright 2024 DeepMind Technologies Limited\n\nAll software is licensed under the Apache License, Version 2.0 (Apache 2.0);\nyou may not use this file except in compliance with the Apache 2.0 license.\nYou may obtain a copy of the Apache 2.0 license at:\nhttps:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nAll other materials are licensed under the Creative Commons Attribution 4.0\nInternational License (CC-BY). You may obtain a copy of the CC-BY license at:\nhttps:\u002F\u002Fcreativecommons.org\u002Flicenses\u002Fby\u002F4.0\u002Flegalcode\n\nUnless required by applicable law or agreed to in writing, all software and\nmaterials distributed here under the Apache 2.0 or CC-BY licenses are\ndistributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,\neither express or implied. See the licenses for the specific language governing\npermissions and limitations under those licenses.\n\nThis is not an official Google product.\n\n[colab-subscriptions]: https:\u002F\u002Fcolab.research.google.com\u002Fsignup\n[flax]: https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fflax\n[gemma]: https:\u002F\u002Fai.google.dev\u002Fgemma\u002Fdocs\u002Fmodel_card\n[gpt2]: https:\u002F\u002Fhuggingface.co\u002Fopenai-community\u002Fgpt2\n[jax]: https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fjax\n[pytorch]: https:\u002F\u002Fpytorch.org\u002F\n[nature-paper]: https:\u002F\u002Fdoi.org\u002F10.1038\u002Fs41586-024-08025-4\n[synthid-colab]: https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002Fgoogle-deepmind\u002Fsynthid-text\u002Fblob\u002Fmain\u002Fnotebooks\u002Fsynthid_text_huggingface_integration.ipynb\n[synthid-pypi]: https:\u002F\u002Fpypi.org\u002Fproject\u002Fsynthid-text\u002F\n[synthid-detector-bayesian]: .\u002Fsrc\u002Fsynthid_text\u002Fdetector_bayesian.py\n[synthid-detector-mean]: .\u002Fsrc\u002Fsynthid_text\u002Fdetector_mean.py\n[synthid-detector-trainer]: .\u002Fsrc\u002Fsynthid_text\u002Ftrain_detector_bayesian.py\n[synthid-mixin]: .\u002Fsrc\u002Fsynthid_text\u002Fsynthid_mixin.py\n[transformers]: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\n[transformers-blog]: huggingface.co\u002Fblog\u002Fsynthid-text\n[transformers-gemma]: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fe55b33ceb4b0ba3c8c11f20b6e8d6ca4b48246d4\u002Fsrc\u002Ftransformers\u002Fmodels\u002Fgemma\u002Fmodeling_gemma.py#L996\n[transformers-gpt2]: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fe55b33ceb4b0ba3c8c11f20b6e8d6ca4b48246d4\u002Fsrc\u002Ftransformers\u002Fmodels\u002Fgpt2\u002Fmodeling_gpt2.py#L1185\n","# SynthID 文本\n\n本仓库提供了针对发表于《自然》杂志的[研究论文][nature-paper]中SynthID文本水印与检测功能的参考实现。此实现并非用于生产环境。核心库已[通过PyPI分发][synthid-pypi]，便于在[Python Notebook示例][synthid-colab]中安装使用。该示例演示了如何将这些工具应用于[Gemma][gemma]和[GPT-2][gpt2]模型。\n\n## 安装与使用\n\n[Colab Notebook][synthid-colab]是一个自包含的参考实现，它：\n\n1. 扩展了来自[Hugging Face Transformers][transformers]的[`GemmaForCausalLM`][transformers-gemma]和[`GPT2LMHeadModel`][transformers-gpt2]类，并引入了一个[混入][synthid-mixin]，以支持对由运行在[PyTorch][pytorch]中的模型生成的文本内容进行水印标记；以及\n1. 检测水印。这可以通过简单的[加权平均检测器][synthid-detector-mean]完成，该检测器无需训练；也可以通过更强大的[贝叶斯检测器][synthid-detector-bayesian]完成，后者需要[训练][synthid-detector-trainer]。如果使用[加权平均检测器][synthid-detector-mean]处理不同长度的文本，我们建议根据经验或理论，在特定的词元长度下计算所需的误报率阈值，或者采用附录A.3.1中描述的加权频率学方法。\n\n该Notebook旨在与Gemma或GPT-2模型一起端到端运行，最佳运行环境如下，其中一些可能需要[Colab订阅][colab-subscriptions]。\n\n*   Gemma v1.0 2B IT：使用具有16GB内存的GPU，例如T4。\n*   Gemma v1.0 7B IT：使用具有32GB内存的GPU，例如A100。\n*   GPT-2：任何运行环境均可，不过高内存CPU或任何GPU会更快。\n\n注意：此实现仅用于参考和研究可重复性目的。由于不同实现中Gemma和Mistral模型存在细微差异，我们预计从本仓库获得的检测能力和困惑度结果会与论文中报告的结果略有波动。此处引入的子类并不适合用于生产系统。请查看[Hugging Face Transformers][transformers-blog]中的官方SynthID文本实现，以获取生产就绪的版本。\n\n注意：在本参考实现中计算G值时所用的`synthid_text.hashing_function.accumulate_hash()`函数，并不提供任何密码学安全性的保证。\n\n### 本地Notebook使用\n\n如果从源码安装，该Notebook也可在本地使用。强烈建议为任何本地使用创建并激活虚拟环境。\n\n```shell\n# 创建并激活虚拟环境\npython3 -m venv ~\u002F.venvs\u002Fsynthid\nsource ~\u002F.venvs\u002Fsynthid\u002Fbin\u002Factivate\n\n# 克隆并安装SynthID文本及Jupyter\ngit clone https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text.git\ncd synthid-text\npip install '.[notebook-local]'\n\n# 启动Jupyter服务器\npython -m notebook\n```\n\n一旦内核运行，导航至.pynb文件即可执行。\n\n### 运行测试\n\n源码安装还包含一个小测试套件，用于验证库是否正常工作。\n\n```shell\n# 创建并激活虚拟环境\npython3 -m venv ~\u002F.venvs\u002Fsynthid\nsource ~\u002F.venvs\u002Fsynthid\u002Fbin\u002Factivate\n\n# 从源码下载并安装带测试依赖的SynthID文本\ngit clone https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text.git\ncd synthid-text\npip install '.[test]'\n\n# 运行测试套件\npytest .\n```\n\n## 工作原理\n\n### 定义水印配置\n\nSynthID文本会根据配置生成唯一的水印，其中最重要的部分是`keys`：一个唯一整数序列，其长度对应于水印或检测模型中的层数。\n\n配置的结构在以下`TypedDict`子类中描述，但实际上，本库中的[混入][synthid-mixin]类使用的是静态配置。\n\n```python\nfrom collections.abc import Sequence\nfrom typing import TypedDict\n\nimport torch\n\n\nclass WatermarkingConfig(TypedDict):\n    ngram_len: int\n    keys: Sequence[int]\n    sampling_table_size: int\n    sampling_table_seed: int\n    context_history_size: int\n    device: torch.device\n```\n\n### 应用水印\n\n水印由一个[混入][synthid-mixin]类应用，该类包装了Transformers中的[`GemmaForCausalLM`][transformers-gemma]和[`GPT2LMHeadModel`][transformers-gpt2]类，从而生成两个具有与Transformers相同API的子类。请注意，本库提供的混入使用的是静态水印配置，因此不适合用于生产环境。\n\n```python\nfrom synthid_text import synthid_mixin\nimport transformers\nimport torch\n\n\nDEVICE = (\n    torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')\n)\nINPUTS = [\n    \"我享受和我的可爱小狗一起散步\",\n    \"我来自纽约\",\n    \"其实这次考试并没有那么难\",\n    \"我觉得他们不可能在这么短的时间内连进两球\",\n]\nMODEL_NAME = 'google\u002Fgemma-2b-it'\nTEMPERATURE = 0.5\nTOP_K = 40\nTOP_P = 0.99\n\n# 初始化标准的Transformers分词器。\ntokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_NAME)\n# 初始化启用SynthID文本功能的模型。\nmodel = synthid_mixin.SynthIDGemmaForCausalLM.from_pretrained(\n    MODEL_NAME,\n    device_map='auto',\n    torch_dtype=torch.bfloat16,\n)\n# 按照常规方式准备输入。\ninputs = tokenizer(\n    INPUTS,\n    return_tensors='pt',\n    padding=True,\n).to(DEVICE)\n# 生成带水印的文本。\noutputs = model.generate(\n    **inputs,\n    do_sample=True,\n    max_length=1024,\n    temperature=TEMPERATURE,\n    top_k=TOP_K,\n    top_p=TOP_P,\n)\n```\n\n### 检测水印\n\n水印检测可采用多种评分函数（参见论文）。本仓库包含论文中所述的均值、加权均值和贝叶斯评分函数的代码。Colab 中提供了这些评分函数的使用示例。\n\n贝叶斯检测器必须先用带水印和无水印的数据进行训练，才能投入使用。每个唯一的水印密钥都需单独训练贝叶斯检测器，且用于此检测器模型的训练数据应独立于系统在生产环境中生成的文本内容，但需能代表该内容的特征与质量。\n\n```python\nimport jax.numpy as jnp\nfrom synthid_text import train_detector_bayesian\n\n\ndef load_data():\n  # 将您的训练数据和测试数据加载到系统中。\n  pass\n\n\ndef process_training_data(split):\n  # 获取所提供分割的 G 值、掩码和标签。\n  pass\n\n\ntrain_split, test_split = load_data()\ntrain_g_values, train_masks, train_labels = process_training_data(train_split)\ntest_g_values, test_masks, test_labels = process_training_data(test_split)\n\ndetector, loss = train_detector_bayesian.optimize_model(\n    jnp.squeeze(train_g_values),\n    jnp.squeeze(train_masks),\n    jnp.squeeze(train_labels),\n    jnp.squeeze(test_g_values),\n    jnp.squeeze(test_masks),\n    jnp.squeeze(test_labels),\n)\n```\n\n贝叶斯检测器训练完成后，使用 `detector.score()` 函数为每个样本生成评分，以指示该文本是否由给定的水印配置生成。评分值介于 0 和 1 之间，越接近 1 表示文本由给定水印生成的可能性越高。您可以根据需要调整接受阈值。\n\n```python\nfrom synthid_text import logits_processing\n\n\nCONFIG = synthid_mixin.DEFAULT_WATERMARKING_CONFIG\n\nlogits_processor = logits_processing.SynthIDLogitsProcessor(\n    **CONFIG, top_k=TOP_K, temperature=TEMPERATURE\n)\n\n# 仅从模型预测中获取生成的文本。\noutputs = outputs[:, inputs_len:]\n\n# 计算序列结束掩码，跳过前 ngram_len - 1 个标记\n# \u003Cbool>[batch_size, output_len]\neos_token_mask = logits_processor.compute_eos_token_mask(\n    input_ids=outputs,\n    eos_token_id=tokenizer.eos_token_id,\n)[:, CONFIG['ngram_len'] - 1 :]\n# 计算上下文重复掩码\n# \u003Cbool>[batch_size, output_len - (ngram_len - 1)]\ncontext_repetition_mask = logits_processor.compute_context_repetition_mask(\n    input_ids=outputs\n)\n\n# 计算隔离生成文本的掩码。\ncombined_mask = context_repetition_mask * eos_token_mask\n# 计算生成文本的 G 值。\ng_values = logits_processor.compute_g_values(input_ids=outputs)\n\n# 根据组合掩码对 G 值评分，并输出每个样本的评分\n# 以指示文本是否由给定水印生成\ndetector.score(g_values.cpu().numpy(), combined_mask.cpu().numpy())\n```\n\n\n## 人类数据\n\n我们发布了人类评估数据，其中我们将带水印的文本与由 Gemma 7B 模型生成的无水印文本进行了对比。数据位于 `data\u002Fhuman_eval.jsonl` 中。要获取用于生成回复的提示，请使用以下代码。\n\n```python\nimport json\nimport tensorflow_datasets as tfds\n\nds = tfds.load('huggingface:eli5\u002FLFQA_reddit', split='test_eli5')\nid_to_prompt = {}\nfor x in ds.as_numpy_iterator():\n  id_to_prompt[x['q_id'].decode()] = x['title'].decode()\n\nfull_data = []\nwith open('.\u002Fdata\u002Fhuman_eval.jsonl') as f:\n  for json_str in f:\n    x = json.loads(json_str)\n    x['question'] = id_to_prompt[x['q_id']]\n    full_data.append(x)\n```\n\n## 引用本工作\n\n```bibtex\n@article{Dathathri2024,\n    author={Dathathri, Sumanth and See, Abigail and Ghaisas, Sumedh and Huang, Po-Sen and McAdam, Rob and Welbl, Johannes and Bachani, Vandana and Kaskasoli, Alex and Stanforth, Robert and Matejovicova, Tatiana and Hayes, Jamie and Vyas, Nidhi and Merey, Majd Al and Brown-Cohen, Jonah and Bunel, Rudy and Balle, Borja and Cemgil, Taylan and Ahmed, Zahra and Stacpoole, Kitty and Shumailov, Ilia and Baetu, Ciprian and Gowal, Sven and Hassabis, Demis and Kohli, Pushmeet},\n    title={面向大规模语言模型输出的可扩展水印技术},\n    journal={自然},\n    year={2024},\n    month={10},\n    day={1},\n    volume={634},\n    number={8035},\n    pages={818-823},\n    issn={1476-4687},\n    doi={10.1038\u002Fs41586-024-08025-4},\n    url={https:\u002F\u002Fdoi.org\u002F10.1038\u002Fs41586-024-08025-4}\n}\n```\n\n## 许可与免责声明\n\n版权所有 2024 DeepMind Technologies Limited\n\n所有软件均依据 Apache 许可协议第 2.0 版（Apache 2.0）授权；您不得在未遵守 Apache 2.0 许可的情况下使用本文件。您可在此处获取 Apache 2.0 许可的副本：\nhttps:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\n所有其他材料均依据知识共享署名 4.0 国际许可协议（CC-BY）授权。您可在此处获取 CC-BY 许可的副本：\nhttps:\u002F\u002Fcreativecommons.org\u002Flicenses\u002Fby\u002F4.0\u002Flegalcode\n\n除非适用法律要求或书面同意，此处依据 Apache 2.0 或 CC-BY 许可分发的所有软件和材料均按“原样”提供，不附带任何明示或暗示的保证或条件。请参阅相关许可，了解这些许可所规定的具体权限和限制。\n\n这不是谷歌的官方产品。\n\n[colab-subscriptions]: https:\u002F\u002Fcolab.research.google.com\u002Fsignup\n[flax]: https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fflax\n[gemma]: https:\u002F\u002Fai.google.dev\u002Fgemma\u002Fdocs\u002Fmodel_card\n[gpt2]: https:\u002F\u002Fhuggingface.co\u002Fopenai-community\u002Fgpt2\n[jax]: https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fjax\n[pytorch]: https:\u002F\u002Fpytorch.org\u002F\n[nature-paper]: https:\u002F\u002Fdoi.org\u002F10.1038\u002Fs41586-024-08025-4\n[synthid-colab]: https:\u002F\u002Fcolab.research.google.com\u002Fgithub\u002Fgoogle-deepmind\u002Fsynthid-text\u002Fblob\u002Fmain\u002Fnotebooks\u002Fsynthid_text_huggingface_integration.ipynb\n[synthid-pypi]: https:\u002F\u002Fpypi.org\u002Fproject\u002Fsynthid-text\u002F\n[synthid-detector-bayesian]: .\u002Fsrc\u002Fsynthid_text\u002Fdetector_bayesian.py\n[synthid-detector-mean]: .\u002Fsrc\u002Fsynthid_text\u002Fdetector_mean.py\n[synthid-detector-trainer]: .\u002Fsrc\u002Fsynthid_text\u002Ftrain_detector_bayesian.py\n[synthid-mixin]: .\u002Fsrc\u002Fsynthid_text\u002Fsynthid_mixin.py\n[transformers]: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\n[transformers-blog]: huggingface.co\u002Fblog\u002Fsynthid-text\n[transformers-gemma]: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fe55b33ceb4b0ba3c8c11f20b6e8d6ca4b48246d4\u002Fsrc\u002Ftransformers\u002Fmodels\u002Fgemma\u002Fmodeling_gemma.py#L996\n[transformers-gpt2]: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\u002Fblob\u002Fe55b33ceb4b0ba3c8c11f20b6e8d6ca4b48246d4\u002Fsrc\u002Ftransformers\u002Fmodels\u002Fgpt2\u002Fmodeling_gpt2.py#L1185","# SynthID Text 快速上手指南\n\n## 环境准备\n- Python 3.7+\n- NVIDIA GPU（推荐配置）：\n  - Gemma 2B 模型：16GB 显存（如 T4）\n  - Gemma 7B 模型：32GB 显存（如 A100）\n  - GPT-2 模型：CPU 或任意 GPU 均可\n- PyTorch（安装时自动处理依赖）\n\n## 安装步骤\n使用清华源加速安装：\n\n```shell\npip install -i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple synthid-text\n```\n\n## 基本使用\n以下示例使用 Gemma 2B 模型生成水印文本：\n\n```python\nimport torch\nfrom synthid_text import synthid_mixin\nimport transformers\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ninput_texts = [\"Hello, world!\"]\nmodel_name = \"google\u002Fgemma-2b-it\"\n\ntokenizer = transformers.AutoTokenizer.from_pretrained(model_name)\nmodel = synthid_mixin.SynthIDGemmaForCausalLM.from_pretrained(\n    model_name, \n    device_map=\"auto\", \n    torch_dtype=torch.bfloat16\n)\ninputs = tokenizer(input_texts, return_tensors=\"pt\", padding=True).to(device)\noutputs = model.generate(**inputs, max_length=100)\nprint(tokenizer.decode(outputs[0]))\n```","某新闻媒体公司内容审核团队每天处理数千条用户投稿，需快速识别AI生成的虚假新闻以保障信息真实性。  \n\n### 没有 synthid-text 时  \n- 无法可靠区分AI生成内容和人工创作，导致虚假新闻误判或漏判  \n- 人工逐条审核耗时，每篇需10-15分钟，团队效率低下且易疲劳出错  \n- 依赖简单关键词过滤，误报率高达30%，真实内容常被错误标记  \n- 现有检测工具对水印不敏感，AI生成内容易被绕过，虚假信息传播风险高  \n\n### 使用 synthid-text 后  \n- 自动添加并检测文本水印，精准识别AI生成来源，准确率提升至95%以上  \n- 集成到审核系统实时扫描，处理速度提升10倍，每篇仅需1分钟  \n- 人工审核仅需抽查可疑内容，团队专注高价值任务，工作负担大幅降低  \n- 水印检测抗干扰性强，误报率降至5%以下，虚假信息传播风险显著减少  \n\nsynthid-text 为内容审核提供了可靠高效的水印检测方案，切实保障了信息生态安全。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Fgoogle-deepmind_synthid-text_2dafb02d.png","google-deepmind","Google DeepMind","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002Fgoogle-deepmind_06b1dd17.png","","https:\u002F\u002Fwww.deepmind.com\u002F","https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind",[82,86],{"name":83,"color":84,"percentage":85},"Python","#3572A5",70.4,{"name":87,"color":88,"percentage":89},"Jupyter Notebook","#DA5B0B",29.6,816,81,"2026-04-04T16:19:00","Apache-2.0","Linux, macOS","需要NVIDIA GPU，Gemma 2B需16GB+，Gemma 7B需32GB+，GPT-2可使用CPU","16GB+",{"notes":98,"python":99,"dependencies":100},"建议使用虚拟环境，首次运行需下载约5GB模型文件","3.8+",[101,102,103,104,105],"torch>=2.0","transformers>=4.30","accelerate","jax","flax",[15],"2026-03-27T02:49:30.150509","2026-04-06T05:17:19.900363",[110,115,120,125],{"id":111,"question_zh":112,"answer_zh":113,"source_url":114},8787,"使用GPT-2模型时出现'pad_token_id must be set in generation_config'错误，如何解决？","设置 model.generation_config.pad_token_id = model.generation_config.eos_token_id。因为GPT-2的generation_config中默认pad_token_id未设置，需显式设置为eos_token_id。维护者建议在Colab中自动处理此问题。","https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text\u002Fissues\u002F12",{"id":116,"question_zh":117,"answer_zh":118,"source_url":119},8788,"水印检测时，水印和非水印文本的均分相同，如何正确设置阈值？","不要使用0.5作为阈值，因为未水印样本会随机分布；建议使用z-scores进行统计分析。检测时必须使用与生成文本相同的tokenizer，否则结果可能不准确。","https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text\u002Fissues\u002F7",{"id":121,"question_zh":122,"answer_zh":123,"source_url":124},8789,"代码实现与论文描述的tournament watermarking不一致，是否只简单提升g=1的token概率？","代码正确实现了tournament watermarking。第一轮token从LLM分布采样，平局时随机选择；分布计算在论文附录E中有详细说明，代码中的update_scores函数对应此分布。","https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text\u002Fissues\u002F17",{"id":126,"question_zh":127,"answer_zh":128,"source_url":129},8790,"在多GPU环境下，模型设备设置不正确，如何处理？","此代码库不支持多GPU配置，仅专注于论文复现，假设用户使用单GPU。建议将device_map设置为'auto'或指定单个GPU（如'cuda:0'），但不要期望多GPU支持。","https:\u002F\u002Fgithub.com\u002Fgoogle-deepmind\u002Fsynthid-text\u002Fissues\u002F14",[131,136,141,146],{"id":132,"version":133,"summary_zh":134,"released_at":135},103653,"0.2.1","Bug fix for #12. Only raise a `ValueError` if `has_eos_stopping_criteria and pad_token_id is None`.","2024-11-14T14:46:07",{"id":137,"version":138,"summary_zh":139,"released_at":140},103654,"0.2","Minor update to ensure Py3.9 compatibility and reliability more generally.\r\n\r\n* #5 fixed a bunch of typos (@jsoref, first time contributor)\r\n* #6 improved the reliability of device comparisons and raises an error when using this with Llama models (@robertsonwang, first time contributor)\r\n* #11 ensures Py3.9 compatibility and adds CI for testing (@RyanMullins)","2024-11-12T15:57:28",{"id":142,"version":143,"summary_zh":144,"released_at":145},103655,"0.1.1","Patch update to remove requirements.txt in favor of the `deps=` property in the pyproject.toml and update the README with permalinks to relevant articles, code, and examples.","2024-11-12T15:51:35",{"id":147,"version":148,"summary_zh":149,"released_at":150},103656,"0.1","Initial release of the SynthID Text code to accompany the [research paper](https:\u002F\u002Fdoi.org\u002F10.1038\u002Fs41586-024-08025-4) in _Nature_.","2024-11-12T15:48:33"]