[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"similar-langchain-ai--agentevals":3,"tool-langchain-ai--agentevals":62},[4,18,26,36,46,54],{"id":5,"name":6,"github_repo":7,"description_zh":8,"stars":9,"difficulty_score":10,"last_commit_at":11,"category_tags":12,"status":17},4358,"openclaw","openclaw\u002Fopenclaw","OpenClaw 是一款专为个人打造的本地化 AI 助手，旨在让你在自己的设备上拥有完全可控的智能伙伴。它打破了传统 AI 助手局限于特定网页或应用的束缚，能够直接接入你日常使用的各类通讯渠道，包括微信、WhatsApp、Telegram、Discord、iMessage 等数十种平台。无论你在哪个聊天软件中发送消息，OpenClaw 都能即时响应，甚至支持在 macOS、iOS 和 Android 设备上进行语音交互，并提供实时的画布渲染功能供你操控。\n\n这款工具主要解决了用户对数据隐私、响应速度以及“始终在线”体验的需求。通过将 AI 部署在本地，用户无需依赖云端服务即可享受快速、私密的智能辅助，真正实现了“你的数据，你做主”。其独特的技术亮点在于强大的网关架构，将控制平面与核心助手分离，确保跨平台通信的流畅性与扩展性。\n\nOpenClaw 非常适合希望构建个性化工作流的技术爱好者、开发者，以及注重隐私保护且不愿被单一生态绑定的普通用户。只要具备基础的终端操作能力（支持 macOS、Linux 及 Windows WSL2），即可通过简单的命令行引导完成部署。如果你渴望拥有一个懂你",349277,3,"2026-04-06T06:32:30",[13,14,15,16],"Agent","开发框架","图像","数据工具","ready",{"id":19,"name":20,"github_repo":21,"description_zh":22,"stars":23,"difficulty_score":10,"last_commit_at":24,"category_tags":25,"status":17},3808,"stable-diffusion-webui","AUTOMATIC1111\u002Fstable-diffusion-webui","stable-diffusion-webui 是一个基于 Gradio 构建的网页版操作界面，旨在让用户能够轻松地在本地运行和使用强大的 Stable Diffusion 图像生成模型。它解决了原始模型依赖命令行、操作门槛高且功能分散的痛点，将复杂的 AI 绘图流程整合进一个直观易用的图形化平台。\n\n无论是希望快速上手的普通创作者、需要精细控制画面细节的设计师，还是想要深入探索模型潜力的开发者与研究人员，都能从中获益。其核心亮点在于极高的功能丰富度：不仅支持文生图、图生图、局部重绘（Inpainting）和外绘（Outpainting）等基础模式，还独创了注意力机制调整、提示词矩阵、负向提示词以及“高清修复”等高级功能。此外，它内置了 GFPGAN 和 CodeFormer 等人脸修复工具，支持多种神经网络放大算法，并允许用户通过插件系统无限扩展能力。即使是显存有限的设备，stable-diffusion-webui 也提供了相应的优化选项，让高质量的 AI 艺术创作变得触手可及。",162132,"2026-04-05T11:01:52",[14,15,13],{"id":27,"name":28,"github_repo":29,"description_zh":30,"stars":31,"difficulty_score":32,"last_commit_at":33,"category_tags":34,"status":17},1381,"everything-claude-code","affaan-m\u002Feverything-claude-code","everything-claude-code 是一套专为 AI 编程助手（如 Claude Code、Codex、Cursor 等）打造的高性能优化系统。它不仅仅是一组配置文件，而是一个经过长期实战打磨的完整框架，旨在解决 AI 代理在实际开发中面临的效率低下、记忆丢失、安全隐患及缺乏持续学习能力等核心痛点。\n\n通过引入技能模块化、直觉增强、记忆持久化机制以及内置的安全扫描功能，everything-claude-code 能显著提升 AI 在复杂任务中的表现，帮助开发者构建更稳定、更智能的生产级 AI 代理。其独特的“研究优先”开发理念和针对 Token 消耗的优化策略，使得模型响应更快、成本更低，同时有效防御潜在的攻击向量。\n\n这套工具特别适合软件开发者、AI 研究人员以及希望深度定制 AI 工作流的技术团队使用。无论您是在构建大型代码库，还是需要 AI 协助进行安全审计与自动化测试，everything-claude-code 都能提供强大的底层支持。作为一个曾荣获 Anthropic 黑客大奖的开源项目，它融合了多语言支持与丰富的实战钩子（hooks），让 AI 真正成长为懂上",159267,2,"2026-04-17T11:29:14",[14,13,35],"语言模型",{"id":37,"name":38,"github_repo":39,"description_zh":40,"stars":41,"difficulty_score":42,"last_commit_at":43,"category_tags":44,"status":17},8272,"opencode","anomalyco\u002Fopencode","OpenCode 是一款开源的 AI 编程助手（Coding Agent），旨在像一位智能搭档一样融入您的开发流程。它不仅仅是一个代码补全插件，而是一个能够理解项目上下文、自主规划任务并执行复杂编码操作的智能体。无论是生成全新功能、重构现有代码，还是排查难以定位的 Bug，OpenCode 都能通过自然语言交互高效完成，显著减少开发者在重复性劳动和上下文切换上的时间消耗。\n\n这款工具专为软件开发者、工程师及技术研究人员设计，特别适合希望利用大模型能力来提升编码效率、加速原型开发或处理遗留代码维护的专业人群。其核心亮点在于完全开源的架构，这意味着用户可以审查代码逻辑、自定义行为策略，甚至私有化部署以保障数据安全，彻底打破了传统闭源 AI 助手的“黑盒”限制。\n\n在技术体验上，OpenCode 提供了灵活的终端界面（Terminal UI）和正在测试中的桌面应用程序，支持 macOS、Windows 及 Linux 全平台。它兼容多种包管理工具，安装便捷，并能无缝集成到现有的开发环境中。无论您是追求极致控制权的资深极客，还是渴望提升产出的独立开发者，OpenCode 都提供了一个透明、可信",144296,1,"2026-04-16T14:50:03",[13,45],"插件",{"id":47,"name":48,"github_repo":49,"description_zh":50,"stars":51,"difficulty_score":32,"last_commit_at":52,"category_tags":53,"status":17},2271,"ComfyUI","Comfy-Org\u002FComfyUI","ComfyUI 是一款功能强大且高度模块化的视觉 AI 引擎，专为设计和执行复杂的 Stable Diffusion 图像生成流程而打造。它摒弃了传统的代码编写模式，采用直观的节点式流程图界面，让用户通过连接不同的功能模块即可构建个性化的生成管线。\n\n这一设计巧妙解决了高级 AI 绘图工作流配置复杂、灵活性不足的痛点。用户无需具备编程背景，也能自由组合模型、调整参数并实时预览效果，轻松实现从基础文生图到多步骤高清修复等各类复杂任务。ComfyUI 拥有极佳的兼容性，不仅支持 Windows、macOS 和 Linux 全平台，还广泛适配 NVIDIA、AMD、Intel 及苹果 Silicon 等多种硬件架构，并率先支持 SDXL、Flux、SD3 等前沿模型。\n\n无论是希望深入探索算法潜力的研究人员和开发者，还是追求极致创作自由度的设计师与资深 AI 绘画爱好者，ComfyUI 都能提供强大的支持。其独特的模块化架构允许社区不断扩展新功能，使其成为当前最灵活、生态最丰富的开源扩散模型工具之一，帮助用户将创意高效转化为现实。",108322,"2026-04-10T11:39:34",[14,15,13],{"id":55,"name":56,"github_repo":57,"description_zh":58,"stars":59,"difficulty_score":32,"last_commit_at":60,"category_tags":61,"status":17},6121,"gemini-cli","google-gemini\u002Fgemini-cli","gemini-cli 是一款由谷歌推出的开源 AI 命令行工具，它将强大的 Gemini 大模型能力直接集成到用户的终端环境中。对于习惯在命令行工作的开发者而言，它提供了一条从输入提示词到获取模型响应的最短路径，无需切换窗口即可享受智能辅助。\n\n这款工具主要解决了开发过程中频繁上下文切换的痛点，让用户能在熟悉的终端界面内直接完成代码理解、生成、调试以及自动化运维任务。无论是查询大型代码库、根据草图生成应用，还是执行复杂的 Git 操作，gemini-cli 都能通过自然语言指令高效处理。\n\n它特别适合广大软件工程师、DevOps 人员及技术研究人员使用。其核心亮点包括支持高达 100 万 token 的超长上下文窗口，具备出色的逻辑推理能力；内置 Google 搜索、文件操作及 Shell 命令执行等实用工具；更独特的是，它支持 MCP（模型上下文协议），允许用户灵活扩展自定义集成，连接如图像生成等外部能力。此外，个人谷歌账号即可享受免费的额度支持，且项目基于 Apache 2.0 协议完全开源，是提升终端工作效率的理想助手。",100752,"2026-04-10T01:20:03",[45,13,15,14],{"id":63,"github_repo":64,"name":65,"description_en":66,"description_zh":67,"ai_summary_zh":67,"readme_en":68,"readme_zh":69,"quickstart_zh":70,"use_case_zh":71,"hero_image_url":72,"owner_login":73,"owner_name":74,"owner_avatar_url":75,"owner_bio":76,"owner_company":77,"owner_location":77,"owner_email":78,"owner_twitter":74,"owner_website":79,"owner_url":80,"languages":81,"stars":93,"forks":94,"last_commit_at":95,"license":96,"difficulty_score":32,"env_os":76,"env_gpu":97,"env_ram":97,"env_deps":98,"category_tags":104,"github_topics":77,"view_count":32,"oss_zip_url":77,"oss_zip_packed_at":77,"status":17,"created_at":106,"updated_at":107,"faqs":108,"releases":139},8571,"langchain-ai\u002Fagentevals","agentevals","Readymade evaluators for agent trajectories","agentevals 是一套专为智能体（Agent）打造的现成评估工具包，旨在帮助开发者深入理解并优化智能体的运行表现。在大模型赋予智能体自由控制流程的同时，其“黑盒”特性也让开发者难以预判某处改动会对后续步骤产生何种影响。agentevals 正是为了解决这一痛点而生，它聚焦于“智能体轨迹”，即智能体在解决问题过程中所经历的中间步骤，提供了一系列评估器和实用工具，让原本模糊的执行过程变得可量化、可分析。\n\n这款工具特别适合正在构建或调试 Agentic 应用的开发者与研究人员使用。无论是验证智能体调用工具的逻辑是否合理，还是检查多步推理是否存在偏差，agentevals 都能提供快速反馈。其核心亮点在于支持基于大模型的“裁判”机制：用户可以利用如 OpenAI o3-mini 等先进模型，自动对智能体的完整交互轨迹进行打分和理由阐述。通过简单的配置，即可将复杂的执行日志转化为清晰的评估报告，为迭代优化提供坚实依据。作为 LangChain 生态的一部分，它与通用评估库 openevals 互补，是提升智能体可靠性的理想起点。","# 🦾⚖️ AgentEvals\n\n[Agentic applications](https:\u002F\u002Flangchain-ai.github.io\u002Flanggraph\u002Fconcepts\u002Fagentic_concepts\u002F) give an LLM freedom over control flow in order to solve problems. While this freedom\ncan be extremely powerful, the black box nature of LLMs can make it difficult to understand how changes in one part of your agent will affect others downstream.\nThis makes evaluating your agents especially important.\n\nThis package contains a collection of evaluators and utilities for evaluating the performance of your agents, with a focus on **agent trajectory**, or the intermediate steps an agent takes as it runs.\nIt is intended to provide a good conceptual starting point for your agent's evals.\n\nIf you are looking for more general evaluation tools, please check out the companion package [`openevals`](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals).\n\n# Quickstart\n\nTo get started, install `agentevals`:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```bash\npip install agentevals\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```bash\nnpm install agentevals @langchain\u002Fcore\n```\n\u003C\u002Fdetails>\n\nThis quickstart will use an evaluator powered by OpenAI's `o3-mini` model to judge your results, so you'll need to set your OpenAI API key as an environment variable:\n\n```bash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n```\n\nOnce you've done this, you can run your first trajectory evaluator. We represent the agent's trajectory as a list of OpenAI-style messages:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT\n\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    prompt=TRAJECTORY_ACCURACY_PROMPT,\n    model=\"openai:o3-mini\",\n)\n\n# This is a fake trajectory, in reality you would run your agent to get a real trajectory\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\n\neval_result = trajectory_evaluator(\n  outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n  'key': 'trajectory_accuracy',\n  'reasoning': 'The trajectory accurately follows the user's request for weather information in SF. Initially, the assistant recognizes the goal (providing weather details), then it efficiently makes a tool call to get the weather, and finally it communicates the result clearly. All steps demonstrate logical progression and efficiency. Thus, the score should be: true.',\n  'score': true\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  type FlexibleChatCompletionMessage,\n  TRAJECTORY_ACCURACY_PROMPT,\n} from \"agentevals\";\n\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT,\n  model: \"openai:o3-mini\",\n});\n\nconst outputs = [\n  { role: \"user\", content: \"What is the weather in SF?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        },\n      },\n    ],\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n  {\n    role: \"assistant\",\n    content: \"The weather in SF is 80 degrees and sunny.\",\n  },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evalResult = await trajectoryEvaluator({\n  outputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n    key: 'trajectory_accuracy',\n    score: true,\n    comment: '...'\n}\n```\n\u003C\u002Fdetails>\n\nYou can see that the evaluator returns a score of `true` since the overall trajectory is a reasonable path for the agent to take to answer the user's question.\n\nFor more details on this evaluator, including how to customize it, see the section on [trajectory LLM-as-judge](#trajectory-llm-as-judge).\n\n# Table of Contents\n\n- [Installation](#installation)\n- [Evaluators](#evaluators)\n  - [Agent Trajectory Match](#agent-trajectory-match)\n    - [Strict match](#strict-match)\n    - [Unordered match](#unordered-match)\n    - [Subset\u002Fsuperset match](#subset-and-superset-match)\n    - [Tool args match modes](#tool-args-match-modes)\n  - [Trajectory LLM-as-judge](#trajectory-llm-as-judge)\n  - [Graph Trajectory](#graph-trajectory)\n    - [Graph trajectory LLM-as-judge](#graph-trajectory-llm-as-judge)\n    - [Graph trajectory strict match](#graph-trajectory-strict-match)\n- [Python Async Support](#python-async-support)\n- [LangSmith Integration](#langsmith-integration)\n  - [Pytest or Vitest\u002FJest](#pytest-or-vitestjest)\n  - [Evaluate](#evaluate)\n\n# Installation\n\nYou can install `agentevals` like this:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```bash\npip install agentevals\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```bash\nnpm install agentevals @langchain\u002Fcore\n```\n\u003C\u002Fdetails>\n\nFor LLM-as-judge evaluators, you will also need an LLM client. By default, `agentevals` will use [LangChain chat model integrations](https:\u002F\u002Fpython.langchain.com\u002Fdocs\u002Fintegrations\u002Fchat\u002F) and comes with `langchain_openai` installed by default. However, if you prefer, you may use the OpenAI client directly:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```bash\npip install openai\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```bash\nnpm install openai\n```\n\u003C\u002Fdetails>\n\nIt is also helpful to be familiar with some [evaluation concepts](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fconcepts) and\nLangSmith's pytest integration for running evals, which is documented [here](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fpytest).\n\n# Evaluators\n\n## Agent trajectory match\n\nAgent trajectory match evaluators are used to judge the trajectory of an agent's execution either against an expected trajectory or using an LLM.\nThese evaluators expect you to format your agent's trajectory as a list of OpenAI format dicts or as a list of LangChain `BaseMessage` classes, and handle message formatting\nunder the hood.\n\nAgentEvals offers the `create_trajectory_match_evaluator`\u002F`createTrajectoryMatchEvaluator` and `create_async_trajectory_match_evaluator` methods for this task. You can customize their behavior in a few ways:\n\n- Setting `trajectory_match_mode`\u002F`trajectoryMatchMode` to [`strict`](#strict-match), [`unordered`](#unordered-match), [`subset`](#subset-and-superset-match), or [`superset`](#subset-and-superset-match) to provide the general strategy the evaluator will use to compare trajectories\n- Setting [`tool_args_match_mode`](#tool-args-match-modes) and\u002For [`tool_args_match_overrides`](#tool-args-match-modes) to customize how the evaluator considers equality between tool calls in the actual trajectory vs. the reference. By default, only tool calls with the same arguments to the same tool are considered equal.\n\n### Strict match\n\nThe `\"strict\"` `trajectory_match_mode` compares two trajectories and ensures that they contain the same messages\nin the same order with the same tool calls. Note that it does allow for differences in message content:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            },\n            {\n                \"function\": {\n                    \"name\": \"accuweather_forecast\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in San Francisco?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80˚ and sunny.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"strict\"\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_strict_match',\n    'score': False,\n    'comment': None,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"What is the weather in SF?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"San Francisco\" })\n      },\n    }, {\n      function: {\n        name: \"accuweather_forecast\",\n        arguments: JSON.stringify({\"city\": \"San Francisco\"}),\n      },\n    }]\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n  { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"What is the weather in San Francisco?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"San Francisco\" })\n      }\n    }]\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"strict\",\n})\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result);\n```\n\n```\n{\n    'key': 'trajectory_strict_match',\n    'score': false,\n}\n```\n\u003C\u002Fdetails>\n\n`\"strict\"` is useful is if you want to ensure that tools are always called in the same order for a given query (e.g. a company policy lookup tool before a tool that requests vacation time for an employee).\n\n**Note:** If you would like to configure the way this evaluator checks for tool call equality, see [this section](#tool-args-match-modes).\n\n### Unordered match\n\nThe `\"unordered\"` `trajectory_match_mode` compares two trajectories and ensures that they contain the same tool calls in any order. This is useful if you want to allow flexibility in how an agent obtains the proper information, but still do care that all information was retrieved.\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\ninputs = {}\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF and is there anything fun happening?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [{\n            \"function\": {\n                \"name\": \"get_weather\",\n                \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n            }\n        }],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [{\n            \"function\": {\n                \"name\": \"get_fun_activities\",\n                \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n            }\n        }],\n    },\n    {\"role\": \"tool\", \"content\": \"Nothing fun is happening, you should stay indoors and read!\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny, but there is nothing fun happening.\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF and is there anything fun happening?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_fun_activities\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            },\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            },\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"Nothing fun is happening, you should stay indoors and read!\"},\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"In SF, it's 80˚ and sunny, but there is nothing fun happening.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"unordered\"\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_unordered_match',\n    'score': True,\n    'comment': None,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"What is the weather in SF and is there anything fun happening?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"SF\" }),\n      }\n    }],\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_fun_activities\",\n        arguments: JSON.stringify({ city: \"SF\" }),\n      }\n    }],\n  },\n  { role: \"tool\", content: \"Nothing fun is happening, you should stay indoors and read!\" },\n  { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny, but there is nothing fun happening.\" },\n] satisifes FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"What is the weather in SF and is there anything fun happening?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_fun_activities\",\n          arguments: JSON.stringify({ city: \"San Francisco\" }),\n        }\n      },\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"San Francisco\" }),\n        }\n      },\n    ],\n  },\n  { role: \"tool\", content: \"Nothing fun is happening, you should stay indoors and read!\" },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n  { role: \"assistant\", content: \"In SF, it's 80˚ and sunny, but there is nothing fun happening.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"unordered\",\n});\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_unordered_match',\n    'score': true,\n}\n```\n\u003C\u002Fdetails>\n\n`\"unordered\"` is useful is if you want to ensure that specific tools are called at some point in the trajectory, but you don't necessarily need them to be in message order (e.g. the agent called a company policy retrieval tool at an arbitrary point in an interaction before authorizing spend for a pizza party).\n\n**Note:** If you would like to configure the way this evaluator checks for tool call equality, see [this section](#tool-args-match-modes).\n\n### Subset and superset match\n\nThe `\"subset\"` and `\"superset\"` modes match partial trajectories (ensuring that a trajectory contains a subset\u002Fsuperset of tool calls contained in a reference trajectory).\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF and London?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [{\n            \"function\": {\n                \"name\": \"get_weather\",\n                \"arguments\": json.dumps({\"city\": \"SF and London\"}),\n            },\n        }, {\n            \"function\": {\n                \"name\": \"accuweather_forecast\",\n                \"arguments\": json.dumps({\"city\": \"SF and London\"}),\n            }\n        }],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF, and 90 degrees and rainy in London.\"},\n    {\"role\": \"tool\", \"content\": \"Unknown.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny. In London, it's 90 degrees and rainy.\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF and London?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF and London\"}),\n                }\n            },\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco, and 90 degrees and rainy in London.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80˚ and sunny. In London, it's 90˚ and rainy.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"superset\", # or \"subset\"\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_superset_match',\n    'score': True,\n    'comment': None,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"What is the weather in SF and London?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"SF and London\" }),\n      }\n    }, {\n      \"function\": {\n        name: \"accuweather_forecast\",\n        arguments: JSON.stringify({\"city\": \"SF and London\"}),\n      }\n    }],\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF, and 90 degrees and rainy in London.\" },\n  { role: \"tool\", content: \"Unknown.\" },\n  { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny. In London, it's 90 degrees and rainy.\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"What is the weather in SF and London?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF and London\" }),\n        }\n      },\n    ],\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco, and 90 degrees and rainy in London.\" },\n  { role: \"assistant\", content: \"The weather in SF is 80˚ and sunny. In London, it's 90˚ and rainy.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"superset\", \u002F\u002F or \"subset\"\n});\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_superset_match',\n    'score': true,\n}\n```\n\u003C\u002Fdetails>\n\n`\"superset\"` is useful if you want to ensure that some key tools were called at some point in the trajectory, but an agent calling extra tools is still acceptable. `\"subset\"` is the inverse and is useful if you want to ensure that the agent did not call any tools beyond the expected ones.\n\n**Note:** If you would like to configure the way this evaluator checks for tool call equality, see [this section](#tool-args-match-modes).\n\n### Tool args match modes\n\nWhen checking equality between tool calls, the above evaluators will require that all tool call arguments are the exact same by default. You can configure this behavior in the following ways:\n\n- Treating any two tool calls for the same tool as equivalent by setting `tool_args_match_mode=\"ignore\"` (Python) or `toolArgsMatchMode: \"ignore\"` (TypeScript)\n- Treating a tool call as equivalent if it contain as subset\u002Fsuperset of args compared to a reference tool call of the same name with `tool_args_match_mode=\"subset\"\u002F\"superset\"` (Python) or `toolArgsMatchMode: \"subset\"\u002F\"superset` (TypeScript)\n- Setting custom matchers for all calls of a given tool using the `tool_args_match_overrides` (Python) or `toolArgsMatchOverrides` (TypeScript) param\n\nYou can set both of these parameters at the same time. `tool_args_match_overrides` will take precendence over `tool_args_match_mode`.\n\n`tool_args_match_overrides`\u002F`toolArgsMatchOverrides` takes a dictionary whose keys are tool names and whose values are either `\"exact\"`, `\"ignore\"`, a list of fields within the tool call that must match exactly, or a comparator function that takes two arguments and returns whether they are equal:\n\n```python\nToolArgsMatchMode = Literal[\"exact\", \"ignore\", \"subset\", \"superset\"]\n\nToolArgsMatchOverrides = dict[str, Union[ToolArgsMatchMode, list[str],  Callable[[dict, dict], bool]]]\n```\n\nHere's an example that allows case insensitivity for the arguments to a tool named `get_weather`:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"san francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in San Francisco?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80˚ and sunny.\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"strict\",\n    tool_args_match_mode=\"exact\",  # Default value\n    tool_args_match_overrides={\n        \"get_weather\": lambda x, y: x[\"city\"].lower() == y[\"city\"].lower()\n    }\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_strict_match',\n    'score': True,\n    'comment': None,\n}\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"What is the weather in SF?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"san francisco\" })\n      },\n    }]\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n  { role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"What is the weather in San Francisco?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"San Francisco\" })\n      }\n    }]\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco.\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"strict\",\n  toolArgsMatchMode: \"exact\",  \u002F\u002F Default value\n  toolArgsMatchOverrides: {\n    get_weather: (x, y) => {\n      return typeof x.city === \"string\" &&\n        typeof y.city === \"string\" &&\n        x.city.toLowerCase() === y.city.toLowerCase();\n    },\n  }\n});\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result);\n```\n\n```\n{\n  'key': 'trajectory_strict_match',\n  'score': true,\n}\n```\n\n\u003C\u002Fdetails>\n\nThis flexibility allows you to handle cases where you want looser equality for LLM generated arguments (`\"san francisco\"` to equal `\"San Francisco\"`) for only specific tool calls.\n\n## Trajectory LLM-as-judge\n\nThe LLM-as-judge trajectory evaluator that uses an LLM to evaluate the trajectory. Unlike the trajectory match evaluators, it doesn't require a reference trajectory. Here's an example:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT\n\nevaluator = create_trajectory_llm_as_judge(\n  prompt=TRAJECTORY_ACCURACY_PROMPT,\n  model=\"openai:o3-mini\"\n)\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\neval_result = evaluator(\n    outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': True,\n    'comment': 'The provided agent trajectory is reasonable...'\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  TRAJECTORY_ACCURACY_PROMPT,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst evaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT,\n  model: \"openai:o3-mini\",\n});\n\nconst outputs = [\n  {role: \"user\", content: \"What is the weather in SF?\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        }\n      }\n    ],\n  },\n  {role: \"tool\", content: \"It's 80 degrees and sunny in SF.\"},\n  {role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst result = await evaluator({ outputs });\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': True,\n    'comment': 'The provided agent trajectory is reasonable...'\n}\n```\n\u003C\u002Fdetails>\n\nIf you have a reference trajectory, you can add an extra variable to your prompt and pass in the reference trajectory. Below, we use the prebuilt  `TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE` prompt, which contains a `reference_outputs` variable:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE\n\nevaluator = create_trajectory_llm_as_judge(\n  prompt=TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n  model=\"openai:o3-mini\"\n)\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80˚ and sunny.\"},\n]\neval_result = evaluator(\n    outputs=outputs,\n    reference_outputs=reference_outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': True,\n    'comment': 'The provided agent trajectory is consistent with the reference. Both trajectories start with the same user query and then correctly invoke a weather lookup through a tool call. Although the reference uses \"San Francisco\" while the provided trajectory uses \"SF\" and there is a minor formatting difference (degrees vs. ˚), these differences do not affect the correctness or essential steps of the process. Thus, the score should be: true.'\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst evaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n  model: \"openai:o3-mini\",\n});\n\nconst outputs = [\n  {role: \"user\", content: \"What is the weather in SF?\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        }\n      }\n    ],\n  },\n  {role: \"tool\", content: \"It's 80 degrees and sunny in SF.\"},\n  {role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  {role: \"user\", content: \"What is the weather in SF?\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"San Francisco\" }),\n        }\n      }\n    ],\n  },\n  {role: \"tool\", content: \"It's 80 degrees and sunny in San Francisco.\"},\n  {role: \"assistant\", content: \"The weather in SF is 80˚ and sunny.\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': true,\n    'comment': 'The provided agent trajectory is consistent with the reference. Both trajectories start with the same user query and then correctly invoke a weather lookup through a tool call. Although the reference uses \"San Francisco\" while the provided trajectory uses \"SF\" and there is a minor formatting difference (degrees vs. ˚), these differences do not affect the correctness or essential steps of the process. Thus, the score should be: true.'\n}\n```\n\u003C\u002Fdetails>\n\n`create_trajectory_llm_as_judge` takes the same parameters as [`create_llm_as_judge`](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals?tab=readme-ov-file#llm-as-judge) in `openevals`, so you can customize the prompt and scoring output as needed.\n\nIn addition to `prompt` and `model`, the following parameters are also available:\n\n- `continuous`: a boolean that sets whether the evaluator should return a float score somewhere between 0 and 1 instead of a binary score. Defaults to `False`.\n- `choices`: a list of floats that sets the possible scores for the evaluator.\n- `system`: a string that sets a system prompt for the judge model by adding a system message before other parts of the prompt.\n- `few_shot_examples`: a list of example dicts that are appended to the end of the prompt. This is useful for providing the judge model with examples of good and bad outputs. The required structure looks like this:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfew_shot_examples = [\n    {\n        \"inputs\": \"What color is the sky?\",\n        \"outputs\": \"The sky is red.\",\n        \"reasoning\": \"The sky is red because it is early evening.\",\n        \"score\": 1,\n    }\n]\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nconst fewShotExamples = [\n  {\n    inputs: \"What color is the sky?\",\n    outputs: \"The sky is red.\",\n    reasoning: \"The sky is red because it is early evening.\",\n    score: 1,\n  }\n];\n```\n\u003C\u002Fdetails>\n\nSee the [`openevals`](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals?tab=readme-ov-file#llm-as-judge) repo for a fully up to date list of parameters.\n\n## Graph trajectory\n\nFor frameworks like [LangGraph](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Flanggraph) that model agents as graphs, it can be more convenient to represent trajectories in terms of nodes visited rather than messages. `agentevals` includes a category of evaluators called **graph trajectory** evaluators that are designed to work with this format, as well as convenient utilities for extracting trajectories from a LangGraph thread, including different conversation turns and interrupts.\n\nThe below examples will use LangGraph with the built-in formatting utility, but graph evaluators accept input in the following general format:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nclass GraphTrajectory(TypedDict):\n    # Only set when specifying reference_outputs\n    inputs: Optional[list[dict]]\n    results: list[dict]\n    steps: list[list[str]]\n    \ndef evaluator(\n    *,\n    inputs: Optional[Union[dict, list]] = None,\n    outputs: GraphTrajectory,\n    reference_outputs: Optional[GraphTrajectory] = None,\n) -> ...\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nexport type GraphTrajectory = {\n  inputs?: (Record\u003Cstring, unknown> | null)[];\n  results: Record\u003Cstring, unknown>[];\n  steps: string[][];\n};\n\nconst evaluator: ({ inputs, outputs, referenceOutputs, ...extra }: {\n    inputs: (string | Record\u003Cstring, unknown> | null)[] | {\n        inputs: (string | Record\u003Cstring, unknown> | null)[];\n    };\n    outputs: GraphTrajectory;\n    referenceOutputs?: GraphTrajectory;\n    [key: string]: unknown;\n}) => ...\n```\n\u003C\u002Fdetails>\n\nWhere `inputs` is a list of inputs (or a dict with a key named `\"inputs\"`) to the graph whose items each represent the start of a new invocation in a thread, `results` representing the final output from each turn in the thread, and `steps` representing the internal steps taken for each turn.\n\n### Graph trajectory LLM-as-judge\n\nThis evaluator is similar to the `trajectory_llm_as_judge` evaluator, but it works with graph trajectories instead of message trajectories. Below, we set up a LangGraph agent, extract a trajectory from it using the built-in utils, and pass it to the evaluator. First, let's setup our graph, call it, and then extract the trajectory:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom agentevals.graph_trajectory.utils import (\n    extract_langgraph_trajectory_from_thread,\n)\nfrom agentevals.graph_trajectory.llm import create_graph_trajectory_llm_as_judge\n\nfrom langgraph.prebuilt import create_react_agent\nfrom langgraph.checkpoint.memory import MemorySaver\nfrom langgraph.types import Command, interrupt\n\nfrom langchain_core.tools import tool\n\n@tool\ndef search(query: str):\n    \"\"\"Call to surf the web.\"\"\"\n    user_answer = interrupt(\"Tell me the answer to the question.\")\n    return user_answer\n\ntools = [search]\n\ncheckpointer = MemorySaver()\ngraph = create_react_agent(\n    model=\"gpt-4o-mini\",\n    checkpointer=checkpointer,\n    tools=[search],\n)\n\ngraph.invoke(\n    {\"messages\": [{\"role\": \"user\", \"content\": \"what's the weather in sf?\"}]},\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n# Resume the agent with a new command, simulating a human-in-the-loop workflow\ngraph.invoke(\n    Command(resume=\"It is rainy and 70 degrees!\"),\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n\n# Extract the trajectory from the first two thread runs\nextracted_trajectory = extract_langgraph_trajectory_from_thread(\n    graph, {\"configurable\": {\"thread_id\": \"1\"}}\n)\n\nprint(extracted_trajectory)\n```\n\n```\n{\n  'inputs': [{\n      '__start__': {\n          'messages': [\n              {'role': 'user', 'content': \"what's the weather in sf?\"}\n          ]}\n      }, \n      '__resuming__': {\n          'messages': [\n              {'role': 'user', 'content': 'It is rainy and 70 degrees!'}\n          ]}\n      ],\n      'outputs': {\n          'results': [\n            {},\n            {\n                'messages': [\n                    {'role': 'ai', 'content': 'The current weather in San Francisco is rainy, with a temperature of 70 degrees.'}\n                ]\n            }\n        ],\n        'steps': [\n            ['__start__', 'agent', 'tools', '__interrupt__'],\n            ['agent']\n        ]\n    }\n}\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { tool } from \"@langchain\u002Fcore\u002Ftools\";\nimport { ChatOpenAI } from \"@langchain\u002Fopenai\";\nimport { createReactAgent } from \"@langchain\u002Flanggraph\u002Fprebuilt\";\nimport { MemorySaver, interrupt } from \"@langchain\u002Flanggraph\";\nimport { z } from \"zod\";\nimport { extractLangGraphTrajectoryFromThread } from \"agentevals\";\n\nconst search = tool((_): string => {\n  const userAnswer = interrupt(\"Tell me the answer to the question.\")\n  return userAnswer;\n}, {\n  name: \"search\",\n  description: \"Call to surf the web.\",\n  schema: z.object({\n      query: z.string()\n  })\n})\n\nconst tools = [search];\n\n\u002F\u002F Create a checkpointer\nconst checkpointer = new MemorySaver();\n\n\u002F\u002F Create the React agent\nconst graph = createReactAgent({\n  llm: new ChatOpenAI({ model: \"gpt-4o-mini\" }),\n  tools,\n  checkpointer,\n});\n\n\u002F\u002F Invoke the graph with initial message\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"what's the weather in sf?\" }] },\n  { configurable: { thread_id: \"1\" } }\n);\n\n\u002F\u002F Resume the agent with a new command (simulating human-in-the-loop)\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"It is rainy and 70 degrees!\" }] },\n  { configurable: { thread_id: \"1\" } }\n);\n\nconst extractedTrajectory = await extractLangGraphTrajectoryFromThread(\n  graph,\n  { configurable: { thread_id: \"1\" } },\n);\n\nconsole.log(extractedTrajectory);\n```\n\n```\n{\n  'inputs': [{\n      '__start__': {\n          'messages': [\n              {'role': 'user', 'content': \"what's the weather in sf?\"}\n          ]}\n      }, \n      '__resuming__': {\n          'messages': [\n              {'role': 'user', 'content': 'It is rainy and 70 degrees!'}\n          ]}\n      ],\n      'outputs': {\n          'results': [\n            {},\n            {\n                'messages': [\n                    {'role': 'ai', 'content': 'The current weather in San Francisco is rainy, with a temperature of 70 degrees.'}\n                ]\n            }\n        ],\n        'steps': [\n            ['__start__', 'agent', 'tools', '__interrupt__'],\n            ['agent']\n        ]\n    }\n}\n```\n\u003C\u002Fdetails>\n\nNow, we can pass the extracted trajectory to the evaluator:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\ngraph_trajectory_evaluator = create_graph_trajectory_llm_as_judge(\n    model=\"openai:o3-mini\",\n)\n\nres = graph_trajectory_evaluator(\n    inputs=extracted_trajectory[\"inputs\"],\n    outputs=extracted_trajectory[\"outputs\"],\n)\n\nprint(res)\n```\n\n```\n{\n  'key': 'graph_trajectory_accuracy',\n  'score': True,\n  'comment': 'The overall process follows a logical progression: the conversation begins with the user’s request, the agent then processes the request through its own internal steps (including calling tools), interrupts to obtain further input, and finally resumes to provide a natural language answer. Each step is consistent with the intended design in the rubric, and the overall path is relatively efficient and semantically aligns with a typical query resolution trajectory. Thus, the score should be: true.'\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { createGraphTrajectoryLLMAsJudge } from \"agentevals\";\n\nconst graphTrajectoryEvaluator = createGraphTrajectoryLLMAsJudge({\n    model: \"openai:o3-mini\",\n})\n\nconst res = await graphTrajectoryEvaluator({\n  inputs: extractedTrajectory.inputs,\n  outputs: extractedTrajectory.outputs,\n});\n\nconsole.log(res);\n```\n\n```\n{\n  'key': 'graph_trajectory_accuracy',\n  'score': True,\n  'comment': 'The overall process follows a logical progression: the conversation begins with the user’s request, the agent then processes the request through its own internal steps (including calling tools), interrupts to obtain further input, and finally resumes to provide a natural language answer. Each step is consistent with the intended design in the rubric, and the overall path is relatively efficient and semantically aligns with a typical query resolution trajectory. Thus, the score should be: true.'\n}\n```\n\u003C\u002Fdetails>\n\nNote that though this evaluator takes the typical `inputs`, `outputs`, and `reference_outputs` parameters, it internally combines `inputs` and `outputs` to form a `thread`. Therefore, if you want to customize the prompt, your prompt should also contain a `thread` input variable:\n\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nCUSTOM_PROMPT = \"\"\"You are an expert data labeler.\nYour task is to grade the accuracy of an AI agent's internal steps in resolving a user queries.\n\n\u003CRubric>\n  An accurate trajectory:\n  - Makes logical sense between steps\n  - Shows clear progression\n  - Is perfectly efficient, with no more than one tool call\n  - Is semantically equivalent to the provided reference trajectory, if present\n\u003C\u002FRubric>\n\n\u003CInstructions>\n  Grade the following thread, evaluating whether the agent's overall steps are logical and relatively efficient.\n  For the trajectory, \"__start__\" denotes an initial entrypoint to the agent, and \"__interrupt__\" corresponds to the agent\n  interrupting to await additional data from another source (\"human-in-the-loop\"):\n\u003C\u002FInstructions>\n\n\u003Cthread>\n{thread}\n\u003C\u002Fthread>\n\n{reference_outputs}\n\"\"\"\n\nevaluator = create_graph_trajectory_llm_as_judge(\n    prompt=CUSTOM_PROMPT,\n    model=\"openai:o3-mini\",\n)\nres = await evaluator(\n    inputs=extracted_trajectory[\"inputs\"],\n    outputs=extracted_trajectory[\"outputs\"],   \n)\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nconst CUSTOM_PROMPT = `You are an expert data labeler.\nYour task is to grade the accuracy of an AI agent's internal steps in resolving a user queries.\n\n\u003CRubric>\n  An accurate trajectory:\n  - Makes logical sense between steps\n  - Shows clear progression\n  - Is perfectly efficient, with no more than one tool call\n  - Is semantically equivalent to the provided reference trajectory, if present\n\u003C\u002FRubric>\n\n\u003CInstructions>\n  Grade the following thread, evaluating whether the agent's overall steps are logical and relatively efficient.\n  For the trajectory, \"__start__\" denotes an initial entrypoint to the agent, and \"__interrupt__\" corresponds to the agent\n  interrupting to await additional data from another source (\"human-in-the-loop\"):\n\u003C\u002FInstructions>\n\n\u003Cthread>\n{thread}\n\u003C\u002Fthread>\n\n{reference_outputs}\n`\n\nconst graphTrajectoryEvaluator = createGraphTrajectoryLLMAsJudge({\n  prompt: CUSTOM_PROMPT,\n  model: \"openai:o3-mini\",\n})\nconst res = await graphTrajectoryEvaluator({\n  inputs: extractedTrajectory.inputs,\n  outputs: extractedTrajectory.outputs,\n});\n```\n\u003C\u002Fdetails>\n\nIn order to format them properly into the prompt, `reference_outputs` should be passed in as a `GraphTrajectory` object like `outputs`.\n\nAlso note that like other LLM-as-judge evaluators, you can pass extra params into the evaluator to format them into the prompt.\n\n### Graph trajectory strict match\n\nThe `graph_trajectory_strict_match` evaluator is a simple evaluator that checks if the steps in the provided graph trajectory match the reference trajectory exactly.\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom agentevals.graph_trajectory.utils import (\n    extract_langgraph_trajectory_from_thread,\n)\nfrom agentevals.graph_trajectory.strict import graph_trajectory_strict_match\n\n\nfrom langgraph.prebuilt import create_react_agent\nfrom langgraph.checkpoint.memory import MemorySaver\nfrom langgraph.types import Command, interrupt\n\nfrom langchain_core.tools import tool\n\n@tool\ndef search(query: str):\n    \"\"\"Call to surf the web.\"\"\"\n    user_answer = interrupt(\"Tell me the answer to the question.\")\n    return user_answer\n\ntools = [search]\n\ncheckpointer = MemorySaver()\ngraph = create_react_agent(\n    model=\"gpt-4o-mini\",\n    checkpointer=checkpointer,\n    tools=[search],\n)\n\ngraph.invoke(\n    {\"messages\": [{\"role\": \"user\", \"content\": \"what's the weather in sf?\"}]},\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n# Resume the agent with a new command, simulating a human-in-the-loop workflow\ngraph.invoke(\n    Command(resume=\"It is rainy and 70 degrees!\"),\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n\n# Extract the trajectory from the first two thread runs\nextracted_trajectory = extract_langgraph_trajectory_from_thread(\n    graph, {\"configurable\": {\"thread_id\": \"1\"}}\n)\n\nreference_trajectory = {\n    # not used for strict match\n    \"results\": [],\n    \"steps\": [[\"__start__\", \"agent\", \"tools\", \"__interrupt__\"], [\"agent\"]],\n}\n\nres = graph_trajectory_strict_match(\n    outputs=extracted_trajectory[\"outputs\"],\n    reference_outputs=reference_trajectory,\n)\n\nprint(res)\n```\n\n```\n{\n  'key': 'graph_trajectory_strict_match',\n  'score': True,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { tool } from \"@langchain\u002Fcore\u002Ftools\";\nimport { ChatOpenAI } from \"@langchain\u002Fopenai\";\nimport { createReactAgent } from \"@langchain\u002Flanggraph\u002Fprebuilt\";\nimport { MemorySaver, interrupt } from \"@langchain\u002Flanggraph\";\nimport { z } from \"zod\";\nimport { extractLangGraphTrajectoryFromThread, graphTrajectoryStrictMatch } from \"agentevals\";\n\nconst search = tool((_): string => {\n  const userAnswer = interrupt(\"Tell me the answer to the question.\")\n  return userAnswer;\n}, {\n  name: \"search\",\n  description: \"Call to surf the web.\",\n  schema: z.object({\n      query: z.string()\n  })\n})\n\nconst tools = [search];\n\n\u002F\u002F Create a checkpointer\nconst checkpointer = new MemorySaver();\n\n\u002F\u002F Create the React agent\nconst graph = createReactAgent({\n  llm: new ChatOpenAI({ model: \"gpt-4o-mini\" }),\n  tools,\n  checkpointer,\n});\n\n\u002F\u002F Invoke the graph with initial message\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"what's the weather in sf?\" }] },\n  { configurable: { thread_id: \"1\" } }\n);\n\n\u002F\u002F Resume the agent with a new command (simulating human-in-the-loop)\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"It is rainy and 70 degrees!\" }] },\n  { configurable: { thread_id: \"1\" } }\n);\n\nconst extractedTrajectory = await extractLangGraphTrajectoryFromThread(\n  graph,\n  { configurable: { thread_id: \"1\" } },\n);\n\nconst referenceTrajectory = {\n  results: [],\n  steps: [[\"__start__\", \"agent\", \"tools\", \"__interrupt__\"], [\"agent\"]],\n}\n\nconst result = await graphTrajectoryStrictMatch({\n  outputs: trajectory.outputs,\n  referenceOutputs: referenceOutputs!,\n});\n\nconsole.log(result);\n```\n\n```\n{\n  'key': 'graph_trajectory_strict_match',\n  'score': True,\n}\n```\n\u003C\u002Fdetails>\n\n# Python Async Support\n\nAll `agentevals` evaluators support Python [asyncio](https:\u002F\u002Fdocs.python.org\u002F3\u002Flibrary\u002Fasyncio.html). As a convention, evaluators that use a factory function will have `async` put immediately after `create_` in the function name (for example, `create_async_trajectory_llm_as_judge`), and evaluators used directly will end in `async` (e.g. `trajectory_strict_match_async`).\n\nHere's an example of how to use the `create_async_llm_as_judge` evaluator asynchronously:\n\n```python\nfrom agentevals.trajectory.llm import create_async_trajectory_llm_as_judge\n\nevaluator = create_async_llm_as_judge(\n    prompt=\"What is the weather in {inputs}?\",\n)\n\nresult = await evaluator(inputs=\"San Francisco\")\n```\n\nIf you are using the OpenAI client directly, remember to pass in `AsyncOpenAI` as the `judge` parameter:\n\n```python\nfrom openai import AsyncOpenAI\n\nevaluator = create_async_llm_as_judge(\n    prompt=\"What is the weather in {inputs}?\",\n    judge=AsyncOpenAI(),\n    model=\"o3-mini\",\n)\n\nresult = await evaluator(inputs=\"San Francisco\")\n```\n\n# LangSmith Integration\n\nFor tracking experiments over time, you can log evaluator results to [LangSmith](https:\u002F\u002Fsmith.langchain.com\u002F), a platform for building production-grade LLM applications that includes tracing, evaluation, and experimentation tools.\n\nLangSmith currently offers two ways to run evals: a [pytest](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fpytest) (Python) or [Vitest\u002FJest](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fvitest_jest) integration and the `evaluate` function. We'll give a quick example of how to run evals using both.\n\n## Pytest or Vitest\u002FJest\n\nFirst, follow [these instructions](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fpytest) to set up LangSmith's pytest runner, or these to set up [Vitest or Jest](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fvitest_jest),\nsetting appropriate environment variables:\n\n\n```bash\nexport LANGSMITH_API_KEY=\"your_langsmith_api_key\"\nexport LANGSMITH_TRACING=\"true\"\n```\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\nThen, set up a file named `test_trajectory.py` with the following contents:\n\n```python\nimport pytest\nimport json\n\nfrom langsmith import testing as t\n\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge\n\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    model=\"openai:o3-mini\",\n)\n\n@pytest.mark.langsmith\ndef test_trajectory_accuracy():\n    outputs = [\n        {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": json.dumps({\"city\": \"SF\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n        {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n    ]\n    reference_outputs = [\n        {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n        {\"role\": \"assistant\", \"content\": \"The weather in SF is 80˚ and sunny.\"},\n    ]\n\n    t.log_inputs({})\n    t.log_outputs({\"messages\": outputs})\n    t.log_reference_outputs({\"messages\": reference_outputs})\n\n    trajectory_evaluator(\n      outputs=outputs,\n      reference_outputs=reference_outputs\n    )\n```\n\nNote that when creating the evaluator, we've added a `feedback_key` parameter. This will be used to name the feedback in LangSmith.\n\nNow, run the eval with pytest:\n\n```bash\npytest test_trajectory.py --langsmith-output\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\nThen, set up a file named `test_trajectory.eval.ts` with the following contents:\n\n```ts\nimport * as ls from \"langsmith\u002Fvitest\";\n\u002F\u002F import * as ls from \"langsmith\u002Fjest\";\n\nimport { createTrajectoryLLMAsJudge } from \"agentevals\";\n\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  model: \"openai:o3-mini\",\n});\n\nls.describe(\"trajectory accuracy\", () => {\n  ls.test(\"accurate trajectory\", {\n    inputs: {\n      messages: [\n        {\n          role: \"user\",\n          content: \"What is the weather in SF?\"\n        }\n      ]\n    },\n    referenceOutputs: {\n      messages: [\n        {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": JSON.stringify({\"city\": \"San Francisco\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in San Francisco.\"},\n        {\"role\": \"assistant\", \"content\": \"The weather in SF is 80˚ and sunny.\"},\n      ],\n    },\n  }, async ({ inputs, referenceOutputs }) => {\n    const outputs = [\n        {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": JSON.stringify({\"city\": \"SF\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n        {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n    ];\n    ls.logOutputs({ messages: outputs });\n\n    await trajectoryEvaluator({\n      inputs,\n      outputs,\n      referenceOutputs,\n    });\n  });\n});\n```\n\nNow, run the eval with your runner of choice:\n\n```bash\nvitest run test_trajectory.eval.ts\n```\n\n\u003C\u002Fdetails>\n\nFeedback from the prebuilt evaluator will be automatically logged in LangSmith as a table of results like this in your terminal:\n\n![Terminal results](https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Flangchain-ai_agentevals_readme_84d888bed60c.png)\n\nAnd you should also see the results in the experiment view in LangSmith:\n\n![LangSmith results](https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Flangchain-ai_agentevals_readme_47a167b71a1c.png)\n\n## Evaluate\n\nAlternatively, you can [create a dataset in LangSmith](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fconcepts#dataset-curation) and use your created evaluators with LangSmith's [`evaluate`](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation#8-run-and-view-results) function:\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom langsmith import Client\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge\n\nclient = Client()\n\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    model=\"openai:o3-mini\",\n)\n\nexperiment_results = client.evaluate(\n    # This is a dummy target function, replace with your actual LLM-based system\n    lambda inputs: \"What color is the sky?\",\n    data=\"Sample dataset\",\n    evaluators=[\n        trajectory_evaluator\n    ]\n)\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { evaluate } from \"langsmith\u002Fevaluation\";\nimport { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from \"agentevals\";\n\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  model: \"openai:o3-mini\",\n  prompt: TRAJECTORY_ACCURACY_PROMPT\n});\n\nawait evaluate(\n  (inputs) => [\n      {role: \"user\", content: \"What is the weather in SF?\"},\n      {\n          role: \"assistant\",\n          content: \"\",\n          tool_calls: [\n              {\n                  function: {\n                      name: \"get_weather\",\n                      arguments: json.dumps({\"city\": \"SF\"}),\n                  }\n              }\n          ],\n      },\n      {role: \"tool\", content: \"It's 80 degrees and sunny in SF.\"},\n      {role: \"assistant\", content: \"The weather in SF is 80 degrees and sunny.\"},\n    ],\n  {\n    data: datasetName,\n    evaluators: [trajectoryEvaluator],\n  }\n);\n```\n\u003C\u002Fdetails>\n\n# Thank you!\n\nWe hope that `agentevals` helps make evaluating your LLM agents easier!\n\nIf you have any questions, comments, or suggestions, please open an issue or reach out to us on X [@LangChainAI](https:\u002F\u002Fx.com\u002Flangchainai).\n","# 🦾⚖️ AgentEvals\n\n[代理式应用](https:\u002F\u002Flangchain-ai.github.io\u002Flanggraph\u002Fconcepts\u002Fagentic_concepts\u002F) 通过赋予大模型对控制流的自由度来解决问题。虽然这种自由度非常强大，但由于大模型的黑箱特性，很难理解代理中某一部分的变化会如何影响下游的其他部分。因此，评估你的代理显得尤为重要。\n\n本包包含一系列评估工具和实用程序，用于评估代理的性能，重点在于**代理轨迹**，即代理在运行过程中所经历的中间步骤。它旨在为你的代理评估提供一个良好的概念性起点。\n\n如果你正在寻找更通用的评估工具，请查看配套包 [`openevals`](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals)。\n\n# 快速入门\n\n要开始使用，首先安装 `agentevals`：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```bash\npip install agentevals\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```bash\nnpm install agentevals @langchain\u002Fcore\n```\n\u003C\u002Fdetails>\n\n本快速入门将使用由 OpenAI 的 `o3-mini` 模型驱动的评估器来评判你的结果，因此你需要将 OpenAI API 密钥设置为环境变量：\n\n```bash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n```\n\n完成上述步骤后，你就可以运行第一个轨迹评估器了。我们以 OpenAI 风格的消息列表来表示代理的轨迹：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT\n\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    prompt=TRAJECTORY_ACCURACY_PROMPT,\n    model=\"openai:o3-mini\",\n)\n\n# 这是一个假想的轨迹，实际应用中你需要运行代理来获取真实的轨迹\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n]\n\neval_result = trajectory_evaluator(\n  outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n  'key': 'trajectory_accuracy',\n  'reasoning': '该轨迹准确地遵循了用户关于旧金山天气信息的请求。起初，助手明确了目标（提供天气详情），随后高效地调用了工具获取天气信息，最后清晰地传达了结果。所有步骤都展现了逻辑上的连贯性和效率。因此，评分应为：true。',\n  'score': true\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  type FlexibleChatCompletionMessage,\n  TRAJECTORY_ACCURACY_PROMPT,\n} from \"agentevals\";\n\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT,\n  model: \"openai:o3-mini\",\n});\n\nconst outputs = [\n  { role: \"user\", content: \"旧金山的天气如何？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        },\n      },\n    ],\n  },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\" },\n  {\n    role: \"assistant\",\n    content: \"旧金山的天气是80华氏度，晴朗。\",\n  },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evalResult = await trajectoryEvaluator({\n  outputs,\n});\n\nconsole.log(evalResult);\n```\n\n```\n{\n    key: 'trajectory_accuracy',\n    score: true,\n    comment: '...'\n}\n```\n\u003C\u002Fdetails>\n\n你可以看到，评估器返回了 `true` 的评分，因为整体轨迹是代理回答用户问题的一个合理路径。\n\n有关此评估器的更多详细信息，包括如何对其进行自定义，请参阅 [轨迹 LLM-as-judge](#trajectory-llm-as-judge) 部分。\n\n# 目录\n\n- [安装](#installation)\n- [评估器](#evaluators)\n  - [代理轨迹匹配](#agent-trajectory-match)\n    - [严格匹配](#strict-match)\n    - [无序匹配](#unordered-match)\n    - [子集\u002F超集匹配](#subset-and-superset-match)\n    - [工具参数匹配模式](#tool-args-match-modes)\n  - [轨迹 LLM-as-judge](#trajectory-llm-as-judge)\n  - [图轨迹](#graph-trajectory)\n    - [图轨迹 LLM-as-judge](#graph-trajectory-llm-as-judge)\n    - [图轨迹严格匹配](#graph-trajectory-strict-match)\n- [Python 异步支持](#python-async-support)\n- [LangSmith 集成](#langsmith-integration)\n  - [Pytest 或 Vitest\u002FJest](#pytest-or-vitestjest)\n  - [评估](#evaluate)\n\n# 安装\n\n你可以这样安装 `agentevals`：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```bash\npip install agentevals\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```bash\nnpm install agentevals @langchain\u002Fcore\n```\n\u003C\u002Fdetails>\n\n对于 LLM-as-judge 评估器，你还需要一个大模型客户端。默认情况下，`agentevals` 会使用 [LangChain 聊天模型集成](https:\u002F\u002Fpython.langchain.com\u002Fdocs\u002Fintegrations\u002Fchat\u002F) 并自带 `langchain_openai`。不过，如果你愿意，也可以直接使用 OpenAI 客户端：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```bash\npip install openai\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```bash\nnpm install openai\n```\n\u003C\u002Fdetails>\n\n此外，熟悉一些 [评估概念](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fconcepts) 以及 LangSmith 的 pytest 集成来运行评估也会很有帮助，相关文档请参见 [这里](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fpytest)。\n\n# 评估器\n\n## 代理轨迹匹配\n\n代理轨迹匹配评估器用于根据预期轨迹或使用大语言模型来判断代理执行过程中的轨迹。这些评估器期望您将代理的轨迹格式化为 OpenAI 格式的字典列表，或者为 LangChain 的 `BaseMessage` 类列表，并在内部自动处理消息格式化。\n\nAgentEvals 提供了 `create_trajectory_match_evaluator`\u002F`createTrajectoryMatchEvaluator` 和 `create_async_trajectory_match_evaluator` 方法来完成这一任务。您可以从以下几个方面自定义其行为：\n\n- 设置 `trajectory_match_mode`\u002F`trajectoryMatchMode` 为 [`strict`](#strict-match)、[`unordered`](#unordered-match)、[`subset`](#subset-and-superset-match) 或 [`superset`](#subset-and-superset-match)，以指定评估器比较轨迹时采用的总体策略。\n- 设置 [`tool_args_match_mode`](#tool-args-match-modes) 和\u002F或 [`tool_args_match_overrides`](#tool-args-match-modes)，以自定义评估器如何判断实际轨迹与参考轨迹中工具调用是否相等。默认情况下，只有调用相同工具且参数相同的工具调用才会被视为相等。\n\n### 严格匹配\n\n`\"strict\"` `trajectory_match_mode` 会比较两条轨迹，确保它们包含相同的消息、顺序一致，并且工具调用也完全相同。需要注意的是，它允许消息内容存在差异：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            },\n            {\n                \"function\": {\n                    \"name\": \"accuweather_forecast\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气怎么样？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"strict\"\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_strict_match',\n    'score': False,\n    'comment': None,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"旧金山的天气如何？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"San Francisco\" })\n      },\n    }, {\n      function: {\n        name: \"accuweather_forecast\",\n        arguments: JSON.stringify({\"city\": \"San Francisco\"}),\n      },\n    }]\n  },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\" },\n  { role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗。\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"旧金山的天气怎么样？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"San Francisco\" })\n      }\n    }]\n  },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\" },\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"strict\",\n})\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result);\n```\n\n```\n{\n    'key': 'trajectory_strict_match',\n    'score': false,\n}\n```\n\u003C\u002Fdetails>\n\n`\"strict\"` 模式适用于需要确保对于给定查询，工具总是以相同顺序被调用的情况（例如，先调用公司政策查询工具，再调用申请员工休假的工具）。\n\n**注意：** 如果您希望配置此评估器检查工具调用是否相等的方式，请参阅 [本节](#tool-args-match-modes)。\n\n### 无序匹配\n\n`\"unordered\"` `trajectory_match_mode` 会比较两条轨迹，并确保它们包含相同的工具调用，顺序不限。这在你希望代理获取正确信息的方式有一定的灵活性，但仍关心是否已检索到所有必要信息时非常有用。\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\ninputs = {}\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？有什么好玩的活动吗？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [{\n            \"function\": {\n                \"name\": \"get_weather\",\n                \"arguments\": json.dumps({\"city\": \"旧金山\"}),\n            }\n        }],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [{\n            \"function\": {\n                \"name\": \"get_fun_activities\",\n                \"arguments\": json.dumps({\"city\": \"旧金山\"}),\n            }\n        }],\n    },\n    {\"role\": \"tool\", \"content\": \"目前没有有趣的活动，建议待在室内看书！\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗，但没有什么好玩的活动。\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？有什么好玩的活动吗？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_fun_activities\",\n                    \"arguments\": json.dumps({\"city\": \"旧金山\"}),\n                }\n            },\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"旧金山\"}),\n                }\n            },\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"目前没有有趣的活动，建议待在室内看书！\"},\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山现在是80华氏度，晴朗，但没有什么好玩的活动。\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"unordered\"\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_unordered_match',\n    'score': True,\n    'comment': None,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"旧金山的天气如何？有什么好玩的活动吗？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"旧金山\" }),\n      }\n    }],\n  },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_fun_activities\",\n        arguments: JSON.stringify({ city: \"旧金山\" }),\n      }\n    }],\n  },\n  { role: \"tool\", content: \"目前没有有趣的活动，建议待在室内看书！\" },\n  { role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗，但没有什么好玩的活动。\" },\n] 符合 FlexibleChatCompletionMessage[]类型;\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"旧金山的天气如何？有什么好玩的活动吗？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_fun_activities\",\n          arguments: JSON.stringify({ city: \"旧金山\" }),\n        }\n      },\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"旧金山\" }),\n        }\n      },\n    ],\n  },\n  { role: \"tool\", content: \"目前没有有趣的活动，建议待在室内看书！\" },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\" },\n  { role: \"assistant\", content: \"旧金山现在是80华氏度，晴朗，但没有什么好玩的活动。\" },\n] 符合 FlexibleChatCompletionMessage[]类型;\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"unordered\",\n});\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_unordered_match',\n    'score': true,\n}\n```\n\u003C\u002Fdetails>\n\n`\"unordered\"` 非常适合用于确保特定工具在轨迹中的某个时刻被调用，但并不一定要求它们按照消息顺序出现的情况（例如，代理在授权举办披萨派对之前，在交互过程中的任意时间点调用了公司政策检索工具）。\n\n**注意：** 如果您希望配置此评估器检查工具调用相等性的方式，请参阅[本节](#tool-args-match-modes)。\n\n### 子集和超集匹配\n\n`\"subset\"` 和 `\"superset\"` 模式用于匹配部分轨迹（确保轨迹包含参考轨迹中工具调用的子集或超集）。\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山和伦敦的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [{\n            \"function\": {\n                \"name\": \"get_weather\",\n                \"arguments\": json.dumps({\"city\": \"旧金山和伦敦\"}),\n            },\n        }, {\n            \"function\": {\n                \"name\": \"accuweather_forecast\",\n                \"arguments\": json.dumps({\"city\": \"旧金山和伦敦\"}),\n            }\n        }],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山气温80华氏度，晴朗；伦敦气温90华氏度，下雨。\"},\n    {\"role\": \"tool\", \"content\": \"未知。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。伦敦则是90华氏度，下雨。\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"旧金山和伦敦的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"旧金山和伦敦\"}),\n                }\n            },\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山气温80华氏度，晴朗；伦敦气温90华氏度，下雨。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。伦敦则是90华氏度，下雨。\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"superset\", # 或 \"subset\"\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_superset_match',\n    'score': True,\n    'comment': None,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"旧金山和伦敦的天气如何？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"旧金山和伦敦\" }),\n      }\n    }, {\n      \"function\": {\n        name: \"accuweather_forecast\",\n        arguments: JSON.stringify({\"city\": \"旧金山和伦敦\"}),\n      }\n    }],\n  },\n  { role: \"tool\", content: \"旧金山气温80华氏度，晴朗；伦敦气温90华氏度，下雨。\" },\n  { role: \"tool\", content: \"未知。\" },\n  { role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗。伦敦则是90华氏度，下雨。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"旧金山和伦敦的天气如何？\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"旧金山和伦敦\" }),\n        }\n      },\n    ],\n  },\n  { role: \"tool\", content: \"旧金山气温80华氏度，晴朗；伦敦气温90华氏度，下雨。\" },\n  { role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗。伦敦则是90华氏度，下雨。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"superset\", \u002F\u002F 或 \"subset\"\n});\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_superset_match',\n    'score': true,\n}\n```\n\u003C\u002Fdetails>\n\n`\"superset\"` 在您希望确保轨迹中至少调用了某些关键工具，但允许代理调用额外工具时非常有用。而 `\"subset\"` 则是其相反模式，在您希望确保代理未调用任何超出预期工具时很有用。\n\n**注意：** 如果您希望配置此评估器检查工具调用是否相等的方式，请参阅 [本节](#tool-args-match-modes)。\n\n### 工具参数匹配模式\n\n在检查工具调用之间的相等性时，上述评估器默认会要求所有工具调用的参数完全相同。您可以通过以下方式配置此行为：\n\n- 通过设置 `tool_args_match_mode=\"ignore\"`（Python）或 `toolArgsMatchMode: \"ignore\"`（TypeScript），将针对同一工具的任何两个工具调用视为等价。\n- 通过设置 `tool_args_match_mode=\"subset\"\u002F\"superset\"`（Python）或 `toolArgsMatchMode: \"subset\"\u002F\"superset\"`（TypeScript），如果一个工具调用包含与同名参考工具调用相比的子集或超集参数，则将其视为等价。\n- 使用 `tool_args_match_overrides`（Python）或 `toolArgsMatchOverrides`（TypeScript）参数为给定工具的所有调用设置自定义匹配器。\n\n您可以同时设置这两个参数。`tool_args_match_overrides` 将优先于 `tool_args_match_mode`。\n\n`tool_args_match_overrides`\u002F`toolArgsMatchOverrides` 接受一个字典，其键是工具名称，值可以是 `\"exact\"`、`\"ignore\"`、必须精确匹配的工具调用中的字段列表，或者一个接受两个参数并返回它们是否相等的比较函数：\n\n```python\nToolArgsMatchMode = Literal[\"exact\", \"ignore\", \"subset\", \"superset\"]\n\nToolArgsMatchOverrides = dict[str, Union[ToolArgsMatchMode, list[str],  Callable[[dict, dict], bool]]]\n```\n\n以下是一个示例，允许对名为 `get_weather` 的工具的参数进行不区分大小写的匹配：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.match import create_trajectory_match_evaluator\n\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"san francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气怎么样？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80˚，晴朗。\"},\n]\n\nevaluator = create_trajectory_match_evaluator(\n    trajectory_match_mode=\"strict\",\n    tool_args_match_mode=\"exact\",  # 默认值\n    tool_args_match_overrides={\n        \"get_weather\": lambda x, y: x[\"city\"].lower() == y[\"city\"].lower()\n    }\n)\n\nresult = evaluator(\n    outputs=outputs, reference_outputs=reference_outputs\n)\n\nprint(result)\n```\n\n```\n{\n    'key': 'trajectory_strict_match',\n    'score': True,\n    'comment': None,\n}\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryMatchEvaluator,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst outputs = [\n  { role: \"user\", content: \"旧金山的天气如何？\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"san francisco\" })\n      },\n    }]\n  },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\"},\n  { role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  { role: \"user\", content: \"旧金山的天气怎么样？\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [{\n      function: {\n        name: \"get_weather\",\n        arguments: JSON.stringify({ city: \"San Francisco\" })\n      }\n    }]\n  },\n  { role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst evaluator = createTrajectoryMatchEvaluator({\n  trajectoryMatchMode: \"strict\",\n  toolArgsMatchMode: \"exact\",  \u002F\u002F 默认值\n  toolArgsMatchOverrides: {\n    get_weather: (x, y) => {\n      return typeof x.city === \"string\" &&\n        typeof y.city === \"string\" &&\n        x.city.toLowerCase() === y.city.toLowerCase();\n    },\n  }\n});\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result);\n```\n\n```\n{\n  'key': 'trajectory_strict_match',\n  'score': true,\n}\n```\n\n\u003C\u002Fdetails>\n\n这种灵活性使您能够处理仅对特定工具调用放宽 LLM 生成参数相等性的情况（例如，将 `\"san francisco\"` 视为等于 `\"San Francisco\"`）。\n\n## 轨迹 LLM 作为裁判\n\nLLM 作为裁判的轨迹评估器使用 LLM 来评估轨迹。与轨迹匹配评估器不同，它不需要参考轨迹。以下是一个示例：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT\n\nevaluator = create_trajectory_llm_as_judge(\n  prompt=TRAJECTORY_ACCURACY_PROMPT,\n  model=\"openai:o3-mini\"\n)\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n]\neval_result = evaluator(\n    outputs=outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': True,\n    'comment': '提供的智能体轨迹是合理的...'\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  TRAJECTORY_ACCURACY_PROMPT,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst evaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT,\n  model: \"openai:o3-mini\",\n});\n\nconst outputs = [\n  {role: \"user\", content: \"旧金山的天气如何？\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        }\n      }\n    ],\n  },\n  {role: \"tool\", content: \"旧金山现在是80华氏度，晴朗。\"},\n  {role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst result = await evaluator({ outputs });\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': True,\n    'comment': '提供的智能体轨迹是合理的...'\n}\n```\n\u003C\u002Fdetails>\n\n如果你有一个参考轨迹，可以在提示中添加一个额外的变量，并传入该参考轨迹。下面我们使用预构建的 `TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE` 提示，其中包含一个 `reference_outputs` 变量：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nimport json\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE\n\nevaluator = create_trajectory_llm_as_judge(\n  prompt=TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n  model=\"openai:o3-mini\"\n)\noutputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n]\nreference_outputs = [\n    {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"San Francisco\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"旧金山现在80华氏度，晴朗。\"},\n    {\"role\": \"assistant\", \"content\": \"旧金山的天气是80˚，晴朗。\"},\n]\neval_result = evaluator(\n    outputs=outputs,\n    reference_outputs=reference_outputs,\n)\n\nprint(eval_result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': True,\n    'comment': '提供的智能体轨迹与参考轨迹一致。两条轨迹都从相同的用户查询开始，随后通过工具调用正确地查询了天气信息。尽管参考轨迹使用了“San Francisco”，而提供的轨迹使用了“SF”，并且在格式上存在细微差异（degrees vs. ˚），但这些差异并不影响过程的正确性或关键步骤。因此，评分应为：真。'\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n  type FlexibleChatCompletionMessage,\n} from \"agentevals\";\n\nconst evaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,\n  model: \"openai:o3-mini\",\n});\n\nconst outputs = [\n  {role: \"user\", content: \"旧金山的天气如何？\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        }\n      }\n    ],\n  },\n  {role: \"tool\", content: \"旧金山现在80华氏度，晴朗。\"},\n  {role: \"assistant\", content: \"旧金山的天气是80华氏度，晴朗。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst referenceOutputs = [\n  {role: \"user\", content: \"旧金山的天气如何？\"},\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"San Francisco\" }),\n        }\n      }\n    ],\n  },\n  {role: \"tool\", content: \"旧金山现在80华氏度，晴朗。\"},\n  {role: \"assistant\", content: \"旧金山的天气是80˚，晴朗。\"},\n] satisfies FlexibleChatCompletionMessage[];\n\nconst result = await evaluator({\n  outputs,\n  referenceOutputs,\n});\n\nconsole.log(result)\n```\n\n```\n{\n    'key': 'trajectory_accuracy',\n    'score': true,\n    'comment': '提供的智能体轨迹与参考轨迹一致。两条轨迹都从相同的用户查询开始，随后通过工具调用正确地查询了天气信息。尽管参考轨迹使用了“San Francisco”，而提供的轨迹使用了“SF”，并且在格式上存在细微差异（degrees vs. ˚），但这些差异并不影响过程的正确性或关键步骤。因此，评分应为：真。'\n}\n```\n\u003C\u002Fdetails>\n\n`create_trajectory_llm_as_judge` 接受与 `openevals` 中的 [`create_llm_as_judge`](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals?tab=readme-ov-file#llm-as-judge) 相同的参数，因此你可以根据需要自定义提示和评分输出。\n\n除了 `prompt` 和 `model` 之外，还提供以下参数：\n\n- `continuous`: 一个布尔值，用于设置评估器是否返回介于0到1之间的浮点分数，而不是二元分数。默认值为 `False`。\n- `choices`: 一个浮点数列表，用于设置评估器可能的分数范围。\n- `system`: 一个字符串，用于通过在提示的其他部分之前添加系统消息来设置裁判模型的系统提示。\n- `few_shot_examples`: 一个示例字典列表，附加到提示的末尾。这对于向裁判模型提供良好和不良输出的示例非常有用。所需的结构如下所示：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfew_shot_examples = [\n    {\n        \"inputs\": \"天空是什么颜色？\",\n        \"outputs\": \"天空是红色。\",\n        \"reasoning\": \"天空是红色是因为现在是傍晚。\",\n        \"score\": 1,\n    }\n]\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nconst fewShotExamples = [\n  {\n    inputs: \"天空是什么颜色？\",\n    outputs: \"天空是红色。\",\n    reasoning: \"天空是红色是因为现在是傍晚。\",\n    score: 1,\n  }\n];\n```\n\u003C\u002Fdetails>\n\n有关最新参数列表，请参阅 [`openevals`](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals?tab=readme-ov-file#llm-as-judge) 仓库。\n\n## 图轨迹\n\n对于像 [LangGraph](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Flanggraph) 这样的将智能体建模为图的框架，用访问过的节点来表示轨迹，而不是消息，可能会更方便。`agentevals` 包含一类称为 **图轨迹** 评估器的评估器，它们专为这种格式设计，并提供便捷的工具来从 LangGraph 线程中提取轨迹，包括不同的对话轮次和中断。\n\n下面的示例将使用 LangGraph 和内置的格式化工具，但图评估器接受以下通用格式的输入：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nclass GraphTrajectory(TypedDict):\n    # 仅在指定参考输出时设置\n    inputs: Optional[list[dict]]\n    results: list[dict]\n    steps: list[list[str]]\n    \ndef evaluator(\n    *,\n    inputs: Optional[Union[dict, list]] = None,\n    outputs: GraphTrajectory,\n    reference_outputs: Optional[GraphTrajectory] = None,\n) -> ...\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nexport type GraphTrajectory = {\n  inputs?: (Record\u003Cstring, unknown> | null)[];\n  results: Record\u003Cstring, unknown>[];\n  steps: string[][];\n};\n\nconst evaluator: ({ inputs, outputs, referenceOutputs, ...extra }: {\n    inputs: (string | Record\u003Cstring, unknown> | null)[] | {\n        inputs: (string | Record\u003Cstring, unknown> | null)[];\n    };\n    outputs: GraphTrajectory;\n    referenceOutputs?: GraphTrajectory;\n    [key: string]: unknown;\n}) => ...\n```\n\u003C\u002Fdetails>\n\n其中 `inputs` 是图的输入列表（或带有键名 `\"inputs\"` 的字典），其每个元素代表线程中一次新调用的开始；`results` 表示线程中每一轮的最终输出；`steps` 则表示每一轮所采取的内部步骤。\n\n### 图轨迹 LLM 作为评判者\n\n此评估器类似于 `trajectory_llm_as_judge` 评估器，但它处理的是图轨迹，而非消息轨迹。下面我们设置一个 LangGraph 智能体，使用内置工具从中提取轨迹，并将其传递给评估器。首先，我们设置图、调用它，然后提取轨迹：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom agentevals.graph_trajectory.utils import (\n    extract_langgraph_trajectory_from_thread,\n)\nfrom agentevals.graph_trajectory.llm import create_graph_trajectory_llm_as_judge\n\nfrom langgraph.prebuilt import create_react_agent\nfrom langgraph.checkpoint.memory import MemorySaver\nfrom langgraph.types import Command, interrupt\n\nfrom langchain_core.tools import tool\n\n@tool\ndef search(query: str):\n    \"\"\"用于上网搜索的工具.\"\"\"\n    user_answer = interrupt(\"告诉我问题的答案。\")\n    return user_answer\n\ntools = [search]\n\ncheckpointer = MemorySaver()\ngraph = create_react_agent(\n    model=\"gpt-4o-mini\",\n    checkpointer=checkpointer,\n    tools=[search],\n)\n\ngraph.invoke(\n    {\"messages\": [{\"role\": \"user\", \"content\": \"旧金山的天气怎么样？\"}]},\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n# 恢复智能体并发出新命令，模拟人机协作流程\ngraph.invoke(\n    Command(resume=\"现在正在下雨，温度是70华氏度！\"),\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n\n# 从前两次线程运行中提取轨迹\nextracted_trajectory = extract_langgraph_trajectory_from_thread(\n    graph, {\"configurable\": {\"thread_id\": \"1\"}}\n)\n\nprint(extracted_trajectory)\n```\n\n```\n{\n  'inputs': [{\n      '__start__': {\n          'messages': [\n              {'role': 'user', 'content': '旧金山的天气怎么样？'}\n          ]}\n      }, \n      '__resuming__': {\n          'messages': [\n              {'role': 'user', 'content': '现在正在下雨，温度是70华氏度！'}\n          ]}\n      ],\n      'outputs': {\n          'results': [\n            {},\n            {\n                'messages': [\n                    {'role': 'ai', 'content': '旧金山目前的天气是阴雨，气温为70华氏度。'}\n                ]\n            }\n        ],\n        'steps': [\n            ['__start__', 'agent', 'tools', '__interrupt__'],\n            ['agent']\n        ]\n    }\n}\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { tool } from \"@langchain\u002Fcore\u002Ftools\";\nimport { ChatOpenAI } from \"@langchain\u002Fopenai\";\nimport { createReactAgent } from \"@langchain\u002Flanggraph\u002Fprebuilt\";\nimport { MemorySaver, interrupt } from \"@langchain\u002Flanggraph\";\nimport { z } from \"zod\";\nimport { extractLangGraphTrajectoryFromThread } from \"agentevals\";\n\nconst search = tool((_): string => {\n  const userAnswer = interrupt(\"告诉我问题的答案。\")\n  return userAnswer;\n}, {\n  name: \"search\",\n  description: \"用于上网搜索的工具。\",\n  schema: z.object({\n      query: z.string()\n  })\n})\n\nconst tools = [search];\n\n\u002F\u002F 创建检查点存储\nconst checkpointer = new MemorySaver();\n\n\u002F\u002F 创建 React 智能体\nconst graph = createReactAgent({\n  llm: new ChatOpenAI({ model: \"gpt-4o-mini\" }),\n  tools,\n  checkpointer,\n});\n\n\u002F\u002F 首次调用图，发送初始消息\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"旧金山的天气怎么样？\"}] },\n  { configurable: { thread_id: \"1\" } }\n);\n\n\u002F\u002F 恢复智能体并发送新命令（模拟人机协作）\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"现在正在下雨，温度是70华氏度！\"}] },\n  { configurable: { thread_id: \"1\" } }\n);\n\nconst extractedTrajectory = await extractLangGraphTrajectoryFromThread(\n  graph,\n  { configurable: { thread_id: \"1\" } },\n);\n\nconsole.log(extractedTrajectory);\n```\n\n```\n{\n  'inputs': [{\n      '__start__': {\n          'messages': [\n              {'role': 'user', 'content': '旧金山的天气怎么样？'}\n          ]}\n      }, \n      '__resuming__': {\n          'messages': [\n              {'role': 'user', 'content': '现在正在下雨，温度是70华氏度！'}\n          ]}\n      ],\n      'outputs': {\n          'results': [\n            {},\n            {\n                'messages': [\n                    {'role': 'ai', 'content': '旧金山目前的天气是阴雨，气温为70华氏度。'}\n                ]\n            }\n        ],\n        'steps': [\n            ['__start__', 'agent', 'tools', '__interrupt__'],\n            ['agent']\n        ]\n    }\n}\n```\n\u003C\u002Fdetails>\n\n现在，我们可以将提取的轨迹传递给评估器：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\ngraph_trajectory_evaluator = create_graph_trajectory_llm_as_judge(\n    model=\"openai:o3-mini\",\n)\n\nres = graph_trajectory_evaluator(\n    inputs=extracted_trajectory[\"inputs\"],\n    outputs=extracted_trajectory[\"outputs\"],\n)\n\nprint(res)\n```\n\n```\n{\n  'key': 'graph_trajectory_accuracy',\n  'score': True,\n  'comment': '整个流程遵循逻辑顺序：对话从用户的请求开始，代理通过其内部步骤（包括调用工具）处理请求，期间会中断以获取更多输入，最后恢复并给出自然语言答案。每一步都符合评分标准中的设计意图，整体路径相对高效，语义上也与典型的查询解决轨迹一致。因此，得分应为：true。'\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { createGraphTrajectoryLLMAsJudge } from \"agentevals\";\n\nconst graphTrajectoryEvaluator = createGraphTrajectoryLLMAsJudge({\n    model: \"openai:o3-mini\",\n})\n\nconst res = await graphTrajectoryEvaluator({\n  inputs: extractedTrajectory.inputs,\n  outputs: extractedTrajectory.outputs,\n});\n\nconsole.log(res);\n```\n\n```\n{\n  'key': 'graph_trajectory_accuracy',\n  'score': True,\n  'comment': '整个流程遵循逻辑顺序：对话从用户的请求开始，代理通过其内部步骤（包括调用工具）处理请求，期间会中断以获取更多输入，最后恢复并给出自然语言答案。每一步都符合评分标准中的设计意图，整体路径相对高效，语义上也与典型的查询解决轨迹一致。因此，得分应为：true。'\n}\n```\n\u003C\u002Fdetails>\n\n需要注意的是，尽管该评估器通常接受 `inputs`、`outputs` 和 `reference_outputs` 参数，但它会在内部将 `inputs` 和 `outputs` 合并成一个 `thread`。因此，如果你希望自定义提示词，你的提示词中也应包含 `thread` 输入变量：\n\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nCUSTOM_PROMPT = \"\"\"你是一位数据标注专家。\n你的任务是评估AI代理在解决用户查询过程中内部步骤的准确性。\n\n\u003C评分标准>\n  准确的轨迹：\n  - 各步骤之间逻辑清晰\n  - 展现明确的进展\n  - 非常高效，工具调用次数不超过一次\n  - 如果有参考轨迹，则语义上与之完全一致\n\u003C\u002F评分标准>\n\n\u003C说明>\n  请对以下对话线程进行评分，判断代理的整体步骤是否逻辑清晰且相对高效。\n  在轨迹中，“__start__”表示代理的初始入口点，“__interrupt__”则表示代理为了等待来自其他来源（例如人工参与环节）的额外数据而中断：\n\u003C\u002F说明>\n\n\u003Cthread>\n{thread}\n\u003C\u002Fthread>\n\n{reference_outputs}\n\"\"\"\n\nevaluator = create_graph_trajectory_llm_as_judge(\n    prompt=CUSTOM_PROMPT,\n    model=\"openai:o3-mini\",\n)\nres = await evaluator(\n    inputs=extracted_trajectory[\"inputs\"],\n    outputs=extracted_trajectory[\"outputs\"],   \n)\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nconst CUSTOM_PROMPT = `你是一位数据标注专家。\n你的任务是评估AI代理在解决用户查询过程中内部步骤的准确性。\n\n\u003C评分标准>\n  准确的轨迹：\n  - 各步骤之间逻辑清晰\n  - 展现明确的进展\n  - 非常高效，工具调用次数不超过一次\n  - 如果有参考轨迹，则语义上与之完全一致\n\u003C\u002F评分标准>\n\n\u003C说明>\n  请对以下对话线程进行评分，判断代理的整体步骤是否逻辑清晰且相对高效。\n  在轨迹中，“__start__”表示代理的初始入口点，“__interrupt__”则表示代理为了等待来自其他来源（例如人工参与环节）的额外数据而中断：\n\u003C\u002F说明>\n\n\u003Cthread>\n{thread}\n\u003C\u002Fthread>\n\n{reference_outputs}\n`\n\nconst graphTrajectoryEvaluator = createGraphTrajectoryLLMAsJudge({\n  prompt: CUSTOM_PROMPT,\n  model: \"openai:o3-mini\",\n})\nconst res = await graphTrajectoryEvaluator({\n  inputs: extractedTrajectory.inputs,\n  outputs: extractedTrajectory.outputs,\n});\n```\n\u003C\u002Fdetails>\n\n为了正确地将其格式化到提示词中，`reference_outputs` 应该以与 `outputs` 相同的 `GraphTrajectory` 对象形式传递。\n\n另外请注意，与其他 LLM-as-judge 评估器一样，你也可以向评估器传递额外参数，以便将其格式化到提示词中。\n\n\n\n### 图形轨迹严格匹配\n\n`graph_trajectory_strict_match` 评估器是一个简单的评估器，用于检查提供的图形轨迹中的步骤是否与参考轨迹完全一致。\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom agentevals.graph_trajectory.utils import (\n    extract_langgraph_trajectory_from_thread,\n)\nfrom agentevals.graph_trajectory.strict import graph_trajectory_strict_match\n\n\nfrom langgraph.prebuilt import create_react_agent\nfrom langgraph.checkpoint.memory import MemorySaver\nfrom langgraph.types import Command, interrupt\n\nfrom langchain_core.tools import tool\n\n@tool\ndef search(query: str):\n    \"\"\"用于上网搜索的工具\"\"\"\n    user_answer = interrupt(\"请告诉我问题的答案。\")\n    return user_answer\n\ntools = [search]\n\ncheckpointer = MemorySaver()\ngraph = create_react_agent(\n    model=\"gpt-4o-mini\",\n    checkpointer=checkpointer,\n    tools=[search],\n)\n\ngraph.invoke(\n    {\"messages\": [{\"role\": \"user\", \"content\": \"旧金山的天气怎么样？\"}]},\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n# 恢复代理执行，并发出新命令，模拟人工参与的工作流\ngraph.invoke(\n    Command(resume=\"现在正在下雨，气温约70华氏度！\"),\n    config={\"configurable\": {\"thread_id\": \"1\"}},\n)\n\n# 从前两次线程运行中提取轨迹\nextracted_trajectory = extract_langgraph_trajectory_from_thread(\n    graph, {\"configurable\": {\"thread_id\": \"1\"}}\n)\n\nreference_trajectory = {\n    # 不用于严格匹配\n    \"results\": [],\n    \"steps\": [[\"__start__\", \"agent\", \"tools\", \"__interrupt__\"], [\"agent\"]],\n}\n\nres = graph_trajectory_strict_match(\n    outputs=extracted_trajectory[\"outputs\"],\n    reference_outputs=reference_trajectory,\n)\n\nprint(res)\n```\n\n```\n{\n  'key': 'graph_trajectory_strict_match',\n  'score': True,\n}\n```\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { tool } from \"@langchain\u002Fcore\u002Ftools\";\nimport { ChatOpenAI } from \"@langchain\u002Fopenai\";\nimport { createReactAgent } from \"@langchain\u002Flanggraph\u002Fprebuilt\";\nimport { MemorySaver, interrupt } from \"@langchain\u002Flanggraph\";\nimport { z } from \"zod\";\nimport { extractLangGraphTrajectoryFromThread, graphTrajectoryStrictMatch } from \"agentevals\";\n\nconst search = tool((_): string => {\n  const userAnswer = interrupt(\"Tell me the answer to the question.\")\n  return userAnswer;\n}, {\n  name: \"search\",\n  description: \"Call to surf the web.\",\n  schema: z.object({\n      query: z.string()\n  })\n})\n\nconst tools = [search];\n\n\u002F\u002F 创建检查点\nconst checkpointer = new MemorySaver();\n\n\u002F\u002F 创建 React 代理\nconst graph = createReactAgent({\n  llm: new ChatOpenAI({ model: \"gpt-4o-mini\" }),\n  tools,\n  checkpointer,\n});\n\n\u002F\u002F 使用初始消息调用图\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"what's the weather in sf?\" }] },\n  { configurable: { thread_id: \"1\" } }\n);\n\n\u002F\u002F 以新命令恢复代理（模拟人机交互）\nawait graph.invoke(\n  { messages: [{ role: \"user\", content: \"It is rainy and 70 degrees!\" }] },\n  { configurable: { thread_id: \"1\" } }\n);\n\nconst extractedTrajectory = await extractLangGraphTrajectoryFromThread(\n  graph,\n  { configurable: { thread_id: \"1\" } },\n);\n\nconst referenceTrajectory = {\n  results: [],\n  steps: [[\"__start__\", \"agent\", \"tools\", \"__interrupt__\"], [\"agent\"]],\n}\n\nconst result = await graphTrajectoryStrictMatch({\n  outputs: trajectory.outputs,\n  referenceOutputs: referenceOutputs!,\n});\n\nconsole.log(result);\n```\n\n```\n{\n  'key': 'graph_trajectory_strict_match',\n  'score': True,\n}\n```\n\u003C\u002Fdetails>\n\n# Python 异步支持\n\n所有 `agentevals` 评估器都支持 Python 的 `asyncio`。按照惯例，使用工厂函数的评估器会在函数名中的 `create_` 后面立即加上 `async`（例如 `create_async_trajectory_llm_as_judge`），而直接使用的评估器则会在名称末尾加上 `async`（如 `trajectory_strict_match_async`）。\n\n以下是异步使用 `create_async_llm_as_judge` 评估器的示例：\n\n```python\nfrom agentevals.trajectory.llm import create_async_trajectory_llm_as_judge\n\nevaluator = create_async_llm_as_judge(\n    prompt=\"What is the weather in {inputs}?\",\n)\n\nresult = await evaluator(inputs=\"San Francisco\")\n```\n\n如果您直接使用 OpenAI 客户端，请记得将 `AsyncOpenAI` 作为 `judge` 参数传入：\n\n```python\nfrom openai import AsyncOpenAI\n\nevaluator = create_async_llm_as_judge(\n    prompt=\"What is the weather in {inputs}?\",\n    judge=AsyncOpenAI(),\n    model=\"o3-mini\",\n)\n\nresult = await evaluator(inputs=\"San Francisco\")\n```\n\n# LangSmith 集成\n\n为了长期跟踪实验，您可以将评估结果记录到 [LangSmith](https:\u002F\u002Fsmith.langchain.com\u002F) 上。LangSmith 是一个用于构建生产级 LLM 应用程序的平台，包含追踪、评估和实验工具。\n\nLangSmith 目前提供两种方式来运行评估：一种是通过 [pytest](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fpytest)（Python）或 [Vitest\u002FJest](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fvitest_jest) 集成，另一种是使用 `evaluate` 函数。下面我们将简要介绍如何使用这两种方法运行评估。\n\n## Pytest 或 Vitest\u002FJest\n\n首先，请按照[这些说明](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fpytest)设置 LangSmith 的 pytest 运行器，或者按照这些说明设置 [Vitest 或 Jest](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fhow_to_guides\u002Fvitest_jest)，并设置相应的环境变量：\n\n\n```bash\nexport LANGSMITH_API_KEY=\"your_langsmith_api_key\"\nexport LANGSMITH_TRACING=\"true\"\n```\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n然后，创建一个名为 `test_trajectory.py` 的文件，内容如下：\n\n```python\nimport pytest\nimport json\n\nfrom langsmith import testing as t\n\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge\n\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    model=\"openai:o3-mini\",\n)\n\n@pytest.mark.langsmith\ndef test_trajectory_accuracy():\n    outputs = [\n        {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": json.dumps({\"city\": \"旧金山\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n        {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n    ]\n    reference_outputs = [\n        {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": json.dumps({\"city\": \"旧金山\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n        {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n    ]\n\n    t.log_inputs({})\n    t.log_outputs({\"messages\": outputs})\n    t.log_reference_outputs({\"messages\": reference_outputs})\n\n    trajectory_evaluator(\n      outputs=outputs,\n      reference_outputs=reference_outputs\n    )\n```\n\n请注意，在创建评估器时，我们添加了一个 `feedback_key` 参数。这将用于在 LangSmith 中为反馈命名。\n\n现在，使用 pytest 运行评估：\n\n```bash\npytest test_trajectory.py --langsmith-output\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n然后，创建一个名为 `test_trajectory.eval.ts` 的文件，内容如下：\n\n```ts\nimport * as ls from \"langsmith\u002Fvitest\";\n\u002F\u002F import * as ls from \"langsmith\u002Fjest\";\n\nimport { createTrajectoryLLMAsJudge } from \"agentevals\";\n\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  model: \"openai:o3-mini\",\n});\n\nls.describe(\"轨迹准确性\", () => {\n  ls.test(\"准确的轨迹\", {\n    inputs: {\n      messages: [\n        {\n          role: \"user\",\n          content: \"旧金山的天气如何？\"\n        }\n      ]\n    },\n    referenceOutputs: {\n      messages: [\n        {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": JSON.stringify({\"city\": \"旧金山\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n        {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n      ],\n    },\n  }, async ({ inputs, referenceOutputs }) => {\n    const outputs = [\n        {\"role\": \"user\", \"content\": \"旧金山的天气如何？\"},\n        {\n            \"role\": \"assistant\",\n            \"content\": \"\",\n            \"tool_calls\": [\n                {\n                    \"function\": {\n                        \"name\": \"get_weather\",\n                        \"arguments\": JSON.stringify({\"city\": \"旧金山\"}),\n                    }\n                }\n            ],\n        },\n        {\"role\": \"tool\", \"content\": \"旧金山现在是80华氏度，晴朗。\"},\n        {\"role\": \"assistant\", \"content\": \"旧金山的天气是80华氏度，晴朗。\"},\n    ];\n    ls.logOutputs({ messages: outputs });\n\n    await trajectoryEvaluator({\n      inputs,\n      outputs,\n      referenceOutputs,\n    });\n  });\n});\n```\n\n现在，使用您选择的运行器运行评估：\n\n```bash\nvitest run test_trajectory.eval.ts\n```\n\n\u003C\u002Fdetails>\n\n预构建评估器的反馈将自动记录在 LangSmith 中，以表格形式显示在您的终端中，如下所示：\n\n![终端结果](https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Flangchain-ai_agentevals_readme_84d888bed60c.png)\n\n您还应在 LangSmith 的实验视图中看到结果：\n\n![LangSmith 结果](https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Flangchain-ai_agentevals_readme_47a167b71a1c.png)\n\n## 评估\n\n或者，您也可以在 LangSmith 中[创建数据集](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Fconcepts#dataset-curation)，并使用您创建的评估器与 LangSmith 的 [`evaluate`](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation#8-run-and-view-results) 函数一起使用：\n\n\u003Cdetails open>\n\u003Csummary>Python\u003C\u002Fsummary>\n\n```python\nfrom langsmith import Client\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge\n\nclient = Client()\n\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    model=\"openai:o3-mini\",\n)\n\nexperiment_results = client.evaluate(\n    # 这是一个示例目标函数，替换为您实际的基于 LLM 的系统\n    lambda inputs: \"天空是什么颜色？\",\n    data=\"示例数据集\",\n    evaluators=[\n        trajectory_evaluator\n    ]\n)\n```\n\n\u003C\u002Fdetails>\n\n\u003Cdetails>\n\u003Csummary>TypeScript\u003C\u002Fsummary>\n\n```ts\nimport { evaluate } from \"langsmith\u002Fevaluation\";\nimport { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from \"agentevals\";\n\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  model: \"openai:o3-mini\",\n  prompt: TRAJECTORY_ACCURACY_PROMPT\n});\n\nawait evaluate(\n  (inputs) => [\n      {role: \"user\", content: \"旧金山的天气如何？\"},\n      {\n          role: \"assistant\",\n          content：\"\",\n          tool_calls：[\n              {\n                  function: {\n                      name：“get_weather”，\n                      arguments：JSON.stringify({city：“旧金山”}),\n                  }\n              }\n         ],\n      },\n      {role：“tool”，content：“旧金山现在是80华氏度，晴朗。”},\n      {role：“assistant”，content：“旧金山的天气是80华氏度，晴朗。”},\n    ],\n  {\n    data：datasetName，\n    evaluators：[trajectoryEvaluator]，\n  }\n);\n```\n\u003C\u002Fdetails>\n\n# 谢谢！\n\n我们希望 `agentevals` 能帮助您更轻松地评估您的 LLM 代理！\n\n如果您有任何问题、意见或建议，请提交一个问题，或通过 X [@LangChainAI](https:\u002F\u002Fx.com\u002Flangchainai) 联系我们。","# Agentevals 快速上手指南\n\n`agentevals` 是一个专为评估 **Agent 轨迹（Agent Trajectory）** 设计的开源工具包。它帮助开发者理解 Agent 在解决问题过程中的中间步骤，判断其执行路径是否符合预期或逻辑正确。\n\n## 环境准备\n\n在开始之前，请确保满足以下要求：\n\n*   **编程语言**：Python 3.8+ 或 Node.js (TypeScript)\n*   **API Key**：如果使用基于 LLM 的评估器（如 `o3-mini`），需要配置 OpenAI API Key。\n*   **依赖库**：\n    *   Python 用户通常需安装 `langchain-openai`（默认包含）或直接使用 `openai`。\n    *   TypeScript 用户需安装 `@langchain\u002Fcore`。\n\n设置环境变量（以 Bash\u002FZsh 为例）：\n```bash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n```\n\n## 安装步骤\n\n根据你使用的开发语言，选择对应的安装命令。\n\n### Python\n```bash\npip install agentevals\n```\n> **提示**：国内用户如遇下载缓慢，可使用清华源加速：\n> ```bash\n> pip install agentevals -i https:\u002F\u002Fpypi.tuna.tsinghua.edu.cn\u002Fsimple\n> ```\n\n若需直接使用 OpenAI 客户端进行评估，建议额外安装：\n```bash\npip install openai\n```\n\n### TypeScript\n```bash\nnpm install agentevals @langchain\u002Fcore\n```\n\n若需直接使用 OpenAI 客户端：\n```bash\nnpm install openai\n```\n\n## 基本使用\n\n以下示例展示如何使用内置的 `o3-mini` 模型作为裁判，对 Agent 的执行轨迹进行准确性评估。\n\n### Python 示例\n\n```python\nimport json\nfrom agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT\n\n# 1. 创建评估器\ntrajectory_evaluator = create_trajectory_llm_as_judge(\n    prompt=TRAJECTORY_ACCURACY_PROMPT,\n    model=\"openai:o3-mini\",\n)\n\n# 2. 准备模拟的 Agent 轨迹 (OpenAI 消息格式)\noutputs = [\n    {\"role\": \"user\", \"content\": \"What is the weather in SF?\"},\n    {\n        \"role\": \"assistant\",\n        \"content\": \"\",\n        \"tool_calls\": [\n            {\n                \"function\": {\n                    \"name\": \"get_weather\",\n                    \"arguments\": json.dumps({\"city\": \"SF\"}),\n                }\n            }\n        ],\n    },\n    {\"role\": \"tool\", \"content\": \"It's 80 degrees and sunny in SF.\"},\n    {\"role\": \"assistant\", \"content\": \"The weather in SF is 80 degrees and sunny.\"},\n]\n\n# 3. 运行评估\neval_result = trajectory_evaluator(outputs=outputs)\n\nprint(eval_result)\n```\n\n**输出结果示例：**\n```text\n{\n  'key': 'trajectory_accuracy',\n  'reasoning': 'The trajectory accurately follows the user's request...',\n  'score': True\n}\n```\n\n### TypeScript 示例\n\n```ts\nimport {\n  createTrajectoryLLMAsJudge,\n  type FlexibleChatCompletionMessage,\n  TRAJECTORY_ACCURACY_PROMPT,\n} from \"agentevals\";\n\n\u002F\u002F 1. 创建评估器\nconst trajectoryEvaluator = createTrajectoryLLMAsJudge({\n  prompt: TRAJECTORY_ACCURACY_PROMPT,\n  model: \"openai:o3-mini\",\n});\n\n\u002F\u002F 2. 准备模拟的 Agent 轨迹\nconst outputs = [\n  { role: \"user\", content: \"What is the weather in SF?\" },\n  {\n    role: \"assistant\",\n    content: \"\",\n    tool_calls: [\n      {\n        function: {\n          name: \"get_weather\",\n          arguments: JSON.stringify({ city: \"SF\" }),\n        },\n      },\n    ],\n  },\n  { role: \"tool\", content: \"It's 80 degrees and sunny in SF.\" },\n  {\n    role: \"assistant\",\n    content: \"The weather in SF is 80 degrees and sunny.\",\n  },\n] satisfies FlexibleChatCompletionMessage[];\n\n\u002F\u002F 3. 运行评估\nconst evalResult = await trajectoryEvaluator({\n  outputs,\n});\n\nconsole.log(evalResult);\n```\n\n**输出结果示例：**\n```text\n{\n    key: 'trajectory_accuracy',\n    score: true,\n    comment: '...'\n}\n```\n\n### 核心功能说明\n*   **轨迹匹配 (Trajectory Match)**：支持严格匹配 (`strict`)、无序匹配 (`unordered`) 以及子集\u002F超集匹配，用于对比实际执行路径与预期路径。\n*   **LLM 作为裁判 (LLM-as-judge)**：利用大模型智能判断轨迹的逻辑性、准确性和效率，而不仅仅是字符串比对。\n*   **灵活配置**：可自定义工具参数匹配模式，适应不同的业务场景。","某电商公司的 AI 研发团队正在优化一个能自动处理“退货 + 换货”复杂流程的智能客服 Agent，该 Agent 需要自主决定调用查询库存、验证订单或生成物流单等多个工具。\n\n### 没有 agentevals 时\n- **黑盒调试困难**：当 Agent 错误地先调用了“生成物流单”而非“验证订单”时，开发者只能靠肉眼逐行检查冗长的日志，难以快速定位逻辑断裂点。\n- **回归测试盲目**：每次更新提示词或修改工具定义后，团队无法量化评估这些改动是否导致其他正常流程（如单纯查询订单）出现退化，只能依赖少量人工抽检。\n- **缺乏中间态洞察**：传统的最终结果评估（只看回复是否正确）忽略了推理过程，导致那些“蒙对答案但路径混乱”的低质量轨迹被误判为合格，埋下隐患。\n- **评估标准不统一**：不同开发人员对“好的执行路径”主观理解不一致，导致代码审查时对 Agent 行为优劣的争论频发，缺乏客观数据支撑。\n\n### 使用 agentevals 后\n- **轨迹级自动诊断**：利用 agentevals 内置的评估器，系统能自动分析 Agent 的每一步工具调用序列，瞬间识别出“未验证订单即发货”的逻辑违规并给出具体原因。\n- **量化回归监控**：在 CI\u002FCD 流水线中集成 agentevals，每次代码提交后自动运行数百个历史案例的轨迹评估，用具体的准确率分数确认新改动未破坏现有能力。\n- **过程质量深察**：不仅关注最终回复，更通过 `trajectory_accuracy` 等指标深入评判推理链条的逻辑性与效率，精准剔除那些步骤冗余或顺序错误的“伪成功”案例。\n- **标准化评估体系**：团队基于 agentevals 提供的预置模板建立了统一的评分标准，让所有成员对 Agent 行为的优劣有了共同的客观语言，大幅减少沟通成本。\n\nagentevals 将原本模糊不可控的 Agent 推理过程转化为可度量、可优化的清晰数据流，让智能体开发从“凭感觉调试”迈向“数据驱动迭代”。","https:\u002F\u002Foss.gittoolsai.com\u002Fimages\u002Flangchain-ai_agentevals_84d888be.png","langchain-ai","LangChain","https:\u002F\u002Foss.gittoolsai.com\u002Favatars\u002Flangchain-ai_8e6aaeef.png","",null,"support@langchain.dev","https:\u002F\u002Fwww.langchain.com","https:\u002F\u002Fgithub.com\u002Flangchain-ai",[82,86,90],{"name":83,"color":84,"percentage":85},"Python","#3572A5",62.6,{"name":87,"color":88,"percentage":89},"TypeScript","#3178c6",36.4,{"name":91,"color":92,"percentage":42},"JavaScript","#f1e05a",550,41,"2026-04-16T06:42:31","MIT","未说明",{"notes":99,"python":97,"dependencies":100},"该工具支持 Python 和 TypeScript。若使用 LLM 作为评判器（LLM-as-judge），需配置 OpenAI API Key 环境变量。默认依赖 LangChain 的 OpenAI 集成，也可选择直接安装 openai 客户端库。",[101,102,103],"langchain-core","langchain-openai","openai",[13,105],"其他","2026-03-27T02:49:30.150509","2026-04-18T00:45:52.111355",[109,114,119,124,129,134],{"id":110,"question_zh":111,"answer_zh":112,"source_url":113},38397,"运行 agentevals 时遇到 '_missing _convertMessagesToOpenAIParams export' 错误怎么办？","该问题已在 versione 0.0.7 中修复，请升级依赖：`npm install agentevals@0.0.7`。\n\n如果暂时无法升级，且你的代码库中未实际使用 OpenAI 模型，可以在测试配置（如 vitest）中添加模拟（mock）作为临时解决方案：\n\n1. 在测试文件中添加以下 mock 代码：\n```typescript\nvi.mock('@langchain\u002Fopenai', async () => {\n  const actual = await vi.importActual\u003Ctypeof import('@langchain\u002Fopenai')>('@langchain\u002Fopenai');\n  return {\n    ...actual,\n    _convertMessagesToOpenAIParams: vi.fn(() => []),\n  };\n});\n```\n2. 在 `vitest.config.ts` 中将 `agentevals` 添加到内联依赖：\n```typescript\ntest: {\n    deps: {\n        inline: ['agentevals']\n    }\n}\n```","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fissues\u002F63",{"id":115,"question_zh":116,"answer_zh":117,"source_url":118},38398,"从 thread id 提取 Graph Trajectory 时出现 'metadata object 缺少 writes 键' 的错误如何解决？","该问题是由 langgraph v0.5 的更新引起的，已在 agentevals 0.0.9 版本中修复。请将库升级到最新版本：\n\n- Python 用户：`pip install --upgrade agentevals` (确保版本 >= 0.0.9)\n- 参考发布页面：https:\u002F\u002Fpypi.org\u002Fproject\u002Fagentevals\u002F0.0.9\u002F","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fissues\u002F52",{"id":120,"question_zh":121,"answer_zh":122,"source_url":123},38399,"create_graph_trajectory_llm_as_judge 的同步和异步调用行为不一致导致失败怎么办？","这是一个已知的同步与异步实现差异导致的 Bug，维护者已确认并将进行修补（patch）。\n\n根本原因是同步版本传递了 `inputs` 参数，而 `_stringify_prompt_param` 函数期望的是消息轨迹而非图轨迹。在修复发布前，建议优先使用异步版本 `create_async_graph_trajectory_llm_as_judge`，或者等待包含该修复的新版本发布。","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fissues\u002F39",{"id":125,"question_zh":126,"answer_zh":127,"source_url":128},38400,"agentevals 是否支持 Python 3.10 或更低版本？","是的，支持。之前 PyPI 上的元数据标注错误导致显示仅支持 Python 3.11+，这已被修正。\n\n该库旨在支持 Python 3.9 及以上版本。请确保安装最新版（0.0.5 或更高），元数据已更正：\n`pip install --upgrade agentevals`","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fissues\u002F26",{"id":130,"question_zh":131,"answer_zh":132,"source_url":133},38401,"在使用 create_trajectory_llm_as_judge() 时，如果省略 model 参数会发生什么？","目前 README 快速开始指南中明确展示了 `model` 参数（例如 `model=\"openai:o3-mini\"`）。\n\n虽然函数签名可能允许省略该参数，但官方文档并未定义默认行为，省略它可能导致静默失败或意外行为。最佳实践是始终显式传递 `model` 参数，格式如 `'openai:gpt-4'` 或 `'anthropic:claude-3-5-sonnet-20241022'`，以确保评估器按预期工作并避免混淆。","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fissues\u002F76",{"id":135,"question_zh":136,"answer_zh":137,"source_url":138},38402,"评估运行时出现 'Could not get the current run tree' 错误但能获取结果，该如何处理？","此错误通常与 LangSmith 的追踪上下文（run tree）初始化有关。虽然评估结果可能正常返回，但该错误表明系统无法正确关联当前的执行树。\n\n解决方法：\n1. 确保已正确设置 `LANGCHAIN_API_KEY` 和 `LANGCHAIN_TRACING_V2=true` 环境变量。\n2. 检查是否在调用评估器之前正确初始化了 LangChain 的回调管理器或使用了 `langchain.traceable` 装饰器。\n3. 如果使用的是自定义 LLM（如 ChatBedrockConverse），请确保其配置与 LangSmith 追踪兼容。\n\n如果该错误不影响业务逻辑且仅需评估分数，可暂时忽略；若需完整追踪链路，请检查上述环境配置。","https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fissues\u002F38",[140,145,150,155,160,165,170,175,180,185,190,195],{"id":141,"version":142,"summary_zh":143,"released_at":144},306585,"js==0.0.7","## 变更内容\n* chore(ci): 更新 CI，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F61 中完成\n* build(deps): 将 npm_and_yarn 组中的 langchain 从 0.3.32 升级至 0.3.37，在 \u002Fjs 目录下进行，由 @dependabot[bot] 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F68 中完成\n* build(deps): 在 npm_and_yarn 组中对 1 个目录下的依赖进行了 8 次更新，由 @dependabot[bot] 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F69 中完成\n* build(deps): 在 uv 组中对 1 个目录下的依赖进行了 6 次更新，由 @dependabot[bot] 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F70 中完成\n* build(deps): 将 npm_and_yarn 组中 \u002Fjs 目录下的 rollup 从 4.34.7 升级至 4.59.0，由 @dependabot[bot] 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F71 中完成\n* build(deps): 将 npm_and_yarn 组中 \u002Fjs 目录下的 minimatch 从 3.1.2 升级至 3.1.5，由 @dependabot[bot] 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F72 中完成\n* fix(js): 修复 22 个 Dependabot 安全告警（严重 + 高危 + 中危），由 @jkennedyvz 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F67 中完成\n* fix(js): 修复来自 langchain 0.x 的导入问题，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F74 中完成\n* release(js): 发布 0.0.7 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F75 中完成\n\n## 新贡献者\n* @dependabot[bot] 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F68 中完成了首次贡献\n* @jkennedyvz 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F67 中完成了首次贡献\n\n**完整变更日志**: https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fjs==0.0.6...js==0.0.7","2026-03-03T02:49:20",{"id":146,"version":147,"summary_zh":148,"released_at":149},306586,"js==0.0.6","## 变更内容\n* 修复（JS）：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F57 中修复 OpenEvals 0.1.0 的类型定义\n* 杂项（JS）：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F59 中放宽依赖版本\n* 发布（JS）：0.0.6 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F60 中发布\n\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fpy==0.0.9...js==0.0.6","2025-09-03T20:36:48",{"id":151,"version":152,"summary_zh":153,"released_at":154},306587,"py==0.0.9","## 变更内容\n* chore(js): 由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F45 中升级 JS openevals 依赖\n* 更新文档中 graphTrajectoryEvaluator 方法的参数（JS），由 @jung-han 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F43 中完成\n* feat(js): 由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F46 中更新 openevals 0.1.0 的类型定义\n* docs: 由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F47 中修复 README 中的拼写错误\n* feat(python): 由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F54 中为 LangGraph 0.5.x 更新图轨迹工具\n* release(py): 0.0.9，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F55 中发布\n\n## 新贡献者\n* @jung-han 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F43 中完成了首次贡献\n\n**完整变更日志**: https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fpy==0.0.8...py==0.0.9","2025-07-24T17:45:04",{"id":156,"version":157,"summary_zh":158,"released_at":159},306588,"py==0.0.8","## 变更内容\n* 修复 (py)：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F42 中标准化调用图轨迹 LLM 作为评判者的评估器的方式\n* 发布 (py)：0.0.8，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F44 中完成\n\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fpy==0.0.7...py==0.0.8","2025-05-22T14:47:03",{"id":161,"version":162,"summary_zh":163,"released_at":164},306589,"py==0.0.7","## 变更内容\n* 发布(js): 0.0.5，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F34 中完成\n* 功能(py): 修复导入问题，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F35 中完成\n* 发布(py): 0.0.6，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F36 中完成\n* 发布(py): 0.0.7，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F37 中完成\n\n\n**完整变更日志**: https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fpy==0.0.6...py==0.0.7","2025-05-03T20:10:49",{"id":166,"version":167,"summary_zh":168,"released_at":169},306590,"js==0.0.5","## 变更内容\n* 文档：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F21 中更新 README\n* 文档：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F22 中更新 README\n* 文档：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F23 中对 README 进行小幅调整\n* 文档：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F24 中优化 Markdown 标题层级\n* 功能（Python、JavaScript）：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F25 中添加测试\n* 修复（Python）：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F27 中修复测试用例\n* 修复（Python）：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F28 中修正最低版本要求\n* 发布（Python）：0.0.5 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F29 中发布\n* 功能（Python、JavaScript）：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F31 中新增子集和超集工具调用参数匹配模式，并更新其他 README 内容\n* 文档：由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F32 中添加 README 更新脚本\n* 发布（Python）：0.0.6 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F33 中发布\n* 发布（JavaScript）：0.0.5 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F34 中发布\n\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fjs==0.0.4...js==0.0.5","2025-05-03T19:33:40",{"id":171,"version":172,"summary_zh":173,"released_at":174},306591,"py==0.0.6","## 变更内容\n* 功能（Python、JavaScript）：新增子集和超集工具调用参数匹配模式，以及其他 README 更新，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F31 中完成。\n* 文档：添加 README 更新脚本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F32 中完成。\n* 发布（Python）：0.0.6 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F33 中发布。\n\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fpy==0.0.5...py==0.0.6","2025-05-03T19:31:09",{"id":176,"version":177,"summary_zh":178,"released_at":179},306592,"py==0.0.5","## 变更内容\n* 发布（JS）：0.0.4，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F20 中完成\n* 文档：更新 README，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F21 中完成\n* 文档：更新 README，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F22 中完成\n* 文档：对 README 进行小幅调整，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F23 中完成\n* 文档：展平 Markdown 标题层级，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F24 中完成\n* 功能（Python、JS）：添加测试，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F25 中完成\n* 修复（Python）：修复测试用例，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F27 中完成\n* 修复（Python）：修正最低版本要求，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F28 中完成\n* 发布（Python）：0.0.5，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F29 中完成\n\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002Fpy==0.0.4...py==0.0.5","2025-03-31T20:29:25",{"id":181,"version":182,"summary_zh":183,"released_at":184},306593,"js==0.0.4","## 变更内容\n* 文档：修复 @os1ma 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F12 中的 README.md 错误\n* 修复(CI)：更新更改文件操作，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F17 中完成\n* 修复(文档)：轻微标点符号修正，由 @Niraj-Fonseka 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F15 中完成\n* 修复(Python, JS)：重构轨迹评估，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F16 中完成\n* 文档：更新 README 文件，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F18 中完成\n* 发布(Python)：0.0.4 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F19 中发布\n* 发布(JS)：0.0.4 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F20 中发布\n\n## 新贡献者\n* @os1ma 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F12 中完成了首次贡献\n* @Niraj-Fonseka 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F15 中完成了首次贡献\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002F0.0.3...js==0.0.4","2025-03-21T21:34:56",{"id":186,"version":187,"summary_zh":188,"released_at":189},306594,"py==0.0.4","## 变更内容\n* docs：修复 @os1ma 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F12 中的 README.md 错误\n* fix(ci)：更新更改文件操作，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F17 中完成\n* fix(docs)：轻微标点符号修正，由 @Niraj-Fonseka 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F15 中完成\n* fix(python,js)：重构轨迹评估，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F16 中完成\n* docs：更新 README 文件，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F18 中完成\n* release(py)：发布 0.0.4 版本，由 @jacoblee93 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F19 中完成\n\n## 新贡献者\n* @os1ma 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F12 中完成了首次贡献\n* @Niraj-Fonseka 在 https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F15 中完成了首次贡献\n\n**完整变更日志**：https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002F0.0.3...py==0.0.4","2025-03-21T21:32:57",{"id":191,"version":192,"summary_zh":193,"released_at":194},306595,"0.0.3","## What's Changed\r\n* Fix create_trajectory_llm_as_judge path in python README by @antonpibm in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F9\r\n* fix(docs): Fix imports in README by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F11\r\n* fix(python,js): Simplify trajectory formatting by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F13\r\n* release(python,js): 0.0.3 by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F14\r\n\r\n## New Contributors\r\n* @antonpibm made their first contribution in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F9\r\n\r\n**Full Changelog**: https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcompare\u002F0.0.2...0.0.3","2025-03-06T17:03:24",{"id":196,"version":197,"summary_zh":198,"released_at":199},306596,"0.0.2","## What's Changed\r\n* small changes by @isahers1 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F1\r\n* changes by @isahers1 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F2\r\n* wip by @isahers1 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F3\r\n* feat: Update deps by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F5\r\n* release(python,js): 0.0.1 by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F6\r\n* feat(python,js): Bump openevals version, refactor runner by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F7\r\n* release(python,js): 0.0.2 by @jacoblee93 in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F8\r\n\r\n## New Contributors\r\n* @isahers1 made their first contribution in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F1\r\n* @jacoblee93 made their first contribution in https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fpull\u002F5\r\n\r\n**Full Changelog**: https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fagentevals\u002Fcommits\u002F0.0.2","2025-02-25T21:08:25"]