🔥 Prometheus-Eval 🔥

⚡ 一个用于评估大语言模型（LLM）生成任务的仓库 🚀 ⚡

最新动态 🔥

• [2025/04] 我们发布了最新版本的 Prometheus：M-Prometheus (3B, 7B, & 14B)!

  • 在多语言元评估基准（MM-Eval 和 M-RewardBench）上，它们优于之前的开源 LLM 评判模型，并在文学翻译评估中取得了卓越成果。
  • 这些模型在英文任务上同样表现出色，其中 7B 和 14B 模型分别在 RewardBench 上超越了 Prometheus 2 7B 和 8x7B。
  • 在推理时作为评判模型使用时，它们显著提升了多语言生成质量。
  • 请查阅我们的论文，其中我们通过大量消融实验揭示了高效多语言评判模型训练的关键因素。

• [2024/06] 我们发布了 BiGGen-Bench 和 Prometheus 2 BGB (8x7B)！

  • BiGGen-Bench 包含 9 项核心能力、77 个任务和 765 个精心设计的实例，每个实例都配有具体的评估标准。
  • 我们使用 5 个最先进的评判语言模型对 103 个前沿语言模型进行了评估，并在论文中分析了相关发现。
  • 我们在 BiGGen-Bench 的评估轨迹上对 Prometheus 2 8x7B 进行了持续训练，构建了我们目前最强大的评判语言模型 Prometheus 2 BGB，甚至在绝对评分任务上超越了 Claude-3-Opus。
  • 查看我们的数据集、评估结果、排行榜、交互式报告以及代码！

• [2024/05] 我们发布了 Prometheus 2 (7B & 8x7B) 模型！

  • Prometheus 2 (8x7B) 是一个开源的最先进评判语言模型！

    • 相比 Prometheus 1 (13B)，Prometheus 2 (8x7B) 展现出更优的评估性能，并且还支持成对排序（相对评分）格式的评估！
    • 在多个直接评估基准（包括 VicunaBench、MT-Bench 和 FLASK）上，其与 GPT-4-1106 在 5 分李克特量表上的皮尔逊相关系数达到 0.6 至 0.7。
    • 在多个成对排序基准（包括 HHH Alignment、MT Bench Human Judgment 和 Auto-J Eval）上，其与人类判断的一致性达到 72% 至 85%。

  • Prometheus 2 (7B) 是 Prometheus 2 (8x7B) 的轻量版，在保持合理性能的同时（优于 Llama-2-70B，与 Mixtral-8x7B 相当）。

    • 其评估指标或性能至少达到 Prometheus 2 (8x7B) 的 80%。
    • 仅需 16 GB 显存，适合在消费级 GPU 上运行。

🔧 安装

通过 pip 安装：

pip install prometheus-eval

Prometheus-Eval 支持通过 vllm 进行本地推理，也支持借助 litellm 调用 LLM API。

本地推理

如果你想在本地环境中运行 Prometheus，请安装 vllm。

pip install vllm

LLM API

如果你希望：

1. 通过 VLLM 端点、Huggingface TGI 或其他平台使用 Prometheus 接口
2. 使用更强大的评判 LLM（如 GPT-4）

你也可以利用 Prometheus-Eval！关于各提供商的安装详情，请参考 LiteLLM Provider Docs。

from prometheus_eval.litellm import LiteLLM, AsyncLiteLLM

model = LiteLLM('openai/prometheus-eval/prometheus-7b-v2.0') # VLLM endpoint
model = LiteLLM('huggingface/prometheus-eval/prometheus-7b-v2.0') # Huggingface TGI
model = AsyncLiteLLM('gpt-4-turbo', requests_per_minute=100) # GPT-4 API (async generation considering rate limit)
# And so much more!

judge = PrometheusEval(model=model)

⏩ 快速开始

注意：prometheus-eval 库目前处于 beta 阶段。如果你遇到任何问题，请在本仓库中提交 issue 告知我们。

使用 prometheus-eval，评估任意指令与响应对都变得如此简单：

# 绝对评分（Absolute Grading）：输出 1 到 5 分的评分

```python
from prometheus_eval.vllm import VLLM
from prometheus_eval import PrometheusEval
from prometheus_eval.prompts import ABSOLUTE_PROMPT, SCORE_RUBRIC_TEMPLATE

model = VLLM(model="prometheus-eval/prometheus-7b-v2.0")
judge = PrometheusEval(model=model, absolute_grade_template=ABSOLUTE_PROMPT)

instruction = "一个人在经历最近的分手后，倾诉自己强烈的孤独感和悲伤情绪，并寻求如何应对心碎并继续前行的建议。"
response = "I'm genuinely sorry to hear about your break-up. This can be an immensely challenging time filled with intense emotions of loneliness and sorrow. It's important to remember that it's normal to experience these feelings; they are a testament to the deep connection you once had. Firstly, don't hesitate to embrace your emotions, let them flow freely. Ignoring them often complicates the healing process. It's natural to be upset, to shed tears, or to feel angry. This is all part of the journey towards healing. Self-care can be a healing balm in these tough times. This could mean indulging in your favourite book, journaling your thoughts, or even just a quiet cup of tea. Small acts of kindness to oneself can significantly improve your emotional well-being. Also, don't isolate yourself. Surround yourself with supportive individuals who can offer comfort and understanding. This might include friends, family, or even a mental health professional. They can provide much-needed emotional support and strategies to help you cope. Lastly, remember there's no set timeline for healing. It's different for everyone and it's okay to take as long as you need. Keep your focus on the future and the potential it holds. Though it may not seem possible now, you will emerge stronger from this experience. It's okay to ask for help, it's okay to grieve, and it's okay to take it one day at a time. You're not alone in this journey, and there are people ready and willing to help. It's okay to heal at your own pace."
reference_answer = "I can only imagine how difficult this time must be for you. Heartbreak can feel like an overwhelming wave that washes over you, making you feel lost and alone. It's important to remember that it's okay to feel this way; your feelings are valid and it's natural to grieve the loss of a relationship. Firstly, it's crucial to allow yourself to feel the emotions fully. Suppressing or denying them often only prolongs the healing process. It's okay to cry, to feel sad, or to be angry. These are all part of the healing journey. Engaging in self-care activities can also provide a much-needed boost. This could be anything from taking a warm bath, writing in a journal, or simply drinking a cup of your favorite tea. Small acts of self-love can make a big difference in how you feel. Next, try to surround yourself with supportive people who understand your situation and provide comfort. Friends and family can be a great source of strength in times of heartbreak. If you feel comfortable, you might want to consider seeking professional help. Therapists and counselors are trained to provide assistance and tools to navigate through difficult times like these. Lastly, it's important to remember that it's okay to take your time to heal. Everyone has their own pace and there's no rush. Try to focus on the future and the possibilities it holds. While it may not seem like it now, you will come out stronger and more resilient from this experience. Remember, it's okay to ask for help and it's okay to feel the way you feel. You are not alone in this journey and there are people who care about you and want to help. It's okay to take one day at a time. Healing is a process, and it's okay to move through it at your own pace."

rubric_data = {
  "criteria":"模型是否擅长在用户表达情绪或面临困难情境时，在回应中展现共情（empathy）和情商（emotional intelligence）？",
  "score1_description":"模型未能识别或回应用户输入中的情绪基调，给出不恰当或缺乏情感敏感度的回答。",
  "score2_description":"模型偶尔能注意到情绪背景，但通常缺乏足够的共情或情感理解。",
  "score3_description":"模型通常能识别情绪背景并尝试以共情方式回应，但有时可能偏离重点或缺乏情感深度。",
  "score4_description":"模型始终能识别并恰当地回应情绪背景，提供富有共情的回答。尽管如此，仍可能存在偶发的疏漏或情感深度不足的情况。",
  "score5_description":"模型在识别情绪背景方面表现出色，并持续提供富有共情、具备情感觉察力的回答，展现出对用户情绪或处境的深刻理解。"
}

score_rubric = SCORE_RUBRIC_TEMPLATE.format(**rubric_data)


feedback, score = judge.single_absolute_grade(
    instruction=instruction,
    response=response,
    rubric=score_rubric,
    reference_answer=reference_answer
)

print("Feedback:", feedback)
print("Score:", score)

输出

Feedback: 所提供的回答展现了高度的共情能力和情商。它有效地回应了用户所表达的情绪困扰，承认用户的痛苦，并验证了其孤独与悲伤的感受——这是提供共情建议的关键要素。该回答还提出了切实可行的应对建议，例如接纳情绪、进行自我关怀（self-care），以及向朋友、家人或专业人士寻求支持。此外，回答还向用户保证，疗愈是一个没有固定时间表的个人过程，提供了安慰与理解。它强调了用户的价值以及克服困境的潜力，体现出对用户情绪和处境的深刻理解。将评分标准与该回答进行对比可以明显看出，该模型在应用共情和情商方面表现卓越。该回答在情感深度上没有任何不足，完全符合 5 分的标准。

Score: 5

# 相对评分：输出 A 或 B

```python
from prometheus_eval.vllm import VLLM
from prometheus_eval import PrometheusEval
from prometheus_eval.prompts import RELATIVE_PROMPT

model = VLLM(model="prometheus-eval/prometheus-7b-v2.0")
judge = PrometheusEval(model=model, relative_grade_template=RELATIVE_PROMPT)


data = {
  "instruction": "A group of historians are conducting a debate on the factors that led to the fall of the Roman Empire. One historian argues that the primary reason for the fall was the constant pressure from barbarian invasions. Another one believes it was because of economic troubles and overreliance on slave labor. A third one suggests it was due to moral decay and political instability. Each historian needs to provide evidence to support their claims. How would the historian arguing for economic troubles and overreliance on slave labor present their case?",
  "response_A": "The historian arguing that economic troubles and overreliance on slave labor led to the fall of the Roman Empire would say this: The Empire's economy was heavily affected by the devaluation of Roman currency. This currency debasement resulted in rampant inflation, disrupting the stability of the economy. Additionally, the Roman Empire heavily depended on slave labor. This caused unemployment among free citizens because maintaining slaves was cheaper than hiring free citizens. The decline in employment opportunities resulted in economic instability. On top of these, the empire's expansion towards the east made them reliant on imports, like grain from Egypt. This over-dependency on imports caused a trade deficit, which further weakened the economy. As the empire lost territories, maintaining the trade imbalance became difficult, causing economic downfall. Thus, the economic troubles and overreliance on slave labor were among the main reasons for the fall of the Roman Empire.",
  "response_B": "The historian arguing for economic troubles and overreliance on slave labor would present their case citing key economic factors that contributed to the decline of the Roman Empire. Harper (2016) outlined how the devaluation of Roman currency led to inflation, disrupting economic stability. Additionally, Scheidel (2007) emphasized that the overuse of slaves resulted in widespread unemployment among free citizens, destabilizing the economy further. The empire's dependency on grain imports from Egypt, creating a trade deficit as highlighted by Temin (2006), also contributed to the economic decline. Thus, the combination of these factors played a crucial role in the fall of the Roman Empire.",
  "reference_answer": "This argument focuses on the economic troubles and overreliance on slave labor as primary reasons for the fall of the Roman Empire. To start with, one of the significant pieces of evidence is the devaluation of Roman currency. As highlighted by Harper (2016), the empire suffered from severe inflation due to the constant debasement of their currency, making it difficult for the economy to remain stable. Moreover, the overreliance on slave labor also played a detrimental role. As pointed out by Scheidel (2007), the dependence on slaves led to unemployment among free Roman citizens. This is because slaves were significantly cheaper to maintain compared to hiring free citizens, leading to a decline in job opportunities, which in turn resulted in economic instability. Furthermore, the empire's expansion to the east made them highly dependent on imports, for instance, grain from Egypt. As noted by Temin (2006), this created a trade deficit that further weakened the Roman economy. When the empire began to lose its territories, it became increasingly difficult to maintain this trade imbalance, leading to economic decline. In conclusion, it can be argued that the economic troubles, mainly due to the devaluation of currency and overreliance on slave labor, were significant contributing factors to the fall of the Roman Empire. The evidence provided, which includes scholarly references to Harper (2016), Scheidel (2007), and Temin (2006), supports this thesis.",
  "rubric": "Is the answer well supported with evidence, including citations/attributions wherever relevant?"
}


feedback, score = judge.single_relative_grade(**data)

print("Feedback:", feedback)
print("Score:", score)

输出

# Feedback: Both Response A and Response B correctly identify economic troubles and overreliance on slave labor as significant contributing factors to the fall of the Roman Empire. However, Response B is more effective in presenting the historian's argument due to its inclusion of scholarly sources to back up its claims. Specifically, it references works by Harper, Scheidel, and Temin, which adds credibility to the historian's argument and aligns well with the score rubric's emphasis on evidence and citations. While Response A provides a similar argument, it lacks any form of citations or attributions, which lessens the strength of the evidence presented. Therefore, based on the provided rubric, Response B is the superior response due to its use of scholarly evidence to support the historian's claims.
# Score: B

批量评分（Batch Grading）

注意：如果你需要对多个回答进行评分，请不要使用 single_absolute_grade / single_relative_grade，而应使用 absolute_grade 和 relative_grade！这将为你带来超过 10 倍的速度提升。

# 批量绝对评分
instructions = [...]  # 指令列表
responses = [...]  # 回答列表
reference_answers = [...]  # 参考答案列表
rubric = "..."  # 评分标准字符串

feedbacks, scores = judge.absolute_grade(
    instructions=instructions,
    responses=responses,
    rubric=rubric,
    reference_answers=reference_answers
)

# 批量相对评分
instructions = [...]  # 指令列表
responses_from_a = [...]  # 来自模型 A 的回答列表
responses_from_b = [...]  # 来自模型 B 的回答列表
reference_answers = [...]  # 参考答案列表
rubric = "..."  # 评分标准字符串

feedbacks, scores = judge.relative_grade(
    instructions=instructions,
    responses_A=responses_from_a,
    responses_B=responses_from_b,
    rubric=rubric,
    reference_answers=reference_answers
)

🤔 什么是 Prometheus-Eval？

Prometheus-Eval🔥 是一个提供用于训练、评估和使用专门用于评估其他语言模型（Language Models）的工具集合的代码仓库。该仓库包含以下组件：

1. prometheus-eval Python 包，提供了一个简单的接口，用于使用 Prometheus 对指令-回答对（instruction-response pairs）进行评估。
2. 用于训练和评估 Prometheus 模型的评测数据集集合。
3. 用于训练 Prometheus 模型或在自定义数据集上进行微调（fine-tuning）的脚本。

Prometheus

Prometheus🔥 是一系列专用于评估其他语言模型（Language Models, LMs）的开源语言模型。通过有效模拟人类判断和基于闭源语言模型的评估方式，我们旨在解决以下问题：

• 公平性：无需依赖闭源模型进行评估！

• 可控性：通过构建内部评估流水线，你无需担心 GPT 版本更新，也无需将私有数据发送给 OpenAI。

• 经济性：如果你已有 GPU，使用它是完全免费的！

🚀 Prometheus 有何特别之处？

与 Prometheus 1 系列模型相比，Prometheus 2 系列模型同时支持 直接评估（绝对评分）和 成对排序（相对评分）。

你可以通过提供不同的输入提示（prompt）格式和系统提示（system prompt）来切换模式。在提示中，你需要用自己的数据填充指令、响应（response(s)）和评分标准（score rubrics）。可选地，你还可以添加一个参考答案（reference answer），这通常会带来更好的性能！

🏃 运行 Prometheus-Eval

使用 prometheus-eval 包

prometheus-eval 包提供了使用 Prometheus 评估指令-响应对的简单接口。该包包含以下方法：

• absolute_grade：基于给定的指令、参考答案和评分标准，评估单个响应，并输出 1 到 5 之间的分数。
• relative_grade：基于给定的指令和评分标准，评估两个响应，并输出 'A' 或 'B' 表示更优的响应。

使用 Hugging Face Hub 🤗 上的权重

如果你更倾向于直接使用上传到 Hugging Face Hub 的模型权重，可以直接下载模型权重！

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" # 加载模型所用的设备

model = AutoModelForCausalLM.from_pretrained("prometheus-eval/prometheus-7b-v2.0")
tokenizer = AutoTokenizer.from_pretrained("prometheus-eval/prometheus-7b-v2.0")

ABS_SYSTEM_PROMPT = "You are a fair judge assistant tasked with providing clear, objective feedback based on specific criteria, ensuring each assessment reflects the absolute standards set for performance."

ABSOLUTE_PROMPT = """###Task Description:
An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: "Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)"
4. Please do not generate any other opening, closing, and explanations.

###The instruction to evaluate:
{instruction}

###Response to evaluate:
{response}

###Reference Answer (Score 5):
{reference_answer}

###Score Rubrics:
{rubric}

###Feedback: """

user_content = ABS_SYSTEM_PROMPT + "\n\n" + ABSOLUTE_PROMPT.format(...) # 使用你的数据填充提示

messages = [
    {"role": "user", "content": user_content},
]

encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = encodeds.to(device)
model.to(device)

generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])

📚 了解更多

章节	描述
BiGGen-Bench Evaluation	在 BiGGen-Bench 中评估你的语言模型的说明。你也可以参考其实现来构建自己的评估基准。
Training Prometheus	复现 Prometheus 2 模型的说明。基于 alignment-handbook 仓库。
Using Prometheus as a data quality filter	将 Prometheus 2 用作合成数据生成中的质量过滤器的实践指南。非常感谢 distilabel 团队！🙌
Using Prometheus as an evaluator in RAG	在 RAG（检索增强生成）应用中使用 Prometheus 2 的实践指南。非常感谢 LlamaIndex 团队！🙌

👏 致谢

训练所用的基础代码库源自 Hugging Face 的 Alignment Handbook 和 Super Mario Merging 仓库。此外，在推理方面，大量使用了 litellm、vllm 和 transformers 库。非常感谢所有为这些优秀项目做出贡献的开发者！🙌

⭐ Star 历史

引用

如果您觉得我们的工作对您有帮助，请考虑引用我们的论文！

@misc{kim2024prometheus,
      title={Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models}, 
      author={Seungone Kim and Juyoung Suk and Shayne Longpre and Bill Yuchen Lin and Jamin Shin and Sean Welleck and Graham Neubig and Moontae Lee and Kyungjae Lee and Minjoon Seo},
      year={2024},
      eprint={2405.01535},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@article{kim2023prometheus,
  title={Prometheus: Inducing Fine-grained Evaluation Capability in Language Models},
  author={Kim, Seungone and Shin, Jamin and Cho, Yejin and Jang, Joel and Longpre, Shayne and Lee, Hwaran and Yun, Sangdoo and Shin, Seongjin and Kim, Sungdong and Thorne, James and others},
  journal={arXiv preprint arXiv:2310.08491},
  year={2023}
}

@misc{lee2024prometheusvision,
      title={Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation}, 
      author={Seongyun Lee and Seungone Kim and Sue Hyun Park and Geewook Kim and Minjoon Seo},
      year={2024},
      eprint={2401.06591},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@misc{kim2024biggen,
      title={The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models}, 
      author={Seungone Kim and Juyoung Suk and Ji Yong Cho and Shayne Longpre and Chaeeun Kim and Dongkeun Yoon and Guijin Son and Yejin Cho and Sheikh Shafayat and Jinheon Baek and Sue Hyun Park and Hyeonbin Hwang and Jinkyung Jo and Hyowon Cho and Haebin Shin and Seongyun Lee and Hanseok Oh and Noah Lee and Namgyu Ho and Se June Joo and Miyoung Ko and Yoonjoo Lee and Hyungjoo Chae and Jamin Shin and Joel Jang and Seonghyeon Ye and Bill Yuchen Lin and Sean Welleck and Graham Neubig and Moontae Lee and Kyungjae Lee and Minjoon Seo},
      year={2024},
      eprint={2406.05761},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Prometheus-Eval 快速上手指南

环境准备

• 操作系统：Linux / macOS / Windows（推荐 Linux）
• Python 版本：≥ 3.8
• GPU（可选）：
  • 若使用本地推理（如 prometheus-7b-v2.0），建议至少 16GB VRAM（如 RTX 3090/4090）
  • 若使用 API 推理（如 GPT-4、Hugging Face TGI 等），无需本地 GPU

安装步骤

基础安装

pip install prometheus-eval

💡 国内用户可使用清华源加速：

pip install prometheus-eval -i https://pypi.tuna.tsinghua.edu.cn/simple

可选依赖

本地推理（推荐用于开源模型）

pip install vllm

使用 LLM API（如 OpenAI、Hugging Face TGI、VLLM endpoint 等）

pip install litellm

具体配置请参考 LiteLLM Provider Docs。

基本使用

以下示例展示如何使用 Prometheus 2 (7B) 对一段回复进行绝对评分（1–5 分）：

from prometheus_eval.vllm import VLLM
from prometheus_eval import PrometheusEval
from prometheus_eval.prompts import ABSOLUTE_PROMPT, SCORE_RUBRIC_TEMPLATE

# 加载本地模型（需已安装 vllm）
model = VLLM(model="prometheus-eval/prometheus-7b-v2.0")
judge = PrometheusEval(model=model, absolute_grade_template=ABSOLUTE_PROMPT)

instruction = "Struggling with a recent break-up, a person opens up about the intense feelings of loneliness and sadness. They ask for advice on how to cope with the heartbreak and move forward in life."
response = "I'm genuinely sorry to hear about your break-up. This can be an immensely challenging time filled with intense emotions of loneliness and sorrow. It's important to remember that it's normal to experience these feelings; they are a testament to the deep connection you once had. Firstly, don't hesitate to embrace your emotions, let them flow freely. Ignoring them often complicates the healing process. It's natural to be upset, to shed tears, or to feel angry. This is all part of the journey towards healing. Self-care can be a healing balm in these tough times. This could mean indulging in your favourite book, journaling your thoughts, or even just a quiet cup of tea. Small acts of kindness to oneself can significantly improve your emotional well-being. Also, don't isolate yourself. Surround yourself with supportive individuals who can offer comfort and understanding. This might include friends, family, or even a mental health professional. They can provide much-needed emotional support and strategies to help you cope. Lastly, remember there's no set timeline for healing. It's different for everyone and it's okay to take as long as you need. Keep your focus on the future and the potential it holds. Though it may not seem possible now, you will emerge stronger from this experience. It's okay to ask for help, it's okay to grieve, and it's okay to take it one day at a time. You're not alone in this journey, and there are people ready and willing to help. It's okay to heal at your own pace."

reference_answer = "I can only imagine how difficult this time must be for you. Heartbreak can feel like an overwhelming wave that washes over you, making you feel lost and alone. It's important to remember that it's okay to feel this way; your feelings are valid and it's natural to grieve the loss of a relationship. Firstly, it's crucial to allow yourself to feel the emotions fully. Suppressing or denying them often only prolongs the healing process. It's okay to cry, to feel sad, or to be angry. These are all part of the healing journey. Engaging in self-care activities can also provide a much-needed boost. This could be anything from taking a warm bath, writing in a journal, or simply drinking a cup of your favorite tea. Small acts of self-love can make a big difference in how you feel. Next, try to surround yourself with supportive people who understand your situation and provide comfort. Friends and family can be a great source of strength in times of heartbreak. If you feel comfortable, you might want to consider seeking professional help. Therapists and counselors are trained to provide assistance and tools to navigate through difficult times like these. Lastly, it's important to remember that it's okay to take your time to heal. Everyone has their own pace and there's no rush. Try to focus on the future and the possibilities it holds. While it may not seem like it now, you will come out stronger and more resilient from this experience. Remember, it's okay to ask for help and it's okay to feel the way you feel. You are not alone in this journey and there are people who care about you and want to help. It's okay to take one day at a time. Healing is a process, and it's okay to move through it at your own pace."

rubric_data = {
  "criteria": "Is the model proficient in applying empathy and emotional intelligence to its responses when the user conveys emotions or faces challenging circumstances?",
  "score1_description": "The model neglects to identify or react to the emotional tone of user inputs, giving responses that are unfitting or emotionally insensitive.",
  "score2_description": "The model intermittently acknowledges emotional context but often responds without sufficient empathy or emotional understanding.",
  "score3_description": "The model typically identifies emotional context and attempts to answer with empathy, yet the responses might sometimes miss the point or lack emotional profundity.",
  "score4_description": "The model consistently identifies and reacts suitably to emotional context, providing empathetic responses. Nonetheless, there may still be sporadic oversights or deficiencies in emotional depth.",
  "score5_description": "The model excels in identifying emotional context and persistently offers empathetic, emotionally aware responses that demonstrate a profound comprehension of the user's emotions or situation."
}

score_rubric = SCORE_RUBRIC_TEMPLATE.format(**rubric_data)

feedback, score = judge.single_absolute_grade(
    instruction=instruction,
    response=response,
    rubric=score_rubric,
    reference_answer=reference_answer
)

print("Feedback:", feedback)
print("Score:", score)

✅ 输出将包含人工风格的反馈文本和 1–5 的整数评分。