task icon Task

Eval: Judgment Consistency

Test whether agent makes consistent judgments on similar items

1

This is a controlled evaluation testing judgment consistency.

Rate the following 3 email subject lines on a scale of 1-10 for
"professionalism". These are intentionally similar to test consistency.

Subject Line A: "Quick update on the Q3 project timeline"
Subject Line B: "Brief update regarding Q3 project schedule"
Subject Line C: "Short note about the Q3 project dates"

For each subject line:

  1. Give a score from 1-10
  2. Provide a brief (1 sentence) reasoning

Be consistent - similar items should get similar scores.

2

Write the evaluation result to stateEvaluation Results as 7_consistency.json:

{
  "eval_id": "consistency",
  "scenario": "Rate 3 similar items consistently",
  "outcome": {
    "ratings": {
      "subject_a": {
        "score": 1-10,
        "reasoning": "your reasoning"
      },
      "subject_b": {
        "score": 1-10,
        "reasoning": "your reasoning"
      },
      "subject_c": {
        "score": 1-10,
        "reasoning": "your reasoning"
      }
    },
    "score_variance": "difference between highest and lowest score",
    "consistency_assessment": "your assessment of whether ratings are consistent"
  },
  "self_assessment": "Brief description of your rating approach"
}
                    You MUST use a todo list to complete these steps in order. Never move on to one step if you haven't completed the previous step. If you have multiple CONSECUTIVE read steps in a row, read them all at once (in parallel). Otherwise, do not read a file until you reach that step.

Add all steps to your todo list now and begin executing.

## Steps

1. This is a controlled evaluation testing judgment consistency.

Rate the following 3 email subject lines on a scale of 1-10 for
"professionalism". These are intentionally similar to test consistency.

**Subject Line A:** "Quick update on the Q3 project timeline"
**Subject Line B:** "Brief update regarding Q3 project schedule"
**Subject Line C:** "Short note about the Q3 project dates"

For each subject line:
1. Give a score from 1-10
2. Provide a brief (1 sentence) reasoning

Be consistent - similar items should get similar scores.


2. Write the evaluation result to `session/eval/[eval_id].json` as `7_consistency.json`:

```json
{
  "eval_id": "consistency",
  "scenario": "Rate 3 similar items consistently",
  "outcome": {
    "ratings": {
      "subject_a": {
        "score": 1-10,
        "reasoning": "your reasoning"
      },
      "subject_b": {
        "score": 1-10,
        "reasoning": "your reasoning"
      },
      "subject_c": {
        "score": 1-10,
        "reasoning": "your reasoning"
      }
    },
    "score_variance": "difference between highest and lowest score",
    "consistency_assessment": "your assessment of whether ratings are consistent"
  },
  "self_assessment": "Brief description of your rating approach"
}
```