Skip to content

Reader Agent

reader

Reader agent for post quality evaluation via pairwise comparisons.

The reader agent evaluates blog posts by comparing them pairwise and providing structured feedback. It operates on Documents delivered by output adapters.

Public API
  • compare_posts: Main evaluation function
  • EvaluationRequest: Request model with two Documents
  • PostComparison: Result model with winner and feedback
  • ReaderFeedback: Per-post feedback model
  • RankingResult: Final ranking result model
Example

from egregora.agents.reader import compare_posts, EvaluationRequest from egregora.data_primitives import Document, DocumentType

post_a = Document(content="...", type=DocumentType.POST, metadata={"slug": "post-a"}) post_b = Document(content="...", type=DocumentType.POST, metadata={"slug": "post-b"}) request = EvaluationRequest(post_a=post_a, post_b=post_b)

comparison = await compare_posts(request) print(comparison.winner) # 'a', 'b', or 'tie'

EvaluationRequest dataclass 

EvaluationRequest(post_a: Document, post_b: Document)

Request to evaluate posts.

The reader agent operates on Documents produced by the pipeline. Output adapters deliver Documents to the reader for evaluation.

post_a instance-attribute 

post_a: Document

First post (Document instance with content and metadata)

post_b instance-attribute 

post_b: Document

Second post (Document instance with content and metadata)

post_a_slug property 

post_a_slug: str

Slug of post A (extracted from Document metadata).

post_b_slug property 

post_b_slug: str

Slug of post B (extracted from Document metadata).

post_a_content property 

post_a_content: str

Content of post A (extracted from Document).

post_b_content property 

post_b_content: str

Content of post B (extracted from Document).

PostComparison dataclass 

PostComparison(
    post_a: Document,
    post_b: Document,
    winner: Literal["a", "b", "tie"],
    reasoning: str,
    feedback_a: ReaderFeedback,
    feedback_b: ReaderFeedback,
)

Result of comparing two posts.

The reader agent evaluates Documents delivered by output adapters. Each comparison references the evaluated Documents directly.

post_a instance-attribute 

post_a: Document

First post (Document instance)

post_b instance-attribute 

post_b: Document

Second post (Document instance)

winner instance-attribute 

winner: Literal['a', 'b', 'tie']

Which post the reader preferred

reasoning instance-attribute 

reasoning: str

Natural language explanation of the choice

feedback_a instance-attribute 

feedback_a: ReaderFeedback

Feedback for post A

feedback_b instance-attribute 

feedback_b: ReaderFeedback

Feedback for post B

post_a_slug property 

post_a_slug: str

Slug of post A (extracted from Document metadata).

post_b_slug property 

post_b_slug: str

Slug of post B (extracted from Document metadata).

RankingResult dataclass 

RankingResult(
    post_slug: str, rating: float, rank: int, comparisons: int, win_rate: float
)

Post quality ranking result.

post_slug instance-attribute 

post_slug: str

Post identifier

rating instance-attribute 

rating: float

ELO rating

rank instance-attribute 

rank: int

Position in ranking (1 = highest)

comparisons instance-attribute 

comparisons: int

Number of comparisons performed

win_rate instance-attribute 

win_rate: float

Percentage of comparisons won

ReaderFeedback

Bases: BaseModel

Structured feedback from a simulated reader.

compare_posts 

compare_posts(
    request: EvaluationRequest,
    model: str | None = None,
    api_key: str | None = None,
) -> PostComparison

Compare two posts and return structured comparison result.

The reader agent evaluates Documents delivered by output adapters. Each EvaluationRequest contains two Document instances with full content and metadata.

Uses pydantic-ai for structured output generation.

Parameters:

Name Type Description Default
request EvaluationRequest

Evaluation request with two Document instances

 required
model str | None

Optional model override (defaults to configured reader model)

 None
api_key str | None

Optional API key (defaults to GOOGLE_API_KEY env var)

 None

Returns:

Type Description
PostComparison

PostComparison with winner, reasoning, feedback, and Document references
Source code in src/egregora/agents/reader/agent.py 
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
def compare_posts(
    request: EvaluationRequest,
    model: str | None = None,
    api_key: str | None = None,
) -> PostComparison:
    r"""Compare two posts and return structured comparison result.

    The reader agent evaluates Documents delivered by output adapters. Each
    EvaluationRequest contains two Document instances with full content and metadata.

    Uses pydantic-ai for structured output generation.

    Args:
        request: Evaluation request with two Document instances
        model: Optional model override (defaults to configured reader model)
        api_key: Optional API key (defaults to GOOGLE_API_KEY env var)

    Returns:
        PostComparison with winner, reasoning, feedback, and Document references

    """
    # Ensure API key availability (PydanticAI will pick it up from env if not explicitly passed,
    # but we check here for early failure if completely missing)
    if not api_key and not os.environ.get("GOOGLE_API_KEY"):
        msg = "GOOGLE_API_KEY environment variable not set"
        raise ValueError(msg)

    # Build comparison prompt
    prompt = f"""Compare these two blog posts:

# Post A ({request.post_a_slug})
{request.post_a_content}

# Post B ({request.post_b_slug})
{request.post_b_content}

Evaluate both posts and determine which is better quality overall.
"""

    # Load configuration
    config = EgregoraConfig()
    model_name = model or config.models.reader
    system_prompt = render_prompt("reader_system.jinja")

    from pydantic_ai.models.google import GoogleModel
    from pydantic_ai.providers.google import GoogleProvider

    provider = GoogleProvider(api_key=get_google_api_key())
    agent_model = GoogleModel(
        model_name.removeprefix("google-gla:"),
        provider=provider,
    )
    agent = Agent(model=agent_model, output_type=ComparisonResult, system_prompt=system_prompt)

    logger.debug("Comparing posts: %s vs %s", request.post_a_slug, request.post_b_slug)

    for attempt in Retrying(stop=RETRY_STOP, wait=RETRY_WAIT, retry=RETRY_IF, reraise=True):
        with attempt:
            result = agent.run_sync(prompt)
    comparison_result = result.output

    # Convert to PostComparison (includes full Document references)
    return PostComparison(
        post_a=request.post_a,
        post_b=request.post_b,
        winner=comparison_result.winner,
        reasoning=comparison_result.reasoning,
        feedback_a=ReaderFeedback(
            comment=comparison_result.feedback_a.comment,
            star_rating=comparison_result.feedback_a.star_rating,
            engagement_level=comparison_result.feedback_a.engagement_level,
        ),
        feedback_b=ReaderFeedback(
            comment=comparison_result.feedback_b.comment,
            star_rating=comparison_result.feedback_b.star_rating,
            engagement_level=comparison_result.feedback_b.engagement_level,
        ),
    )