Home › Self Learning Agents › Article 3
Part 3 Generative AI

Automated Tests as Reward Signal

Image: AI-generated with Beuys

Automated Tests as Reward Signal

How test frameworks become the "environment" for learning code agents. From binary pass/fail to differentiated multi-signal reward – and why the feedback pipeline Agent → Code → Test → Score → Agent forms the core of every self-learning system.

1. Tests as Environment

In the previous article, we built the RL framework for code agents: State, Action, Reward, Policy. But one central question remained open: Where does the reward signal come from in practice?

The answer is surprisingly straightforward: Automated tests.

"Tests are the first line of feedback and defense against coding mistakes, refactoring mistakes, and so on." — Fabrizio Romano, Learn Python Programming, 4th Ed., p. 472

In the world of code agents, test suites take on the role that the physical environment plays in classical RL. The agent generates code (action), the tests evaluate the result (environment), and the test outcome becomes the reward signal:

Agent → Generate code → Run tests → Result → Compute reward → Agent

This is not an abstract concept – it is exactly the loop that TDD (Test-Driven Development) has practiced for decades:

"TDD is a software development methodology that is based on the continuous repetition of a very short development cycle." — Fabrizio Romano, Learn Python Programming, 4th Ed., p. 472

The difference: In TDD, a human sits in the loop. In self-learning agents, the loop is automated – and the test result becomes a formal reward signal.

2. The Problem with Binary Feedback

The simplest approach: All tests pass → Reward 1.0. At least one test fails → Reward 0.0.

def binary_reward(suite: SuiteResult) -> float:
    return 1.0 if suite.all_passed else 0.0

This works – but it's a sparse reward signal. Why this is problematic:

Scenario Pass Rate Binary Reward
0 of 10 tests pass 0% 0.0
9 of 10 tests pass 90% 0.0
10 of 10 tests pass 100% 1.0

An agent with 9 out of 10 passing tests receives the same feedback as one with 0 out of 10. From an RL perspective, this is fatal: The agent cannot detect a gradient toward improvement.

"A key innovation in RLHF is its approach to handling the high cost of human feedback. Rather than requiring constant human oversight, RLHF allows for asynchronous and sparse feedback." — Labonne, M. et al. (2025), LLM Engineer's Handbook, p. 331

What is acceptable for human feedback – sparsity – becomes an unnecessary bottleneck with automated tests. We have detailed information. We just need to use it.

3. Structured Test Results

The first step toward better reward: Treat test results not as booleans, but as structured data.

class TestStatus(Enum):
    PASSED = "passed"
    FAILED = "failed"
    ERROR = "error"
    SKIPPED = "skipped"

class TestSeverity(Enum):
    CRITICAL = 3
    MAJOR = 2
    MINOR = 1

@dataclass(frozen=True)
class TestResult:
    name: str
    status: TestStatus
    severity: TestSeverity = TestSeverity.MAJOR
    message: str = ""
    duration_ms: float = 0.0

Each individual test now carries three dimensions of information: - Status: Passed, failed, error, or skipped - Severity: How critical is this test? (A failed login test weighs more than a missing tooltip) - Context: Error message and execution duration

Aggregation into a suite yields a rich data structure:

@dataclass(frozen=True)
class SuiteResult:
    tests: tuple[TestResult, ...]
    suite_name: str = "default"

    @property
    def pass_rate(self) -> float:
        active = self.total - self.skipped
        return self.passed / active if active > 0 else 0.0

Note: Skipped tests are excluded from the pass rate. A skip is not a statement about code quality – it's a statement about test scope.

4. From Single Signal to Reward Vector

With structured test results, we can extract four different reward signals:

4.1 Pass Rate: Proportional Instead of Binary

def pass_rate_reward(suite: SuiteResult) -> float:
    return suite.pass_rate

Much better: 9/10 → 0.9, 5/10 → 0.5. The agent now sees a gradient.

4.2 Severity-Weighted: Critical Tests Count More

def severity_weighted_reward(suite: SuiteResult) -> float:
    active = [t for t in suite.tests if t.status != TestStatus.SKIPPED]
    if not active:
        return 0.0
    total_weight = sum(t.severity.value for t in active)
    passed_weight = sum(
        t.severity.value for t in active
        if t.status == TestStatus.PASSED
    )
    return passed_weight / total_weight

When a CRITICAL test (weight 3) fails and a MINOR test (weight 1) passes, the weighted score is lower than with a simple pass rate. This reflects reality: A missing tooltip is less severe than a broken login.

4.3 Progression: Rewarding Improvement

def progression_reward(
    current: SuiteResult,
    previous: SuiteResult | None,
) -> float:
    if previous is None:
        return current.pass_rate
    return current.pass_rate - previous.pass_rate

The delta between the current and previous iteration. An agent that improves from 3/10 to 7/10 receives a positive signal (+0.4), even though not all tests pass yet. An agent that regresses from 8/10 to 5/10 receives a negative signal (−0.3).

4.4 Speed: Faster Tests as Bonus

def speed_score(suite: SuiteResult, target_ms: float = 1000.0) -> float:
    if not suite.tests:
        return 0.0
    total_ms = sum(t.duration_ms for t in suite.tests)
    return math.exp(-total_ms / target_ms)

Exponential decay relative to a target value. Code that passes all tests quickly is better than code that passes all tests slowly. This prevents brute-force solutions and rewards efficient implementations.

5. Multi-Signal Reward Shaping

Each individual signal has blind spots. The pass rate ignores severity levels. Progression ignores the absolute state. The speed score ignores correctness. Only the combination yields a complete picture.

@dataclass(frozen=True)
class RewardSignal:
    pass_rate: float
    severity_score: float
    progression: float
    coverage_score: float
    speed_score: float
    total: float

The weighting is configurable:

@dataclass(frozen=True)
class RewardWeights:
    pass_rate: float = 0.35
    severity: float = 0.25
    progression: float = 0.20
    coverage: float = 0.10
    speed: float = 0.10

The default weighting reflects priorities: - Pass Rate (35%): The most important metric – does the code work? - Severity (25%): Critical failures weigh more heavily - Progression (20%): Improvement is rewarded, even with incomplete solutions - Coverage (10%): Bonus for broad test coverage - Speed (10%): Bonus for efficient solutions

"Reward hacking: Models may exploit loopholes in the reward function." — Vaid, S. & Lund, B. (2025), LLM Design Patterns, p. 452

The weighting is deliberately not focused on a single metric. Reward hacking – exploiting weaknesses in the reward function – is made harder by multi-signal scoring. An agent that only optimizes pass rate while writing catastrophic code is penalized by the severity and speed scores.

6. The Feedback Pipeline

The reward signal alone is not enough. The agent needs structured feedback that tells it what went wrong and where to start fixing.

@dataclass(frozen=True)
class FeedbackEntry:
    iteration: int
    submission: CodeSubmission
    suite_result: SuiteResult
    reward: RewardSignal
    failing_tests: tuple[tuple[str, str], ...]

Each iteration produces a FeedbackEntry with: 1. The submitted code – what did the agent try? 2. The complete test result – which tests passed, which didn't? 3. The multi-signal reward – how good was the attempt overall? 4. The failing tests with messages – concrete hints for correction

The build_feedback function orchestrates the computation:

def build_feedback(
    submission: CodeSubmission,
    suite: SuiteResult,
    previous_suite: SuiteResult | None = None,
    coverage: float = 0.0,
    target_ms: float = 1000.0,
    weights: RewardWeights | None = None,
) -> FeedbackEntry:
    reward = compute_reward_signal(
        suite, previous_suite, coverage, target_ms, weights
    )
    failing = extract_failing_tests(suite)
    return FeedbackEntry(
        iteration=submission.iteration,
        submission=submission,
        suite_result=suite,
        reward=reward,
        failing_tests=failing,
    )

7. The Agent-Test Loop

All components together form the Agent-Test Loop – the central architecture of self-learning code agents:

@dataclass
class AgentTestLoop:
    test_runner: Callable[[str], SuiteResult]
    agent_fn: Callable[[list[FeedbackEntry], str | None], str]
    max_iterations: int = 10
    reward_weights: RewardWeights = field(default_factory=RewardWeights)

    def run(self, task: str | None = None) -> list[FeedbackEntry]:
        history: list[FeedbackEntry] = []
        previous_suite: SuiteResult | None = None

        for i in range(1, self.max_iterations + 1):
            code = self.agent_fn(history, task)
            submission = CodeSubmission(
                code=code, iteration=i,
                strategy=_infer_strategy(history),
            )
            suite = self.test_runner(code)
            entry = build_feedback(
                submission, suite,
                previous_suite=previous_suite,
                weights=self.reward_weights,
            )
            history.append(entry)
            previous_suite = suite

            if suite.all_passed:
                break

        return history

The architecture is deliberately minimal and compositional:

The loop stops under two conditions: 1. All tests pass – the agent has solved the task 2. Max iterations reached – the agent is not converging

Strategy Inference

A subtle detail: The strategy is automatically derived from the history:

def _infer_strategy(history: list[FeedbackEntry]) -> str:
    if not history:
        return "initial"
    last = history[-1]
    if last.suite_result.errors > 0:
        return "fix_errors"
    if last.suite_result.failed > 0:
        return "fix_failures"
    return "refactor"

This strategy inference provides the agent with a hint about what kind of change is sensible next – without restricting its freedom of choice.

8. Learning Curves and Convergence

The feedback history is not only input for the agent – it is also a diagnostic tool for the quality of reward shaping.

@dataclass(frozen=True)
class LearningPoint:
    iteration: int
    pass_rate: float
    total_reward: float
    cumulative_reward: float
    tests_fixed: int

The learning curve shows: - Pass rate over iterations: Is it monotonically increasing? → Good reward signal - Cumulative reward: Is it growing steadily? → Stable learning process - Tests fixed per iteration: Are new tests being fixed each round? → Progress, not oscillation

def convergence_rate(history: list[FeedbackEntry]) -> float | None:
    for entry in history:
        if entry.suite_result.all_passed:
            return float(entry.iteration)
    return None

The convergence rate – at which iteration all tests pass – is the ultimate metric. Lower values = better agent or better reward shaping.

A moving average of the reward smooths outliers and reveals the overall trend:

def reward_trend(history: list[FeedbackEntry], window: int = 3) -> list[float]:
    rewards = [e.reward.total for e in history]
    smoothed = []
    for i in range(len(rewards)):
        start = max(0, i - window + 1)
        chunk = rewards[start:i + 1]
        smoothed.append(sum(chunk) / len(chunk))
    return smoothed

9. Comparison: Reward Strategies in Practice

Strategy Strength Weakness Use Case
Binary Simple, clear Sparse, no gradient Final validation
Pass Rate Gradient present Ignores severity levels Quick prototyping
Severity-Weighted Prioritizes critical tests Requires severity annotation Production code
Progression Rewards improvement Unstable with oscillation Learning phase
Multi-Signal Robust overall picture More complex configuration Production use

The recommendation: Start with pass rate, switch to multi-signal once the agent is fundamentally working. Binary reward only as final acceptance.

10. Conclusion

Automated tests are the natural reward signal for code agents:

The core architecture – AgentTestLoop with interchangeable test runner and agent function – is deliberately kept minimal. It enforces no particular test technology and no specific LLM. The intelligence resides not in the loop, but in the reward shaping and the feedback format.

The next article in the series covers Memory & Experience Replay for Agents – how agents store successful strategies, retrieve them for similar tasks, and learn across tasks.

Sources & Resources

→ Full source code: src/

📄 As PDF