What Does "Self-Learning" Mean for AI Agents?

Distinguishing classical machine learning from LLM-based agents. Why a large language model alone does not "learn" – and which three pillars actually make an agent system self-learning.

1. The Illusion of Learning

Large Language Models (LLMs) like GPT-4, Claude, or Gemini create the impression that they "learn." An LLM-based agent answers questions, writes code, analyzes data. Yet between the inference of a pretrained model and actual learning lies a fundamental difference.

"Intelligence is the ability to adapt to change." — Stephen Hawking

An LLM does not modify its weights during usage. Every API call starts with the same model – regardless of how many times the agent has previously worked on a similar task. Without external mechanisms, there is no memory, no adaptation, no learning.

The central question is therefore not: Can GPT-4 learn? – but rather: What architecture around an LLM enables real, continuous learning?

2. Classical ML vs. LLM Agents

❌ Classical Supervised Learning: Learning Through Training

In classical machine learning, "learning" means adapting internal parameters (weights) through an optimization process:

Training Data → Model → Prediction → Loss → Gradient → Weight Update

This cycle repeats over thousands to millions of iterations. The result: a model that generalizes from data. But after training, the weights are frozen. The model cannot learn from new experiences without retraining.

✅ LLM Agent: Learning Through Architecture

An LLM agent can become self-learning – not through weight updates, but through a feedback architecture that surrounds the model:

Task → Agent (LLM) → Action → Environment → Feedback → Agent (adapted)
            ↑                                              |
            └──────── Memory ◄─────────────────────────────┘

The LLM remains identical. What changes: the context, the strategy, the stored knowledge. Learning happens not inside the model, but around it.

3. The Three Pillars of Self-Learning Agents

Three mechanisms transform a static LLM agent into a self-learning system:

Pillar 1: Feedback Loops

The agent receives structured feedback on its actions and adjusts its behavior in the next attempt.

def feedback_loop(agent, task, environment, max_iterations=5):
    """Iterative feedback loop: Agent learns from environment feedback."""
    for i in range(max_iterations):
        action = agent.generate(task)
        result = environment.evaluate(action)

        if result.success:
            return action, result

        task = enrich_with_feedback(task, result.feedback)

    return None, result

The pattern follows the principle of Reinforcement Learning (RL): The agent selects an action, receives an evaluation (reward), and adjusts its strategy. The crucial difference from classical RL: The "policy" is not a neural network with trainable weights, but a prompt that is contextually extended.

Pillar 2: Persistent Memory

Without memory, the agent forgets every experience after the session. Persistent memory stores successful strategies and makes them retrievable for similar tasks.

def retrieve_relevant_experience(memory_store, current_task, top_k=3):
    """Finds similar past experiences via semantic search."""
    embedding = embed(current_task.description)
    similar = memory_store.search(embedding, top_k=top_k)
    return [
        {
            "task": exp.task,
            "solution": exp.solution,
            "feedback": exp.feedback,
            "success": exp.success
        }
        for exp in similar
    ]

Two memory types work together:

Long-term memory implements Experience Replay – a concept from Deep Reinforcement Learning (DQN, Mnih et al. 2015). Instead of learning only from the current episode, the agent draws from a pool of past experiences.

Pillar 3: Self-Critique

The agent evaluates its own output before submitting it. A separate evaluation instance identifies weaknesses and triggers corrections.

def self_critique(agent, output, task):
    """Agent evaluates own output and corrects if needed."""
    critique_prompt = f"""
    Task: {task}
    Generated Output: {output}

    Critically evaluate the output:
    1. Does it fully meet the requirements?
    2. What weaknesses or errors exist?
    3. How can the output be improved?
    """
    critique = agent.generate(critique_prompt)

    if critique.has_issues:
        improved = agent.generate(
            task,
            previous_output=output,
            critique=critique
        )
        return improved

    return output

This pattern is inspired by DeepSeek GRM (Generative Reward Model) – a system where the model learns to evaluate its own outputs and improve them on-the-fly. In the LLM agent world, this becomes a two-stage process:

1. Generation: Agent produces output
2. Critique: Separate prompt evaluates the output
3. Correction: Agent improves based on the critique

4. Current Research: Three Approaches

Academic research provides three relevant concepts that sharpen the understanding of "self-learning" for LLM agents:

Sakana AI: Transformer Squared Model

Sakana AI demonstrates a system that dynamically adjusts its weights at runtime – without retraining. The translation to LLM agents: dynamic prompt selection. Instead of using a fixed system prompt, the agent selects context-dependently from a pool of strategies:

STRATEGIES = {
    "conservative": "Generate minimal, safe code. Avoid complexity.",
    "exploratory": "Try new approaches. Use modern libraries.",
    "defensive": "Focus on error handling and edge cases.",
}

def select_strategy(task, history):
    """Selects strategy based on task type and past success."""
    if history.recent_failures > 2:
        return STRATEGIES["conservative"]
    if task.complexity == "low":
        return STRATEGIES["exploratory"]
    return STRATEGIES["defensive"]

SVFT: Singular Value Fine Tuning

SVFT decomposes weight matrices via SVD (Singular Value Decomposition) into components that can be individually tuned up or down – skill dials for specific capabilities. For LLM agents, this inspires a skill modulation approach:

SKILL_WEIGHTS = {
    "code_generation": 0.8,
    "debugging": 0.6,
    "testing": 0.7,
    "architecture": 0.5,
}

def modulate_skills(skill_weights, feedback):
    """Adjusts skill weighting based on feedback."""
    updated = dict(skill_weights)
    for skill, score in feedback.skill_scores.items():
        if skill in updated:
            # Exponential moving average
            updated[skill] = 0.7 * updated[skill] + 0.3 * score
    return updated

DeepSeek GRM: Generative Reward Model

DeepSeek GRM combines generation and evaluation in one model. It learns from errors, critiques itself, and improves performance on-the-fly. The concept is directly applicable to LLM agents (see Pillar 3: Self-Critique).

5. What "Self-Learning" Does Not Mean

A clear delineation prevents misunderstandings:

6. The Architecture: From LLM to Learning System

The three pillars together form an architecture that goes beyond pure LLM inference:

                    ┌─────────────┐
                    │    Task     │
                    └──────┬──────┘
                           ▼
              ┌────────────────────────┐
              │   Long-term Memory     │──── Retrieve similar experiences
              └────────────┬───────────┘
                           ▼
              ┌────────────────────────┐
              │    LLM Agent           │──── Strategy selection
              │    (Generation)        │
              └────────────┬───────────┘
                           ▼
              ┌────────────────────────┐
              │    Self-Critique       │──── Evaluate output
              └────────────┬───────────┘
                           ▼
              ┌────────────────────────┐
              │    Environment         │──── Tests, metrics, reviewer
              │    (Feedback)          │
              └────────────┬───────────┘
                           ▼
              ┌────────────────────────┐
              │    Memory Update       │──── Store episode
              └────────────┬───────────┘
                           ▼
                    ┌─────────────┐
                    │   Result    │
                    └─────────────┘

Every component is replaceable. The LLM can be GPT-4, Claude, or an open-source model. The memory can use FAISS, ChromaDB, or a simple JSON file. The environment can consist of automated tests, human feedback, or a combination.

7. Conclusion

"Self-learning" for LLM agents does not mean the same as in classical ML. There are no gradient updates, no backpropagation at runtime. Instead, learning emerges through a feedback architecture built on three pillars:

1. Feedback loops close the circle between action and evaluation
2. Persistent memory makes experiences available across sessions
3. Self-critique increases quality before submission

Current research (Sakana AI, SVFT, DeepSeek GRM) provides concepts that – translated to LLM agents – lead to dynamic strategy selection, skill modulation, and integrated self-evaluation.

The next article in the series covers Reinforcement Learning for Code Agents – the formal framework that places this feedback architecture into a mathematically grounded structure.

Sources & Resources

• Mnih, V. et al. (2015): Human-level control through deep reinforcement learning. Nature 518, 529–533. – Experience Replay as a core technique for Deep RL.
• Sakana AI (2024): Transformer²: Self-adaptive LLMs. – Dynamic weight adaptation at runtime.
• SVFT (2024): Singular Value Fine-Tuning: Few-shot Adaptation of LLMs. – SVD-based skill modulation.
• DeepSeek (2025): DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. – Self-improvement through generative reward model.
• Russell, S. & Norvig, P. (2020): Artificial Intelligence: A Modern Approach. 4th Edition. – Standard reference for RL fundamentals and agent architecture.

→ Full code: src/