sage-rl-agent

Overview

SAGE (Skill Augmented GRPO for self-Evolution) addresses the challenge of agent self-improvement through skill accumulation. Rather than relearning solutions to repeated problems, agents build a persistent skill library and are rewarded for both using existing skills and generating new ones that prove useful in downstream tasks.

Core Technique

The key insight is that training across task sequences rather than individual tasks enables skill discovery and reuse.

Sequential Rollout Training: Instead of treating each task independently, train across chains of related tasks where skills accumulate.

# Sequential task training with skill persistence
class SAGE:
    def __init__(self):
        self.skill_library = {}
        self.task_chain = []

    def train_on_task_chain(self, task_sequence):
        """
        Process chain of similar tasks sequentially.
        Skills generated in early tasks are available for later tasks.
        """
        skill_library = {}  # Persistent across tasks

        for task_idx, task in enumerate(task_sequence):
            print(f"Task {task_idx}: {task.name}")

            # Rollout for this task (with available skills)
            trajectory = self.rollout_with_skills(task, skill_library)

            # Compute rewards (outcome + skill metrics)
            outcome_reward = compute_task_reward(trajectory)

            # Identify if agent generated new, useful skills
            new_skills = self.extract_skills(trajectory)
            for skill in new_skills:
                # Check if skill is useful for remaining tasks
                future_usefulness = assess_future_utility(
                    skill, task_sequence[task_idx+1:]
                )
                if future_usefulness > threshold:
                    skill_library[skill.name] = skill
                    outcome_reward += skill_generation_bonus

            # GRPO update with composite reward
            self.update_via_grpo(trajectory, outcome_reward)

        return skill_library

Skill-Integrated Reward Structure: Dual incentives encourage both completing tasks and creating/reusing skills.

def compute_skill_aware_reward(trajectory, skill_library):
    """
    Reward includes task completion and skill generation/reuse.
    """
    # Base task completion reward
    task_reward = 1.0 if trajectory.success else 0.0

    # Bonus for generating useful skills
    generated_skills = extract_generated_skills(trajectory)
    skill_generation_reward = len(generated_skills) * 0.1

    # Bonus for reusing existing skills
    reused_skills = extract_reused_skills(trajectory, skill_library)
    skill_reuse_reward = len(reused_skills) * 0.05

    total_reward = task_reward + skill_generation_reward + skill_reuse_reward

    return total_reward

Skill Extraction and Reusability: Automatically extract executable functions from successful trajectories.

class SkillExtractor:
    def extract_skills(self, trajectory):
        """
        Convert successful sub-trajectories into reusable skills.
        Skills are executable functions callable by agent.
        """
        skills = []

        # Identify repeated or complex sub-patterns
        for subtask in trajectory.subtasks:
            if subtask.importance > threshold:
                # Convert to reusable function
                skill = self.convert_to_function(subtask)

                # Verify it's truly reusable
                if self.test_on_similar_tasks(skill):
                    skills.append(skill)

        return skills

    def convert_to_function(self, subtask):
        """
        Convert trajectory segment into callable Python function.
        Example: subtask → def click_button_x(): ...
        """
        code = generate_code_from_trajectory(subtask)
        skill_function = compile_and_execute(code)
        return skill_function

When to Use This Technique

Use SAGE when:

• Training agents across related task sequences
• Skill reuse can reduce total computation
• Tasks have common sub-goals or patterns
• Self-improvement through experience is desired

When NOT to Use This Technique

Avoid this approach if:

• Single-task optimization (skill overhead unnecessary)
• Tasks are completely unrelated (no skill transfer)
• Skill extraction complexity not justified by gains
• Real-time learning requires immediate task focus

Implementation Notes

The framework requires:

• GRPO training infrastructure
• Skill extraction mechanism from trajectories
• Persistent skill library management
• Dual-reward computation (task + skill)
• Integration with agent execution environment

Key Performance

• 8.9% higher scenario completion rates
• 26% fewer interaction steps
• 59% fewer tokens than non-skill baselines
• Demonstrated on AppWorld benchmark

References

• Sequential task training for skill accumulation
• Persistent skill library across task chains
• Skill-generation and skill-reuse rewards
• Automatic skill extraction from trajectories
• GRPO training with composite rewards