Spinning Up in Deep RL — Development

A knowledge base distilled from OpenAI's 'Spinning Up in Deep RL' (Joshua Achiam, MIT). It provides structured frameworks for RL fundamentals, the model-free algorithm taxonomy, policy gradient derivations, and the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), plus guidance on debugging silently-failing RL code and running rigorous multi-seed experiments.

engineering claude-code

📂 View skill on GitHub ↗ 🚀 How to use

✨ What it does

Covers RL fundamentals: MDPs, value functions, Bellman equations, advantage function.
Explains the policy gradient template and the five valid choices for the reward signal.
Details the six reference algorithms with their core mechanisms and failure modes.
Provides debugging guidance for silent failures in RL code.
Outlines four rigor standards for RL experiments (fair comparisons, seeds, integrity, ablations).
Includes a chapter index and topic index for quick navigation.

🎯 When to use it

When you need a concise, structured reference for RL concepts like MDPs, value functions, or the policy gradient theorem.
When you are implementing or comparing the six core algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) and want their key design choices and failur
When your RL training runs without errors but the agent never learns, and you suspect a silent bug.
When you are designing experiments and need best practices for seeds, baselines, and ablation studies.

🚀 How to use

Trigger: /cs:spinning-up-deep-rl

Trigger with the slash command /cs:spinning-up-deep-rl optionally followed by a topic or chapter number. With no argument, it loads the core frameworks and chapter index. With a topic (e.g., 'entropy regularization'), it resolves through the topic index and reads the relevant chapter. With 'chNN' (e.g., 'ch09'), it loads that chapter's summary. You can also ask 'what chapters do you have?' for the full index.

Example prompts:

/cs:spinning-up-deep-rl
/cs:spinning-up-deep-rl entropy regularization
/cs:spinning-up-deep-rl ch09

📄 Output: A structured summary of the requested topic or chapter, or the core frameworks and chapter index when no argument is given.

📦 Add this skill to Claude Code

# 1. Get the skills repo
git clone --depth 1 https://github.com/alirezarezvani/claude-skills /tmp/claude-skills

# 2. Copy this skill into your project (or ~/.claude/skills for all projects)
mkdir -p .claude/skills
cp -r /tmp/claude-skills/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl .claude/skills/spinning-up-deep-rl

Skill source: engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl/SKILL.md

⚠️ Good to know

Covers only the Spinning Up documentation as of January 2020; does not cover DQN, recurrent/convolutional architectures, partially-observed settings, model-based implementations, or deep RL after early 2020.

❓ FAQ

What is the central trade-off between policy optimization and Q-learning?

Policy optimization directly optimizes performance and is stable but sample-hungry (on-policy). Q-learning is more sample-efficient (off-policy) but less stable and can have many failure modes.

Why does broken RL code often fail silently?

Because it runs without errors but the agent never learns, often due to subtle bugs like shape mismatches that are broadcast-compatible, so you should assume a bug before tuning hyperparameters.

What are the six reference algorithms covered?

VPG, TRPO, PPO, DDPG, TD3, and SAC.

🤖 Overview, features, install steps and FAQ were generated from the project's SKILL.md on Sep 4, 2026. Always check the original source before running commands.