A/B Test Setup — Business

This skill guides the design of statistically valid A/B tests, from hypothesis formation through sample size calculation, variant design, execution, and analysis. It produces structured experiment briefs, sample size estimates, QA checklists, and results analysis reports.

marketing claude-code

📂 View skill on GitHub ↗ 🚀 How to use

✨ What it does

Guides hypothesis creation with a structured framework.
Provides a sample size calculator script and quick reference tables.
Offers metrics selection guidance (primary, secondary, guardrail).
Includes pre-launch QA checklist and analysis checklist.
Documents common mistakes and proactive triggers for testing.
Outputs experiment briefs, QA checklists, and results reports.

🎯 When to use it

When planning or designing an A/B test or experiment.
When debating between two versions of copy, design, or a CTA.
When a landing page or campaign is underperforming and you need a structured test plan.
When discussing pricing page changes that need controlled testing.
When you need to calculate required sample size or test duration.

🚀 How to use

Trigger by asking to design an A/B test or experiment, or by mentioning 'A/B test', 'split test', 'experiment', 'test this change', etc. The skill first checks for product marketing context, then asks about test context, current state, and constraints. Provide details like baseline conversion rate, traffic volume, and the change you're considering. Example prompts:

I want to test a new headline on my pricing page. My current conversion rate is 5% and I get about 2000 visitors per day. Can you help me design the test?

Design an A/B test to see if changing the CTA button color increases signups.

📄 Output: Deliverables include an experiment brief, sample size calculation, pre-launch QA checklist, results analysis report, and a test backlog.

📦 Add this skill to Claude Code

# 1. Get the skills repo
git clone --depth 1 https://github.com/alirezarezvani/claude-skills /tmp/claude-skills

# 2. Copy this skill into your project (or ~/.claude/skills for all projects)
mkdir -p .claude/skills
cp -r /tmp/claude-skills/marketing-skill/skills/ab-test-setup .claude/skills/ab-test-setup

Skill source: marketing-skill/skills/ab-test-setup/SKILL.md

⚠️ Good to know

Requires a clear hypothesis and defined metrics; does not implement tracking or run the test itself.

❓ FAQ

What is the recommended sample size for a test with a 5% baseline and a 20% lift?

According to the quick reference table, you need about 8,200 visitors per variant for a 20% lift from a 5% baseline.

Why should I not peek at results before the test is complete?

Peeking and stopping early leads to false positives and wrong decisions. Pre-commit to the sample size and trust the process.

What should I do if my test results are not statistically significant?

If there is no significant difference, you may need more traffic or a bolder test. The result is inconclusive, not a failure.

🤖 Overview, features, install steps and FAQ were generated from the project's SKILL.md on Sep 4, 2026. Always check the original source before running commands.