Z-test for two proportions (A/B conversion rates) — Development

This skill runs hypothesis tests, analyzes A/B experiment results, calculates sample sizes, and interprets statistical significance with effect sizes. It helps validate whether observed differences are real, size experiments correctly before launch, and interpret results with confidence.

engineering claude-code

📂 View skill on GitHub ↗ 🚀 How to use

✨ What it does

Runs Z-test for proportions, t-test for means, and Chi-square test for categorical data.
Calculates sample size for proportion and mean tests with power and MDE tradeoffs.
Computes confidence intervals for proportions and means.
Provides plain-English verdicts and effect size interpretations (Cohen's d, h, Cramér's V).
Flags validity threats like peeking, multiple comparisons, and underpowered tests.

🎯 When to use it

When an A/B test has completed and you need to determine if the observed difference is statistically significant.
Before launching an experiment to calculate the required sample size per variant.
When someone shares a result and asks 'is this significant?' or 'what does this mean?'.

🚀 How to use

Trigger by asking to analyze experiment results, size an experiment, or interpret statistical significance. Provide sample sizes, observed values, and baseline rates. Use the provided scripts with command-line arguments. Example prompts:

Did our test win? Control: 5000 users, 250 conversions; Treatment: 5000 users, 310 conversions.
How big should our test be? Baseline conversion 5%, MDE 20%, alpha 0.05, power 0.80.

📄 Output: Significance reports, sample size reports, confidence interval reports, and plain-English conclusions.

📦 Add this skill to Claude Code

# 1. Get the skills repo
git clone --depth 1 https://github.com/alirezarezvani/claude-skills /tmp/claude-skills

# 2. Copy this skill into your project (or ~/.claude/skills for all projects)
mkdir -p .claude/skills
cp -r /tmp/claude-skills/engineering/statistical-analyst/skills/statistical-analyst .claude/skills/statistical-analyst

Skill source: engineering/statistical-analyst/skills/statistical-analyst/SKILL.md

⚠️ Good to know

Assumes frequentist statistics; not suitable for Bayesian inference or multi-armed bandit analysis. Requires clean, independent data; not for clustered or heavy-tailed data without preprocessing.

❓ FAQ

What tests does this skill support?

It supports Z-test for proportions, two-sample t-test for means, and Chi-square test for categorical data.

How does the skill handle multiple comparisons?

It flags when more than 3 metrics are evaluated and suggests Bonferroni-adjusted thresholds.

What should I do if my test is underpowered?

A non-significant result is inconclusive; the skill recommends extending the test or killing it if underpowered.

🤖 Overview, features, install steps and FAQ were generated from the project's SKILL.md on Sep 4, 2026. Always check the original source before running commands.