a-b-test-design
Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use `test-scenario`.
Resumen
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome
Archivos 1 archivo
Instálala a mano
Deska ejecuta esto por ti, para cada agente que elijas. Para ejecutarlo tú en una terminal:
npx skills add owl-listener/designer-skills --skill a-b-test-designMás de owl-listener
- presentation-deckowl-listenerStructure a design presentation for a specific audience and decision. Use when presenting internally. For a portfolio narrative use `case-study`; for the written argument use `design-rationale`.2,800 instalaciones en skills.sh0
- typography-scaleowl-listenerCreate a modular type scale with size, weight, and line-height relationships. Use when establishing typographic structure. For line length only use `readable-measure`; for judging type on an existing screen use `critique-typography` (visual-critique).2,300 instalaciones en skills.sh0
- visual-hierarchyowl-listenerEstablish hierarchy through size, weight, colour, spacing, and position so the eye lands in the intended order. Use when composing new work. For judging an existing screen, use `critique-visual-hierarchy` (visual-critique).2,300 instalaciones en skills.sh0
- color-systemowl-listenerBuild a product colour system — tonal scales, semantic roles, and contrast compliance. Use when defining or rebuilding colour from scratch. For dark-mode adaptation use `dark-mode-design`; for chart palettes use `data-visualization`; for multi-brand token architecture use `theming-system` (design-systems).2,200 instalaciones en skills.sh0
- dark-mode-designowl-listenerAdapt an existing palette to dark mode — surface elevation, contrast rebalancing, and desaturation rules. Use when you already have a light palette to translate. For building the base palette first, use `color-system`.2,200 instalaciones en skills.sh0
