Experiment Design Template

---
title: "Experiment Design Template"
methodology: "universal"
complexity: "advanced"
owner: "mirichard"
updated: "2026-10-08"
domain: "Uncertainty"
primary_principles: ["evidence-based-decisions", "risk-optimization"]
secondary_principles: ["quality-by-design", "continuous-learning"]
principle_rationale: "Defines controlled tests with explicit metrics and risks so learning quality supports reliable product and investment decisions."
tags: ["lean-startup", "experiment-design", "hypothesis-testing", "validated-learning"]
---

# Experiment Design Template

## Overview
This template helps you design and execute systematic experiments to validate business hypotheses using the scientific method. Proper experiment design ensures reliable results and actionable insights.



## When to Use
- Use when a high-risk assumption requires controlled testing with defined variables, sample boundaries, and decision thresholds.
- Apply before running A/B tests, interviews, or pilot experiments that need traceable metrics and reproducible setup choices.
- Run this template when multiple teams contribute to one experiment and ownership, timing, and analysis criteria must be explicit.

## When NOT to Use
- Do not use when success conditions are retrofitted after seeing results instead of fixed before execution.
- Do not report precise findings without source clarity, measurement method, and uncertainty context (see [Metric without source](../../../docs/principles/anti-patterns.md#metric-source)).
- Do not run concurrent scope changes that invalidate the control assumptions used by the experiment design.

## Pairs Well With
- [Hypothesis-Driven Planning Template](hypothesis-driven-planning.md)
- [Pivot Decision Framework](pivot-decision-framework.md)

Decision context model reference: 800-801-context-assessment-model.md
## Experiment Planning Framework

### 1. Experiment Definition

#### Basic Information
```
Experiment Name: [Descriptive title]
Experiment ID: [Unique identifier, e.g., EXP-001]
Date Created: [Date]
Owner: [Person responsible]
Status: [Planning/Running/Completed/Cancelled]
```

#### Hypothesis Being Tested
```
Hypothesis ID: [Link to hypothesis registry]
Hypothesis Statement: [Copy from hypothesis-driven planning]

Key Assumption: [The most critical assumption being tested]
Risk Level: [High/Medium/Low - if wrong, what's the impact?]
```

### 2. Experiment Objectives

#### Primary Objective
- [ ] **What are you trying to learn?**
  - [Specific learning objective]

#### Secondary Objectives
- [ ] **Additional insights you hope to gain**
  - [Additional learning objective 1]
  - [Additional learning objective 2]

#### Success Definition
```
The experiment will be considered successful if:
[Specific, measurable outcome criteria]

The hypothesis will be validated if:
[Conditions that would prove the hypothesis true]

The hypothesis will be invalidated if:
[Conditions that would prove the hypothesis false]
```

### 3. Target Audience

#### Primary Audience
```
Customer Segment: [Target segment for this experiment]
Demographics: [Age, gender, income, location, etc.]
Psychographics: [Attitudes, interests, behaviors]
Sample Size Needed: [Number of participants]
Recruitment Method: [How will you find participants?]
```

#### Inclusion Criteria
- [ ] Criterion 1: [e.g., Must be in target age range]
- [ ] Criterion 2: [e.g., Must have experienced the problem]
- [ ] Criterion 3: [e.g., Must have decision-making authority]

#### Exclusion Criteria
- [ ] Criterion 1: [e.g., Already using competitor solution]
- [ ] Criterion 2: [e.g., Too familiar with our company]

### 4. Experiment Design

#### Experiment Type
- [ ] **A/B Test**: Compare two versions (A vs B)
- [ ] **Multivariate Test**: Test multiple variables simultaneously
- [ ] **User Interview**: Qualitative feedback collection
- [ ] **Survey**: Quantitative data collection
- [ ] **Landing Page Test**: Measure interest/conversion
- [ ] **Prototype Test**: User interaction with mockup
- [ ] **Concierge MVP**: Manual service delivery
- [ ] **Wizard of Oz**: Fake automation with manual backend
- [ ] **Beta Test**: Limited release to early users
- [ ] **Observation Study**: Watch natural behavior

#### Variables Being Tested
```
Independent Variable(s): [What you're changing/testing]
- Variable 1: [e.g., Price point - $10 vs $20]
- Variable 2: [e.g., Feature set - Basic vs Premium]

Dependent Variable(s): [What you're measuring]
- Primary Metric: [e.g., Conversion rate]
- Secondary Metrics: [e.g., Time to purchase, user satisfaction]

Control Variables: [What you're keeping constant]
- [e.g., Same target audience, same time period]
```

### 5. Metrics and Measurement

#### Primary Metrics
```
Metric 1: [Name of metric]
- Definition: [How is it calculated?]
- Current Baseline: [What's the current rate/value?]
- Target: [What would indicate success?]
- Measurement Method: [How will you collect this data?]
```

#### Secondary Metrics
```
Metric 2: [Name of metric]
- Definition: [How is it calculated?]
- Target: [What would be good?]
- Measurement Method: [How will you collect this data?]

Metric 3: [Name of metric]
- Definition: [How is it calculated?]
- Target: [What would be good?]
- Measurement Method: [How will you collect this data?]
```

#### Qualitative Measurements
- [ ] User feedback and comments
- [ ] Behavioral observations
- [ ] Customer interview insights
- [ ] Support ticket patterns
- [ ] Social media sentiment

### 6. Implementation Plan

#### Pre-Experiment Setup
```
Week -2:
- [ ] Finalize experiment design
- [ ] Set up tracking and analytics
- [ ] Create experimental materials (landing pages, prototypes, etc.)
- [ ] Recruit participants (if needed)

Week -1:
- [ ] Test all systems and tracking
- [ ] Train team members involved
- [ ] Prepare data collection templates
- [ ] Double-check experiment parameters
```

#### Experiment Execution
```
Duration: [How long will the experiment run?]
Start Date: [When will it begin?]
End Date: [When will it conclude?]

Daily Tasks:
- [ ] Monitor data collection
- [ ] Check for technical issues
- [ ] Document observations
- [ ] Track participant recruitment/retention

Weekly Tasks:
- [ ] Review interim results
- [ ] Assess if changes needed
- [ ] Update stakeholders
- [ ] Document learnings
```

#### Post-Experiment Activities
```
Week +1:
- [ ] Collect final data
- [ ] Clean and analyze results
- [ ] Compare to success criteria
- [ ] Identify unexpected findings

Week +2:
- [ ] Create experiment report
- [ ] Present findings to stakeholders
- [ ] Update business model/strategy
- [ ] Plan next experiments
```

### 7. Resource Requirements

#### Human Resources
- [ ] Experiment Designer: [Name] - [Hours/week]
- [ ] Data Analyst: [Name] - [Hours/week]
- [ ] Developer: [Name] - [Hours/week]
- [ ] Customer Researcher: [Name] - [Hours/week]

#### Budget Requirements
- [ ] Advertising/Marketing: $[Amount]
- [ ] Tools/Software: $[Amount]
- [ ] Participant Incentives: $[Amount]
- [ ] Development/Design: $[Amount]
- **Total Budget**: $[Amount]

#### Technical Requirements
- [ ] Analytics setup (Google Analytics, Mixpanel, etc.)
- [ ] A/B testing platform (Optimizely, VWO, etc.)
- [ ] Survey tools (Typeform, SurveyMonkey, etc.)
- [ ] Prototype tools (Figma, InVision, etc.)

### 8. Risk Assessment

#### Potential Risks
```
Risk 1: [Description]
- Probability: [High/Medium/Low]
- Impact: [High/Medium/Low]
- Mitigation: [How to prevent/address]

Risk 2: [Description]
- Probability: [High/Medium/Low]
- Impact: [High/Medium/Low]
- Mitigation: [How to prevent/address]
```

#### Ethical Considerations
- [ ] Participant consent obtained
- [ ] Data privacy protected
- [ ] No deceptive practices
- [ ] Fair treatment of all participants
- [ ] Clear communication about experiment

### 9. Data Collection Plan

#### Quantitative Data
```
Data Source 1: [e.g., Website Analytics]
- Metrics: [What will be measured]
- Collection Method: [How data is gathered]
- Storage: [Where data will be stored]
- Access: [Who can access the data]

Data Source 2: [e.g., Survey Responses]
- Metrics: [What will be measured]
- Collection Method: [How data is gathered]
- Storage: [Where data will be stored]
- Access: [Who can access the data]
```

#### Qualitative Data
- [ ] Customer interview recordings/transcripts
- [ ] User behavior observations
- [ ] Support ticket feedback
- [ ] Social media comments
- [ ] Expert evaluations

#### Data Quality Checks
- [ ] Regular data validation
- [ ] Duplicate detection
- [ ] Outlier identification
- [ ] Missing data handling
- [ ] Bias detection

### 10. Analysis Plan

#### Statistical Analysis
```
Statistical Test: [t-test, chi-square, ANOVA, etc.]
Significance Level: [Usually 0.05]
Sample Size Calculation: [Power analysis results]
Confidence Interval: [Usually 95%]
```

#### Analysis Timeline
- [ ] **Day 1-2**: Data cleaning and preparation
- [ ] **Day 3-4**: Descriptive statistics and visualization
- [ ] **Day 5-6**: Hypothesis testing and significance analysis
- [ ] **Day 7**: Report writing and recommendations

#### Analysis Questions
1. **Primary Question**: [Main question the experiment answers]
2. **Secondary Questions**: 
   - [Additional analysis questions]
   - [Segment-specific insights]
   - [Unexpected patterns to investigate]

## Experiment Types Guide

### 1. A/B Testing
**Best for**: Testing two versions of a feature, design, or message

#### Setup Template
```
Control (A): [Current version/baseline]
Variant (B): [New version being tested]
Traffic Split: [50/50 or specify percentages]
Primary Metric: [Conversion rate, click-through rate, etc.]
Minimum Sample Size: [Based on power analysis]
Test Duration: [Time needed for statistical significance]
```

### 2. Landing Page Experiments
**Best for**: Testing demand, messaging, and value propositions

#### Setup Template
```
Page Purpose: [What action you want visitors to take]
Traffic Source: [Google Ads, social media, email, etc.]
Value Proposition Tested: [Core message being validated]
Call-to-Action: [Specific action: sign up, pre-order, etc.]
Success Metric: [Conversion rate to CTA]
```

### 3. Customer Interview Experiments
**Best for**: Deep qualitative insights and problem validation

#### Setup Template
```
Interview Type: [Problem discovery, solution validation, etc.]
Duration: [15-45 minutes typical]
Sample Size: [15-30 interviews for patterns]
Interview Guide: [Prepared questions and topics]
Recording Method: [Audio, video, notes]
Analysis Method: [Thematic analysis, affinity mapping]
```

### 4. Survey Experiments
**Best for**: Quantitative validation with larger samples

#### Setup Template
```
Survey Tool: [Typeform, SurveyMonkey, Google Forms]
Distribution Method: [Email, social media, ads]
Target Response Rate: [Typically 5-15% for cold outreach]
Question Types: [Multiple choice, rating scales, open-ended]
Survey Length: [5-15 minutes maximum]
Incentive: [Gift card, discount, early access]
```

### 5. Prototype Testing
**Best for**: Usability and feature validation

#### Setup Template
```
Prototype Fidelity: [Low/Medium/High]
Testing Method: [Moderated/unmoderated]
Task Scenarios: [Specific user journeys to test]
Success Metrics: [Task completion rate, time to complete]
Tools: [Figma, InVision, Marvel, etc.]
Participant Recruitment: [How to find testers]
```

### 6. Concierge MVP
**Best for**: Service validation with manual delivery

#### Setup Template
```
Service Description: [What you'll deliver manually]
Customer Touchpoints: [All interaction points]
Manual Processes: [What you'll do behind scenes]
Success Metrics: [Customer satisfaction, retention]
Scaling Plan: [How to automate if successful]
Resource Requirements: [Time and people needed]
```

## Common Experiment Mistakes to Avoid

### 1. Sample Size Too Small
❌ Running test with 50 visitors per variant
✅ Calculate minimum sample size using power analysis (typically 1000+ per variant)

### 2. Stopping Test Too Early
❌ Calling results after 2 days because one variant is winning
✅ Run until statistical significance AND practical significance achieved

### 3. Testing Multiple Things
❌ Changing headline, button color, and price simultaneously
✅ Test one variable at a time to isolate impact

### 4. Ignoring Seasonality
❌ Running test during atypical period (holidays, events)
✅ Account for seasonal patterns and external factors

### 5. Leading Questions in Surveys
❌ "How much do you love our amazing new feature?"
✅ "How would you rate your experience with the new feature?"

### 6. Confirmation Bias
❌ Only measuring metrics that might show positive results
✅ Include metrics that could show negative results too

## Experiment Documentation

### Experiment Log
| ID | Name | Hypothesis | Start Date | End Date | Status | Result | Impact |
|----|------|------------|------------|----------|---------|---------|---------|
| EXP-001 | [Name] | [Hypothesis] | [Date] | [Date] | [Status] | [Outcome] | [Learning] |

### Experiment Report Template
```
# Experiment Report: [Name]

## Executive Summary
- **Hypothesis Tested**: [Brief statement]
- **Key Finding**: [Main result in one sentence]
- **Recommendation**: [What should be done next]
- **Impact**: [Business implications]

## Experiment Details
- **Duration**: [Start - End dates]
- **Sample Size**: [Total participants]
- **Method**: [Type of experiment]
- **Primary Metric**: [Main measurement]

## Results
- **Primary Metric Result**: [Actual vs. target]
- **Statistical Significance**: [p-value, confidence interval]
- **Effect Size**: [Practical significance]
- **Secondary Findings**: [Other insights]

## Insights and Learnings
- **What Worked**: [Positive findings]
- **What Didn't Work**: [Negative findings]
- **Surprises**: [Unexpected results]
- **Customer Feedback**: [Key quotes/themes]

## Recommendations
- **Immediate Actions**: [What to do now]
- **Next Experiments**: [Follow-up tests needed]
- **Business Model Updates**: [Strategy changes]
- **Long-term Implications**: [Strategic considerations]

## Supporting Data
- [Charts, graphs, statistical tables]
- [Customer quotes and feedback]
- [Detailed methodology]
```

---

*Remember: The quality of your experiments determines the quality of your learning*
*Good experiments lead to confident decisions and faster progress toward product-market fit*