Observational Studies vs Experiments

Understanding the difference and implications

Observational Studies vs Experiments

The Fundamental Difference

Observational Study: Observe/measure without intervention
Experiment: Actively impose treatment to observe effect

Key distinction: In experiments, researchers control the explanatory variable; in observational studies, they don't.

Observational Studies

Definition: Researchers observe individuals and measure variables without assigning treatments.

Characteristics:

  • No manipulation by researcher
  • Observe "natural" conditions
  • Can show association but NOT causation

Examples:

  • Survey asking smokers about health
  • Study comparing test scores by study method (students choose method)
  • Medical records analysis of diet and disease

Types:

Sample Survey: Collect data from sample at one point in time
Retrospective Study: Look back at historical data
Prospective Study: Follow subjects forward in time

Advantages:

  • Can study variables that can't be manipulated ethically
  • Often cheaper and faster
  • Observes real-world behavior

Disadvantages:

  • Cannot establish causation (correlation ≠ causation!)
  • Confounding variables
  • Less control over conditions

Experiments

Definition: Researchers deliberately impose treatments and observe responses.

Characteristics:

  • Researcher assigns treatments
  • Control over conditions
  • Can establish cause-and-effect

Examples:

  • Randomly assign students to study methods, compare test scores
  • Give some patients medicine, others placebo
  • Test fertilizer by applying different amounts to randomly selected plots

Key Components:

Explanatory Variable (Factor): What researcher manipulates
Treatments: Different values/levels of explanatory variable
Response Variable: Outcome measured
Experimental Units: Individuals/items receiving treatments

Advantages:

  • Can establish causation (if well-designed)
  • Control over conditions
  • Can isolate effect of treatment

Disadvantages:

  • May be unethical (can't randomly assign smoking!)
  • Artificial setting may not reflect real life
  • Often expensive and time-consuming

Why Experiments Can Show Causation

Three criteria for causation:

  1. Association: Variables are related
  2. Time order: Cause precedes effect
  3. No plausible alternative explanation

Experiments satisfy all three:

  • Random assignment controls for confounding
  • Researcher imposes treatment before measuring response
  • Controlled conditions eliminate alternatives

Observational studies can show association but struggle with #3 (confounding variables provide alternative explanations)

Confounding Variables

Confounding Variable: Variable related to both explanatory and response variables that distorts the relationship.

Example:

  • Observational study: Coffee drinkers have higher heart disease rates
  • Confounding variable: Smoking (coffee drinkers more likely to smoke; smoking causes heart disease)
  • Conclusion: Can't tell if coffee causes heart disease or if it's the smoking!

Experiments address confounding through randomization (discussed in Experimental Design topic)

Association vs Causation

Association (Correlation): Variables are related
Causation: One variable causes changes in another

All causation involves association, but NOT all association implies causation!

Famous example: Ice cream sales and drowning deaths are associated (both higher in summer), but ice cream doesn't cause drowning. Confounding variable: weather/temperature.

Choosing Between Observational and Experimental

Use Observational Study when:

  • Can't manipulate variable ethically (smoking, genetics)
  • Want to study relationships in natural setting
  • Experiment not feasible

Use Experiment when:

  • Want to establish causation
  • Can ethically manipulate variable
  • Need controlled conditions

Ethical Considerations

Cannot experiment on:

  • Harmful treatments (cigarette smoking)
  • Variables you can't control (age, gender, genetics)
  • Situations where withholding beneficial treatment would be harmful

Must use observational studies for many important questions!

Scope of Inference

Generalization (to population): Requires random sampling
Causation (cause-effect): Requires random assignment (experiment)

Four scenarios:

| Random Sample? | Random Assignment? | Can Infer Causation? | Can Generalize? | |----------------|-------------------|---------------------|-----------------| | Yes | Yes | Yes | Yes | | Yes | No | No | Yes | | No | Yes | Yes | No | | No | No | No | No |

Best case: Random sample + random assignment (rare!)
Worst case: Convenience sample + observational (common but weakest)

Quick Reference

Observational Study:

  • Observe without intervention
  • Shows association
  • Cannot establish causation
  • Confounding variables problematic

Experiment:

  • Impose treatments
  • Can show causation
  • Random assignment controls confounding
  • May not reflect real-world conditions

Remember: The gold standard for establishing causation is a well-designed, randomized experiment. Observational studies can suggest relationships but cannot prove cause-and-effect!

📚 Practice Problems

1Problem 1easy

Question:

Classify each study as observational or experimental: a) Researchers randomly assign patients to receive either a new drug or placebo b) Scientists measure air pollution levels and asthma rates in different cities c) Psychologists observe children's behavior in a playground

💡 Show Solution

Step 1: Understand the distinction Experiment: Researchers IMPOSE treatments on subjects Observational: Researchers OBSERVE without intervention

Step 2: Analyze each study

a) Patients assigned to drug or placebo

  • Researchers IMPOSE treatment (drug vs placebo)
  • Random assignment by researchers
  • Manipulation of explanatory variable Classification: EXPERIMENT

b) Measure pollution and asthma rates

  • Researchers OBSERVE existing conditions
  • No manipulation of pollution levels
  • Just measuring what naturally occurs Classification: OBSERVATIONAL STUDY

c) Observe children's behavior

  • Researchers WATCH without interference
  • No manipulation of children or environment
  • Passive observation Classification: OBSERVATIONAL STUDY

Answer: a) Experiment (random assignment to treatments) b) Observational study (measuring existing conditions) c) Observational study (passive observation)

2Problem 2easy

Question:

True or False: "If an observational study finds a strong association between two variables, you can conclude one causes the other." Explain your answer.

💡 Show Solution

Step 1: Answer FALSE - absolutely FALSE

Step 2: Why this is false Association ≠ Causation Correlation ≠ Causation Strong association does NOT imply cause-and-effect

Step 3: The three possible explanations for association

When X and Y are associated, three possibilities:

  1. X causes Y (what we might want to conclude)
  2. Y causes X (reverse causation)
  3. Z causes both X and Y (confounding)

Without an experiment, cannot distinguish between these!

Step 4: Famous examples

Example 1: Shoe size and reading ability in children

  • Strong positive association
  • Larger shoes → better reading
  • Does shoe size CAUSE reading ability? NO!
  • Confounder: AGE (older kids have bigger feet AND read better)

Example 2: Number of firefighters and fire damage

  • Strong positive association
  • More firefighters → more damage
  • Do firefighters CAUSE damage? NO!
  • Confounder: FIRE SIZE (bigger fires draw more firefighters AND cause more damage)

Example 3: Chocolate consumption and Nobel Prizes (real published finding!)

  • Countries that eat more chocolate have more Nobel laureates per capita
  • Does chocolate make you smarter? Probably not!
  • Confounders: Wealth, education systems, culture

Step 5: What you CAN conclude from observational studies

Can conclude: ✓ Variables are associated/correlated ✓ Variables move together ✓ Knowing one helps predict the other ✓ There's a relationship worth investigating

Cannot conclude: ✗ One causes the other ✗ Changing one will change the other ✗ The relationship is causal

Step 6: How to establish causation

ONLY through:

  • Randomized controlled experiments
  • Or very strong evidence from multiple observational studies with:
    • Temporal sequence (cause before effect)
    • Dose-response relationship
    • Biological plausibility
    • Consistency across studies
    • No plausible confounders

Answer: FALSE. Strong association from observational studies does NOT prove causation. Confounding variables could cause both variables, or reverse causation could occur. Example: shoe size and reading ability are strongly associated, but age is the confounder - older children have bigger feet AND read better. Only randomized experiments can establish cause-and-effect.

3Problem 3medium

Question:

Can you conclude cause-and-effect from an observational study? Why or why not? Give an example.

💡 Show Solution

Step 1: Direct answer NO - observational studies generally CANNOT establish causation Can only show association/correlation

Step 2: Explain why not The problem: CONFOUNDING VARIABLES

  • Other variables might cause both the explanatory and response variable
  • Can't distinguish between correlation and causation
  • No control over lurking variables

Step 3: Classic example - Ice cream and drowning

Observation: Cities with high ice cream sales have high drowning rates

Possible (wrong) conclusion: Ice cream causes drowning

Reality: CONFOUNDING VARIABLE = Temperature/Summer

  • Hot weather → people buy ice cream (association)
  • Hot weather → people go swimming → more drownings (causation)
  • Ice cream and drowning are correlated but not causal

Step 4: Another example - Coffee and heart disease

Observational finding: Coffee drinkers have higher heart disease rates

Cannot conclude: Coffee causes heart disease

Why? Possible confounders:

  • Smoking (coffee drinkers may smoke more)
  • Stress (stressed people drink more coffee AND have heart issues)
  • Sleep deprivation
  • Diet differences

Step 5: When can observational studies suggest causation?

Rarely, with very strong evidence:

  • Smoking and lung cancer (overwhelming evidence from many studies)
  • Dose-response relationship
  • Temporal sequence (cause before effect)
  • Biological plausibility
  • Consistency across many studies

But still can't PROVE causation without experiment

Answer: NO. Observational studies cannot establish cause-and-effect because of confounding variables. Can only show association. Example: ice cream sales and drowning are associated, but both are caused by hot weather (confounder), not each other. Only randomized experiments can establish causation.

4Problem 4medium

Question:

A researcher wants to study whether homework improves test scores. Compare an observational approach vs. an experimental approach. Which would be better for establishing causation?

💡 Show Solution

OBSERVATIONAL APPROACH:

Design:

  • Observe students naturally
  • Record how much homework they do (voluntary)
  • Measure test scores
  • Compare scores of high-homework vs low-homework students

Problems:

  1. Selection bias

    • Studious students do more homework naturally
    • They might score better anyway (motivated, better study habits)
  2. Confounding variables

    • Parental involvement
    • Prior knowledge
    • Intelligence/ability
    • Study skills
    • Time available
    • Teacher quality
  3. Reverse causation possible

    • Maybe students who understand material better choose to do more homework
    • Causation could go either way

Cannot conclude: Homework causes better scores (only association)

EXPERIMENTAL APPROACH:

Design:

  • Take a class of students
  • RANDOMLY assign half to required homework, half to no homework
  • Keep everything else the same (same teacher, material, class time)
  • Compare test scores

Advantages:

  1. Random assignment balances confounders

    • Both groups have similar motivation, ability, backgrounds
    • Differences wash out on average
  2. Control over treatment

    • Researcher dictates homework amount
    • Not student choice
  3. Can isolate homework effect

    • Only systematic difference between groups is homework
    • If scores differ → homework caused it

CAN conclude: Homework causes score differences

PRACTICAL/ETHICAL CONSIDERATIONS:

Observational:

  • Easier to conduct
  • No ethical issues
  • More realistic (natural behavior)
  • Cannot establish causation

Experimental:

  • Can establish causation
  • Harder to implement (need cooperation)
  • Ethical concern: denying homework to some students
  • May not generalize (artificial setting)

BEST APPROACH:

For causation: EXPERIMENTAL is better Must use random assignment to establish cause-effect

However: Might need to use observational if:

  • Experiment is unethical
  • Experiment is impractical
  • Want to study natural behavior

Could do BOTH:

  • Observational to explore relationships
  • Experimental to test causation

Answer: EXPERIMENTAL approach is better for establishing causation. Random assignment of students to homework vs. no-homework groups controls for confounding variables, allowing causal conclusions. Observational approach only shows association because studious students might naturally do more homework AND score better for other reasons (confounders).

5Problem 5hard

Question:

Why can randomized experiments establish cause-and-effect while observational studies cannot? Explain the role of random assignment.

💡 Show Solution

Step 1: The key difference EXPERIMENTS use random assignment OBSERVATIONAL STUDIES do not

Step 2: What random assignment does

Random assignment means:

  • Subjects randomly placed into treatment groups
  • Done by researcher, not by choice or natural circumstances
  • Creates groups that are similar EXCEPT for the treatment

Magic of randomization:

  • Balances known confounders (age, gender, health, etc.)
  • Balances UNKNOWN confounders (genetic factors we don't know about)
  • Makes groups comparable at the start

Step 3: How this enables causal inference

Before treatment: Groups are essentially equivalent After treatment: Groups differ Only difference: The treatment itself Conclusion: Treatment CAUSED the difference

Step 4: Example - Testing a new drug

EXPERIMENT (can show causation):

  1. Take 200 patients with headaches
  2. RANDOMLY assign 100 to new drug, 100 to placebo
  3. Random assignment balances:
    • Age, gender, severity, stress, diet, genetics, etc.
  4. Measure headache improvement
  5. If drug group improves more → drug CAUSED improvement

OBSERVATIONAL STUDY (cannot show causation):

  1. Let patients CHOOSE whether to take new drug
  2. Those who choose drug might differ:
    • More severe headaches (more desperate)
    • More health-conscious
    • Better insurance
    • Different expectations
  3. Measure improvement
  4. If drug group improves more → could be:
    • The drug works
    • They had different headaches to begin with
    • Placebo effect from expectation
    • Better overall health habits Cannot separate these!

Step 5: Mathematical perspective

Observational: Treatment group ≠ Control group (systematically different) Experimental: Treatment group ≈ Control group (random differences only)

With random assignment: E(confounders | treatment) = E(confounders | control) Without: E(confounders | treatment) ≠ E(confounders | control)

Answer: Random assignment in experiments creates equivalent groups that differ ONLY in treatment, allowing causal conclusions. Observational studies lack random assignment, so groups may differ in many ways (confounders), making it impossible to determine if the explanatory variable caused the outcome or if confounding variables did.