Psychiatry Meta-Analysis Guide

Table of Contents

1. Why Meta-Analysis Is Critical in Psychiatry
2. Defining Your Psychiatric Research Question (PICO)
3. Choosing the Right Effect Size
4. Handling Rating Scale Differences
5. The Placebo Effect Problem
6. Data Extraction Checklist for Psychiatric Studies
7. Handling Heterogeneity in Psychiatric Trials
8. Subgroup and Network Meta-Analysis
9. Step-by-Step: Psychiatry Meta-Analysis in MetaReview
10. Common Pitfalls in Psychiatric Meta-Analysis

1. Why Meta-Analysis Is Critical in Psychiatry

Psychiatric disorders affect hundreds of millions of people worldwide, yet individual trials frequently produce conflicting results. Effect sizes are typically small to moderate (SMD 0.2–0.5 for antidepressants), placebo response rates are high, and dozens of competing treatments exist without head-to-head comparisons. Meta-analysis addresses these challenges by:

• Increasing statistical power — Small drug-placebo differences become detectable only when pooling thousands of patients across trials.
• Resolving the "do antidepressants work?" debate — Kirsch et al. (2008) and Cipriani et al. (2018, Lancet) reached different conclusions partly due to methodological choices in their meta-analyses. Understanding these choices is essential.
• Comparing treatments that were never directly compared — Network meta-analysis (NMA) allows ranking of 21 antidepressants or 15 antipsychotics despite most trials only comparing one drug to placebo.
• Quantifying publication bias — Turner et al. (2008, NEJM) showed that 94% of published antidepressant trials were positive, while FDA records revealed only 51% were truly positive. Meta-analysis with bias correction provides a more honest estimate.

2. Defining Your Psychiatric Research Question (PICO)

A precise PICO framework prevents scope creep and unmanageable heterogeneity. Psychiatric populations are highly variable, so specificity at this stage saves enormous effort later.

Disorder-Specific Considerations

3. Choosing the Right Effect Size

This is the most critical decision in psychiatric meta-analysis. Because studies use different rating scales for the same construct, Standardized Mean Difference (SMD) is the default choice — unlike most other medical fields where Mean Difference (MD) is common.

J(df) is the small-sample correction factor, approximately 1 − 3/(4df − 1)

4. Handling Rating Scale Differences

Rating scale heterogeneity is the defining challenge of psychiatric meta-analysis. Unlike blood pressure or tumor size, psychiatric symptoms are measured through subjective scales with different items, scoring ranges, and rater perspectives.

Major Depression Scales

Other Key Psychiatric Scales

Practical Recommendations

• Use SMD whenever studies use more than one scale — this is nearly always the case in psychiatry
• Prefer change scores over endpoint scores when available, as they reduce between-subject variability
• Never convert HAMD-17 to HAMD-21 by adding estimated points for the extra 4 items — this introduces noise
• Run sensitivity analysis restricting to the most commonly used scale (usually HAMD-17) to check robustness

5. The Placebo Effect Problem

No other medical field faces the placebo challenge quite like psychiatry. The placebo response in antidepressant trials typically ranges from 30% to 50%, dwarfing the drug-specific effect and creating unique analytical difficulties.

Key Facts About the Psychiatric Placebo Response

• Magnitude: Placebo arms in MDD trials show a mean improvement of ~10 points on HAMD-17. Drug arms improve ~13 points. The drug-specific effect is only ~3 points.
• Temporal trend: Placebo response has increased over decades. Trials from the 1980s showed lower placebo response than trials from the 2010s, creating a confound when pooling across eras.
• Failed trials: Approximately 50% of antidepressant trials "fail" (do not separate drug from placebo), not because drugs are ineffective but because placebo response was unusually high in that trial.
• Site variability: Multi-site trials show enormous variability in placebo response across sites (20%–70%), often driven by rater training differences and patient selection.

How the Placebo Effect Impacts Your Meta-Analysis

6. Data Extraction Checklist for Psychiatric Studies

Psychiatric trials have unique reporting patterns. Use this checklist to ensure complete, consistent extraction.

Study Characteristics

• First author, publication year, journal, NCT registration number
• Study design (RCT, crossover, open-label), phase, sponsor (industry vs academic)
• Region/country, number of sites

Patient Population

• Diagnosis and diagnostic criteria (DSM-IV, DSM-5, ICD-10, ICD-11)
• Baseline severity (mean HAMD or MADRS score, severity category)
• Episode characteristics (first episode vs recurrent, duration of current episode)
• Comorbidities (anxiety, substance use, personality disorder)
• Sample size per arm (ITT and completer populations)
• Age, sex, treatment history (treatment-naive vs treatment-resistant)

Intervention Details

• Drug name, dose (fixed vs flexible dosing), titration schedule
• For psychotherapy: modality (CBT, IPT, DBT), number and frequency of sessions, therapist qualifications
• Placebo type (pill placebo, psychological placebo, waitlist)
• Trial duration (acute phase: typically 6–12 weeks for MDD)
• Concomitant medications allowed (especially benzodiazepines, sleep aids)

Outcomes Data

• Primary scale and version (HAMD-17 vs HAMD-21 vs MADRS)
• Mean and SD of change scores (or endpoint scores) per arm
• Response and remission rates (events/total per arm)
• Dropout rates (all-cause and AE-related, per arm)
• Analysis method: LOCF vs MMRM vs observed cases

7. Handling Heterogeneity in Psychiatric Trials

Psychiatric meta-analyses almost always show moderate to high heterogeneity (I² 40%–80%). Understanding and accounting for the sources is essential for credible results.

Sources of Heterogeneity Specific to Psychiatry

Recommended Strategies

1. Always use random-effects models — Fixed-effect is almost never appropriate in psychiatry given the clinical diversity
2. Pre-specify subgroup analyses by drug class, severity, scale type, and trial duration
3. Meta-regression on baseline severity, publication year, and placebo response rate (if ≥10 studies)
4. Sensitivity analysis — Leave-one-out; restrict to HAMD-17 only; exclude studies with high risk of bias; MMRM-only vs all studies

8. Subgroup and Network Meta-Analysis

Subgroup Analysis

Pre-specified subgroup analyses are essential for interpreting psychiatric meta-analyses. Common subgroup variables include:

• Drug class: SSRIs (fluoxetine, sertraline, escitalopram, paroxetine, citalopram) vs SNRIs (venlafaxine, duloxetine) vs atypicals (mirtazapine, bupropion, vortioxetine)
• Severity: Mild vs moderate vs severe depression (based on baseline HAMD or MADRS)
• Age group: Adolescents vs adults vs elderly (antidepressant efficacy and safety differ by age)
• Treatment modality: Pharmacotherapy vs psychotherapy vs combination
• Scale type: Clinician-rated vs self-rated instruments

Network Meta-Analysis (NMA)

NMA is arguably the most impactful analytical method in psychiatric research. Because head-to-head trials between active drugs are rare, NMA uses indirect comparisons through a common comparator (usually placebo) to rank multiple treatments simultaneously.

The Cipriani 2018 Landmark Study

Cipriani et al. (Lancet, 2018) conducted the largest NMA of antidepressants ever performed: 522 double-blind RCTs, 116,477 patients, 21 antidepressants. Key findings:

• All 21 antidepressants were more effective than placebo for acute MDD treatment
• Most effective: amitriptyline, mirtazapine, venlafaxine, escitalopram
• Best tolerated (lowest dropout): agomelatine, fluoxetine, escitalopram, citalopram
• Overall best balance of efficacy and acceptability: escitalopram, sertraline, mirtazapine, venlafaxine

NMA Requirements and Assumptions

9. Step-by-Step: Psychiatry Meta-Analysis in MetaReview

Step 1: Select Effect Measure

Open MetaReview and choose the appropriate measure:

• SMD (Hedges' g) for continuous symptom scores across different scales (most common)
• Mean Difference if all studies use the same scale version
• Odds Ratio for response/remission rates

Step 2: Enter Study Data

For SMD analysis: enter Study name, Year, Mean, SD, and N for both Treatment and Control arms.

For OR analysis: enter Study name, Year, Events and Total for both arms.

Step 3: Assign Subgroups

Use the Subgroup column to label studies by drug class (SSRI, SNRI, atypical), severity level, or scale type. This enables stratified forest plots with Q-between tests.

Step 4: Run the Analysis

Click "Run Meta-Analysis". Results appear within seconds:

• Forest plot with individual and pooled SMD (or OR)
• Subgroup forest plot with Q-between significance test
• Funnel plot for publication bias assessment
• Heterogeneity statistics (I², Q, τ²)

Step 5: Advanced Diagnostics

• Egger's / Begg's test — Quantitative publication bias assessment (critical for psychiatric data)
• Trim-and-Fill — Estimate impact of missing negative studies
• Leave-one-out sensitivity analysis — Check whether any single trial drives the result
• Meta-regression — Test baseline severity or publication year as moderators

Step 6: Export Report

Generate an HTML or DOCX report with all figures, tables, auto-generated Methods paragraph (PRISMA 2020 format), and narrative interpretation including SMD-to-clinical-meaning conversion.

10. Common Pitfalls in Psychiatric Meta-Analysis

Pitfall 1: Ignoring Publication Bias

Turner et al. (2008, NEJM) demonstrated that publication bias inflates antidepressant effect sizes by approximately 32%. Relying only on published literature produces an overly optimistic estimate of drug efficacy.

Solution: Search ClinicalTrials.gov and FDA medical reviews for unpublished data. Use funnel plots, Egger's test, and Trim-and-Fill. Always report both corrected and uncorrected pooled estimates.

Pitfall 2: The Kirsch Debate — Clinical vs Statistical Significance

Kirsch et al. (2008) reported a pooled SMD of −0.32 for SSRIs vs placebo, arguing this was below the NICE threshold of 0.50 for clinical significance. Others argued the 0.50 threshold was arbitrary and that even SMD = 0.30 translates to meaningful symptom relief for many patients.

Solution: Report SMD alongside clinically interpretable metrics: NNT (number needed to treat), HAMD-point equivalents (SMD × SD_placebo), and response/remission rate differences.

Pitfall 3: Mixing Scales Without Using SMD

Pooling raw mean differences from HAMD-17 (range 0–52) and MADRS (range 0–60) produces nonsensical results because a 3-point change means entirely different things on each scale.

Solution: Always use SMD (Hedges' g) when combining data from different rating scales. This is not optional — it is a fundamental methodological requirement.

Pitfall 4: LOCF vs MMRM Confusion

Many older trials report only LOCF results. LOCF carries forward the last observed score for patients who drop out, which can bias results in either direction. MMRM uses all available data without imputation and is now the regulatory standard.

Solution: Prefer MMRM data when available. Record the analysis method for each study. Run a sensitivity analysis excluding LOCF-only studies.

Pitfall 5: Not Accounting for Active Placebo

Standard pill placebos may be "unblinded" because patients can tell they are not experiencing side effects. Studies using active placebos (substances that mimic side effects without therapeutic action) show smaller drug-placebo differences.

Solution: Record whether the trial used an active or inert placebo. Consider subgroup analysis or at minimum discuss as a limitation.

Pitfall 6: Allegiance Effects

The researcher's theoretical allegiance (favoring psychotherapy vs pharmacotherapy, or one drug over another) can influence trial outcomes through subtle methodological choices. Industry-sponsored trials tend to favor the sponsor's drug.

Solution: Record sponsor type and declared conflicts of interest. Run subgroup analysis by sponsor (industry vs academic vs government). Use GRADE to downgrade certainty if most evidence comes from a single sponsor.