Skip to Main Content

Psychiatry Portal: Study Design & Appraisal

Critical Appraisal Worksheets [click files to download - IM Journal Club uses blue McMaster sheets]

Types of Follow Up

Intention to Treat - Once Randomized, Always Analyzed; participants included even if they didn’t fully adhere to protocol

Per Protocol – only patients who complete the entire clinical trial according to protocol are counted towards final results. Can lead to attrition bias if dropout was uneven. Tends to give an "ideal scenario" of how treatment works

Intention-to-Treat resources

Blinding & Allocation

Blinding – two or more parties in a trial are kept unaware of which treatment arms the participants have been assigned to

  • Can be blinded at different levels (unblended; single blinded with patient; or double blinded with clinician and patient unaware of treatment)

Allocation – person randomizing patient doesn’t know what the next treatment allocation will be

  • Prevents selection bias

Types of Error

Type 1 Error: False Positive Rate (detects an effect that isn’t present)

Type 2 Error: False Negative Result (failure to detect an effect that is present)

Example: The Boy who cried wolf

Common Types of Bias

  • Selection Bias – select sample/data wrong, usually because data is easy to access; accidentally work with specific subset of patients rather than the whole (sample is unrepresentative of the whole population)
  • Self-selection Bias – found in polling, surveys; Subcategory of selection bias; Let the subjects of analyses select themselves – less proactive people will be excluded
  • Recall Bias – common in interviews/surveys. Respondent doesn’t remember things correctly (or can’t); not about having a good or bad memory – humans have selective memory by default
  • Observer Bias - Researcher subconsciously projects their own expectations onto the research
  • Funding Bias (sponsorship bias) - Scientific studies are biased in a way that supports the financial sponsor of the research
  • Survivorship Bias - Researcher focuses only on the part of the data set that already went through some kind of pre-selectin process – and missing the data points that fell of during this process because they’re not visible anymore
  • Cognitive Bias
  • Hindsight Bias – even the greatest findings seem very trial looking back on them a few days later
  • Confirmation Bias – carrying pre-conceptions in to research & looking only to confirm it
  • Belief Bias – someone is so sure of gut feelings that they ignore results of a data research project
  • Curse of knowledge – assuming someone else has the same background knowledge you 
  • Omitted Variable Bias - Occurs when you are leaving out one or more important variables from your model – especially often regarding predictive analytics
  • Cause-Effect Bias – correlation does not imply causation
  • Publication Bias - When the likelihood of a study being published is affected by the findings of the study.
  • Language bias - English language has been the predominant language in medical research. Studies publishing positive results might also be more likely to publish in English. Reading and using only English language research could provide a biased assessment of a topic, and can lead to biased results
  • Confounding Variable Bias – has direction (positive – biased away from null, overestimates effect; or negative – biased toward null, underestimates effect)
    • Confounding Variable - Extra variables in a study that weren’t accounted for

Propensity score matching – Drawing causal conclusions from observational studies where the “treatment” was not randomly assigned

  • Approximates a random trial to match controls with experimental subjects; control group is given a propensity score and matched with at least 1 participant. 
    • Bipartate matching – equivalent to sampling without a replacement
    • Non-bipartate matching – sampling with replacement (reuse a member, such as using control as a match for two or more treatment group participants)
  • Can be used to control confounding; other methods include stratification, regression adjustment, and weighting.

(more info: https://catalogofbias.org/biases/)

Types of Questions & study design

DETERMINING THE BEST STUDY TO SEARCH FOR DURING YOUR LITERATURE SEARCH.
Type of Question Description (What is it?) Study Design (when SR not available)
Diagnosis

An evaluation of a test, screening or other assessment such as history or physical exam

Prospective, blind comparison to a gold standard.

RCT > controlled clinical trial (CCT) > cohort > case control > case series

Therapy

An evaluation of a therapeutic or preventive intervention

RCT > CCT > cohort >case control > case series

Prognosis An evaluation of clinical outcomes over time cohort > case control > case series
Etiology/Harm

An evaluation of a therapeutic, preventive, screening or diagnostic intervention, or a non-therapeutic exposure or behavior

cohort > case control > case series

Prevention

An evaluation of the effectiveness of an intervention or exposure in preventing morbidity and mortality

RCT > CCT > cohort > case control > case series
Cost

An evaluation of cost versus benefit of a treatment or procedures

Economic analysis, cost-benefit analysis

 

Contact the Darnall Medical Library (DML)

For WRNNMC Healthcare
Providers and Staff
For WRNNMC Patients
  • TRICARE Appointment Line: 855-227-6331
  • Walter Reed NMMC Main Phone: 301-295-4611
  • TRICARE Health Services (includes phone numbers)

Researching a Medical Condition?
Click the button below!

Our Partners

JAMAevidence

JAMAevidence

Helps decision makers identify the best available evidence by providing guides to the systematic consideration of the validity, importance, and applicability of claims about the assessment of health problems and the outcomes of health care.

Statistics for the Non-Statistician

 

Disclaimer: The appearance of hyperlinks does not constitute endorsement by the Department of Defense, Defense Health Agency of non-U.S. Government sites or the information, products, or services contained therein. Although the Darnall Medical Library, WRNMMC may or may not use these sites as additional distribution channels for Department of Defense information, it does not exercise editorial control over all of the information that you may find at these locations. Such hyperlinks are provided consistent with the stated purpose of this website.

Darnall Medical Library | Walter Reed NMMC | Building 1, Room 3458 | 8955 Wood Road | Bethesda, MD 20889 | 301-295-1184/85 | Open Monday-Friday, 0700-1630

After-hours access to the library is available to WRNMMC Staff via the CDO at 301-295-4611.