Designing Deliberative Methods for Combining Heterogeneous Evidence: A Systematic Review and Qualitative Scan

Full Report (PDF, 1.17 MB)


M.J. Dobrow
Department of Health Policy, Management & Evaluation, University of Toronto, Cancer Services & Policy Research Unit, Cancer Care Ontario

R. Chafe
Department of Health Policy, Management & Evaluation, University of Toronto

H.E.D. Burchett 
London School of Hygiene & Tropical Medicine

A. Culyer
Department of Health Policy, Management & Evaluation, University of Toronto, Department of Economics & RelatedStudies, University of York

L. Lemieux-Charles
Department of Health Policy, Management & Evaluation, University of Toronto

Main Messages

  • Deliberative methods are generating increasing interest among policy-makers, managers, practitioners and researchers.
  • Deliberative methods are evolving rapidly, resulting in greater variation of approaches and ongoing conceptual confusion. However, with few true exemplars, it is unclear what progress is actually being made.
  • Based on the assumption that better policy guidance requires a combination of heterogeneous evidence, the aim of this study was to examine: (1) how/when deliberative methods are used to combine heterogeneous evidence; and (2) what is known about the effectiveness of deliberative methods for combining heterogeneous evidence.
  • We conducted a systematic review of health and non-health literature to provide guidance on this area. We also conducted a qualitative scan, including direct observation and interviews with individuals responsible for exemplar deliberative processes.
  • We identified numerous examples where deliberative methods are used in policy guidance processes. However, there were only a handful of examples explicitly using deliberative methods to combine heterogeneous evidence, with no work directly assessing their effectiveness.
  • The health sector has more established deliberative processes than other sectors. However, work in the field of environmental policy provided important insights on the role of deliberative methods for combining heterogeneous evidence.
  • Despite the lack of evidence to address the second research question, the findings of the systematic review and qualitative scan did provide important insights on the role that deliberative methods play in combining heterogeneous evidence. Drawing on these findings, we identified three key factors that affect how deliberative methods contribute to the combining of heterogeneous evidence, including:
    • Decision proximity: reflects the proximity of the deliberative process to a specific decision context (distal-general or proximal-specific).
    • Deliberative approach: reflects either representative (democratic-deliberative) or technical (analytic-deliberative) approaches to combining evidence.
    • Evidence use: reflects varying formality and explicitness in identifying, interpreting and combining heterogeneous evidence (informal-implicit or formal-explicit).