Approaches to Evidence Synthesis in Systematic Reviews of the CPSTF


Good afternoon my name is Marie Rienzo
and I want to welcome you to the NIH Office of Disease Prevention’s Mind the
Gap webinar series this series explores research design, measurement, intervention,
data analysis, and other methods of interest to prevention science our goal
is to engage the prevention research community and thought-provoking
discussions to promote the use of the best available methods and to support
the development of better methods before we begin I have some housekeeping items
to submit questions during the webinar there are two options first you may
submit questions via WebEx by clicking on the question mark in the WebEx
toolbar please direct your questions to all panelists second you may participate
by Twitter and submit questions using the hashtag #NIHMTG at the conclusion of
today’s talk we will open the floor to questions that have been submitted by a
WebEx and Twitter lastly you would appreciate your feedback about today’s
webinar upon closing the WebEx meeting you will be directed to a website to
complete an evaluation we would appreciate your feedback it will help us
improve this webinar series at this time I’d like to introduce Dr. David M Murray
Associate Director for Prevention and Director of the Office of Disease
Prevention Thank you Marie today’s speaker is Dr. David Hopkins who’s a
medical officer at the Community Guide Branch of the Center’s for Disease Control and
Prevention he joined the Community Guide in 1997 and has led systematic reviews
of population-based interventions in tobacco prevention and control, in
asthma, diabetes, cardiovascular disease prevention, physical activity promotion,
and immunizations he has also worked closely with Community Guide researchers,
CDC partners, and members of the Community Preventive Services Task Force
on methods for both conducting systematic reviews public health
interventions and prioritizing future community guide topic areas and
intervention reviews Dr. Hopkins is coming to us today from Portland Oregon
it’s my pleasure to introduce him to the webinar Good afternoon thank you for that introduction and the challenging opportunity to summarize 22 years of
task force debate in a 40 minute webinar first a disclaimer CDC provides staffing
and an administrative support for the Community Preventative Services Task
Force but their assessments of the evidence findings and recommendations do
not necessarily represent the official position of CDC here’s what I’d like to
cover in this webinar first they introduce the Community Guide and the Community Preventive Services Task Force and their perspective on an
evidence-based for public health I will highlight a few of the systematic review
methods used as support task force recommendations and additional findings
I’ll use a study on inclusion of practice based evidence to quantify
observational study designs in our work I’ll point out a few of the process
steps used to conduct systematic reviews of economic evidence and end with a few
persistent challenges it’s possible that I will gloss over a couple of
the slides in this presentation and identify some judgment tools that we use
or that the task force uses which sort of blur the line between systematic
review methods and process guidance for drawing conclusions and recommendations
regarding use the Community Guide which is primarily now a web resource houses
sets of intervention reviews across the range of public health topics our focus
is on interventions appropriate for communities and health care system to
improve health our approach includes at least initially a broad consideration of
evidence on effectiveness and and on other issues important to program
planners and decision makers the reviews produced support the conclusions and
findings of the Task Force the community Preventive Services Task Force which I
will now refer to as the Task Force is an independent non-federal volunteer panel
of experts recruited from public health and health care research and practice
Task Force members provide oversight on each systematic review project and guide
to develop into methods and process steps as a group the Task Force
evaluates each completed review draws conclusions produces recommendations and
additional findings and highlights evidence gaps both to promote an agenda
for further research and to inform decision-making the current 15 members
of the task force are outlined on this slide as well
since their inception in 1996 the Task Force has produced systematic reviews in
18 prioritized topic areas and they have 234 active findings every five years the
Task Force engages in a prioritization process for topic area work those to add
new intervention reviews in existing topic areas and to identify new topic
areas for exploration as staffing and resources allow here is their current
unranked priority topic areas which I’ve separated to identify the topics that
are candidates for expanded work at the top and new areas at the bottom this
diagram identifies a standard set of issues considered in each community
guide project the primary focus is an assessment of the evidence on
effectiveness of the intervention for all health or adequately health link to
outcomes for some interventions such as policies or programs directed at
determinants of health there may be several different pathways and distinct
sets of outcomes we search for evidence or information on postulated or
potential harms additional benefits and considerations on implementation studies
are also organized to examine applicability or generalizability of the
evidence to important settings populations and intervention
characteristics we examine evidence and information to determine if the
intervention is effective or potentially useful in advancing health equity for
prioritized interventions with evidence on effectiveness the task force may
direct a follow-up economic review to assess the evidence on cost benefits and
comparisons of the two ideal goal for the task forces to
produce prioritize sets of intervention reviews within each topic this is the
current set for the topic area of health equity within which the task force
approved an initial focus in the subtopic of education programs and
policies a set of intervention reviews such as this set provides options to
local decision-makers and in this case span the gamut from pre-k through high
school Task Force conclusions range from
recommendations for use of an intervention based on strong or
sufficient evidence to recommendations against use of the intervention based on
strong or insufficient evidence although we have only produced two of
those in our history if the task force finds that the evidence doesn’t support
a conclusion on effectiveness they draw an insufficient evidence finding frequently Task Force reviews and
recommendations complement the work of the U.S. Preventive Services Task Force
whose scope includes preventive services and treatments appropriate for primary
care or primary care referral settings and providers here’s an example the U.S.
Preventive Services Task Force recommends that clinicians ask all
adults about tobacco use advise them to stop using tobacco and provide
behavioral interventions and FDA-approved pharmacotherapy to support
cessation and adults who use tobacco the Community Preventive Services Task Force
has produced 21 intervention reviews appropriate for communities and health
care systems in this topic area and I pulled out four with recommendations for
use for this example of the four recommended interventions
the first two smoke-free policies and mass media campaigns our community level
interventions that are effective at reducing tobacco use in the population
and in part they may do so by increasing demand for clinical cessation services
provider reminder systems are an effective intervention that a healthcare
system may implement to increase identification of patients who smoke and
this can contribute to initiating the U.S. Preventive Services Task Force
recommendation for assessment brief advise cessation counseling and
medications finally quit lines are effective community level intervention
and provide a community accessible referral preventive service that could
be used instead of health system based counseling or in addition to it the
complan area complimentary nature of our work is deliberate it reflects ongoing
communication between the two task forces and stakeholder interest in
increasing the delivery and receipt of recommended preventive services by the
U.S. Preventive Services Task Force let’s turn now to how the community guide and
the task force think about public or population health interventions and the
range of evidence to be considered in their assessments on effectiveness based
on their current and future topic areas and the set of education interventions
we just went over should be clear that the Task Force has a very expansive view
of interventions to improve population health specific types of interventions
range from programs and services to public and health system policies a
scale of implementation for a specific intervention can be very broad and this
can contribute to a highly variable body of evaluation evidence both the
interventions and the available evaluations can be complicated including
potentially important differences and components of the intervention settings
target populations and level of resources used these differences can
affect the effectiveness of the intervention or the findings of the
study evaluating the intervention Task Force considers needs to be critical
questions for them to attempt to answer using the available evidence does the
intervention work, how well, is that impact meaningful, for whom does it work,
under what conditions does the intervention have effectiveness our
utility for influencing disparities in advancing health equity what is the cost
does it provide value are there issues related to implementation that would be
useful for decision-makers to know in advance
in general the task force takes the position the tentative answers to these
questions based on the available evidence still provide utility the
decision makers and program implementers Tom Frieden our former director of CDC
published this his framework for public health action in the American Journal of
Public Health in 2010 his argument nicely captures the foundational
perspective of the Task Force on the importance of moving public health
awareness and action down in this case his health impact pyramid well there are
important and effective interventions in all strata of the pyramid and in fact we
community guide has looked at interventions in at least four of these
five criteria strata interventions contributing to healthier social and
physical environments and programs and policies influencing determinants of
health are likely to have the greatest impact on population health incorporating this argument into the
mandate of the Task Force which is charged with finding and evaluating the
available evidence and you have one rationale for a broad consideration of
the evidence at the top of our pyramid randomized trials are appropriately both
the high standard and a common design for evaluation of intervention
effectiveness at the base of the pyramid the available evidence may not
exclusively consist of but typically depends on natural experiments
observational designs and cross-sectional comparisons additional
reasons for a broad consideration of study designs is to retain evidence or
information for assessments of generalizability or applicability across
settings populations and intervention characteristics this additional evidence
might help the task force to determine components or combinations associated
with effectiveness or to assist to document or suggest effect modification
depending on the intervention and outcomes under review these decisions
come at the cost of including studies with potential additional and important
sources of bias which we may or may not be able to identify and may or may not
be able to discount when evaluating the full body of evidence which consists of
the range of different study designs with different strengths weaknesses this slide provides one example of the
value of retaining observational studies and a Community Guide review this is a
plot of change in influenza vaccination coverage among workers before and after
implementation of a worksite program coverage rates for each study are lined
up for the years before the on-site program and after the on-site program
was implemented each study found a relatively stable baseline and a
meaningful post-intervention improvement the team did not identify in the studies
and could not generate possible alternative explanations for this
observed effect and Task Force found this evidence to be adequate to support
a conclusion and guidance to the field there are important implications of this
willingness to consider a broad range of study designs as a result most community
guide reviews will retrieve a mixed body of evidence which requires additional
categorization and subset assessments and those subsets can be based on a very
small number of studies most reviews will not be good candidates for meta
analyses at least in whole and the task force will be faced with having to weigh
subsets of the evidence and develop overall determinations on whether the
body of evidence indicates a consistent and meaningful effect now let’s look at
some of the process steps built into the typical Community Guide review to enable
an assessment of population based interventions there are a couple of process steps in which
the Community Guide reviews differ slightly from those produced by other
organizations although the basic approach is similar given the breadth of
what we’re look at very different topic areas very different interventions we
depend on input and oversight from a recruited team of in this in most
cases one or two Task Force members two to four member participants from CDC or
NIH partners and four to six recruited subject matter experts this group
prioritizes the intervention for review within a topic and sets up each specific
intervention review we’re going to look at some additional elements in later
slides and this includes study design inclusion decisions quality assessment
applicability assessment and task force judgment tables and those are coming up
I will start with study design inclusion and exclusion now this is this process
tab is standard in all systematic reviews and because we look at a very
broad range of topics and a very broad range of different types of intervention
the criteria for inclusion in one Community Guide review can be very
different from another however in the simplest terms most projects will
start with the discussion about observational studies are in until the
team and in some cases the Task Force argues them out this slide identifies
how we categorize study design suitability for the Community Guide in the left column the types of designs included in each category in the
middle column and typical design inclusion deliberations for different
types of intervention reviews that we typically engage in on the right looking
at study design suitability I’ll note that that category includes both
randomized trials and controlled before/after designs although it’s quite
likely and the next step which is quality assessment that studies based on
these designs will differ in their quality assessment
team deliberations are most commonly triggered over inclusion decisions about
before/after designs and health system intervention reviews where they are
frequently excluded and cross-sectional designs for policy interventions where
they are included about half the time however its routine for uncontrolled
before/after designs the part of community program type interventions and
policy reviews turning now to the assessment of study
quality our process differs somewhat from those of other review organizations
such as Cochrane, AHRQ, U.S. Preventive Services Task Force guidance
admittedly our tool while flexible is is old and the field I think has moved
convincingly in a more rigorous direction in the areas that I’ve
underlined in on this slide for example our quality assessment tool assigns
limitations for threats to internal and external validity although these are
primarily focused on the adequacy of the description our reporting and the field
in general as favors keeping those separate our tool has basic judgment
prompt prompts which teams use as a starting point for a topic in
interventions specific set of criteria and decision rules and that’s a common
process where you use two evaluators for each study that’s a common process we
total up limitations to assign an overall score and both of those aspects
are not used by other organizations we exclude studies based on total quality
limitations and that’s a process and other organizations that move to more
sensitivity analysis as opposed to exclusion we like most of the other
organization set up assessments looking down a body of evidence of included
studies for each of our quality categories in order to explore whether
the body of included studies had the same recurring quality issue here’s a slide by slide comparison of
the quality assessment categories in our evaluation form in the Cochrane risk of
bias table while Cochrane methods are rigorous clearly described we don’t
we’d make use of them if if we were looking at randomized controlled trials
routinely applying this tool to the types of study that task force routinely
considers would not provide much information to distinguish between
better and worse albeit low-quality studies
here’s a standard Community Guide display of a mixed body of evidence with
studies categorized by overall quality score and study design suitability this
is not an uncommon distribution for a community-based program type review for
the community guide in which the team retain consideration of observational
studies synthesis of results here would involve an outcome by outcome assessment
of study findings with a comparison internally of results from the higher
quality evidence such as the studies of greatest suitability of design here to
the lower qualitative evidence here’s an example of a display plot for a non
meta-analysis that is a typical form that is used in community guide reviews
and the example here is from our review of mass media campaigns to reduce
tobacco use among youth study or study arm effect estimates are plotted usually
arranged on some important variable this display enables several things an
assessment of consistency of intervention effect from the included
studies an assessment of the magnitude of the intervention effect usually
retained in absolute measures and a comparison of the evidence for the
studies of greatest suitability which are positioned at the top of this
display and least suitability at the bottom the preferred approach here would
have been have strata specific median and inch or quartile intervals as
opposed to the overall measure but this is a common display used in Community
Guide reviews for presentations in the team into the Task Force this is our current evidence assessment
table which the Task Force uses to organize their consideration of the
evidence and to draw a conclusion I’m not going to go over the details of the
table but I will outline the columns so there’s the final Task Force rating for
the strength of evidence either for or against use of the intervention in the
first column then a requirement for this is just outlining the types of
suitability of study designs moving from strongest to at least suitable than the
required quality of execution going from good to fair required number of studies
of that combination in order to use that row for a determination of the strength
of evidence overall assessment of the distribution of study results for
outcomes that are going to be the basis for the taskforce conclusion on
effectiveness whether it’s consistent or not and the overall assessment of the
population health impact based on findings from the studies due today it
is the impact meaningful and a public health or population health sense I will
point out that the use of this table is to start at the top and work down the
table this approach ensures that consideration of results from the
stronger subset of designs are considered first if you can answer
question of effectiveness base on studies a great a suitability design has
the potential to be a strong body of evidence if you have to dip down into
the studies of moderate and least suitable design which includes most of
the observational evidence maximizes sufficient finding on effectiveness if
those two subsets differ in the conclusion that’s a concern that needs
to be raised at the team or Task Force level about especially when the stronger
body of evidence has the mixed results and in the face of a favorable results
from the studies of moderately suitability of design given that the evidence or information
on external validity is an important issue for the target audiences of our
work Community Guide reviews capture evaluate and consider findings on
applicability of the evidence as part of setting up the review the coordination
team identifies important characteristics and makes an initial
judgment on the likely generalizability with respect to that factor call this an
a priori assessment of the importance of the factor and this helps us as I’ll
outline coming up in the next few slides the assessment of applicability boils
down to identifying factors for stratified assessment comparing the
subset findings studies pool because they had that factor to the overall
findings of the review they have to be cautious here as subsets are typically
small the studies are likely to have important differences from one another
strong conclusions from this process are infrequent but teams do try to answer
the following questions are these interventions effective across most or
all of the examined settings and populations so that task force can
suggest broad applicability are there gaps settings or populations that were
not examined at all or with only a couple of studies and if so should
differences in an impact such as inconsistent effects or gaps be
identified as task force findings beyond just a statement of fact and the call
for additional research here is where we find utility and revisiting our initial
judgments especially in the absence of evidence from the review
and this is a guidance table on options for the generation and placement of
findings on applicability based on how important we thought the factor wasn’t
setting up the review the evidence available on that factor quantified
whether or not the subset results are in line with the overall results we get
range of options for potential findings regarding add factor and placement of
statements in the Task Force recommendations summary this slide provides an example of the
summation of the process although it’s a fairly boring one the next slide
presents the final piece of the task force evidence assessment table a set of
options for adjusting the strength of evidence or the overall Task Force
conclusion findings on applicability evidence are concerns on potential harms
and recurring quality flaws across the body of evidence for example might
prompt the Task Force to downgrade a recommendation and this set of tables
provides guidance on the type of downgrade and the criteria for we’re gonna look at a related and more
detailed study in a minute but this slide summarizes the range of Task Force
systematic review findings for the 234 reviews that are still active on our
website across all active topic areas the Task Force was found just over a
third of the reviews to have strong body of evidence of effectiveness and from
that evidence decision table I went over a bit ago this indicates that the
evidence includes and the findings are driven by studies using great
comparative designs or an assessed as the greatest suitability of design
almost a quarter of findings are recommendations based on sufficient
evidence of effectiveness which encompasses reviews in which the
evidence of effectiveness is driven by results from observational studies it
also includes some reviews which may otherwise have had a strong body of
evidence but the Task Force had concerns about issues and prompted a downgrade
39% of reviews provided the Task Force with insufficient evidence to determine
whether or not the intervention is effective in our first decade the
primary driver of this finding was lack of evidence but in our second decade the
more common driver has been inconsistent results across the body of evidence to
date the Task Force has only issued two recommendations against use of an
intervention first was on the basis of evidence of harms and the second was a
demonstration of effect opposite from the primary outcome on
effectiveness despite our conceptual foundation and twenty years of experience in the
consideration of a relatively broad range of study designs and evidence we
frequently receive complaints that our methods are too restrictive and requests
that we consider her expand or focus on practice based evidence which can be
difficult to define but it’s often described as effectiveness evaluation of
a program service or policy as it’s actually implemented in the field
definitions may emphasize use of participatory approaches to engage
stakeholders evaluations of ongoing programs and policies using study
designs that place greater emphasis on external validity use of system science
methods to understand context our practice based networks to conduct the
study a few years ago staff at the Community Guide set out to document the
inclusion of what we could define broadly as practice based evidence in systematic
reviews produced for what was then 20 Community Guide topic areas the research
goals of the our study included developing operational definitions to
categorize studies and practice based our research-based evidence and then did
examine the inclusion of these types of intervention studies across reviews in
our topic areas in addition the team wanted to characterize and compare the
studies in each category by design intervention type setting study location
and quality a flowchart on the right indicates the team dissembled and
evaluated 3,656 studies from 202 intervention reviews across those 20
topic areas the operational definitions were really basic and
this flowchart identifies the final categorization use of randomization in
the study design was to find this identifying research-based evidence and
absence of randomization identified practice based evidence research-based
evidence was further broken down based on the unit of allocation whether it be
individual small groups of individuals or large groups such as work sites or
schools our conceptual generalizability spectrum is at the bottom with our two
primary categories at opposite ends and group allocated research-based evidence
in the spec along the spectrum in between the larger the univ a location
our perception was the closer that approaches practice-based evidence I’m gonna just point out a couple of
things in each of the next three slides starting here with the proportion of
evidence that was identified this practice base our research based for a
selected set of our 20 topic areas I’m gonna start at the bottom across all
20 topics those 3656 studies practice based
evidence and compass 54% of that evidence and research-based evidence the
remainder group based group allocated research based evidence was 17% reliance on practice based evidence was highest in the ten reviews
from alcohol prevention or management indicating high proportion of policy
type reviews in that that topic area and the smallest incorporation of practice
based evidence was for cancer the cancer reviews were heavily focused on health
system interventions to improve receipt of recommended U.S. Preventive Services
screening procedures next slide has findings on the
characteristics of the included studies I’m gonna just focus on four items to no
surprise the dominant source of evidence for policy intervention reviews is
practice based evidence so the somewhat less dominate was use of practice based evidence for evaluation of intervention set in community
settings and the last issue we’ll look at is moderate and least suitability of
study designs and community guide work identify observational evidence the time
series and the before after studies and this comprised across the first 3,000
studies in 202 reviews 37% of the evidence that was concerned so I’m gonna turn now to a one fairly
unique area of community guide process which is follow up systematic reviews on
economic evidence for intervention which the Task Force has found evidence of
effectiveness Community Guide methods for conducting systematic reviews of
economic evidence but take a full webinar and I’m gonna get to this in a
bit but we’re currently doing a major overhaul of the process right now I’m
gonna give you a quick walk through following the basic steps of the
systematic review outlined on the left we’ll start with prioritization task
force has always favored restricting economic reviews to projects in which
the task force finds evidence of effectiveness and provides a
recommendation regarding use and the setting economic evidence might be
useful to decision-makers to choose between effective intervention options
and to appreciate the potential start-up costs and timing and sources of benefits
due to staffing limitations economic projects are now prioritized we have an
economics team they conduct the reviews typically keep making use of both the
task force and the coordination and team from the effectiveness review
the economic review generally adopts the scope of the effectiveness review do you
limit the studies considered to those conducted in high-income countries which
is something effectiveness review does as well so that evidence is
relevant to U.S. decision-makers we generally take a societal perspective on
costs and benefits although some reviews may adopt a payer or health system
perspective the economic team conducts a
supplementary search for evidence adding economic terms to the
effectiveness review search strategy and expands the search to include economic
relevant databases the review also includes economic information
that’s identified by the effectiveness review team and this is occasionally
hidden within an effectiveness paper typically a data on the costs of the
intervention but it might not get flagged if you were trying to search for
it in a database search the economic abstraction includes
intervention components and component costs health system related expenditures
which might change in response to intervention impact and outcome data on
attributable costs and benefits our evaluation of the quality of
economic evidence is focused on identifying the important drivers of
intervention costs and intervention attributable benefits across the body of
evidence and then rating the completeness of the capture for each
study a good study for example will include the most important contributors
to cost of the intervention let’s say staffing costs and the most important
contributors to benefits let’s say change in ED visits or hospitalizations
or mortality in synthesizing the results of the review the economic team attempts
to standardize estimates although this can be a challenge estimates from the
included studies are categorized and summarized using medians
and interquartile intervals for intervention costs change in system
expenditures if that’s relevant and median cost-effectiveness estimate
cost-benefit studies are on relatively uncommon so these are often presented
narratively we’re unlikely to have enough for a median
Task Force reports the findings of the economic review for each of these
categories in a section of the rationale statement they have the option to add a
finding to follow their effectiveness recommendation and there’s criteria for
this although those are now being subject to modification currently it’s
two or more good quality estimates on cost-effectiveness using a very
conservative threshold for cost-effectiveness are two or more good
quality benefit to cost estimates I say the Task Force economics methods
committee is currently working on changes but both the quality assessment
to expand the quality assessment of economic evidence and to essentially
adopt a evidence assessment table similar to the one we have for
effectiveness review to guide conclusions that would represent
findings to accompany the effectiveness recommendation and they’re doing this
because we’re increasingly encountering more economic evidence at least in some
areas for example we did an economic review of team-based care for blood
pressure control and that economic review included 31 studies
I’m going to wrap up with a short list of persistent challenges Task Force members serve for 5 years
and we’re in a period of relatively significant turnover right now new
members bring new perspectives on the consideration of observational studies
and the meaningfulness of intervention attributable changes or magnitudes of
effect in the last year we’ve revisited and revised the evidence assessment
table that is used by the Task Force we’ve revised our process steps in the
consideration of harms and then where I’ve already mentioned this overhaul on
our methods for conducting economic reviews typical database search poll for
a broad search for evidence and information on an intervention has
increased substantially since I started with the Community Guide citation pulls
for a broad searcher now frequently 30,000 citations which can make search
screening a labor of love it’s now routine to encounter multiple published
systematic reviews related to but rarely right on the mark of our own research
questions we’ve added an initial search and screen for review papers and our
pilot testing a process for evaluating existing existing reviews for use by the
Task Force either directly or with a limited amount of work finally where the
evidence exists U.S. Preventive Services Task Force says now crafting
recommendations for interventions which might follow up a clinical interaction
and these primary care are primary care referral services include things like
exercise programs which might be found in community settings the the U.S.
Preventive Services Task Force recommendations are important as they
have coverage consideration implications it’s just an additional challenge for us
at the community preventive services to minimize duplication of effort and to
avoid the potential that we’d adopt different findings for similar
interventions with that I’m going to wrap up and open
it up for questions and discussion Thank you Dr. Hopkins for a terrific
webinar I’m gonna start with a question that’s related to the material that you
presented most recently is there a quality assessment or critical appraisal
tool that you would recommend for assessing the quality of systematic
reviews of economic evaluations of economic evaluations I do not know
the answer to that and that would be quite an achievement I think so there’s
room for the person who raised the question to do a little work and develop
something suppose turning to some more general questions that was one that
was specific to the economic analysis that you just spoke about but turning to
more general things you you often mention the U.S. Preventive Services Task
Force even though your presentation was focused on the Community Preventive
Services Task Force can you summarize just briefly what the major differences
in the methods are that those two groups use and doing their systematic reviews
we we understand that they address different kinds of interventions we know
that but could you briefly summarize the differences in methods that they use as
they approach their review yeah I’ll I’ll throw out a couple differences
their process for generating nominations for topics to be looked at
prioritization of work and actual conduct of the systematic reviews is
very different they contract out reviews to EPCs
they have a oversight from those EPCs frequently use consultants to help them set up their intervention reviews and provide subject
matter expertise they’re similar in that regard in regards to the specific
conduct of the review there they are much more restricted in consideration of
study designs to the more classic comparative designs our CTs
may be controlled before/afters for the effectiveness review for consideration
of harms it is quite common for their reviews to expand the consideration of
study designs to capture things like case reports or other formats by which
harms might be identified from the literature in regards to the quality
assessment they have a guidance tool that is study design specific which is
something we should have thought of years ago and in our upcoming revisits
are almost certainly going to adopt as well in addition they do provide an
overall quality grade assessment for the studies that are included in their
review and in the narrative portion of their descriptions of the evidence they
will often organize their descriptions of the evidence on effectiveness identifying
studies based on their quality score they separate a internal validity and
external ability so they have a quality grade for both of those two they’re
several other issues but I’ll leave it at that you know thank you those are
important distinctions I know the U.S. Preventive Services Task Force publishes
regularly papers describing the methods that they used does the Community
Preventive Services Task Force do the same thing no and we should so that’s
something that’s on perhaps on the list of things to start doing yes and my
guess is that the economics overhaul is going to contribute to to a paper that will initiate a set of methods papers
yeah how do the two Task Forces stay abreast of what each other is
working on and and coordinate at one point I know that was there was someone
who served on both panels and that was that was one obvious way but that’s not
always possible so how do you regularly stay in touch of what with what each
other is doing so there we’re liaisons to their meetings
we attend their meetings and at times when there’s the things that we’re
working on they’re very close together we have questions about what they’re
going to work on which is increasingly becoming an issue because of of their
aggressive move into primary care referable recommendations will have
telephone conference calls with them for us it’s been routine for us to to get
one of their retiring members and to join our Task Force so we’ve had
two coming to us and one going going to them and that helps keep the lines
of communication open and sitting on our Task Force we have somebody who’s
understands the differences between the two groups in terms of decision making
and in consideration of evidence our office is in a very enviable spot
because we attend the meetings of both groups so we we stay very much informed
with what both panels are doing and it’s important work for each panel addressing
different but but clearly related issues the Community Preventive Services Task
Force considers cost I don’t think the U.S. Preventive Services Task Forces does
and I believe that’s correct and I believe they’re they that’s specifically
not something they look at right we consider costs but it’s a separate
process from the determination of effectiveness and we
provide information that might be useful to decision-makers but there’s no are
our reviews in contrast to the U.S. Preventive Services Task Force where
coverage consideration is related to the recommendations our reviews and
recommendations are primarily advice to the
the field and we’re overjoyed when people make use of especially CDC and
NIH partners one of the interests of my office is in encouraging the institutes
and centers at NIH to support research to advance the work of the two Task
Forces so if we know that one of the Task Force’s has said that hey we are
lacking evidence on this or that we may try to stimulate conversations across
the campus or at least in some of the ICs to try to stimulate additional work
that could you know over a period of years eventually address the gaps take a
few minutes and tell me as well as the others listening what kinds of
things are missing as you are reviewing evidence for various topics is it
evidence of long-term effectiveness is it information on cost what what are
some of the kinds of things that are commonly missing that you really like to
have so given our interest in information from a study on that could
be useful for external ability or applicability reporting of population
characteristics even at the community level is extremely useful so knowing
that a study was done in a low-income community is useful for us in that isn’t
routine isn’t always reported although many the community-based programs will
will make that effort in regards to the practice-based evidence the similar
things man the study design might be weaker but that can be complemented by
additional details about the intervention implementation of the
intervention and targeted our target populations
which would provide the evidence that we can make use of I’d like to call for
longer-term studies I’d like to call for randomized control designs in the in the
topics that we look at but those those may not be feasible in some of the topic
areas but there’s no we’re not we certainly wouldn’t be opposed to
stronger bodies stronger designs used to evaluate types of interventions okay
well certainly our office is all in favor of using the best possible methods
to evaluate prevention research whether it’s community based or something that’s
done out of primary care so we would we would agree with you and we are
regularly encouraging our colleagues around the campus to support that kind
of work as well as longer term follow-up which we think is important one of the
points you mentioned in response to an earlier question was that the methods
that the two Task Forces use to select topics are different
can you remind us how does the Community Preventive Services Task Force
choose the topics that it decides it’s going to work on you showed us a list
from I think 2015 of topics that you’re currently working on so presumably in
about 2020 you’d prepare you have a new list yeah how do you decide what’s going
to be on it so the last time we take we take public input our nominations and I
believe we had website submissions although that was that was quite a
challenge the primary source is it’ll be typically start with Healthy People 2030
2030 topic areas and then the task force is
going to prioritize topic areas and so to Healthy People 2030 is likely
to outline most of the potential topic areas that the Task Force will
consider will provide information to accompany what what are possible avenues
of exploration for topic areas that they may not be
familiar with and and then our existing body of work if it’s a topic area that
we do already have an existing body of work on they can make decisions and in
whether or not that body work it’s adequate and we should use our limited
resources and go into a new topic area or if that new topic area might not be
ready for prime time yet and that the top the issues are new and it’s unlikely
that there’s a body of evidence from a community perspective to inform use of
the field does the task force or any other group that you’re aware of
monitor implementation of interventions that the task force has identified as
effective you know once the task force may see it
makes its recommendation is there any kind of a sort of post recommendation
monitoring program not a surveillance system I think at best we
take anecdotal evidence and our fairly aggressive in looking for what we call
community action examples where people have used the community guide at the
local level to decide which areas to invest their own limited resources
and our partnering efforts that’s about as activists a search
for uses of the Community Guide that I’m aware do you have some examples of how
the Task Force’s findings have likely influenced a greater or lesser use of a
particular prevention program in communities it is I sit in the Oregon
Health Department and much of the chronic disease portion of the Oregon
Health Department operates under CDC program support and for some of the
topics like physical activity and tobacco use but mostly physical activity
and asthma the community guide reviews are often a part of the set of you
should examine these and consider these for your program activities based
on a CDC support how often does the has the Task Force had to go back to a
recommendation that it’s issued and downgraded from something that you
recommend to something that you decide well the evidence really isn’t
sufficient or maybe you need to change the position and say no longer recommend
does that does that happen often it does not happen often older reviews get
archived so they go obsolete either because the intervention isn’t as
active an active area of interest or more commonly because we lack the
resources to to update some of our original work I think we’ve revisited
one intervention review but I think it was going the other way from an
insufficient evidence to a conclusion on effectiveness I am not aware of
intervention reviews that we have revisited and downgraded well that that
would be a good indication that the judgment that the task force’s racine
reaching is sound and based on sufficient evidence that it doesn’t get
turned around later that does happen that does happen occasionally with the
U.S. Preventive Services Task Force where new studies come out and they decide
that something is maybe the new studies disagree with the existing studies and
they’ll so they’ll decide well there’s no longer sufficient evidence so it does
happen there I know that the Community Task Force issues insufficient evidence
statements are you often able to turn those around as you have additional
information coming out yeah although so like in our first decade most of the
insufficient evidence assessments of the Task Force were based on small numbers
so like wanting to study reviews and several of those and initial updates
were converted because additional evidence had accumulated more recently
insufficient evidences is a more common result of a completely mixed body of
evidence which may be saying something in terms of the effectiveness of the
intervention or approach when the body of evidence does not get clarified over
over time we have a question from someone who works in a local health care
yet health care district as a community advocate wondering how they can support
local intervention projects what are some ways that they can train themselves
to be current in the sorts of activities that the Task Force is recommending excellent question my guess is there’s
no easy answer it’s in the state of Oregon it’s it’s definitely
opportunities are for partnering with State Health Department program which
may have funding for specific topic areas and are frequently looking for for
partners to implement programs that they want to promote as a methodologist I was
paying particular attention to the kinds of study designs that you were talking
about on some of your earlier slides like nine ten eleven and that in that
area the kinds of designs that you consider and which ones you consider to
be more important I didn’t see stepped wedge on the list anywhere is I saw you
did you have a webinar on that not long ago so yeah I not aware of that I’ve not
encountered that yet we see certainly applications of that
methodology in hospital settings which may not be as relevant for you but there
we also see them in community settings which would be relevant for you so it’s
irrelevant to of course it’s a type of group randomized trial but a different
sort and one that I think probably has a lot of appeal in communities who don’t
want to have a control on can you say more about the trade offs of the
explosion of the literature leading to using existing reviews and and how
the weaknesses in the detailed information interviews affects the
Community Guide process another excellent question so certainly the
presence of existing systematic reviews is has to be by us considered seriously
as a potential alternative to doing them ourselves if the reviews high-quality
because just at the length of time it takes to produce a review in that
situation and us our current piloting efforts are exploring what the
trade-offs are that is the Task Force willing to make in terms of absence of
information and it’s been pretty rocky with our initial reviews
and that they are initially open to considering the review but as you look
under the hood you become concerned by the kinds of stuff that you’re used to
seeing but as but isn’t there and it’s not available in the in a 4,000 word
summary of a systematic review which is and may not be available in the summary
evidence tables for the studies included in the review typically for our pilot
programs it’s very common for us in considering a systematic review to to
determine that the minimum effort is to pull all the intervention reviews, studies included in the review and do additional abstraction
which detracts from the potential utility although it’s still probably a
significant time savings so that issue of the scope of the reviews that we have
that we find not being exactly what the scope or the research questions for our
review that’s a big problem for us so that routinely the reviews that we encounter
have just one slightly different scope issue so we’re looking at use of health
for physical activity in older adults and it’s an its narrowed to a specific
population the interventions of interest to us the outcomes of interest to us but
if we were doing our own review be open to a broader range of a study population
may be interested in adults or may be so the full range of adults so those kinds
of decisions that have to be made on whether or not to make use of an
existing system and a review or routine in addition to the fact that once you
get under the hood you invariably want additional details from the included
studies even in an up-to-date systematic review Thank you very much Dr.
Hopkins I’m I’m afraid we’ve exhausted the hour that we had available to us but
I want to thank you very much for your presentation today and I’ll turn this
back to Marie Rienzo thank you Dr. Murray and thank you to everyone who
participated in today’s webinar on the Mind the Gap website prevention.nih.gov/mindthegap you’ll find several resources for this talk including the
slides a list of references we’ll also be posting a recording of today’s
webinar on our website next week you will receive an email with a link to the
recording a when it’s available thank you

Leave a Reply

Your email address will not be published. Required fields are marked *