Combining quantitative and qualitative evidence: why, how and when?


Joann Starks: Good afternoon, everyone. I’m
Joann Starks of SEDL in Austin, Texas and I will be moderating today’s webinar entitled
“Combining Quantitative and Qualitative Evidence in Systematic Reviews: Why, How,
and When.” It is the final presentation in a series of four webinars that make up
an online workshop on qualitative research synthesis. I want to thank my colleague Ann
Williams for her logistical and technical support for today’s session. The webinar
is offered through the Center on Knowledge Translation for Disability and Rehabilitation
Research (KTDRR), which is funded by the National Institute on Disability and Rehabilitation
Research. The Center on KTDRR is sponsoring a Community of Practice on Evidence for Disability
and Rehabilitation, or D&R, Research. Evidence in the field of disability and rehabilitation
often includes studies that follow a variety of qualitative research paradigms. Such evidence
is difficult to summarize using traditional, systematic research review procedures. The
goal of this series of web-based workshops is to introduce D&R researchers to the methodology
of qualitative evidence reviews. Participants will be provided a state-of-the-art overview
on current approaches and will learn to apply those to the literature base. Ongoing innovative
initiatives at review-producing institutions will be highlighted. Today, our speaker is
Dr. James Thomas, Associate Director of the EPPI-Centre and Professor of Social Research
and Policy, University College London. His research interests include systematic reviewing,
methods for research synthesis and research more broadly, the use of new information technologies
in social research, evidence-informed policy and practice, and health promotion in public
health. He specializes in developing methods for research synthesis, in particular, for
qualitative and mixed methods reviews and in using emerging information technology such
as text mining and research. He leads the module on synthesis and critical appraisal
on the EPPI-Centre’s Master’s Program in Research for Public Policy and Practice
and development on the Centre’s in-house reviewing software, EPPI-Reviewer. Welcome,
James, and thank you for agreeing to conduct this session today on combining quantitative
and qualitative evidence. If you’re ready, please go ahead. James Thomas: Thank you Joann,
and thank you for asking me to join you in this webinar. Okay, I’ll move on to my first
slide. I’m going to talk about combining qualitative and quantitative evidence, so
it’s rounding off this series on qualitative synthesis and synthesizing qualitative research.
I’m thinking about how we can use qualitative research with other types of research and
what useful function those kinds of reviews can play. So I’m going to talk through some
of the why’s, why we might to do this, why should we integrate these different types
of literature in systematic review? And, I’m going to talk about the policy context, which
is what I know, and that will help us locate what we’re trying to do with these reviews
in a sort of a practical situation. I’m then going to talk a little bit about the
mixed method literature on primary research and how that relates to systematic reviews,
because there’s a temptation in the systematic review community to think that we’re still
inventing methods from scratch. And actually there’s a lot going on in primary research
that we can learn from. Then I’m going to talk through one example in particular of
one type of mixed methods synthesis and then refer you to some other sorts of research
synthesis using mixed methods, which you might want to pick up in later reading.
So, first of all, thinking about what the problem is, say, what’s the problem
that doing a mixed method synthesis might solve? The context that I’m thinking about,
the context I work in, [is in policy]. One of the things that is important to bear in
mind when engaged in this type of work is that research is only one small factor in
the range of influences that go into policy development. So we’ve got all of these different
factors here. We’ve got professional experience and expertise, we’ve got political judgments,
we’ve got resources, values, habits, tradition, pragmatics, and lobbyists and pressure groups,
and research evidence is sometimes this little outside voice here trying to get a voice at
the table. So what we’ve been trying to do is to promote the use of research and get
it heard in this quite noisy environment of policy making. Iain Chalmers, who’s one
of the pioneers in this area, has written a number of papers on this. This one is a
particularly interesting one because it sets the scene very well and he reminds us that
policymakers and practitioners who intervene in the lives of other people, even for what
intents or purposes, not infrequently do more harm than good. One of the examples that he
talks about in that paper is Dr. Benjamin Spock’s book, Baby and Child Care, which
my parents had when I was a baby. So this is one of the single largest bestselling books
of 40 or so years ago, 50 years ago. It sold over 50 million copies and it has well-intentioned
advice in it, that it’s preferable to accustom a baby to sleep on its stomach from the start
if he is willing. This was fairly reasoned advice from Dr. Spock but it wasn’t actually
based on any evidence. This was simply advice based on what seems to be a good idea. However,
we now know that there’s positively harmful evidence that actually, the risk of cot death
is reduced if babies are not put on their tummy to sleep. So there was then a campaign
in the 80’s and into the 90’s, which has dramatically reduced the incidence of sudden
infant death. This is really an example of a well-intentioned intervention, if you like,
in that book which actually did more harm than good, and once you’ve got some policy
and some intervention based on evidence, that actually we get a little bit more good than
harm. So that’s the message that Iain Chalmers is portraying in that paper, and it is well
worth reading. I’ll add to that, is this move in medicine towards what is called evidence-based
medicine, which sometimes sounds more certain and more based, if you like, on evidence than
it actually is. Dave Sackett and colleagues have written this very widely cited paper,
which aims to say what evidence-based medicine is and what it isn’t. They’ve described
it very succinctly as the conscientious explicit and judicious use of current best evidence
in making decisions. So it’s not necessarily expecting that you can read off the answer
to a given problem from the research but it’s about making use of that evidence, and the
current best evidence, in order to make decisions. The question then is of course, well, how
do we do that? This slide here illustrates what happens in an Internet Minute. The growth
of information on the Internet let alone anywhere else is staggering and we’re all suffering
in a way from this deluge of data. How can we make sense of so much data? There’s just
so much of it. So this is where systematic reviews themselves come in. And in the words
of Ben Goldacre in his Bad Pharma book, he describes systematic reviews as being distinct
from simply looking at the research literature and sort of traditional literature reviews.
So instead of just mooching through the research literature, but basically picking out papers
here and there to support our preexisting beliefs, or simply the ones that happen to
be sitting there in our filing cabinet, we take a scientific and a systematic approach
for the very process of looking for scientific evidence and ensuring that our evidence is
as complete and representative as possible of all the research that has ever been done.
So that’s systematic reviews. Speaking mainly from a fairly clinical standpoint, now when
we have been looking at translating that kind of a model of identifying and synthesizing
the evidence-base for use in public health policy, we found it very challenging. And
there are three main reasons for this. Things are more complex in public health. We find
that the context is complex, the questions the policymakers ask us are complex, and the
data themselves are complex. I’ll talk a little bit about each of those now. So to
begin with when we’re thinking about the context of intervention as a context of intervening
in public health, there are two aspects of this. There’s what I call complicated complex
and there’s complex complex. So the Medical Research Council in the UK has defined what
a complex intervention looks like and it’s basically something with several interacting
components and many of the problems of analysis, et cetera, relate to being able to standardize
but also to identify which are the most important components of a multi-component intervention.
That’s one idea of what complex is. So it’s a multi-component intervention and it’s
difficult to standardize. However, some would say that that is just a complicated issue
really, that’s not genuinely complex. So there’s a whole literature on complexity,
which is much more around conceptualizing interventions as dynamic processes, which
operate in difficult to predict ways. For example, they may have virtuous circles and
feedback loops within the [cycle] of change. They also may have non-linear step changes.
So you might think that an intervention with one dose might not do very much, a little
bit more might not do very much. The response to a particular amount of intervention might
be expected to increase as the amount of intervention increases. But actually, some of the ideas
around complexity are that you get non-linear step changes. So what happens is you keep
increasing the quantity or the intensity of intervention and you don’t see any particular
results until suddenly you reach a particular threshold, and then suddenly other things
kick in and things start to happen. This is all very well sort of to think about in abstract
but actually then thinking about how you might model that statistically is really quite a
challenge. Finally, there are what you might think of as multiple routes to effectiveness.
So for example, you might be looking at whether or not you need to train, for example, the
people who are delivering the intervention. Some studies may well report results that
are better if you’ve trained the people who are delivering the intervention, and that’s
fine, and you can detect that statistically, so long as the ones which don’t train their
intervention providers do worse. Sometimes you get the situation where actually what’s
going on in the intervention is more complicated than that, where you may train the intervention
providers in some situations and it makes a difference, and in others you may not train
them. Maybe you’ve just recruited more authoritative, more experienced providers, and so they don’t
need training but unless you’ve got that level of detail and that level of understanding
about what’s going on, you can’t actually detect that in the synthesis, in the systematic
review. So there are many levels in which looking at what’s going on in the environment,
on the ground, is more complex in public health. In addition, we find that policymakers just
ask us pretty complex questions that don’t just ask “What’s the impact of this intervention?”
Usually what they ask are questions, which start with a given problem. They start with
“What can we do about…?” “What we can do about [IFDE]?” for example. They
don’t just look at “What can we do about…?,” they’re interested in, “Okay, so we’ve
got one particular approach to intervention, but who might this be most acceptable for,
who might it be appropriate for – we’re interested, we might have a particular service,
how can we get people to sign up for this service?” So they ask some complicated questions
around to what extent and in what ways does the person who delivers this intervention
affect the outcomes, for example; who does it work for and why; and what works to achieve
this outcome, for whom and what circumstances? So we get complex and we get compound questions,
which don’t easily map on to particular research literatures. As I’ve said, our
questions, they start from a given problem, usually a known outcome or population and
they identify a range of answers. They have multiple components and what is quite common
in a lot of the reviews that we do is that the real question is what causes variation
in outcome? It’s less around “what are we going to get if we do this,” because
I think people’s understanding has advanced to a point now but we don’t expect to get
a single response, if you like, to a particular intervention. We get a range of outcomes and
a range of responses from different people and what policymakers are interested in, is
understanding what those courses of variation are. So we do systematic reviews which don’t
aim to come to a single answer. What we’re really looking to be able to do is explain
and understand what is going on in between different interventions, which might start
to help people to understand for whom and why a particular intervention has the effect
that it does or doesn’t have. Finally, one of the challenging areas about doing research
and policy is that policymakers seek to blend – as I’ve put on this final bullet point
here – they blend the micro, the macro perspective. So we find that we’re interested in impacting
outcomes at a sort of population level or at a sort of local population level. In order
to understand and pick and to identify some of the sort of explanatory issues that I’ve
been talking about, we’re needing to blend this macro perspective of influences, so a
fairly wide population level, with the micro perspective of actually what’s going on
in individual people’s lives, which means that they do or they don’t respond to a
particular intervention strategy. So I’ve got a little bit more on this and I’ve taken
this from Julia Brannen’s paper, which you can download from the NCRM website. It’s
well worth a read if you’re interested in an introduction to mixed methods research.
So we’ve got these two perspectives, and obviously there’s a continuum between the
two. There’s a micro perspective where what we’re wanting to do is emphasize the agency
of those who we’re studying, we are wanting to understand their experiences, their interpretations,
the meanings that they draw from particular phenomena. Then the macro perspectives, you
might sort of think of large scale surveys, cohort studies, maybe large trials which are
concerned with looking at patterns across the population and trends and help in terms
of sort of service planning overall and thinking about more structural explanations. And often
what we find is that actually we need to do a little bit of both in a lot of the reviews
that we do. So we’ve covered complexity of context and the complexity of the types
of questions that are asked. Finally, I’ve got a quick slide on the complexities that
are caused by the data. This really is summarized up in one slide which is there are just no
replications, or at least there are almost no replications, of intervention studies in
many areas of social research. What I mean by that is that whenever someone tries or
conducts a trial of a particular intervention strategy, it’s very rare that someone else
will then come along and then try and do that exact same intervention in a different or
a similar context. Usually people will vary the intervention in some way. They’ll take
that as their starting point and they’ll add things or they’ll remove things, or
they’ll just try a completely different strategy altogether. There are very, very
few replication studies. There are some, there’s no doubt there are a few, but on the whole,
the culture is in this area that you don’t just replicate what somebody else has done.
Sometimes it’s actually quite difficult to get funding to do that. People think, “Well,
someone has already tried that. What’s the value in trying it again?” The problem is,
in terms of research synthesis, is that a lot of the methods that we have for synthesis
in terms of statistical meta-analysis really rely on there being replications. The expectation
and the premise behind those methods is that we’ve got some homogeneity there, we’ve
got similarity of people or interventions and outcomes so that we can aggregate, we
can put things together, we can put similar with similar and like with like, and get a
greater confidence in what actually the effect is of that intervention. The problem in our
area is that we don’t really know what that intervention is because what we end up with
in the review are multiple examples of similar types of interventions, all of which differ
in smaller or larger ways from one another. To summarize, if you want to go in terms of
synthesis, we wouldn’t start necessarily from the methods that we’ve got. What we
find is that standard systematic review methods can’t cope with some of the complexities
that I’ve identified here. So we need to use appropriate methods that are able to cope
better with complexity, that actually help us to tackle some of those issues around explanation
and operate what I called “small n scenarios.” What I mean by that is that if we’ve got
say, 10 studies which are looking at a particular strategy, we might have 10 studies there but
maybe only two or three of them are actually comparable enough to think of as being the
same intervention. So actually what we’ve got are multiple examples, small numbers of
comparable interventions. How do we think about how we understand the differences and
the similarities in those situations? So it won’t be a surprise for you to hear that
mixed methods reviews are a potential solution to some of these problems. I’m going to
talk a little bit about the mixed methods literature and a little bit about mixed methods
heritage. Mixed methods reviews address complex, compound questions that I’ve talked about.
What we do with them is we use different types of evidence in a sort of dialectical fashion
to grapple with complexity. I’m going to come back to that and it will be one of the
examples that I’ll give so I won’t dwell on it now. What we can also do with these
types of reviews is mitigate some of the impact of the lack of replication that I was just
talking about, helping us to blend this micro and the macro perspectives to helping us to
think in terms of the large scale of large data sets but also think about how individual
people’s experiences and understandings help us to understand what is going on and
understand what’s going on in the bigger picture. Some of this thinking’s already
taking place in the primary mixed methods literature. So there’s lots of writing on
this slide. I’ve given you some references but I do recommend that paper by Julia Brannen,
which I’ve highlighted a few slides ago as being a good introduction to it. So mixed
methods is defined as a type of research where the researcher or the investigator mixes or
combines qualitative and quantitative research techniques, methods, approaches, or data,
concepts, language into one study. So what we’re thinking about is looking across the
whole of the different types of research evidence that are available, which we can identify
which is about a particular issue and how we can pull them together to get a sort of
stronger picture of really what is going on. People have talked about mixed methods research
as being expansive and creative, being – certainly one of the principles is being inclusive and
pluralistic. There’s quite a freedom when you’re conducting or at least designing
methods that you’re going to use in the mixed method study, to be quite creative.
There’s obviously statistical rules which you’ll have come across in the past, but
at the same time, when you’re designing a mixed method study, it’s a creative process
where you think about what types of knowledge, what types of evidence might be available,
and how you might best be able to put those different pieces of evidence together still
in a robust and a systematic way. We don’t compromise on those sorts of principles. So
again, the Julia Brannen paper picks up on why mixed methods fits quite well with evidence-informed
policy and practice. It fits quite well in terms of the political currency as Martyn
Hammersley’s called it, in practical enquiry. It’s really sort of thinking about solutions
to practical problems. While the sort of more scientific, maybe disciplinary-based research
might require closer attention to a justification for methods used on the types of data generated,
which is one of the things I’m going to come back to as we think about paradigms,
but that kind of demand is relaxed in an enquiry which is aimed at solving a practical problem
because actually what we’re wanting to do here is identify the best evidence that we
can get to inform a particular decision. That’s also in line with that slide I showed nearer
the beginning on evidence-based medicine, which is about the judicious use of current
best evidence. It’s not really specifying actually what that evidence should be, it’s
simply thinking about well, what is the best evidence that we can use? There’s obviously
some downsides to this and one of them identified here is that researchers have less leeway
to define their own research questions and follow their own ideas. There are pros and
cons to being engaged in this type of activity. There are a number of different reasons why
we might combine different sorts of data and different results. I’ve talked a little
bit about the need to explain, the need to elaborate on a generic finding to understand
what situation or what context that might apply to. That’s one of the reasons that
we use mixed methods literature. Also an important concept is initiation where what we’re doing
here is thinking about how we can take data from one context or one type of literature
to drive an analysis, what I’ve called here an empirically-driven analysis. Again, what
happens thereafter is explaining findings. So it’s initiating a new analysis based
on evidence and based on a perspective which you might have gleaned from a particular literature.
I’ve mentioned this issue about making use of all the evidence, which is called complementarity.
One of the really key issues that we find that mixed methods helps us to deal with is
this issue on contextualization. What we’re doing an awful lot of the time in evidence
synthesis is thinking about, okay, so we’ve got evidence from this context offered from
the US, for example, so lots of research that’s carried out in the US; how does that actually
apply in other countries, other health systems? How can we take evidence that’s been generated
from one situation and use it and apply it to a decision in another? So contextualisation
and re-contextualisation is an important use of mixed methods. The final bullet point here
I think is a really critical one for mixed method synthesis as opposed to mixed methods
primary research because it enables us to see the problem from different directions,
different perspectives. I’m going to talk about that, right now in fact. So, mixed methods
reviews have quite a lot in common with mixed methods in primary research but the really
key difference between the two, I think, is the social nature of research activity. This
slide is from Kuhn, back in the 1960’s, who first wrote The Structure of Scientific
Revolutions. What he did was he highlighted the way in which researchers tend to coalesce
around the particular areas of shared understanding for their investigation. What we’ve got
in the background here is sort of a little network graph which shows people clustered
around a similar research topic. They don’t just cluster around the similar research topic,
what they do is they cluster around the research topic, that they have a shared understanding
about the ways in which it’s legitimate to conduct research around that research topic.
So when you get clustering like that, shared understanding of what’s important, how we
should investigate it, and how we should justify our claims, it describes this as being research
paradigms. On the next slides, we’ll talk about that a little bit more. A paradigm is
a social construct which is around thinking about research activity that’s being conducted
and by communities of researchers who have a shared understanding around what they’re
interested in studying, how the concepts they’ve got within what they’re studying are related
to one another, and how they should be done, the methods and the tools and knowledge claims
that can be made on the basis of the research that they’re conducting. Even within the
discipline, what we find is that different methodological approaches then, in terms of
qualitative, quantitative, the paradigm split here, can reflect quite different fundamental
understandings about what’s important and also about ethics around whose voice should
be heard. What we find when we’re doing mixed methods research synthesis as opposed
to primary research where the mixed methods all in the same study is when you’re doing
a synthesis, what you identify are these clusters, these communities of researchers who have
coalesced around particular topics. But because we’re looking at research activity as a
whole, what we do is we can identify different communities of researchers who are talking
and investigating the same issue, but because they’ve got different perspectives and different
priorities in the way in which they investigate whatever it is, we actually have quite different
perspectives and different understandings around what’s important, how something should
be investigated, et cetera. They’re still looking at the same issue and this is what
I’m going to come back to in the example that you’ll see. They’re looking at the
same issue from different directions and so this can really give us some quite important
insights and by building up different perspectives and using the different perspectives in opposition
to one another almost, to understand what’s going on in different situations, we can start
to understand and explain why particular research might be finding the findings that it does.
This is much more concrete when I get into the example. So now I’m going to get on
to the practical part of this webinar, where we’re going to think about how mixed methods
reviews are actually conducted. So Pierre Pluye and Quan Hong produced this really nice
paper, which they called “Combining the Power of Stories and the Power of Numbers,”
and I recommend it for reading because it’s a nice and clear summary of some of the ways
in which you might want to think about what we’re doing in a mixed methods review. They
say there are three overall ways – and many people have said this so they wouldn’t claim
that they were the only ones – but there are three overarching ways in which we combine
different types of research in a systematic review. We think about the sequence in which
we’re combining qualitative and quantitative, that is qualitative and quantitative data.
Whether we do it sequentially for a start or whether we do it at the same time; there
are sequential designs and there are convergent designs. Within those two overarching ways
of thinking about combining different types of research, there are then different characteristics
of the different types of synthesis that you can do there. One of the main differences
when in a sequential design is whether what we are wanting to do is seek explanation,
we want them to explain some of the findings that we’re seeing in the research, or we’re
exploring so we’re opening up the issues. You might be developing like a taxonomy or
something like that where, rather than saying this is the reason this is happening, we’re
wanting to identify a list of understandings or a list of phenomena. Then the convergent
designs are those where the qualitative, quantitative data are transformed before the synthesis
takes place. So what I’m going to do now is, I’m going to take you through a worked
example in a little bit of detail, which is in the sequential explanatory area. Then I’m
going to refer you to some other reading for the other types. One of the things that I
wanted to talk about in this slide, and I’m not going to spend a long time on it because
I know that Karin has already covered this, is configuration and aggregation. I think
this is important here because it really starts to get at what we’re doing in the area of
synthesis, what we’re doing in the analysis rather than saying we’re doing qualitative
or quantitative analyses which actually the – especially the word qualitative just covers
such a vast range of types and approaches and stances to doing research that it really
– if someone says they’ve done a qualitative analysis, it actually is really quite difficult
to know what they’ve done. So what we’ve been thinking in terms of – and there are
other people here, we’ve listed some papers for you – is think about actually what are
we trying to achieve in the analysis, what’s the difference, what’s actually going on
in the synthesis? So, here, we’ve got this heuristic of aggregation and configuration.
Aggregation refers to adding up, aggregating, to putting findings from similar research
studies together towards a review question. So if you’ve come across Cochrane systematic
reviews, a meta-analysis is a good example of an aggregation. What we’re wanting to
do here is to be able to get the direction and the size of the effect, the size of the
effect size and as much as possible, the direction and then the degree of confidence that we’ve
got in that finding. The more studies we have to put in our analysis, we have to aggregate,
or the more stones we’ve got in our cairn here, the more confident we are that that
actually is the place that it is. So in a meta-analysis, what you would see is the more
studies – so long as we have not been violating any of the assumptions underpinning our meta-analysis
– what you’ll end up with smaller and smaller confidence intervals around an effect. In
contrast to this, configuration is around arranging research findings and it’s around,
saying okay, we’ve got this study, which tells us about this piece of the picture.
We’ve got this study here, which tells us about these people for example. We’ve got
these studies here, which tell us about this approach to intervention. What we can do with
the configurative type of approach is we can build up a picture of what the research is
telling us potentially as across quite a wide area of research. You don’t simply get aggregation
or configuration in a review. Often you get both and a lot of the mixed methods reviews
that we do, we explicitly have both aggregation and configuration. So Karin has talked about
this, I’m not going to talk about this in any more detail than this, other than to say
that one of the key aspects of doing a configurative review where we’re wanting to understand
and build up a picture of research activity and what research is telling us across a wide
area, is that sometimes you’ve got aggregation and configuration going on at the same time.
So we might do separate analyses or if you visualize separate piles of paper for interventions
which have got the same approach, in the same context or the same people. We might have
several different piles of paper. So we would aggregate within each pile of paper and then
we would configure between them. So that would start to help us to understand variation between
the different study findings. Now I’m going to talk you through the first type of mixed
method synthesis, which is called the sequential explanatory design. So sequenced, we’ve
got one, a qualitative and a quantitative separate synthesis, followed by what we call
a mixed method synthesis where they come together. The paper which outlines some of these methods
is there at the bottom of the slide. So here’s the overall design of the study. This systematic
review was around the barriers to and facilitators of fruit and vegetable intake amongst children
aged four to ten, and that was by [people] who were busy at the time developing policy
for interventions, partly interventions in schools. They were interested in the barriers
and facilitators of fruits and veg they have to eat. So we did what we call the mapping
exercise where we’ve searched quite broadly and we identified a lot of different studies.
We then took that back to the policy team who said, “Actually this is the area we’re
most interested in,” and then we’ve then identified two broad areas of research that
we were going to look at. We’ve got what’s called the “views” studies, we had eight
of them. The views studies are what you might think of as qualitative studies. They were
focus groups and interviews with children in which we’re talking to them about their
healthy eating and in this case, their overall health with something about fruits and vegetable
in them. On the left hand side there, we’ve got the trials, which are typical evaluation
studies, but there was an intervention that was evaluating a controlled trial, we had
33 of them. We did a meta-analysis with them and then we brought all of these studies together,
there at the bottom of the screen, the trials and the views mixed methods synthesis. That’s
the structure of the study. We have a qualitative synthesis, we have a meta-analysis, and then
we have a mixed methods synthesis. I’ll take you through the various stages of this
now. First of all, I’m highlighting what the meta-analysis looked like. This is one
particular analysis, it’s a forest plot, for those of you familiar with forest plots.
Each of the rows in the graph is one study. This is for fruits and vegetable intake. Anything
to the right of this zero line, the little red dot to the right is a positive result,
and then the little red dot to the left is a negative result. There is one trial that
managed to reduce fruits and vegetable intake with the children that they were intervening
with. The little horizontal lines that go through the red line are the confidence intervals.
So you can see that broadly speaking, most of the studies had a positive effect but there’s
quite a lot of difference, there’s quite a lot of heterogeneity there. They were all
similar studies. Most of them were involving schools and intervening in schools but beyond
saying, overall they seem to be able to promote the consumption of another half a portion
or quarter of a portion a day, there’s not really very much that we can say about this
because they were all quite different. So that was the quantitative, single quantitative
synthesis on its own. It tells us that things work overall but we don’t really understand
why because they’re all quite different to one another. So now we move to the qualitative
synthesis. I’ve got the pattern of the mosaic behind here because we’re doing a configurative
synthesis here. If you come across other methods, obviously you’ve been through two other
presentations on qualitative synthesis, so we won’t talk about this at great length.
The methods I presented in this paper here on the left side, here are source data with
the text of the documents, the material like the concepts, the understandings, the reports
by the children. The real key methods about doing this type of synthesis, which I’m
sure you’ve already heard, is around translation. It’s around identifying concepts which translate
well or translate between different studies to identify those studies where there’s
agreement and sometimes, that’s quite rare, you get disagreement. Usually what happens
when you’ve got a disagreement then you’ve got an opportunity to start to understand
and explore why there was disagreement between studies. So there were three stages really
in the synthesis though they were iterative so it looks very clean and neat. I have one,
two, and three here but sometimes there’s a little bit of interaction between stages
one and two and two and three. First of all, we treat essentially the documents as data.
We treat those primary research study reports as data. We go through them applying methods
of qualitative data analysis that you might well apply to interview transcripts for example.
So we code the text, we develop descriptive themes, and we identify findings. Then once
we’ve been through and identified these findings and we’ve identified these themes,
we analyze the themes, we group the themes so that we end up with really quite a sort
of a detailed understanding of how the different concepts in the studies relate to one another.
Then at that point then, we have quite a nice synthesis but it didn’t directly help us
to get the two groups of studies to speak to one another. So to do the mixed methods
synthesis, we then undertook stage three, which was around generating what we called
“analytical themes” in light of the review question. To begin with, we went through all
of the different studies, we coded the themes described in our data extraction. One of them
was bad food is nice, good food is awful. We developed descriptive codes, we have 36
descriptive codes. Then we group them as I’ve said and we developed the synthesis on this
summary of 13 descriptive themes which helps us to understand really what was going on
in the relationships between the concepts in those studies. This is a screenshot of
what it looked like at the time. So we had the data which was the primary studies and
we had the codes that we were generating as we went through and we coded the studies line
by line as I said. We might use NVivo or any other package to do this; this is from EPPI-Reviewer.
The other thing that we did then was we represented some of the concepts pictorially and diagrams
help us to understand the relationships between them. Once we’ve done that, we had quite
a nice synthesis but it didn’t really completely address the review question because it was
a synthesis of those studies in their own terms. What we’ve then needed to do is to
do some analytic work to infer what the recommendations, what the implications for interventions are
from those studies and in order to generate the data that we needed to undertake the mixed
methods synthesis. So it’s easier to see that in reality than in abstract here. So
we went back to our review questions, what were the children’s perceptions of and attitudes
to healthy eating? What do they think stops them eating healthily? What do they think
helps them eat healthily? What could be done to promote their healthy eating? Then what
we ended up with was a list of recommendations for interventions which we then re-analyzed
or identified really what the themes which underpinned them were. So, some of the key
themes were around children’s perceptions and understandings of health and who is responsible
for health. They were very clear that they didn’t see it as their role to be interested
in their health and that that was an adult responsibility. Also that when people would
say to them, “Eat this, it’s good for you,” that that wasn’t necessarily very
relevant or credible to their lives. The future didn’t really exist at a real way. So saying
“This is not going to be good for you when you’re 40” just had no salience for a
child aged between four and 10. The implications for interventions then, thinking about what
those themes were telling us was really around, okay, so health is clearly a problematic subject
but children sort of value taste far more than health in terms of their decision making.
So let’s talk about food being tasty rather than being healthy. As I showed you in that
earlier quotation, kind of the opposite was starting to happen in that whenever someone
says to children, “Well, eat this, it’s good for you” the children were automatically
starting to think, “Well, I’m automatically not going to like it. It’s awful.” So
that was a [possibly less] harmful way of thinking about why a child should eat a particular
food. They should eat food because they think they’ll enjoy eating them and they’re
tasty rather than they’re healthy. So equally the implication for interventions is that
health emphasis, the emphasis on health messages should be reduced and that fruits and vegetable
shouldn’t be promoted basically in the same way as the same thing in the same intervention.
The children were really quite clear that fruits and vegetables had quite different
meanings in ways for the children and they were quite keen on some sorts of fruit but
just liked some sorts of vegetables. They saw them as being quite different, and so
lumping them altogether in the same intervention, again, didn’t seem to be in line with the
way that the children saw the world. So we have these findings that were coming out of
the qualitative synthesis which had quite clear messages for interventions – not that
that was the point at this part of the analysis. We wanted to identify some key features of
interventions which according to the perspectives and the world view of the children would be
more likely to encourage them to eat healthily than not. So then what we did was we took
our recommendations for interventions here in this cross-synthesis matrix. Here are three
of them on the left here – don’t promote fruits and vegetables in the same way, et
cetera. Then we looked at the quantitative studies, the trials, and looked to see which
of those were in line and which were not in line with the findings from the qualitative
synthesis. We’ve got on the right-hand side here the outcome evaluations and we separated
them into the good quality and the other, so the ones that we wanted to rely on, the
ones – those that we didn’t. So our first message, “Don’t promote fruits and vegetables
in the same way,” we had absolutely no trials which really took that kind of message on
board, or which really based their interventions on that. All of them, where they did have
fruits and vegetables in the same intervention, promoted them in the same way. They’re all
around “five a day, grab five, eat five, it’s good for you,” et cetera. Then moved
on – we looked at branding fruits and vegetables as exciting or child-relevant and reducing
the health emphasis in messages to promote fruits and vegetables, particularly those
which concern future health. Here we did find some trials, we have found five soundly evaluated
in the bottom row and six Other. So we have some data here, but then what we were able
to do is to conduct some statistical sub-group analysis to look to see whether based on the
findings from the qualitative studies, if following the advice, if you like, from the
children from this synthesis, we were able to identify and understand differences in
the findings of the trials. So did those trials which didn’t go on about health being the
reason that children should eat this, did they actually do better than those that did?
Methodologically, this method of synthesis across the term “study types” preserves
the integrity of the different types of studies. So it doesn’t try and get qualitative research
to do what qualitative research can’t do, it gets it to do what it’s really good at.
Likewise, it doesn’t try and make the trials do things that trials can’t do. It keeps
the different types of research as methodologically distinct and uses their strengths. It allows
us to explore heterogeneity in ways which we may well not have imagined in advance.
It facilitates this analytical explanation, it starts to help us to understand what’s
going on in the different types of interventions that we had in the review. It also protects
against data dredging because it’s a theoretically informed framework for exploring differences
between studies. It’s not like a free-for-all, it’s not something where we can just go
around looking and dredging the data for statistically significant differences. We had some hypotheses
that came from the qualitative studies and those were the hypotheses wich were then tested
in the interventions and in the sub-group analyses we then conducted. Epistemologically,
in terms of thinking about how this helps us to know, if you like, better than we did
before is helping us to integrate qualitative estimates of benefits and harm with qualitative
understanding from people’s lives. What this is really helping us to do is to generate
cross-paradigm knowledge. So remember a few slides ago, I was talking about the way in
which research is conducted in little clusters of researchers. Many of the trials were either
conducted by people from the same institutions or there were people who were collaborating
across institutions or they cited one another. There’s a strong network of people who are
investigating this field. Not surprisingly then the studies have a similar approach and
that’s probably one of the reasons why we saw that there were so many similarities between
some of the trials in terms of the fact that they didn’t treat fruits and vegetables
differently, for example; they had the similar approach there. Likewise, there are people
doing qualitative research among children. Often not always simply around the formation
of healthy eating but certainly around the children’s understandings with their lives
and their own health. They’ve got their own way of understanding what’s important
and how to investigate the world. Those two schools of thought, the trialist school of
thought and the research and sociology of childhood, some I suppose came from that perspective.
Traditionally, they don’t tend to talk to one another. So what we were able to do in
this analysis, we find it’s another analysis synthesis as well, is think about how knowledge
coming from these quite different perspectives and schools of thoughts can actually be used
to generate new understanding. That’s what I mean by this cross-paradigm knowledge being
generated here. So in terms of the mixed methods synthesis heuristics that I talked about earlier,
in the meta-analysis we aggregated the findings across the studies and talked about them in
terms of being a single paradigm. The qualitative synthesis then aggregated within the concepts
and then configured between them to help us to build up that picture of recommendations
for interventions mostly within that, using a single paradigm stance again. Then finally,
the mixed methods synthesis aggregated the findings within each recommendation in a single
paradigm and then configured findings between them in a dialectical paradigms stance. That’s
what I’ve referred to several times now, this dialectical idea where we’ve got research
coming from quite different traditions. So we’ve got the research from the trials,
which is literature which has been produced often by people who’ve been either collaborating
with one another from the same institution or between institutions or following a similar
school of thought. So a lot of the papers cite one another even if they don’t have
the same authors. There’s a body of research which is around investigation of healthy eating
among children. People go to the same conferences, they cite one another and there’s a school
of thought which is developing and evolving in this area. And there’s another school
of thought which is around investigating the lives of children not just in terms of their
health but their social lives and other expects of their lives too. The two often don’t
meet. So there are different networks of researchers who were investigating in this case what we’ve
identified as overlapping topic which is children’s healthy eating which is eating anyway. By
using the different schools of thought, the different research traditions, and the different
methods, we were able to generate new what was called cross-paradigm knowledge to help
us to start to understand why some of the interventions in our analysis have different
results to one another. So there are a couple of references there. If you want to follow
up either on thematic synthesis or the mixed methods, synthesis methods. So I’m now just
going to take you through the other types of synthesis, mixed methods synthesis but
I’m not going to give you detailed worked examples here but I can refer you to further
reading. So first of all, there’s another type of sequential synthesis which is generally
called sequential exploratory design. So rather than trying to explain the differences that
we’re seeing in our case, between the different studies, what it’s wanting to do is to generate
new hypotheses or knowledge gaps or as I’ve said, develop a new typology. So there’s
one synthesis that I can think of which developed the typology or reasons why people did or
did not participate in clinical trials. There’s a reference here which is on pediatric medication
error which we conducted here this year where we used a mixed methods design to explore
differences in trials. Finally, there’s this convergent method. Convergent approaches
really have to begin by transforming data. So if you’ve got qualitative data and quantitative
data, they need to be transformed in some way in order for them to be comparable and
combined at the same time. There’s no sequential stage here, everything goes together in one
pool, but either the qualitative data needs to be quantitized or the quantitative data
needs qualitized, if you like in, order for them to be pooled together. So when we’re
thinking about convergent qualitative designs, now these are where we have quantitative data
and qualitative data and what happens is that quantitative data are transformed into qualitative
findings, the sort of themes and concepts, configurations, et cetera, and patterns. Then
they’re combined on this sort of unequal playing field with qualitative data. One of
the most common techniques is thematic synthesis, thematic analysis, and more complex data transformation
happens within a realist synthesis where we’re thinking about theory as well. So I’ll refer
you here to the RAMESES project which has got some training materials on realist synthesis.
So the opposite of that realist convergent QUAN where what we’re doing is we’re going
to conduct quantitative analysis, so what we need to do here is turn any qualitative
findings into numbers and then we can combine them with the quantitative. These are fairly
rare actually in practice but there are plenty of worked examples around. So some of the
typical methods which use a convergent quantitative design are Bayesian synthesis or Bayesian
networks. Qualitative comparative analysis, Charles Ragin has a theoretic way of conducting
analysis, would be an example of that. I’ve got some further readings for you there, so
there’s a methodological paper that we were involved in writing here on qualitative comparative
analysis and systematic reviews. Then Gavin Stewart and colleagues have done a paper on
the uses of Bayesian networks as tools in research synthesis and that’s an example
of a convergent quantitative design. All right, so I feel the last bit is a whistle-stop to
our different options but I didn’t want to overwhelm you with detailed examples. If
you’re interested in those, then I would start with the Pluye paper on the power of
words and numbers and then follow up the leads and the areas of mixed methods synthesis that
you’re interested in. What I wanted to take you through here was the way in which mixed
methods research synthesis have a dynamic and an evolving heritage. So it’s not fixed
and that’s part of the excitement and the challenge of working in this area. We’re
involved in addressing these complex, these compound questions and complex areas, and
blending this micro and macro perspectives, something which I hope came through in the
worked example there. So we have the micro perspective of the children’s views and
the focus groups and the interviews, and then we have the macro perspectives provided by
the quite – some of them are quite large trials. Then what we did was we blended them in order
to start to explain differences in approach within those interventions. What the mixed
method synthesis also did here was really to reflect the state of current knowledge
more faithfully than simply focusing on one or the other. By bringing them together, we
overcame some of those gaps that you see caused by some of these paradigm divisions within
qualitative and quantitative. These methods are still evolving, and it’s a really rewarding
and interesting field to work in. So I look forward to your questions. Joann Starks: Thank
you very much, James, for today’s presentation. I also want to thank everyone for participating
this afternoon. We hope you found the session to be helpful as it wraps up our current four-part
series on qualitative research synthesis. Here on the last slide is a link to a brief
evaluation form and we would really appreciate your input. We will also be sending an e-mail
with the link for the evaluation to everyone who registered. On this final note, I would
like to conclude today’s webinar with a big thank you to our speaker Dr. James Thomas
from myself, Ann Williams, and all of the staff at the KTDRR. We appreciate the support
from NIDRR to carry out the webinars and other activities. We look forward to your participation
in future events. Good afternoon.

One Reply to “Combining quantitative and qualitative evidence: why, how and when?”

Leave a Reply

Your email address will not be published. Required fields are marked *