Methods for a Qualitative Systematic Review

Joann Starks: Good afternoon, everyone. I
am Joann Starks of SEDL in Austin, Texas, and I will be moderating today’s webinar
entitled Methods for Qualitative Evidence Synthesis. It is the second in a series of
four webinars that make up an online workshop on Qualitative Research Synthesis. I want
to thank my colleague, Ann Williams, for her logistical and technical support for today’s
session. The webinar is offered through the Center on Knowledge Translation for Disability
and Rehabilitation Research, KTDRR, which is funded by the National Institute on Disability
and Rehabilitation Research. The KTDRR is sponsoring a community of practice on evidence
for disability and rehabilitation or D&R research. Evidence in the field of disability and rehabilitation
often includes studies that follow a variety of qualitative research paradigms. Such evidence
is difficult to summarize using traditional, systematic research review procedures. The
goal of this series of web-based workshops is to introduce D&R researchers to the methodology
of qualitative evidence reviews. Participants will be provided a state-of-the-art overview
on current approaches and will learn to apply those to the literature base. Ongoing, innovative
initiatives at review-producing institutions will be highlighted. Today, our speaker is
Michael Saini, Associate Professor at the Factor-Inwentash Faculty of Social Work, University
of Toronto, and holds the Endowed Factor-Inwentash Chair of Law and Social Work. His professional
interests include research policy and practice with children and families involved with child
welfare, family law, and alternative dispute programs. He is actively involved in the Cochrane
collaboration and the Campbell Collaboration as an author of systematic reviews and is
a member of Campbell’s Process and Implementation Methods Group. He has over 50 publications,
including books, book chapters, government reports, systematic reviews, qualitative reviews
and peer-reviewed journal articles. He is the co-author with Dr. Aron Shlonsky of a
book entitled “Systematic Synthesis of Qualitative Research.” Welcome and thank you for agreeing
to conduct the session today on Methods for Synthesizing Qualitative Evidence. If you’re
ready, please go ahead. Michael Saini: Thanks, Joann. I’m really excited to be here to
present to you today the methods for qualitative evidence synthesis. So let’s talk about
specifically qualitative synthesis. So now we’ve introduced you to the concepts related
to qualitative research of the primary studies. Qualitative synthesis is simply taking those
primary studies and bringing them together in a way that you tell a larger story about
those experiences that was in the individual primary studies. So it’s a method within
a family of systematic reviews that soaks us on either aggregation, integration or interpretation
of the data. To aggregate primary studies that are qualitative, you would simply look
at the number of times that views and comments or themes are presented and you would add
them up across studies and that would give you an indication of how pronounced those
themes may be. Integration, you perhaps then would just collect those themes and you would
integrate those themes into a larger theme. Then the interpretation, you may look at those
themes and look at the context in which these themes were derived and come up with some
interpretations about what does this mean, what new knowledge do we have by bringing
these islands of primary studies together. What’s the big picture that gets – that
emerges from this process? So similar to systematic reviews, qualitative synthesis should continue
to strive to be transparent, systematic and rigorous in its methods. Now there is some
debate about this because of the qualitative nature of qualitative synthesis. Some would
say that those methods need to be fluid and they need to be moved around and changed around
depending on the context or the best fit, the qualitative methods. I think you can have
some flexibility. I think that flexibility is wanted and needed to be able to develop
a qualitative synthesis that still relates to the qualitative primary studies. I think
it’s also important that we document, that we’re transparent, that people can follow
our logic of how we got to the results, how do we interpret these findings. They should
be able to go back and see what was our research question, which studies do we include and
not include and why do we include them and not include them. What methods do we use to
integrate or aggregate or to interpret? What methods were included? What methods were not
included? All of these decision points, which I’ll talk about in a moment, are all really
important to consider. By doing so, we enhance the utilization value of these qualitative
studies and the power to be able to tell a rich and compelling story. There are a growing
number of examples of qualitative synthesis within the field of disabilities and rehabilitation.
So here are a couple of them that I found just doing a quick search of qualitative synthesis
within the disability and rehabilitation field. So you see that there is one on disruption,
disbelief and resistance: a meta-synthesis of disability in the workplace. There is another
one on youth with disabilities. There is another one on students with disabilities and another
one with caregivers. I think the possibilities are endless. Considering how much qualitative
research is now out there in the electronic databases and other sources, there is so much
rich qualitative research that if it’s just sitting there for us to pick and to be able
to use in a way that tells that larger rich story. So questions specifically related to
the interventions could be: How do people experience the identified outcomes? Why does
an intervention work or not work? For whom? In what circumstances? Who is not at the table?
Who has not been included in these quantitative studies? What groups of people have been omitted
from having voice to tell us whether or not these interventions would work or not work?
What aspects of intervention are valued? Maybe, for instance, it’s the distance to the intervention.
Perhaps, it may be the accessibility issues or the lack of accessibility to the interventions.
So knowing something about transportation could be very useful for us, knowing something
about the uptake of an intervention and ultimately then, the effectiveness of that intervention.
What factors facilitate or hinder the successful implementation of the program? How can a particular
intervention be adapted for a large-scale rollout? If you were to take a very controlled,
random-controlled trial and wanted to roll that out across jurisdictions, what do we
need to know before we begin rolling out that program? Let me just show you two examples
of the integration of qualitative research in the evidence-based practice framework as
it relates to systematic reviews. The first one is a study by Palinkas in 2008. They wanted
to understand the evidence-based treatment plan that was integrated within agencies,
within the child mental health centre. They found that the trainers struggled with the
implementation of the treatment. So even though they are measuring the effectiveness of the
treatment approach, there was a range of experiences about whether or not the actual intervention
was working. So then they went further into the analysis by doing qualitative research
and they found that there were many barriers to the implementation of implementing this
treatment plan. So people said that there was a lag time between the initial training
and the use of the treatment in practice. So they received the treatment and then a
year after that, then the treatment was then included and then evaluated. By that time,
they forgot about those core elements of the treatment. So they started to create their
own ideas on what that treatment should look like. There was a lack of consistency in the
competency in the use of administering the treatment. The clinician engagement with the
project was not always consistent as well and therefore, the absolute effect of the
treatment may show that it is effective, but the relative effect suggests that that relative
effect of whether or not it is beneficial may be influenced by these other factors such
as the implementation itself of how the treatment was implemented. This example from Sibthorpe,
I think, is a great example of looking at which question is the right question. In this
example, they wanted to look at hazardous drinking among an Indigenous Australian Community
Medical Centre. So they initiated a random-controlled trial and they had good funding to do this.
Their goal was to have 200 people in the intervention, 200 in the control, and then they would look
at hazardous drinking. Interestingly though, only 10 participants agreed to participate.
None of them indicated they had a drinking problem. So therefore, none of them could
be included into this study. The authors had to actually give back the money to the funders
saying, “I’m sorry, but we were unable to fulfill our objective of doing this random-controlled
trial.” Now what’s interesting is that they had – they’ve done a qualitative
study first to understand and to explore the experiences of the participants about their
experiences of hazardous drinking. They would have found that the patients themselves were
too embarrassed and did not want to approach or deal with their drinking within this study.
So there are barriers to the implementation of having folks come into the survey. Those
who were screened, although the authors, to give them credit, included aboriginal health
workers to help with the inclusion and the assignment, they, because of their connections
with the participants, did not want to disclose that certain people had drinking problems.
So, therefore, no one had a drinking problem even though the study was about hazardous
drinkers. Now have they done some qualitative interviews or some focus groups with community
members and explore how do we go about – this is an issue that’s been identified in their
community. We want to figure out how best to proceed had we – had they spent the time
and did some interviews and focus groups. They perhaps could have come up within strategies
to have – to best implement such an approach. So qualitative synthesis in its various forms
should always be faithful to the primary studies that focus on transparency, that focus on
the original studies and tracking those experiences from those primary studies right to the synthesis
is critical to ensure that the overall quality of synthesis is credible and that it relates
back to those questions. So in terms of capacity to do qualitative synthesis, you need to ensure
that you have the experience and the advice of the qualitative research synthesis experts
to help you ensure that you have a process. I think just going through these webinars,
then you also end up with a level of expertise that can help you develop the specific methods
that you want to conduct to do these qualitative synthesis. So I think you’ll then be able
to share your expertise with your team. Perhaps, you have to look at whether additional training
may be required. Maybe you will want to read some of the books that are out there, some
of the other webinars that are being produced in terms of educating the qualitative synthesis.
The budget and cost for additional time and resources, it does take – it can take quite
a bit of time to conduct a qualitative synthesis properly because you have to do an information
retrieval that’s exhaustive, that tries to identify all of the qualitative studies
and then you need the resources and the time to be able to sift and sort all of those studies
and then begin the analysis, which then leads into the write-up. So you can see that there
are many different steps and it does take time and it does cost – it can cost quite
a bit, depending on your setup. So if you’re attached to a university, for instance, those
costs may not be as high because there are built-in resources like a librarian who can
volunteer their time. If you’re doing this on your own, then you may want to think about
how best to connect to those other resources to offset some of those costs. You’ll need
to think about whether or not you have access to the appropriate databases and journals.
I think anybody in academia, one of the most fortunate things about being in academia is
the access to electronic databases. I just need to log into my account at the university
and I have access to millions of journals across sectors. So I can go into journals
specifically related to disabilities and rehabilitation and have access to most of the research that’s
out there currently in the electronic databases. If you don’t have access to a university
library, then that can be much more challenging. There could be a significant cost to you if
you need to purchase articles online and so you’ll need to think about how best to get
to those databases in the cheapest way. Again, this is why I think having some collaboration
with university researchers is really important because then you have an access, both of the
expertise, but also of the journals themselves. I used to, when I first started doing qualitative
synthesis, do most of my own information retrieval searching, but I’ve recently been hiring
librarians. They’re wonderful at being able to track down this information in a systematic
way. I’ve been really happy with the work of the librarians. Not only do they give me
the literature that I’m looking for, but they also set up updates. So as new literature
becomes available, I get an email, letting me know of those new studies and then I can
go get those studies. So I think if you’re going to develop a qualitative synthesis team,
you should really think about including a librarian as part of your group. If there
are specific challenges of the systematic review process that you’re thinking that
you’re going to be confronted by, maybe it’s choosing a method, maybe it’s choosing
an approach, maybe it is trying to figure out how best to deal with different types
of methods, then think about who you could have on your team that’s going to be able
to help you through this process. There are also some resources that you can review. There
are PowerPoint presentations of these webinars. There are books. What else can you bring in
to educate yourself on – before starting this process? So I’m going to just quickly
now take you through some of the methodological considerations and including and focusing
specifically on information retrieval strategy, which is basically the literature review.
Then I’ll also talk about the screening process and then the critical appraisal. The
critical appraisal basically means how do you appraise the quality of the primary studies?
There is lots of debate about how to do that and I’ll talk about that in a moment. Then
choosing appropriate method and then whether or not a review should be a stand-alone or
mixed-method design. So a mixed method integration of quantitative and qualitative primary studies
and – which provides you both the information from the meta-analysis and information about
coming from the qualitative synthesis and that combination provides you a certain picture
of the story or there are other times that you just want to do a stand-alone qualitative
synthesis. I showed you some of those examples that are just qualitative relevant to the
areas of disabilities and rehabilitation. I’ve done just stand-alone because I wanted
to look at the views of specific groups. I wanted to look at, for instance, the children’s
views of separation divorce and looking at what input they had within the separation
process. So that made sense then to go get all of the qualitative studies that it included
children’s views and bring them into a stand-alone review. I also looked at this intervention
in child welfare called family group conferencing, where we wanted to know both the effectiveness
of the interventions or whether it was effective. Also, we wanted to look at people’s experience
as being part of this approach. So in that example, we used an integrated approach, where
we did both the quantitative and the qualitative. So we’d look at and we present in our book
this notion of the family of systematic reviews, this idea that qualitative synthesis is not
better or worse than meta-analysis. It’s different. It’s for a different research
question with a different approach, much like if you’re going to do a rapid evidence assessment
or scoping review or if you’re going to look at the level of evidence, so what type
of evidence out there on a specific issue, all of these approaches or all reviews and
always in the umbrella of a systematic review. So by doing so, by including qualitative synthesis
within the umbrella of families – within the umbrella of systematic reviews so that
family approach, then we don’t consider qualitative synthesis as being in the shadows
of quantitative, that we don’t consider that a systematic review of random-controlled
trials is the gold standard and that the qualitative synthesis is its subordinate, much like the
debate of quantitative versus qualitative in the primary studies. We’re saying that
that integration of the two and the various purposes for doing qualitative or quantitative
are important to consider and this gives us a richer understanding of the complete evidence
that’s out there. So going back to the definition of evidence-based practice, it is based on
the best available evidence. Sometimes, it’s qualitative and sometimes, it’s quantitative.
By also putting it within the family of systematic reviews, we’re also suggesting that whether
it’s quantitative review or qualitative review, both of them should be transparent.
Both of them should be rigorous in their approach and both of them should be consistent. You
should be able to trace back from beginning to end of a review, whether it’s quantitative
or qualitative and be able to identify the critical decision points that were made throughout
the process. So this next slide, the family of systematic reviews just basically illustrates
what I’ve been talking about. So at the very top, the most important part of beginning
a systematic review, whether it’s quantitative or qualitative, whether it’s a rapid evidence
assessment or a scoping review, it is the research question. The research question will
guide the process. If you are looking to explore much like in the research primary studies
of qualitative research, if you are looking to explore the views, the cultural setting,
the perceptions, the experiences, then a qualitative synthesis then would be perhaps your best
option. If you’re looking to look at the effectiveness of an intervention, then perhaps
the quantitative way would be the best way to go. So your research question guides process
and then you move into the context of the research process. Is this going to be a comprehensive
search of all the literature that’s out there? Do you have the capacity to do this?
Do you have the resources to go get all of the studies that had been out there on a specific
topic area? Is your intention to be comprehensive in its focus? If so, then you would go through
the left side of the screen, which is that comprehensive information retrieval strategy
process, which would include the full comprehensive search of electronic databases and other resources
to help you figure out which study should be included and not included. If you’re
looking at more provisional results, so you’re not looking at a complete comprehensive picture
of all of the research that’s out there but you’re trying to give an overview, then
perhaps then you will be doing a brief information retrieval strategy. So perhaps, you’ll only
look at peer-reviewed studies and you won’t bring in government reports. You won’t bring
in dissertations. You won’t bring in other resources that may be also connected to a
comprehensive approach. Here, you’re going to be having some specific questions that
you wanted to address. So you may be using this brief approach because it’s more specific
to your research question. Here then, you have a rapid evidence assessment, which basically
is the same as the systematic review, whether it be quantitative or qualitative, except
you are just focusing on the peer-reviewed data and not the non-peer-reviewed, not the
fugitive stuff that’s hard to find. If you’re going to use the scoping review, then there,
all you’re doing is mapping out the evidence. I also suggest though that a scoping review
can be part of a more comprehensive approach. So you see that error of going back to the
comprehensive. I think before you even do a quantitative or qualitative review, you
may want to do a scoping review of the literature. I’ve now published a couple of scoping reviews
and people seem to be interested and I seem to be – I’m finding really interesting
information that comes out of this, just knowing about the type of research that’s out there
on a specific issue. So I wanted to look at the issue of separation and divorce for parents
with children with autism. I wanted to know. I didn’t know whether I’d be looking at
interventions or qualitative information or cross-sectional studies. So I did the scoping
review. I found every single study that was out there and then I mapped it to look at,
“What do we know about this area? What studies have been done? What methods do they use?”
Then I took the qualitative studies and went further into my analysis and we’re currently
now doing a meta analysis specifically focused on those qualitative studies to look at the
experiences of parents to the various transitions of diagnosis, treatment and maintenance. So
on the left side where it says type of study, my initial screen included all studies. Only
when I started to extract this data, I then divided up my quantitative and my qualitative.
So I had two bins. Then by doing that, of course, you have to look at those that are
on both quantitative and qualitative. So much like the process of finding the best research
question for a qualitative primary study, the same would apply for determine the research
question for a systematic review. You first want to identify the research question that’s
going to address the overall target population, the phenomenon that you’re looking to explore
or the intervention perhaps, if you’re looking at a combination of qualitative and quantitative.
So my example of the family group conferencing intervention that was the intervention that
I was looking at, but I kept it open to both effectiveness studies and also studies that
looked at the views and experience of those who’ve been through the intervention. So
the components of the questions help you determine the type of study that you’re looking for.
Is this a research question that will only get at qualitative studies? The question around
the views of children post separation divorce, I just wanted to know about the reviews. So
I was not interested in cross-sectional or experimental studies. I was specifically looking
at only studies where children were included in interviews about their experiences. It
also helps you to the databases and the sources to search. So if I’m looking at a psychological
question, there are specific databases that I’m going to want to include such as psych
info, med line, sociological abstracts. If I have a question that is more education-based,
then I would probably want to include ERIC, which is another electronic database which
holds mostly education-related titles and abstracts. Again, if these databases are not
familiar to you, then again, all the more reason why including a librarian within this
process can be really helpful. Also, the research terms. An example, we were doing a systematic
review on cyber reviews. We wanted to look at what interventions had been developed for
cyber reviews. We spent a lot of time trying to find studies and we could not find anything,
although we were pretty sure that there were studies out there. Going back to the research
question and looking at how we are asking the question, we realized that what we were
asking for in our research question was cyber reviews, but what we should have been asking
about is cyber safety because we were looking at prevention studies and not studies where
cyber reviews have actually occurred. So then by changing the search terms to include cyber
safety rather than cyber reviews, we were then able to uncover much more research that
was – that had been completed and much more in line with what we were expecting to find.
So, again, going back to that diagram of whether it’s comprehensive or whether it is more
brief and focused, we want to look at the breadth and scope of the information retrieval
process. Are we looking to locate all studies or are we looking to find a certain view?
If we’re using the methodology such as a grounded theory approach, where we’re looking
at just getting a sample, a theoretical sample from the qualitative studies that had been
completed, then that’s going to have an implication of how we search the literature.
We want to do this in collaboration with the service users, with those expert panels that
I talked about to understand what do we need to know about? How do we go about finding
it? What should be the expected time frames for completion? What resources do we need
to put in place to ensure that we get all this done? This is where the scoping or the
mapping of the evidence can be really helpful. So the information retrieval process, if it’s
going to be comprehensive, then it should be transparent. It should focus on both published
and unpublished studies, if it’s going to be comprehensive. If you’re going to do
a rapid evidence assessment, then maybe that’s just going to be peer-reviewed and not the
unpublished. When you’re locating the qualitative studies special attention should be made to
be more to trying to find these studies. If you’re doing quantitative searches, then
our filters are much more developed in electronic databases to be able to define studies that
are RCTs so we can use those type of filters like randomized and controlled and compare.
Although some filters have been developed for qualitative research, it’s – they’re
not as good and therefore, we really need to be close to the literature review process
to ensure that they research them appropriately. Of course, there has been a debate about whether
or not qualitative evidence should be exhaustive, comprehensive or whether they should be more
flexible. I really do think it depends on the method that we’re choosing. So, again,
if we are using a grounded theory approach or something to – that type of approach,
then we would be looking at a – more of a theoretical sample or more of a proposal
sampling approach. If we’re looking at more of a systematic approach, then we’d be using
a comprehensive approach for identifying the search terms, identifying the electronic databases,
going through each of those databases with those search terms and making clear decision
points about whether studies should be included or excluded. So here is an example of a qualitative
filter that has been created. I’ve used this just to test this out. I’m not convinced
that this is the best approach to use, but I do provide this to you in case you want
to try this out. Again, I’d like to look at all of the evidence and not filter by study
design and only filter by target population and the experience that I am trying to capture
and then make decision based on titles and abstracts because I’m not always confident
that this is the best way to filter down to qualitative studies. So I think the qualitative
filters can be too limiting and our focus on the word “systematic” within systematic
synthesis suggests that we are really suggesting that you’d be more comprehensive in your
approach and that that approach is transparent in the sense that you are documenting all
of your decision points. So you’re documenting which electronic databases you use, the search
terms, when you did it, and that accountability and that transparency provides the details
for somebody else to look at how you went about, including some studies and not others.
Your inclusion criteria is based on making clear decision points about what studies should
be included and not included and we’ve suggested a multiple level screening process where you
go from a liberal screening so you include everything that is relevant to your research
question and that is a study, whether it be quantitative and qualitative, and then you
go into more specific screening based on the type of design. So then at that point, you
would then screen and put it in a separate bin all studies that seem to be qualitative.
When you do extraction for the qualitative and quantitative, you want to do that separately,
and for the qualitative extraction, so bringing in extraction just basically means bringing
in the information that is relevant to your qualitative synthesis. So it could be some
demographic information. It would be the themes. It would be the quotes. There is no standard
template for doing this. So there are many different examples out there. One of them
that I like using is just simply NVivo. That type of approach of bringing in your primary
studies into the NVivo software program, which a qualitative management program, allows you
then to code and to highlight specific parts of each primary study text and then you then
create themes that way which then allows you to code the data to link it to other documents
to then search across themes that you’ve created. That organization can be really important,
especially if you have a qualitative synthesis with a number of qualitative studies included.
So now you have all of the data, let’s say, into NVivo. You have extracted some common
themes and you also want to look at the appraisal of each study to see whether or not it adds
to the overall interpretation of the findings. This is different than quantitative research
because in quantitative research, you may be using the checklist of some form and looking
at the overall rating of quality for that quantitative study and then perhaps even using
that into a moderator analysis, whether it’s high quality one, moderate quality two or
low quality three, and that one, two and three would be added into your analysis. In qualitative
research, we are not developing checklists to put numbers around quality. We’re using
the considerations of quality to build our overall interpretation of those primary studies.
So looking at the quality of the evidence is important because it does add to our overall
understanding of the applicability and relevance of these primary studies in our overall analysis
by looking at issues around credibility and transparency and whether or not these primary
studies made a contribution to the wider knowledge of the specific issue that you’re interested
in. So within the critical appraisal, it’s really important to have a non-hierarchical
view of evidence, specifically around the different research designs that may be used
for qualitative research, unlike from a quantitative standpoint where the RCT is a gold standard,
everything falls from there. When we look at the critical appraisal within qualitative
studies, there is no common approach, but there are some key questions to be asked.
So why is the research clear? Did it use a clear statement? I focus a lot on the importance
of the research question guiding the process. Was the qualitative methodology appropriate?
So if they’re looking at the relationship among concepts, did they focus – did they
use, let’s say, a more of a grounded theory approach? Or if they’re looking at the culture
nuances, did they use an ethnographic view of the information to bring in those cultural
understandings? Was the research design appropriate to address the aims of the research? Was there
a good nest of fit with the questions and the methodologies? Was the recruitment strategy
appropriate for the aims for the research? If they’re wanting to understand the rich
experiences of a select few people but then they had a sample of 50, are they able then
to really capture those experiences if they only spent 20 minutes with those 50 people?
Or would you rather have seen and spent two to three hours with maybe 10 people? Was the
data collected in a way that addressed the research question? Were they able to achieve
a good understanding of those emerging experiences based on the methods used? Has the relationship
between the research and the participants been adequately considered? So, again, we’re
not looking at the research as being objective. We understand that the researcher is subjective.
There should be some transparency around that as well. Ethical considerations are important
to consider. The data analysis, is it sufficiently rigorous? There’s a common approach of analyzing
data and that’s just doing a thematic representation. You don’t get the nuances. You don’t get
the richness of that analysis process. You want to see that the quotes, for instance,
support the themes being generated, that tells you that there is sufficient rigor in the
analysis process if you, as the reader, can say, “Oh, yes, I read that quote and that
makes sense to what the theme is being presented by the researcher.” Is there a clear statement
of findings and then the overall value of the research study itself? So our quality
checklist is an appraisal tool that we developed. It has 25 dimensions of quality and includes
such things as rigor, credibility, dependability, confirmability, the notions that we’ve talked
about. We also included four other dimensions to appraise the authenticity, fairness, promotion
of justice. When these are the central components of a qualitative study where specifically,
participants are brought into research studies that are qualitative to empower their involvement
and to bring action. So, for instance, the participatory action research, the reason
why one would bring in participants into a participant action research design is so that
they can share in the full production of the research process from beginning to the very
end to sharing the information, to being part of the dissemination process. So in those,
goals are set out at the very beginning of a research design where promotion of justice
is central to the objectives of the research, then there need to be a clear demonstration
that the participants were part of the process right from the very beginning. Now, of course,
if you are doing a grounded theory study and the objective is not to empower and not to
promote action, then perhaps the promotion of justice would not be central to your analysis
around quality. That’s why it’s important to use the criteria in a flexible way, depending
on the purposes and the overall goals of the research study that you’re assessing. So
our QRC appraisal form was piloted here at the Factor-Inwentash Faculty of Social Work
in 2008. We looked at the inter-agreement scores, and then we made modifications. We
also developed the user guide in our book and chapter, I believe, seven. We have a section
on qualitative appraisal and we also include a working example of using the appraisal form,
using two qualitative interviews of family group conferencing. So as summary of the challenges,
as we’ve talked about, the first is the research question, whether this should be
an iterative or a predetermined focus. I think that perhaps the best way to start off would
be, again, focusing on your intention of doing this qualitative synthesis. In some ways,
if you’re looking at more of a predetermined research question and you want to know specifically
about some experience or some specific perception, then you may want to start off with that predetermined.
Even in those cases, because you’re looking at qualitative research, because it’s so
rich in complexity, I think you still need to give yourself an option of changing that
question or modifying that question as you go through. That’s different than a quantitative
study where you would have a predetermined question at the very beginning. So does A
costs as B? You should have all of your moderator variables that may impact the A cost as B,
clearly set out and be theoretically linked at the very beginning. So you’re not just
putting a whole bunch of stuff into SPSS or some statistical software and see what comes
out to be statistically significant because that’s considered, in quantitative terms,
that seems to be referred to as dredging. So you would just find what is statistically
significant and there is no theoretical basis for choosing those variables. Whereas in qualitative,
it’s much more iterative. It’s much more contextual. So you are very much focused then
on being flexible in your methods, rather than rigid. The information retrieval process,
theoretical or comprehensive, in our approach, we have developed a comprehensive because
if you’re going to spend that much time searching for qualitative research, it’s
really interesting to find out the – what’s out there and how much information is out
there and not limit yourself to a select group of studies. Then the critical appraisal, whether
it’s interpretation versus screening, if you’re screening out, I think you need to
have a really good rationale for doing so and I can’t think of a good rationale for
screening out even based on poor quality because I do think that’s part of the interpretation
process. Then choosing appropriate methods, I’ve given you three ways of thinking about
this: aggregative, integrative and interpretive and having the sense of what you’re trying
to do with this synthesis prior to your start, whether you’re trying to summarize, whether
you’re trying to build a story line or whether you’re trying to generate new information
or new theories can really help you in making those decisions. Then issues around transferability
versus generalizability, so connecting the islands of evidence and being clear even in
your write-up because I think you have to educate other people that these 20 studies
should not be generalized to the large population, that it really is contextual, that it’s
still based on those 20 studies with the participants’ experience within those 20 studies and so
it does not allow us to make inferences now because we’ve simply connected these qualitative
studies. It really moves into helping us understand the applicability, the relevancy and the lessons
learned from these studies to start pointing us in much clear directions. For your reference,
I’ve included the cover of the book that I’ve been talking about throughout the webinar.
It’s Systematic Synthesis of Qualitative Research that I published with Aron Shlonsky
in 2012. If you want to email me for that information or any other questions you have
about this webinar, I’d be happy to respond to you. I really do appreciate you taking
the time to listen and to read through these PowerPoints and I am hoping that this creates
more discussion. Again, I don’t have all the answers. I think though moving in this
direction of finding ways to include qualitative research is a right set for it and I encourage
you to begin this journey or to continue on this journey and I wish you all the best in
doing so. Joann Starks: Well, thank you very much, Dr. Saini, for a very informative presentation.
I also want to thank everyone for participating today. We hope you found the session to be
interesting and helpful and that you’ll join us for the next two webinars. Here on
the final slide, we have a link to a brief evaluation form, and we’d really appreciate
your input. We will also be sending an email with the link for the evaluation to everyone
who registered. On this final note, I would like to conclude today’s webinar with a
big thank you to our speaker, Dr. Michael Saini from myself, Ann Williams and all of
the staff at the KTDRR. We appreciate the support from NIDRR to carry out the webinars
and other activities. We look forward to your participation in the following sessions. Thank

Leave a Reply

Your email address will not be published. Required fields are marked *