Metodoloˇ skizvezki,Vol.17,No.2,2020,1–29
Qualityguidelinesformixedmethodsresearchinintervention
studies: Aconceptualmodel
GizelaKopaˇ c,ValentinaHlebec
∗
UniversityofLjubljana,FacultyofSocialSciences,Ljubljana,Slovenia
Abstract
Amongresearchers’manyinvestigationsoftheuseofmixedmethodsininterventionstudies,
more recent discussions especially concern the roles played in such studies by qualitative
research, intervention phases, procedures, and integration (Gallo and Lee, 2016; Woolcock,
2018;O’Cathain,2018;CreswellandPlanoClark,2018). Onecanfindthebasicprocedures
to follow while realizing a mixed methods experimental design (Creswell and Plano Clark,
2018),practicalguidance(O’Cathain,2018)forusingqualitativeresearchwitharandomized
control trial (RCT), and a mixed methods appraisal tool for appraising the methodological
qualityofRCTs,non-randomizedstudies,andmixedmethods–MMAT(Hongetal.,2018).
However, no model exists to assess the quality of mixed methods research in intervention
studies,particularlyexperimentalandquasi-experimentalresearchincomplexinterventions.
Our aim is to develop such a theoretical model. Today, the number of interventions relying
on mixed methods methodology is growing exponentially. A theoretical model is called
for to help assess the quality of mixed methods research in intervention studies, and in this
respect our aim is to: (1) provide an overview of guidelines, recommendations, models,
andqualitycriteriaformixedmethodsresearch;(2)overviewtheguidelinesforintervention
studies; (3) give a summary of guidelines and models for mixed methods research in such
studies; (4) evaluate the mentioned guidelines, models, and quality criteria; (5) identify and
describe the key elements of these guidelines, models, and quality criteria; and (6) develop
a theoretical model for assessing the quality of mixed methods research designs used in
interventionstudies.
1. Introduction
Mixed methods research has become a substantive and rising methodological force see-
ingevergreaterpopularityinalltypesofresearch,includinginterventionresearch(Creswell
and Plano Clark, 2018). In this paper, we develop a universal model for the quality assess-
mentofmixedmethodsresearchininterventionstudies. Weinvestigaterelevantdiscussions
about the value of mixed methods for intervention research. An overview of guidelines,
∗
Correspondingauthor
Emailaddresses: gizela.kopac@gmail.com(GizelaKopaˇ c),valentina.hlebec@fdv.uni-lj.si
(ValentinaHlebec)

2 Kopaˇ candHlebec
recommendations,models,andqualitycriteriaformixedmethodsresearchandintervention
studies is then provided. This is evaluated in terms of the components of these guidelines
and models. The outcome of this evaluation is the construction of a conceptual model. The
articleisdividedintothreeparts: (1)discussionsonthevalueofmixedmethodsininterven-
tion research; (2) an overview and evaluation of guidelines, recommendations, models, and
quality criteria for mixed methods research and intervention studies; and (3) the construc-
tion of a conceptual model to assess the quality of mixed methods research in intervention
studies.
The investigation of mixed methods research in intervention studies is in fact not new.
Many authors have considered the value of mixed methods for intervention research. Dis-
cussions led by Sandelowski (1996), Lewin et al. (2009), O’Cathain et al. (2013), Drabble
and O’Cathain (2015), and Grissmer (2016) focused on a combination of a qualitative ap-
proach and RCTs. Margatete Sandelowski (1996) looked at how the qualitative component
can be included and applied in RCTs to understand how RCT findings may be applied in
a real-word clinical setting. Similarly, Lewin et al. (2009) examined the use of qualitative
methods in association with RCTs of complex healthcare interventions while O’Cathain et
al. (2013) also concentrated on the relationship between qualitative research and RCTs,
where296articlesinwhichqualitativeresearchandRCTwereusedwereidentified. Onthe
otherhand,DrabbleandO’Cathain(2015)exploredthevalueofmixedmethodsintervention
evaluations in RCTs to understand an intervention’s effectiveness. Finally, Grissmer (2016)
exploredthepurposeofmixedmethodsineducationalandsocialinterventionswhereRCTs
are used and found that mixed methods are powerful tools when coupled with an RCT to
understand why the effects of an intervention occurred (or not), how, and why they vary
amongtheparticipants.
Other authors like Protheroe et al. (2008), Song et al. (2010), Farquhar et al. (2011),
Bamberger (2012), Hesse-Biber (2012), Farquhar et al. (2013), Boeije et al. (2015), Brown
et al. (2015), Jimenez et al. (2018), Woolcock (2018), and Palinkas et al. (2019) concen-
trated more on the contribution of mixed methods to different intervention and evaluation
phases. For example, Protheroe et al. (2008) investigated the benefits of mixed methods
for evaluating a complex intervention and emphasized possible discrepancies in quantita-
tive and qualitative results. Meanwhile, Song et al. (2010) looked at how mixed methods
intervention studies were conducted in the health sciences before developing the following
components of mixed methods intervention research programs: developing the intervention
research program’s theoretical foundation; the pilot test of the efficacy, feasibility, accept-
ability, treatment conditions, and measurements of the outcomes; assessing the intervention
efficacy, and testing the intervention’s effectiveness. Farquhar et al. (2011) examined how
and why mixed methods are used to develop and evaluate complex interventions in pallia-
tive care. Use of the mixed methods approach was explained by Bamberger (2012) in an
impact evaluation together with its essential elements, while also presenting the design of
a mixed methods impact evaluation in the context of complex interventions. Hesse-Biber
(2012)introducedthevalueofmixedmethodsresearchtoassesstheeffectivenessofclinical
interventions or which specific interventions are best suited for adoption in a research trial
withinclinicalsettings.
Investigating the use of mixed methods approaches as part of complex intervention de-
velopment and evaluation (Farquhar et al., 2013) as well as identifying the methodological
challenges of mixed methods intervention evaluations in complex interventions (Boeije et
al., 2015) and the role of mixed methods in the effectiveness, acceptability, feasibility, and
implementation of both the intervention and evaluation study (Boeije et al., 2015) have be-

Qualityguidelinesformixedmethodsresearchininterventionstudies 3
come important aspects of the scholarly enquiry into the value of mixed methods research
in intervention studies. In the same year, Brown et al. (2015) explored the environments
in which health interventions occur, investigated the evaluation of interventions, and ascer-
tained that the role of mixed methods in such evaluations is to consider the intervention’s
contentandcontext. Therewerealsoinvestigationsabouttherolesofmixing,matching,and
the use of qualitative methods to improve quantitative impact evaluations (Jimenez, 2018)
and the importance of qualitative methods in complex intervention evaluations (Woolcock,
2018). In 2019, Palinkas et al. (2019) found that mixed methods designs had become ever
more common in healthcare for evaluating the process and outcomes of an intervention,
program, or policy effectiveness and investigated how mixed methods have been used in
healthcareinterventions.
Inaddition,Protheroeetal. (2008),Lewinetal. (2009),Farquharetal. (2011),Farquhar
et al. (2013), and Noyes et al. (2019) delved into the value of mixed methods in complex
interventions. While in 2009 Lewin et al. (2009) looked more at the use of qualitative
methodsalongsiderandomizedcontrolledtrialsincomplexhealthcareinterventions,in2011
Farquhar et al. investigated the development and evaluation of complex interventions in
palliative care. Noyes et al. (2019) determined that mixed methods hold the potential to
increase understanding of how complex interventions work, for whom, and how complex
health systems respond and adapt. Albright et al. (2013) also considered the use of mixed
methods in intervention research and established that one benefit of mixed methods is that
it includes both quantitative methods to examine the intervention content (the ‘what’) and
qualitativemethodsforexploringthecontext(the‘why’and‘how’).
Creswell and Plano Clark (2007) studied the use of mixed methods in intervention re-
search, calling this design an “embedded design”, whereby the researcher must embed a
qualitative component into a quantitative design, like in an experimental design. They pre-
sented an “embedded experimental model”, a variant of the embedded design that is the
variant of this design most commonly relied on. This model embeds qualitative data within
anexperimentaldesign(suchasatrueexperimentorquasi-experiment)withthedatacollec-
tionprocessoccurringbefore,during,oraftertheintervention. In2011,CreswellandPlano
Clark renamed it an “embedded experimental mixed methods design”. They presented the
proceduresforpreparinganembeddeddesign: (1)designingtheoverallexperimentandde-
ciding on why qualitative data need to be included; (2) collecting and analyzing qualitative
data to enhance the experimental design: (3) collecting and analyzing quantitative outcome
data for the experimental groups; and (4) interpreting how the qualitative results enhanced
the experimental procedures and/or understanding of the experimental outcomes. Creswell
(2015) relabeled this design an “experimental intervention design” in which the investiga-
tor gathers qualitative data in some phase during the experiment, either before, during, or
after the trial. Integration consists of embedding the qualitative data into the experimental
design. Finally, in 2018, Creswell and Plano Clark conceptualized this type of design as
a “mixed methods experimental (intervention) design” that constitutes one of the complex
mixedmethodsdesigns.
2. Guidelines,recommendations,andmodelsformixedmethods
The researcher is informed by guidelines, recommendations, and models in the research
processregardinghowsomethingshouldbedone. Theyareastructuredtoolforresearchers
to rely on while conducting or reporting the research process. Different study types have
their own guidelines for conducting or reporting the research process. The mixed meth-
ods research process lacks methodological rigor in intervention studies. Researchers need

4 Kopaˇ candHlebec
transparentandclearinformationonhowtousemixedmethodsmethodologyinintervention
research. Weintroduceguidelines,recommendations,andmodelswhichcanbefoundinrel-
evantmixedmethodsliterature. Wedividethemintothreegroups: (1)generalguidelines;(2)
guidelines in specific disciplines; and (3) guidelines in health, psychological, educational,
and evaluation-focused associations. Our aim is to consider the individual components of
theseguidelines,recommendations,andmodelsbeforebuildingourconceptualmodel.
2.1. Generalguidelines,recommendations,andmodelsformixedmethodsresearch
Many discussions about mixed methods research can be found with respect to guide-
lines, recommendations, and models. While Onwuegbuzie and Teddlie (2003) described
and illustrated seven stages of the mixed data analysis process: data reduction, data display,
datatransformation,datacorrelation,dataconsolidation,datacomparison,anddataintegra-
tion, Maxwell and Loomis (2003) and Maxwell (2012) presented an interactive model of
researchdesignconsistingof: purposes,conceptualframework,researchquestions,research
methods (sampling, data collection, data analysis), and validity. They attempted to show
the value of this interactive research design concept for understanding mixed methods re-
search. Tashakkori and Teddlie (1998) discussed mixed methodology, examined sampling
issues, data collection procedures and data analysis techniques, and considered ways to uti-
lizemixedapproaches. Theauthorsexploredthecriteriaaresearchercanrelyonforamixed
methodsdesign(TashakkoriandTeddlie,2010)andmixedmethodsresearchquestions,sam-
pling issues in mixed methods research, along with the analysis of mixed methods data and
integration(TashakkoriandTeddlie,2009).
The framework for sampling designs in mixed methods research, along the way de-
scribing sampling schemes and discussing sample size, was presented by Onwuegbuzie
and Collins (2007). They listed four possibilities: identical sampling, parallel sampling,
nested sampling, and multilevel sampling. Specific guidelines for practice as one of the
methodological domains include: purpose, mixed methods questions, data analysis (inte-
gratedanalysis,jointandinteractiveanalysis),andintegratedmixedmethodsanalysisstrate-
gies(datatransformationanddatacomparison)(Greene,2008). Leech(2012)describedhow
researchershavelittleguidancewhenitcomestowritingupamixedmethodsresearchstudy
and thus offered recommendations for such an endeavor, including: purpose/rationale, re-
search questions, the mixed methods design, and the integration. Guest (2012) presented
a list of six descriptive dimensions for mixed methods research: timing of the interface,
purpose of the interface, theoretical orientation, purpose of the research, number of points
of interface or degree of integration, as well as the relative importance of qualitative data
vs. quantitative data (weighting). He stressed that while every dimension is important for a
study’s conceptualization, the timing and purpose of the integration are crucial with respect
todescribingastudy.
Guest and Fleming (2014) outlined some of the most common mixed method design
typologies and stressed the importance of three dimensions: timing, purpose, weighting.
They presented general guidance in the design of a mixed methods study and a simple set
of instructions on the basic steps: the research question/problem, justification for the mixed
methodsresearch,datacollectionandanalysis,mixedmethodsdesign,anddataintegration.
On the other hand, Fetters and Freshwater (2015) presented an hourglass model of mixed
methods research. At the top of the hourglass model one finds the broader methodologi-
cal context, the purpose of the mixed methods, and the research questions. In the middle
of model lies the mixed methods design, sampling, measurement procedures, data analysis,
data integration, and software applications. At the bottom of this model there are: a dis-

Qualityguidelinesformixedmethodsresearchininterventionstudies 5
cussion of the mixed methods integration, field-specific implications, and methodological
implications for mixed methods. The authors conclude that integration is a challenge and
thatguidanceexiststoassistintegration.
Creswell(2015)discussessixstepsindesigningamixedmethodsstudy: (1)theresearch
question;(2)typesofdatacollectionandanalysis;(3)thereasonsforusingmixedmethods;
(4) diagrams, procedures, and choice of design; (5) the mixed method study’s aim or pur-
pose;and(6)thequantitative,qualitative,andmixedmethodsquestions. Healsointroduces
a checklist of elements to include during a mixed methods manuscript preparation: the type
of mixed methods design, the rationale, the mixed methods study’s aim and purpose, the
quantitative, qualitative, and mixed methods research questions, the quantitative and qual-
itative data collection and analysis, and the integration of quantitative and qualitative data.
While Graff (2017) looks at the emergence, purpose, and characteristics of mixed methods
researchandidentifiesthefollowingcharacteristics: mixedmethodsdesigns,researchques-
tions, sampling, data collection, data analysis, mixing qualitative and quantitative phases,
andadetaileddiscussionofpurposiveprobabilitysamplingtechniques,Schoonenboomand
Burke Johnson (2017) show how to design a high-quality mixed methods research study,
proposing seven major design dimensions: purpose, theoretical drive, timing (simultane-
ity and dependency), point of integration, typological vs. interactive design approaches,
plannedvs. emergentdesign,anddesigncomplexity. Theyalsoproposemultiplesecondary
dimensions. The authors believe the most important decision in mixed methods research is
todeterminethepoint(s)ofintegrationandhowtheresultsaretobeintegrated.
CreswellandPlanoClark(2018)considerhowtodesignandconductmixedmethodsre-
search and list the following steps: (1) research problem; (2) research purpose; (3) research
question; (4) rationale; (5) collecting qualitative and quantitative data; (6) sampling proce-
dures/site(s); (7) participants; (8) sample size; (9) participant recruitment; (10) data sources
identification; (11) data analysis (statistical software); and (12) integration. They highlight
thecrucialstepsanddescribethemindetail. Theyoutlinethreecoremixedmethodsdesigns
(convergent, explanatory sequential, exploratory sequential) and four prominent types of
complex mixed methods designs (mixed methods experimental (intervention), mixed meth-
odscasestudy,mixedmethodsparticipatory-socialjustice,mixedmethodsprogramevalua-
tion).
2.2. Guidelines, recommendations, and models for mixed methods research in specific dis-
ciplines
Many authors have presented guidelines, recommendations, and models for a specific
discipline. Creswell, Fetters, and Ivankova (2004) established the criteria for analyzing
mixed methods studies in primary care research and then used them to evaluate mixed
methods investigations in primary care research journals. They identified the following
study characteristics: content area, rationale for mixing, forms of data collection (quantita-
tive/qualitative), analytical procedure (quantitative/qualitative), characteristics of the design
(priority, implementation), integration, and type of design model. They concluded these
characteristics are rigorous elements which primary care investigators may rely on while
designing their studies. On the other hand, Collins, Onwuegbuzie, and Sutton (2006) cre-
ated a framework for conducting mixed research studies called the RAP model – a 13-step
modelforconductingmixedmethodsresearchinspecialeducationandbeyond,dividedinto
three stages: research formulation, research planning, and research implementation. Re-
searchformulationentailsthe: (1)mixedgoalofthestudy;(2)mixedresearchobjective;(3)
rationale for mixing; (4) purpose of mixing; and (5) mixed research question(s). Research

6 Kopaˇ candHlebec
planning involves: (1) the mixed sampling design; (2) the mixed research design; (3) col-
lecting quantitative and qualitative data; and (4) analyzing quantitative and qualitative data,
namely quantitative and qualitative analysis techniques. Research formulations consist of:
(1) validating the mixed research findings; (2) interpreting the findings; (3) writing up the
mixedresearchreport;and(4)reformulatingthemixedresearchquestion(s).
GRAMMS – the Good Reporting of a Mixed Methods Study checklist was developed
for health services research (O’Cathain, Murphy, and Nicholl, 2008). Their study analyzed
research reports of 75 health services in England between 1994 and 2004, with the analysis
leading the study to produce the GRAMMS framework. This six-item framework contains:
ajustificationforusingmixedmethods,thedesign(purpose,priority,sequenceofmethods),
a description of each method (sampling, data collection, analysis), where and how integra-
tion occurred, the limitation of each method, and insights from the mixing or integrating
of methods. Schifferdecker and Reed (2009) investigated the application of mixed methods
in medical education research. They outlined the main steps in designing, analyzing, and
publishing: identifythestudydesign;decideontheprominenceofeachdatatypeinthedata
collection, analysis, and results; develop sampling strategies for the data collection; decide
how and when data are to be collected, analyzed, integrated, or compared; explore tools
(softwareprograms)ormethodstointegratethequantitativeandqualitativedata;andreview
mixed methods research articles. Leech and Onwuegbuzie (2010) examined the application
ofmixedmethodsinthecounselingfieldandbeyondanddetectedadearthofmixedresearch
studiesintherelatedliterature. Theyproposedseveralguidelines: samplesize,sampleinfor-
mation, the sampling scheme for the quantitative/qualitative research, the sampling scheme
for the mixed methods research, the type of generalization, the mixed research design, the
quantitativeresearchdesign,andthequalitativeresearchdesign.
Mixed methods research has also been introduced as a relevant approach to music ther-
apy practice containing core characteristics, recommendations, and key components: data
collection and analysis of qualitative and quantitative data, design (RCT, ethnography), re-
cruitment,sampling,andintegration(Bradt,Burns,andCreswell,2013). Venkatesh,Brown,
andBala(2013)reviewedthecurrentstateofmixedmethodsresearchwhilealsoinvestigat-
ingthevalueandpurposeofmixedmethodsandprovidingguidelinesforconductingmixed
methods research in the area of information systems. These guidelines include several ele-
ments: purpose,design,dataanalysis,integration,andmeta-inferences.
OnwuegbuzzieandCorrigan(2014)consideredthevalueofmixedresearchstudiesinhu-
manresourcedevelopmentandanalyzedempiricalarticles. Theylistevidence-basedguide-
lines for conducting and reporting mixed research that are based on Collins, Onwuegbuzie,
andSutton’s(2006)13-stepmodel,withtheguidelinesbeingdividedintofourdomains: re-
searchformulation,researchplanning,researchimplementation,andresearchdissemination.
These elements entailed are: determine the mixing rationale, establish the mixed purposes,
present mixed research questions, select the sampling design, specify the type of sampling
scheme, outline the mixed research design, collect data, analyze the data, and create the
mixeddataanalysisstrategy.
On the other hand, Wisdom, Cavaleri, Onwuegbuzie, and Green (2012) described the
frequency of mixed methods in published health services research and concluded that such
researchprovidesamorecomprehensivepictureofhealthservicesthaneithermethodalone.
They established that researchers who use mixed methods should follow rigorous method-
ologies. Theycreatedalistofmixedmethodsmethodologicalcomponentsofhealthservices
research: integration of qualitative and quantitative components, a specified sequence of
methods, areas of consistency between the methods stated, areas of inconsistency between

Qualityguidelinesformixedmethodsresearchininterventionstudies 7
the methods stated, the stage of integration specified, priority of the methods specified, the
statedpurposeofmixingmethods,andastatementonthelimitationsofthemixedmethods.
Venkatesh (2016) proposed the latest guidelines for conducting mixed methods research
in the information systems field. He developed a model of decision choice for conducting
mixedmethodsresearchwhichcontainstheseelements: researchquestion,purposeofmixed
methods research, mixed methods design, sampling design strategies, quantitative/qualita-
tive data collection techniques, mixed methods analysis, integration, meta-inferences, and
assessingthequalityofmeta-inferences.
In terms of sampling, Onwuegbuzie and Collins (2007) focused more on a model for
choosing samples for mixed research (identical sampling, parallel sampling, nested sam-
pling, multilevel sampling), whereas Leech and Onwuegbuzie (2010) gave a detailed de-
scription of sampling (sample size, sample information, sampling scheme for quantitative/-
qualitative research, the sampling scheme for mixed methods research). The guidelines for
music therapy research of Bradt et al. (2013) also include recruitment as an important ele-
ment. Wisdometal. (2012)inguidelinesforhealthservicesresearchstressedtheimportance
ofintegrationandsuggested: theintegrationofqualitativeandquantitativecomponents,stat-
ingtheareasofbothconsistencyandinconsistencyamongthemethods,whilespecifyingthe
stage of integration, the priority of methods, and purpose of mixing. Guidelines for mixed
methodsininformationsystems(Venkatesh2016)alsoincludeassessingthequalityofmeta-
inferences.
2.3. Guidelines,recommendations,andmodelsformixedmethodsresearchinassociations
Mixed methods guidelines, recommendations, and models have also been developed
by various health, psychological, educational, and evaluation-focused associations. The
U. S. National Institute of Health (2018) examined criteria for the design and evaluation
of mixed methods research, proposing three already mentioned guidelines (Creswell and
PlanoClark,2018;O’Cathain,Murphy,andNicholl,2008;SchiffendeckerandReed,2008).
Creswell and Plano Clark (2018) stressed the importance of: research purpose, research
question, collecting qualitative and quantitative data, sampling procedures/site(s), partici-
pants, sample size, recruitment, data sources, data analysis (statistical software), and inte-
gration. O’Cathain, Murphy, and Nicoll (2008) highlighted the importance of: the justifi-
cation for using mixed methods, the design, a description of each method (sampling, data
collection, analysis), where and how integration occurred, the limitation of each method,
and insights from the mixing/integrating of methods. Schifferdecker and Reed (2008) em-
phasizedthevalueof: thestudydesign,prominenceofeachdatatypeinthedatacollection,
analysis,andresults,samplingstrategies,howandwhendataarecollected,andthesoftware
programmingtoolsormethodsforintegratingthequantitativeandqualitativedata.
Levitt et al. (2018) as a team at the American Psychological Association developed
MMARS – Mixed Methods Article Reporting Standards. These standards were developed
for the psychology field and contain: the mixed methods’ objective, aim, and goal, the type
of mixed methods design, the participants or other data sources; separate descriptions of
samples if they vary; participant recruitment; participant sampling or selection; data collec-
tion;recordingandtransformingthedata;dataanalysis,andintegration/mixing.
JalongoandSaracho(2016)investigatedtheuseofmixedmethodsineducationresearch
and found that researchers should carefully consider the best way to write their report to
ensure that it includes all necessary information. They designed guidelines for reporting a
mixed methods studyin education research with these components: state the research ques-
tion(s),bothquantitativeandqualitative;assesstheappropriatenessofusingmixedmethods

8 Kopaˇ candHlebec
research given the study’s purposes; choose a specific mixed methods research design and
supply the rationale for that choice; gather both quantitative and qualitative data; analyze
the quantitative data with the appropriate statistical test and analyze the qualitative data us-
ing the most suitable qualitative analysis; interpret the data from a quantitative, qualitative,
and blended perspective; legitimize the data by demonstrating how it simultaneously meets
quantitativeandqualitativecriteriaforquality;andmakeconclusionsthatemanatefromthe
insightsaffordedbythemixedmethodsapproach.
The American Evaluation Association (Glover-Kudon, 2010) introduced steps involved
intherationaleandpurposeofmixed-methodsresearch,dataprioritization,implementation
sequence,dataintegration,strengths,andchallenges. Allfourofthementionedassociations
stresstheimportanceofthe: purpose/justification,researchquestion,mixedmethodsdesign,
sampling, methods (data collection and data analysis), and integration. A detailed descrip-
tion of sampling is given by the American Psychological Association and the American
EducationalResearchAssociation.
These three groups of guidelines, recommendations, and models permitted an insight
into the options researchers have available when relying on mixed methods methodology.
We also obtained a list of components that make up these guidelines. Some components
repeat, but in the sections below we look in detail and extract components of relevance to
ourconceptmodel.
3. Qualitycriteriaformixedmethods
Quality is a recent theme in mixed methods literature. Tashakkori and Teddlie (2009)
stated that high-quality quantitative and qualitative components are necessary but not suffi-
cient for a quality mixed methods study. The question is whether there should be separate
quality appraisal criteria for quantitative and qualitative methods or mixed methods criteria
(O’Cathain, 2010). Ivankova (2014) concluded that, although many mixed methods quality
frameworks exist, there are no agreed criteria for evaluating mixed methods studies. Doyle
et al. (2016) determined that quality appraisal is a more common topic in debates of recent
timesonthedevelopmentofmixedmethodsandestablishedthatattentionisfocusedondata
analysis,integration,andqualityappraisal. FabreguesandMolina-Azorin(2017)statedthat
qualityisoneofthemoststronglydebatedtopicsinrecentmixedmethodsliteratureandthat
a growing number of authors is interested in conceptualizing and operationalizing the qual-
ity of mixed methods research. These authors suggest that mixed methods research should
beappraisedaccordingtoitsownsetofqualitycriteriagiventhatithasanumberoffeatures
thatdistinguishitfromsingle-methodresearch.
Several researchers have proposed quality criteria to assess the quality of mixed meth-
ods (O’Cathain, 2008; Bryman, 2014; Creamer, 2018; Hong et al., 2018). For example,
O’Cathain et al. (2008) devised a quality assessment framework consisting of six individ-
ual components, each entailing the assessment: of studies’ success, of the studies’ mixed
methodsdesigns,ofthequantitativecomponentofmixedmethodsstudies,ofthequalitative
component of mixed methods studies, of integration in mixed methods studies, and of the
inferences established in completed reports of mixed methods studies. Each of these six
assessmentsiscomposedofqualityquestionsofferingtheoptions“yes”,“yes,butimprove-
mentsarepossible”,“no”,“notenoughinformation”,and“notapplicable”.
• Assessment of the success of mixed methods studies entails the following questions:
“Is the quantitative component feasible?”, “Is the qualitative component feasible?”,
“Have both the qualitative and quantitative components been completed?”, “Is the

Qualityguidelinesformixedmethodsresearchininterventionstudies 9
mixed methods design feasible?”, “Were some qualitative methods planned but not
executed?”, “Were some qualitative methods planned but not executed?”, and “Did
themixedmethodsdesignworkinpractice?”.
• Assessment of the mixed methods design of studies contains the following questions:
“Is the use of mixed methods research justified?”, “Is the design for mixing meth-
ods described?” “Priority? Purpose? Sequence? Stage of integration?”, “Is the de-
sign clearly communicated?”, “Is the design appropriate for addressing the research
questions?”, and “Has the design’s rigor been considered (proposal) or adhered to
(report)?”.
• Assessment of the quantitative and qualitative component of mixed methods studies
attracts the following questions: “Is the role of each method clear?”, “Is each method
described in sufficient detail?”, “Is each method appropriate for addressing the re-
search question?”, “Is the approach to the sampling and analysis appropriate for its
purpose?”, “Is there expertise among the applicants/authors?”, “Is there expertise on
the team to undertake each method?”, “Have issues of validity been addressed for
each method?”, “Has the rigor of any method been compromised?”, “Is each method
sufficiently developed for its purpose?”, and “Is the (intended) analysis sufficiently
sophisticated?”.
• Assessment of integration in mixed methods studies raises the following questions:
“Is the type of integration stated?”, “Is the type of integration appropriate to the de-
sign?”, “Has enough time been allocated for the integration?”, “Is the approach to
the integration detailed in terms of working together as a team?”, “Does the dissem-
ination strategy detail how the mixed methods will be reported in final reports and
peer-reviewed publications?”, “Are the personnel who are to participate in the inte-
gration clearly identified?”, “Did appropriate members of the team participate in the
integration?”,“Isthereevidenceofcommunicationwithintheteam?”,and“Hasrigor
beencompromisedbytheprocessofintegration?”.
• Assessment of the inferences made in completed reports of mixed methods studies
brings the following questions: “Is there clarity about which results have emerged
from which methods?”, “Are the inferences appropriate?”, and “Are the results of all
themethodsconsideredsufficientlyintheinterpretation?”.
Bryman (2014) determined that most quality frameworks are too large and overly com-
plex and therefore presented his own list of six core criteria for assessing the quality of
mixed methods research, namely: implementation of the quantitative and qualitative com-
ponents of the mixed methods project in a technically competent manner; transparency; the
link of the mixed methods to the research questions; being explicit about the nature of the
mixed methods design employed; and a rationale for the use of the mixed methods research
and integration. Bryman’s criteria are comparable with the components of the guidelines
for mixed methods and O’Cathain’s framework for a quality assessment. Both Bryman and
O’Cathain expose the quantitative and qualitative components, the mixed methods design,
and the integration. In our conceptual model, we shall include five quality criteria taken
from Bryman: quantitative and qualitative components, research questions, mixed methods
design,rationale,andintegration.
Elisabeth Creamer (2018) described criteria for the mixed methods evaluation rubric
(MMER) and emphasized the importance of identifying and explaining any inconsistencies

10 Kopaˇ candHlebec
between the qualitative and quantitative data and the need to explain any inferences (Ta-
ble 1). She distinguishes “framework” and “rubric” and concludes that they are both used
forevaluationqualityandbothspecifycriteriaforevaluatingquality. However,arubricuses
numbers and establishes levels of quality. On the other side, a framework does not specify
levels of quality, is typically dichotomous, and used for evaluating the presence or absence
ofacriterion.
Table1: Thecriteriaformixedmethodsevaluationrubric(Creamer,2018)
Evaluation
criteria
0 1 2 3
Rationalefor
usingmixed
methods
Norationale Rationalewith
nocitations
Rationalewith
anycitations
Rationalewith
citationsfrom
mixedmethods
literature
Research
questions
Noresearch
questions
General
research
question(s)
Atleastone
quantitative
andone
qualitative
research
question
Specificmixed
method
research
question
Phasesof
mixing
Nomixing Mixinginone
phaseonly
Mixingintwo
phases
Mixinginthree
ormorephases
Degreeof
comparisons
Nocomparison Mention
comparison,
butnoevidence
ofcomparison
Comparisonfor
similarities
only
Comparisonfor
similaritiesand
differences
Hongetal. (2018)conceptualizedacriticalappraisaltoolforassessingthemethodolog-
ical quality of mixed methods studies and proposed the following methodological quality
criteria: (1) Is there an adequate rationale for the mixed methods design to address the re-
search question?; (2) Are the different components of the study effectively integrated so as
to answer the research question?; (3) Are the outputs of the integration of the qualitative
and quantitative components adequately interpreted?; (4) Are the divergences and inconsis-
tencies between quantitative and qualitative results adequately addressed?; and 5) Do the
differentcomponentsofthestudyadheretothequalitycriteriaofeachtraditionofthemeth-
ods involved? Possible responses were: “Yes”, “No”, “Can’t tell”, and “Comments”. In
ourconceptualmodel,weshallrelyonallfivemethodologicalqualitycriteria: rationale,in-
tegration, interpretation of outputs of the integration, divergencies and inconsistencies, and
adherencetothequalitycriteria.
These four groups of quality criteria provided us with an insight into different quality
criteriainmixedmethodsmethodology. Wealsoobtainedalistofthecomponentsthatmake
up these quality criteria. Some components repeat. We shall look in detail at where various
qualitycriteriaoverlapandextractcomponentsofrelevancetoourconceptualmodel.

Qualityguidelinesformixedmethodsresearchininterventionstudies 11
4. Guidelinesforinterventionstudies
Interventions are treatments or strategies for improving outcomes and making a differ-
ence (Mazurek Melnyk and Morrison-Beedy, 2012). Experimental research is the type of
research that allows conclusions to be made on whether cause-and-effect relationships exist
between an intervention or treatment and an outcome. A randomized controlled trial is a
type of experimental design that requires three components: an intervention or treatment, a
comparison or control group, and the random assignment of participants to experimental or
comparison/control groups (Mazurek Melnyk and Morrison-Beedy, 2012). A quasi experi-
ment is a study in which the independent variable is manipulated, but where there is a lack
ofatleastoneoftheothertwopropertiesofatrueexperiment(i.e.,randomassignmentora
comparison/controlgroup)(MazurekMelnykandMorrison-Beedy,2012).
Several researchers have proposed guidelines for intervention studies. Schulz et al.
(2010) developed the Consolidated Standards of Reporting Trials – CONSORT – which is
a guideline for randomized trials. CONSORT contains 25 checklist items. It provides guid-
ance for reporting all randomized controlled trials, although its aim is to provide guidance
for individually randomized, two-group, parallel trials; namely, the most common design
type. TheCONSORTchecklistisdividedintofivetopics:
• introduction–backgroundandobjectives;
• methods–trialdesign,participants,interventions,outcomes,samplesize,randomiza-
tion,allocation,implementation,blinding,statisticalmethods;
• results–participantflow,recruitment,baselinedata,numbersanalyzed,outcomesand
estimation;
• discussion–limitations,generalizability,interpretation;and
• otherinformation.
Hoffmann et al. (2014) proposed the Template for Intervention Description and Repli-
cation – TIDieR. The TIDieR checklist’s purpose is to encourage authors to describe in-
terventions in sufficient detail to enable their replication. The checklist is composed of
recommendeditemstousewhiledescribinganintervention. Accordingtotheauthors,most
TIDieRitemsarerelevantformostinterventionsandapplicabletoeverysimpleintervention.
TheTIDieRchecklistcontains12items:
• briefname: providethenameoraphrasethatdescribestheintervention;
• why: describe any rationale, theory, or goal of the elements essential to the interven-
tion;
• what (materials): describe any physical or informational materials used in the inter-
vention;
• what(procedures): describeeachprocedure,activity,and/orprocessusedintheinter-
vention;
• whoprovided: foreachcategoryofinterventionprovider;
• how: describethemodesofdelivery;

12 Kopaˇ candHlebec
• where: describethetype(s)oflocation(s)wheretheinterventionoccurred;
• whenandhowmuch: describethenumberoftimestheinterventionwasdeliveredand
overwhatperiodoftime;
• tailoring: iftheinterventionwasplannedtobepersonalized;
• modifications: iftheinterventionwasmodifiedduringthecourseofthestudy;
• howwell(planned): ifinterventionadherenceorfidelitywasassessed;and
• howwell(actual): ifinterventionadherenceorfidelitywasassessed.
On the other side, Borek et al. (2015) suggested GB-BCIs – a checklist to improve
the reporting of group-based behavior-change interventions, which also acts as a guideline
for intervention research. Borek et al. (2015) developed a checklist of elements for the
adequate reporting of GB-BCIs. The checklist consists of 26 essential elements that cover
theinterventiondesign,interventioncontent,participants,andfacilitators.
Besides the MMAT critical appraisal tool for appraising the methodological quality of
mixed methods studies (described in Section 3), Hong developed an appraisal tool for as-
sessingrandomizedcontrolledtrialsandnon-randomizedstudies. Possibleresponsesarethe
sameasformixedmethodsstudies: “Yes”,“No”,“Can’ttell”,and“Comments”.
• For randomized controlled trials, the authors propose five methodological quality cri-
teria: Wastherandomizationperformedappropriately?;Arethegroupscomparableat
the baseline?; Are there complete outcome data?; Are the outcome assessors blinded
totheinterventionprovided?;Didtheparticipantsadheretotheassignedintervention?
• Forappraisingthemethodologicalqualityofnon-randomizedstudies,theauthorspro-
pose five methodological quality criteria: Are the participants representative of the
targetpopulation?;Arethemeasurementsappropriateregardingboththeoutcomeand
intervention (or exposure)?; Are there complete outcome data?; Are the confounders
accounted for in the design and analysis?; During the study period, was the interven-
tionadministered(orexposedoccurred)asintended?
All checklists, templates, and critical appraisal tools for intervention studies are impor-
tant for our conceptual model. In the next sections we consider where elements of these
checklists, templates, and appraisal tools repeat and overlap and then extract elements for
theconceptualmodel.
5. Guidelinesandmodelsformixedmethodsresearchininterventionstudies
In recent years, authors have investigated the use of mixed methods in intervention re-
search. They have proposed basic procedures while implementing a mixed methods experi-
mentaldesign(CreswellandPlanoClark,2018)andpracticalguidanceforusingqualitative
researchwithanRCT(O’Cathain,2018).
Creswell and Plano Clark (2018) conceptualized basic procedures in realizing a mixed
methods experimental design that can be divided into three steps: determine how the quali-
tativedatawillbeusedintheexperiment;conductthequantitativeexperiment,andestablish
howthequalitativefindingsenhancetheexperiment. Inthefirststepofestablishinghowthe
qualitative data are to be used in the experiment, the researcher ascertains why qualitative

Qualityguidelinesformixedmethodsresearchininterventionstudies 13
informationisneeded,howitwillbeusedandwhetherthequalitativedatawillbecollected
before, during, or after the intervention (or in all three phases). In the second step of the
quantitative experiment, the researcher specifies the conceptual model guiding the design,
assignsparticipantsto theexperimentalandcontrolgroups, designsandimplementsthe ex-
perimental treatment, measures the outcome variables, analyzes the quantitative data using
descriptivestatistics,inferentialstatistics,andeffectsizestoanswerthequantitativeresearch
questions, collects and analyzes the qualitative data where it was placed in the experiment
toanswerthequalitativeresearchquestions,andundertakeprocedurestointegratethequan-
titative and qualitative results based on the reason for including the qualitative data. In the
thirdstepofestablishinghowthequalitativefindingsenhancetheexperiment,theresearcher
gives specific evidence of use of the qualitative findings and interprets how the qualitative
findingsenhancetheexperiment.
The basic procedures presented by Creswell and Plano Clark are valuable for our con-
ceptual model. We will include: the rationale for using qualitative research, measurement
of qualitative and quantitative methods (time of measurement), participants (assignment to
experimental and control groups), collection and analysis of qualitative and quantitative
methods, and integration. The authors introduced basic procedures for experimental de-
signs generally without providing specific instructions for an experimental design and any
instructions on how to construct mixed methods in experimental designs. There are no pro-
cedures to guide how to use mixed methods in quasi-experimental designs. Interventions
canbeverycomplexandtheauthorssuggestedgeneralguidelinesonhowtousequalitative
methods in experimental designs, yet they do not consider that some interventions may be
verycomplex.
O’Cathain(2018)identifiedsevenpracticalguidelinesforusingqualitativeresearchwith
an RCT: the rationale for the qualitative research, research questions, methods of data col-
lection, participants, sampling, analysis, and integration with the RCT. Rationale means de-
scribing why the qualitative research will add value to the RCT (O’Cathain, 2018). The re-
searchquestionisusedtoguidetheapplicationofthequalitativeresearchinfeasibilitystud-
ies. O’Cathain (2018) suggests that the author should consider a broad range of questions,
prioritizetheinitialquestions,andallowforthepossibilityofnewquestionstoemerge. The
author focuses on research questions addressed by qualitative research. O’Cathain (2018)
concludes that researchers employ different data collection methods as included along with
RCTs. The author states that researchers often use methods in combination and that the
selection of methods depends on the research questions. Regarding participants, O’Cathain
stated that who is included in the data collection depends on the research question, inter-
vention, context, and available resources. With respect to sampling, the author explored
whichtypeofsamplingisemployedandfoundthatpurposivesamplingandmaximumvari-
ation sampling are commonly used and that the type of RCT can influence the sampling.
Sampling exists of participants, groups, sites, and stage of intervention. O’Cathain (2018)
stated that the number of people included in qualitative research depends on the phase of
evaluation, the aims of the qualitative research, the size of the RCT, data saturation, and
the availableresources. Many approaches to analyzing qualitative data withRCTs are men-
tionedbyO’Cathain(thematicanalysis,iterativeapproach,interpretativephenomenological
analysis, content analysis). Further, O’Cathain (2018) mentioned integration in a phased
sequentialdesignandintegrationinasingle-phaseconcurrentdesignandreviewedthreein-
tegration techniques: integrating findings using an adapted triangulation protocol, working
between the findings and the data sets by ‘following the thread/s’ and combining data using
amixedmethodsmatrix. Inourconceptualmodel,weshallincludeallsevenofO’Cathain’s



Qualityguidelinesformixedmethodsresearchininterventionstudies 15
studies, the integration in mixed methods studies, and of the inferences drawn in completed
reports of mixed methods studies. We then chose quality questions to be included on our
list: “Is the use of mixed methods research justified?”, “Is the design for mixing methods
described? Priority? Purpose? Sequence? Stage of integration?”, “Is the design clearly
communicated?”, “Is the design appropriate for addressing the research questions?”, “Is
eachmethoddescribedinsufficientdetail?”,“Iseachmethodappropriateforaddressingthe
researchquestion?”,“Istheapproachtosamplingandanalysisappropriateforitspurpose?”,
“Is the type of integration stated?”, “Is the type of integration appropriate to the design?”,
“Is there clarity about which results have emerged from which methods?”, and “Are the
inferencesappropriate?”.
Five of Bryman’s quality criteria were included on the list: quantitative and qualita-
tive components, research questions, mixed methods design, rationale, and integration. The
mixed methods evaluation rubric (Creamer, 2018) is important for our conceptional model.
We also take the logic of the rubric into account; namely, it uses numbers and establishes
levels of quality. Thus far, we have just encountered frameworks in our previous guidelines
because the quality evaluation was dichotomous and the presence or absence of a criterion
was being evaluated. In our list, we included all four of Creamer’s mixed methods evalu-
ation criteria: rationale, research questions, phases of mixing, degree of comparisons. Our
conceptual model relies on all five methodological quality criteria from the MMAT model:
rationale,integration,interpretationofoutputsoftheintegration,divergenciesandinconsis-
tencies,andadherencetothequalitycriteria.
Guidelines for intervention studies were also included in the conceptual model. From
CONSORT, we added the following items to the comprehensive list: trial design, partici-
pants, interventions, outcomes, sample size, randomization, implementation, blinding, sta-
tistical methods, participant flow, recruitment, baseline data, numbers analyzed, outcomes,
and estimation. From TIDieR, items added to the list are: brief name, why (rationale, the-
ory or goal), what (materials, procedures), who provided, how (modes of delivery), where,
when and how much, tailoring, modifications, and how well. Our list also includes several
items from the GB-BCIs model: intervention design, intervention content, participants, and
facilitators. Quality criteria for randomized and non-randomized studies contain the items:
randomization,groups(intervention,control),participants,andmeasurements.
6.2. Dataextraction
We recorded the frequency of each item across all guidelines. Items most frequently
included in the guidelines and quality criteria for mixed methods related to rationale/ jus-
tification, research question, mixed methods design, sampling, methods, and integration.
Table 2 shows the number of general guidelines, specific guidelines, guidelines in associa-
tionsandqualitycriteriaformixedmethodsandthenumberofitemsmostoftenincludedin
theseguidelinesandqualitycriteria.
Itemsfromfourguidelinesforinterventionstudieswerealsoaddedtothecomprehensive
list. There are not many overlapping item. We will include the following items in the
conceptualmodel: trialdesign,participants,groups(interventionandcontrol)interventions,
intervention content, sample size, randomization, statistical methods, recruitment, baseline
data,measurements,numbersanalyzed.
Our next step was to include the extracted items on the list in the conceptual model. We
madeaplanforhowtoorganizetheitemsingroups. Werevisedthechecklistandalsoadded
itemsforquantitativeandqualitativemethodsandgeneralinformationabouttheresearch.

16 Kopaˇ candHlebec
Table2: Thenumberofitemsmostfrequentlyincludedinguidelines,recommendations,models,
andqualitycriteriaformixedmethods
General
guidelines,rec-
ommendations
andmodels
(N=13)
Specific
guidelines,rec-
ommendations
andmodels
(N=10)
Guidelinesin
associations
(N=4)
Qualitycriteria
formixed
methods
(N=4)
Rationale/
justification
10 7 4 4
Research
question
9 5 3 4
Mixedmethods
design
10 9 4 4
Sampling 7 7 3 4
Methods(data
collection,data
analysis)
9 9 4 4
Integration 11 9 4 4
7. Conceptualmodelforassessingmixedmethodsininterventionstudies–qualitycri-
teria
We constructed the conceptual model and divided it into three sections: (1) topic; (2)
checklist items; and (3) item description. The model contains five topics: (1) research –
11 items which describe general data about the research; (2) intervention – 12 items which
describe the intervention; (3) quantitative methods – 21 items which describe quantitative
methods; (4) qualitative methods – 22 items which describe qualitative methods; and (5)
mixedmethods–16itemswhichdescribemixedmethods. Eachitemisavariable(nominal,
ordinalornumeric).
Each topic consists of checklist items (Table 3). The first topic “Research” contains the
followingitems: typeofresearch,fieldofstudy,purpose/aim/goaloftheresearch,numberof
studies,numberofphases,methods,ethicalapproval,informedconsent,rewards/incentives,
researchdurationandflowchatofthestudydesign. Thesecondtopic“Intervention”contains
theitems: typeofstudy,interventionphase,typeofintervention,typeofexperiment,typeof
RCT,interventionstandards,numberofgroups,interventiongroup,size,controlgroup,size,
randomization,allocation. Thethirdtopic“Quantitativemethods”containstheitems: meth-
ods,instruments,numberofmeasurements,numberofpre-interventionmeasurements,num-
berofbetween-interventionmeasurements,numberofpost-interventionmeasurements,time
betweenend-of-interventionandpost-interventionmeasurements,datacollectioninstrument
1, data collection instrument 2, data collection instrument 3. “Qualitative methods” is the
fourthtopicandcontainstheitems: methods,instruments,numberofmeasurements,number
of pre-intervention measurements, number of between-intervention measurements, number
of post-intervention measurements, time between end-of-intervention and post-intervention
measurements,datacollectioninstrument1,datacollectioninstrument2,datacollectionin-
strument 3. “Mixed methods” is the fifth topic in our conceptual model and contains the:

Qualityguidelinesformixedmethodsresearchininterventionstudies 17
mixed methods, mixed methods design, description, rationale/justification for mixed meth-
ods,mixedmethodsresearchquestion,levelofmixedmethods,levelofinteractionbetween
quantitativeandqualitativestrands,mixingofmethods,priority,timing,integration,integra-
tionwhere,integrationhow,datatransformation,andunits.
Each checklist item is either a categorical, ordinal, or numerical variable. Under the
itemdescriptionChecklistitemsintheconceptualmodelwecanfindwhicharethecategor-
ical variables that have two, three, or more values. The checklist items which present the
time of measurements are ordinal variables. Numeric variables are the number of groups,
interventiongroups,andcontrolgroups.
Inthenextresearchstep,weplantoundertakeasystematicreviewofarticlesinSpringer-
Link, PubMed, and Web of Science which use mixed methods methodology in intervention
studies. We will apply our conceptual model as we conduct this systematic review. The
conceptual model will serve as a coding scheme with quantitative and qualitative data. At
the end, we will analyze these data and obtain results which will amount to a quality as-
sessment of the use of mixed methods in intervention studies. The results of the conceptual
model will be used to develop guidelines for researchers who employ mixed methods in
interventionstudies.
Table3: Componentsoftheconceptualmodel
Topic Itemdescription
Research
1.1 Typeofresearch 1=protocol//2=researcharticle//3=pilotstudy
1.2 Fieldofstudy 1=health//2=psychology//3=education
1.3 Purpose/aim/goal of
research
1.4 No. ofstudies 1 = one intervention, one phase // 2 = one intervention,
multi-stage//3=twodifferentstudies
1.5 No. ofphases 1=one//2=two//3=three
1.6 Methods 1 = single method qual // 2 = single method quant // 3 =
multi method qual // 4 = multi method quant // 5 = mixed
methods
1.7 Ethicalapproval 1=yes//0=no
1.8 Informedconsent 1 = yes // 0 = no If 1=yes 1=verbal, 2=written, 3=verbal
andwritten
1.9 Rewards,incentives 1=yes//0=no
1.10 Researchduration
1.11 Flowchart of study
design
1=yes//0=no
Intervention
2.1 Typeofstudy 1 = intervention // 2 = evaluation // 3 = intervention and
evaluation
continued...

18 Kopaˇ candHlebec
Topic Itemdescription
2.2 Interventionphase 1=pilotintervention//2=interventiondevelopment//3=
interventionimplementation//4=processevaluation//5=
interventionevaluation
2.3 Typeofintervention 1=experiment//2=quasi-experiment//3=notwritten
2.4 Type of experiment
(if experimental de-
sign)
1 = randomized controlled trial // 2 = parallel design // 3
=multiple-baselinedesign//4=crosssectionalresearch//
5 = prospective study // 6 = longitudinal design // 7 = ret-
rospective study // 8 = pretest–posttest design // 9 = before
andafterdesign//10=observationalstudy//11=stepped-
wedgedesign
2.5 Type of RCT (if
RCT)
1=explanatoryRCT//2=pragmaticRCT//3=feasibility
RCT // 4 = pilot RCT // 5 = exploratory RCT // 6 = cluster
RCT//7=Nof1trials//8=factorialRCT//9=adaptive
RCT
2.6 Intervention stan-
dards
1=CONSORT//2=REAIM//9=NOTSPECIFIED
2.7 No. ofgroups 1=1//2=2//3=3
2.8 Interventiongroup 1=1//2=2//3=3
2.9 Size 1=yes//2=no
2.10 Controlgroup
2.11 Size 1=yes//2=no
2.12 Randomization 1 = simple randomization // 2 = block randomization // 3
= stratified block randomization // 4 = dynamic (adaptive)
randomallocation
2.13 Allocation 1=randomallocation//2=non-randomallocation
Quantitativemethods
3.1 Methods 1=primary//2=secondary
3.2 Instruments 1=survey//2=dataarchives//3=documents
3.3 No. of measure-
ments
1=1//2=2//3=3//4=4//5=5//6=6ormore
3.4 No. of pre-
intervention mea-
surements
1=1//2=2//3=3//4=4//5=5//6=6ormore
3.5 No. of between-
intervention mea-
surements
1=1//2=2//3=3//4=4//5=5//6=6ormore
3.6 No. of post-
intervention mea-
surements
1=1//2=2//3=3//4=4//5=5//6=6ormore
continued...

Qualityguidelinesformixedmethodsresearchininterventionstudies 19
Topic Itemdescription
3.7 Time between end
of intervention and
post intervention
measurements (in
months)
1=oneweek//2=twoweeks//3=threeweeks//4=one
month
3.8 Data collection in-
strument1
1 = survey // 2 = data archives // 3 = chart audits // 4 =
analytics//5=other
3.9 Data collection in-
strument2
1 = survey // 2 = data archives // 3 = chart audits // 4 =
analytics//5=other
3.10 Data collection in-
strument3
1 = survey // 2 = data archives // 3 = chart audits // 4 =
analytics//5=other
Participants1
3.11 Recruitmentstrategy 1 = recruiting via personal and professional connections //
2 = recruiting via fliers // 3 = recruiting via advertisements
//4=recruitingviaemailsandletters
3.12 Recruitment dura-
tion
3.13 Samplingsite 1 = school district // 2 = municipality // 3 = rural based lo-
cation//4=city-basedlocation//5=hospital//6=general
practices//7=carefacilities//8=state//9=other
3.14 Unitsofobservation 1=individuals//2=groups
3.15 Samplingstrategy 1=probabilitysampling//2=non-probabilitysampling
3.16 Inclusioncriteria 1=yes//0=no
3.17 Exclusioncriteria 1=yes//0=no
3.18 Data collection in-
struments
1 = survey // 2 = data archives // 3 = chart audits // 4 =
analytics//5=other
3.19 Data analysis
method
1=descriptivestatistics//2=inferentialstatistics
3.20 Data analysis soft-
ware
1=SPSS//2=R//3=STATA//4=other
3.21 Statisticalpowercal-
culations(3.2
1=yes//2=no
Participants2
3.22 Recruitmentstrategy 1 = recruiting via personal and professional connections //
2 = recruiting via fliers // 3 = recruiting via advertisements
//4=recruitingviaemailsandletters
3.23 Recruitment dura-
tion
3.24 Samplingsite 1 = school district // 2 = municipality // 3 = rural-based lo-
cation//4=city-basedlocation//5=hospital//6=general
practices//7=carefacilities//8=state//9=other
continued...

20 Kopaˇ candHlebec
Topic Itemdescription
3.25 Unitsofobservation 1=individuals//2=groups
3.26 Samplingstrategy 1=probabilitysampling//2=non-probabilitysampling
3.27 Inclusioncriteria 1=yes//0=no
3.28 Exclusioncriteria 1=yes//0=no
3.29 Data collection in-
struments
1 = survey // 2 = data archives // 3 = chart audits // 4 =
analytics//5=other
3.30 Data analysis
method
1=descriptivestatistics//2=inferentialstatistics
3.31 Data analysis soft-
ware
1=SPSS//2=R//3=STATA//4=other
3.32 Statisticalpowercal-
culations
1=yes//2=no
Participants3
3.33 Recruitmentstrategy 1 = recruiting via personal and professional connections //
2 = recruiting via fliers // 3 = recruiting via advertisements
//4=recruitingviaemailsandletters
3.34 Recruitment dura-
tion
3.35 Samplingsite 1 = school district // 2 = municipality // 3 = rural-based lo-
cation//4=city-basedlocation//5=hospital//6=general
practices//7=carefacilities//8=state//9=other
3.36 Unitsofobservation 1=individuals//2=groups
3.37 Samplingstrategy 1=probabilitysampling//2=non-probabilitysampling
3.38 Inclusioncriteria 1=yes//0=no
3.39 Exclusioncriteria 1=yes//0=no
3.40 Data collection in-
struments
1 = survey // 2 = data archives // 3 = chart audits // 4 =
analytics//5=other
3.41 Data analysis
method
1=descriptivestatistics//2=inferentialstatistics
3.42 Data analysis soft-
ware
1=SPSS//2=R//3=STATA//4=other
3.43 Statisticalpowercal-
culations
1=yes2=no
Qualitativemethods
4.1 Methods 1=primary//2=secondary
4.2 Instruments 1 = interview // 2 = focus group // 3 = observation // 4 =
casestudy//5=fieldnotes//6=workshop
4.3 No. of measure-
ments
1=1//2=2//3=3//4=4//5=5//6=6ormore
continued...

Qualityguidelinesformixedmethodsresearchininterventionstudies 21
Topic Itemdescription
4.4 No. of pre-
intervention mea-
surements
1=1//2=2//3=3//4=4//5=5//6=6ormore
4.5 No. of between- in-
tervention measure-
ments
1=1//2=2//3=3//4=4//5=5//6=6ormore
4.6 No. of post- in-
tervention measure-
ments
1=1//2=2//3=3//4=4//5=5//6=6ormore
4.7 Time between end
of intervention and
post intervention
measurements (in
months)
1=oneweek//2=twoweeks//3=threeweeks//4=one
month
4.8 Data collection in-
strument1
1 = interview // 2 = focus groups // 3 = observation // 4 =
other
4.9 Data collection in-
strument2
1 = interview // 2 = focus groups // 3 = observation // 4 =
other
4.10 Data collection in-
strument3
1 = interview // 2 = focus groups // 3 = observation // 4 =
other
Participants1
4.11 Recruitmentstrategy 1 = recruiting via personal and professional connections //
2 = recruiting via fliers // 3 = recruiting via advertisements
//4=recruitingviaemailsandletters
4.12 Recruitment dura-
tion
4.13 Samplingsite 1 = school district // 2 = municipality // 3 = rural-based
location//4=city-basedlocation//5=hospital6=general
practices//7=carefacilities//8=state//9=other
4.14 Unitsofobservation 1=individuals//2=groups
4.15 Samplingstrategy 1=purposefulsampling//2=conveniencesampling
4.16 Inclusioncriteria 1=yes//0=no
4.17 Exclusioncriteria 1=yes//0=no
4.18 Data collection in-
struments
1 = interview // 2 = focus groups // 3 = observation // 4 =
other
4.19 Data analysis
method
1=contentanalysis//2=narrativeanalysis//3=discourse
analysis//4=groundedtheory
4.20 Data analysis soft-
ware
1=Atlas.ti//2=Nvivo//3=MAXQDA//4=other
4.21 No. ofraters 1=1//2=2//3=3ormore
4.22 Interrateragreement 1=yes//2=no
continued...

22 Kopaˇ candHlebec
Topic Itemdescription
Participants2
4.23 Recruitmentstrategy 1 = recruiting via personal and professional connections //
2 = recruiting via fliers // 3 = recruiting via advertisements
//4=recruitingviaemailsandletters
4.24 Recruitment dura-
tion
4.25 Samplingsite 1 = school district // 2 = municipality // 3 = rural-based lo-
cation//4=city-basedlocation//5=hospital//6=general
practices//7=carefacilities//8=state//9=other
4.26 Unitsofobservation 1=individuals//2=groups
4.27 Samplingstrategy 1=purposefulsampling//2=conveniencesampling
4.28 Inclusioncriteria 1=yes//0=no
4.29 Exclusioncriteria 1=yes//0=no
4.30 Data collection in-
struments
1 = interview // 2 = focus groups // 3 = observation // 4 =
other
4.31 Data analysis
method
1=contentanalysis//2=narrativeanalysis//3=discourse
analysis//4=groundedtheory
4.32 Data analysis soft-
ware
1=Atlas.ti//2=Nvivo//3=MAXQDA//4=other
4.33 No. ofraters 1=1//2=2//3=3ormore
4.34 Interrateragreement 1=yes//2=no
Participants3
4.35 Recruitmentstrategy 1 = recruiting via personal and professional connections //
2 = recruiting via fliers // 3 = recruiting via advertisements
//4=recruitingviaemailsandletters
4.36 Recruitment dura-
tion
4.37 Samplingsite 1 = school district // 2 = municipality // 3 = rural-based lo-
cation//4=city-basedlocation//5=hospital//6=general
practices//7=carefacilities//8=state//9=other
4.38 Unitsofobservation 1=individuals//2=groups
4.39 Samplingstrategy 1=purposefulsampling//2=conveniencesampling
4.40 Inclusioncriteria 1=yes//0=no
4.41 Exclusioncriteria 1=yes//0=no
4.42 Data collection in-
struments
1 = interview // 2 = focus groups // 3 = observation // 4 =
other
4.43 Data analysis
method
1=contentanalysis//2=narrativeanalysis//3=discourse
analysis//4=groundedtheory
4.44 Data analysis soft-
ware
1=Atlas.ti//2=Nvivo//3=MAXQDA//4=other
4.45 No. ofraters 1=1//2=2//3=3ormore
continued...

Qualityguidelinesformixedmethodsresearchininterventionstudies 23
Topic Itemdescription
4.46 Interrateragreement 1=yes//2=no
Mixedmethods
5.1 Mixedmethods 1 = true mixed methods // 2 = quasi mixed methods // 3 =
partiallymixedmethods
5.2 Mixed methods de-
sign
5.3 Description If1=yesname: If0=nodescriptive
5.4 Rationale/justification
formixedmethods
1=yes//0=no
5.5 Mixed methods re-
searchquestion
1=yes//0=no
5.6 Mixedmethods 1 = ON THE LEVEL OF UNITS (within phases, on same
participants qual and quant) // 2 = ON THE LEVEL OF
DESIGN (between phases, one phase is qual, one phase is
qual)//3=both
5.7 Level of interaction
between quan and
qual
1=independent//2=interactive
5.8 Mixingofmethods 1 = on the level of design // 2 = on the level of units // 3 =
both
5.9 Priority 1=quan//2=qual//3=equal
5.10 Timing 1=concurrent//2=sequential
5.11 Integration 1=yes//0=no
5.12 Integrationwhere 1=datacollection//2=analysis//3=interpretation//4=
combination
5.13 Integrationhow 1=merge//2=link//3=transform//4=other
5.14 Datatransformation 1=yes//2=no
If 1=yes 1=qual v quant 2=quanti v qual 3=both 4=none
5=notreported6=notapplicable
5.15 Units 1 = identical units // 2 = identical groups // 3 = partially
identical//4=different
5.16 Units 1=thesamelevelofunits//2=differentlevelofunits
8. Conclusion
Thispaperdescribesthreeguidelinesavailabletotheresearcherwhileusingmixedmeth-
odsresearchininterventionstudies: basicproceduresinimplementingamixedmethodsex-
perimentaldesign(CreswellandPlanoClark,2018),practicalguidanceforusingqualitative
research with an RCT (O’Cathain, 2018), and a mixed methods appraisal tool for apprais-
ingthemethodologicalqualityofrandomizedcontrolledtrials,non-randomizedstudies,and
mixed methods – MMAT (Hong et al., 2018). While on one hand these guidelines are too
general since they do not take different experimental and quasi-experimental designs into

24 Kopaˇ candHlebec
account, on the other hand they are too specific because they describe in too much detail a
certain experimental design – a randomized controlled trial. They also lack the relationship
betweenmixedmethodsresearchandinterventionstudies.
These three shortcomings explain the reason for developing a conceptual model for
mixed methods in intervention studies so as to overcome the deficiencies of the existing
guidelines. Thispaperdescribesthedevelopmentofaconceptualmodelbasedonthecurrent
guidelines,recommendations,models,andqualitycriteriaformixedmethods,guidelinesfor
intervention studies, and the existing limited guidelines for mixed methods in intervention
studies. The new conceptual model improves on the existing guidelines by containing all
experimental and quasi-experimental designs along with the relationship between mixed
methods research and intervention studies. In the next step, we shall assess the quality of
mixedmethodsresearchininterventionstudies.
We made an overview of guidelines, recommendations, models, and quality criteria for
mixed methods research and guidelines for intervention research. A checklist of all items
in these guidelines was then prepared. We recorded the frequency of each item across all
guidelines and extracted the items most frequently included in the guidelines and quality
criteria. In the next step, the extracted list items were included in a conceptual model. We
also included items for quantitative and qualitative methods and general information about
theresearch.
The conceptual model is the outcome of the overview and evaluation of guidelines, rec-
ommendations,models,andqualitycriteriaformixedmethodsandinterventionstudies. Itis
dividedintothreesectionsandcontainfivetopics: (1)research–11itemsdescribinggeneral
data about the research; (2) intervention – 12 items describing the intervention; (3) quanti-
tative methods – 21 items describing the quantitative methods; (4) qualitative methods – 22
items describing the qualitative methods; and (5) mixed methods – 16 items describing the
mixedmethods. Eachitemisavariable(nominal,ordinalornumeric).
Our conceptual model extends previous models in four respects, namely, it can be used:
(1) for all types of experimental and quasi-experimental designs; (2) for complex interven-
tions with many participants, sampling sites, and components; (3) in all types of disciplines
(health science, psychology, education, etc.); and (4) in all phases of intervention (develop-
ment,feasibility,implementationorprocessevaluation).
Another important contribution of our conceptual model is the assessment of mixed
methods research in intervention studies. We plan to make a systematic review of inter-
ventionsusingmixedmethodsresearchinthreedatabases: SpringerLink,PubMed,andWeb
ofScience. Resultsofthisassessmentwillhelpcreateguidelinesformixedmethodsresearch
ininterventionstudies.
The study has several limitations. We constructed the conceptual model on the basis of
relevant literature. It is possible that some guidelines for mixed methods in certain disci-
plines were not included because mixed methods methodology use is increasingly popular.
Second, guidelines for mixed methods may be published in the journals of associations that
we did not review. Another limitation is that the literature review only considered sources
in English and it is possible that guidelines have also been developed in other languages. A
review of guidelines for mixed methods in other languages and for other associations and
disciplines is essential to confirm that all important elements of the guidelines, recommen-
dations,andmodelsareincludedinourconceptualmodel.
Theoutcomeoftheevaluationoftheexistingguidelinesandmodelwastheconstruction
of a conceptual model. This model will be utilized to assess the quality of mixed methods
research in intervention studies to increase the methodological quality of mixed methods

Qualityguidelinesformixedmethodsresearchininterventionstudies 25
methodologyinfutureinterventionresearch.
References
[1] AlbrightK.,GechterK.,andKempeA.(2013): Importanceofmixedmethodsinprag-
matictrialsanddisseminationandimplementationresearch.AcademicPediatric,13(5),
400–407.
[2] Bamberger, M. (2012): Introduction to mixed methods in impact evaluation. Impact
Evaluation Notes, 3. The Rockefeller Foundation. http://www.interaction.org/
impact-evaluation-notes
[3] Boeije, H.R., Drabble, S.J., and O’Cathain, A. (2015): Methodological challenges of
mixedmethodsinterventionevaluations.Methodology,11(4),119–125.
[4] Borek,A.J.,Abraham,C.,Smith,J.R.,Greaves,C.J.,andTarrant,M.(2015): Acheck-
list to improve reporting of group-based behaviour-change interventions. BMC Public
Health,15(1),963.
[5] Bradt, J., Burns, D.S., and Creswell, J.W. (2013): Mixed methods research in music
therapyresearch.JournalofMusicTherapy,50(2),123–148.
[6] Brown,K.M.,Elliott,S.J.,Leatherdale,S.T.,andRobertson-Wilson,J.(2015): Search-
ingforrigourinthereportingofmixedmethodspopulationhealthresearch: Amethod-
ologicalreview.HealthEducationResearch,30(6),811–839.
[7] Bryman, A. (2014): June 1989 and beyond: Julia Brannen’s contribution to mixed
methodsresearch.InternationalJournalofSocialResearchMethodology, 17(2), 121–
131.
[8] Collins,K.M.,Onwuegbuzie,A.J.,andSutton,I.L.(2006): Amodelincorporatingthe
rationaleandpurposeforconductingmixed-methodsresearchinspecialeducationand
beyond.LearningDisabilities: AContemporaryJournal,4(1),67–100.
[9] Creamer, E.G. (2018): An introduction to fully integrated mixed methods research.
ThousandOaks,CA:Sage.
[10] Creswell, J. and Plano Clark, V. (2007): Designing and conducting mixed methods
research.ThousandOaks,CA:Sage.
[11] Creswell, J. and Plano Clark, V. (2011): Designing and conducting mixed methods
research.ThousandOaks,CA:Sage.
[12] Creswell, J.W. (2015): A concise introduction to mixed methods research. Thousand
Oaks,CA:Sage.
[13] Creswell,J.W.andPlanoClark,V.L.(2018): Designingandconductingmixedmethods
research.ThousandOaks,CA:Sage.
[14] Creswell,J.W.,Fetters,M.D.,andIvankova,N.V.(2004): Designingamixedmethods
studyinprimarycare.TheAnnalsofFamilyMedicine,2(1),7–12.

26 Kopaˇ candHlebec
[15] Doyle, L., Brady, A.M., and Byrne, G. (2016): An overview of mixed methods re-
search.JournalofResearch,14(2),175–185.
[16] Drabble, S.J. and O’Cathain, A. (2015): Moving from randomized controlled trials to
mixed methods intervention evaluations. In S. Hesse-Biber and R.B. Johnson (Eds):
The Oxford Handbook of Multimethod and Mixed Methods Research, 406–425. New
York,NY:OxfordUniversityPress.
[17] Fabregues and Molina-Azorin (2017): Addressing quality in mixed methods research:
A review and recommendations for future agenda. Quality & Quality, 51(6), 2847–
2863.
[18] Farquhar,M.C.,Ewing,G.,andBooth,S.(2011): Usingmixedmethodstodevelopand
evaluate complex interventions in palliative care research. Palliative Medicine, 25(8),
748–757.
[19] Farquhar, M., Preston, N., Evans, C.J., Grande, G., Short, V., Benalia, H., Higginson,
I.J., and Todd, C. (2013): Mixed methods research in the development and evaluation
of complex interventions in palliative and end-of-life care: Report on the MORECare
ConsensusExercise.JournalofPalliativeMedicine,16(12),1550–1560.
[20] Fetters,M.D.andFreshwater,D.(2015): Publishingamethodologicalmixedmethods
researcharticle.JournalofMixedMethodsResearch,9(3),203–213.
[21] Gallo, J.J and Lee, S.Y. (2016): Mixed methods in behavioral intervention research.
In L.N. Gitlin and S.J. Czaja (Eds.): Behavioral intervention research – designing,
evaluatingandimplementing,195–212.NewYork,NY:Springer.
[22] Glover-Kudon, R. (2010): Guidelines for conducting a high-quality mixed-methods
dissertation. American Evaluation Association Annual Conference. https://eval.
connectedcommunity.org/HigherLogic/System/DownloadDocumentFile.
ashx?DocumentFileKey=39606eed-cb0c-4dd3-8807-27533ce85300
[23] Graff, J.C. (2017): Mixed methods research. In Hall, H.R. and L.A. Roussel (Eds.):
Evidence-based practice: An integrative approach to research, administration, and
practice,47–66.Burlington,MA:Jones&Barlett.
[24] Greene,J.(2008): Ismixedmethodssocialinquiryadistinctivemethodology? Journal
ofMixedMethodsResearch,2(1),7–22.
[25] Grissmer,D.W.(2016): Aguidetoincorporatingmultiplemethodsinrandomizedcon-
tolled trials to assess intervention effects. Washington, DC: American Psycolocigal
Association.
[26] Guest, G. (2012): Describing mixed methods research. Journal of Mixed Methods Re-
search,7(2),141–151.
[27] Guest, G. and Fleming, P.J. (2014): Mixed methods research. In G. Guest and E.
Namely(Eds.): PublicHealthResearchMethods,581–610.LosAngeles,CA:Sage.
[28] Hesse-Biber, S. (2012): Weaving a multimethodology and mixed methods praxis into
randomised controlled trials to enhance credibility. Qualitative Inquiry, 18(10), 876–
889.

Qualityguidelinesformixedmethodsresearchininterventionstudies 27
[29] Hoffmann, T.C., Glasziou P.P., Ruairidh, M., Moder, D., Barbour, V., Johnston, M.,
Lamb,S.,Dixon-Woods,M.,andWyatt,J.C.(2014): Betterreportingofinterventions:
Template for intervention description and replication (TIDieR) checklist and guide.
BritishMedicalJournal,7(348),g1687.
[30] Hong, Q.N., Gonzales-Reyes, A., and Pluye, P. (2018): Improving the usefulness of
a tool for appraising the quality of qualitative, quantitative and mixed methods stud-
ies, the Mixed Methods Appraisal Tool (MMAT). Journal of Evaluation in Clinical
Practice,24(3),459–467.
[31] Ivankova, N.V. (2014): Mixed methods applications in action research. From methods
tocommunityaction.ThousandOaks,CA:Sage.
[32] Jalongo, M.R. and Saracho, O.N. (2016): Writing for publications: Transitions and
toolsthatsupportscholars’success.Switzerland: Springer.
[33] Jimenez, E., Waddington, H., Goel, N., Prost, A., Pullin, A., White, H., Lahiri. S., and
Narain,A.(2018): Mixingandmatching: usingqualitativemethodstoimprovequanti-
tative impact evaluations(Ies) and systematic reviews (Srs)of development outcomes.
JournalofDevelopmentEffectiveness,10(4),400–421.
[34] Leech, N.L. (2012): Writing mixed research reports. American Behavioral Scientist,
56(6),866–881.
[35] Leech, N.L. and Onwuegbuzie, A.J. (2010): Journal article reporting standards for
qualitativeprimary,qualitativemeta-analytic,andmixedmethodsresearchinpsychol-
ogy: The APA Publications and Communications Board task force report. American
Psychologist,73(1): 26–46.
[36] Lewin, S., Glenton, C., and Oxman, A.D. (2009): Use of qualitative methods along-
siderandomisedcontrolledtrialsofcomplexhealthcareinterventions: Methodological
study.BritishMedicalJournal,339,b3496.
[37] Levitt, H.M., Bamberg, M., Creswell, J.W., Frost, D.M., Josselson, R., and Suarez-
Orozco, C. (2018): Journal article reporting standards for qualitative primary, qualita-
tive meta-analytic, and mixed methods research in psychology: The APA Publications
andCommunicationsBoarttaskforcereport.AmericanPsychologist,73(1),26–46.
[38] Maxwell, J.A. and Loomis, D. (2003): Mixed methods design: An alternative ap-
proach. In A. Tashakkori and C. Teddlie (Eds.): Handbook of mixed methods in social
andbehavioralresearch,241–271.ThousandOaks,CA:Sage.
[39] Maxwell, J.A. (2012): A realist approach for qualitative research. Thousand Oaks,
CA:Sage.
[40] NIHOfficeofBehavioralandSocialSciences(2018): Bestpracticesformixedmethods
researchinthehealthsciences.Bethesda,MA:NationalInstitutesofHealth.
[41] Noyes, J., Booth A., Moore, G., Flemming K., Tuncalp, O., and Shakibazadeh, E.
(2019): Synthesising quantitative and qualitative evidence to inform guidelines on
complex interventions: Clarifying the purposes, designs and outlinining some meth-
ods.BritishMedicalJournalGlobalHealth,4,e000893.

28 Kopaˇ candHlebec
[42] O’Cathain,A.,Murphy,E.,andNicholl,J.(2008): Thequalityofmixedmethodsstud-
ies in health services research. Journal of Health Services Research & Policy, 13(2),
92–98.
[43] O’Cathain,A.,Murphy,E.,andNicholl,J.(2010): Threetechniquesforintegratingfor
integratingdatainmixedmethodsstudies.BritishMedicalJournal,341,1147–1150.
[44] O’Cathain, A., Thomas, K.J., Drabble, S.J., Rudolph, A., and Hewison, J. (2013):
What can qualitative research do for randomised controlled trials? A systematic map-
pingreview.BritishMedicalJournal,3,e002889.
[45] O’Cathain,A.(2018): Apracticalguidetousingqualitativeresearchwithrandomized
controlledtrials.NewYork,NY:OxfordUniversityPress.
[46] Onwuegbuzie, A. J. and Teddlie, C. (2003): A framework for analyzing data in mixed
methodsresearch.InA.TashakkoriandC.Teddlie(Eds.): Handbookofmixedmethods
insocialandbehavioralresearch,351–383.ThousandOaks,CA:Sage.
[47] Onwuegbuzie,A.J.andCollins,K.M.(2007): Atypologyofmixedmethodssampling
designsinsocialscienceresearch.QualitativeReport,12(2),281–316.
[48] Onwuegbuzzie, A.J. and Corrigan, J.A. (2014): Improving the quality of mixed re-
searchreportsinthefieldofhumanresourcedevelopmentandbeyond: Acallforrigor
asanethicalpractice.HumanResourceDevelopmentQuarterly,25(3),273–299.
[49] Palinkas et al. (2019): Innovations in mixed methods evaluations. Annual Review of
PublicHealth,40(1),423–442.
[50] Protheroe, J., Bower, P., and Chew-Graham, C. (2008): The use of mixed methodol-
ogy in evaluating complex interventions: Identifying patient factors that moderate the
effectsofadecisionaid.FamilyPractice,24(6),594–600.
[51] Sandelowski, M. (1996): Using qualitative methods in intervention studies. Research
inNursingandHealth,19(4),359–364.
[52] Schifferdecker,K.E.andReedV.A.(2009): Usingmixedmethodsresearchinmedical
education: Basicguidelinesforresearchers.MedicalEducation,43(7),637–644.
[53] Schulz, K.F., Altman, D.G., and Moher, D. (2010): CONSORT 2010 Statement: Up-
datedguidelinesforreportingparallelgrouprandomisedtrials.JournalofClinicalEpi-
demiology,63(8),834–840.
[54] Song, M., Happ, M.B., and Sandelowski, M. (2010): Development of a tool to as-
sessfidelitytoapsycho-educationalintervention.JournalofAdvancedNursing,66(3),
673–682.
[55] Schoonenboom, J. and Burke Johnson, R. (2017): How to construct a mixed methods
design.K¨ olnerZeitschriftf¨ urSoziologieandSozialpsychologie,69(7),107–131.
[56] Tashakkori, A. and Teddlie, C. (1998): Mixed methodology: Combining qualitative
andquantitativeapproaches.ThousandOaks,CA:Sage.

Qualityguidelinesformixedmethodsresearchininterventionstudies 29
[57] Tashakkori,A.andTeddlie,C.(2003): Handbookofmixedmethodsinsocial&behav-
ioralresearch.ThousandOaks,CA:Sage.
[58] Tashakkori, A. and Teddlie, C. (2009): Foundations of mixed methods research: Inte-
grating quantitative and qualitative approaches in the social and behavioral sciences.
ThousandOaks,CA:Sage.
[59] Tashakkori, A. and Teddlie, C. (2010): Handbook of mixed methods in social and
behavioralresearch.ThousandOaks,CA:Sage.
[60] Venkatesh,V.,Brown,S.A.,andBala,H.(2013): Bridgingthequalitative–quantitative
divide: Guidelinesforconductingmixedmethodsresearchininformationsystems.MIS
Quarterly,36(1),21–54.
[61] Venkatesh, V., Brown, S. A., and Sullivan, Y.W. (2016): Guidelines for conducting
mixed-methods research: An extension and illustration. Journal of the Association for
InformationSystems,17(7),435–495.
[62] Wisdom, J.P., Cavaleri, M.A., Onwuegbuzie, A.J., and Green, C.A. (2012): Method-
ological reporting in qualitative, quantitative, and mixed methods health services re-
searcharticles.HealthServicesResearch,47(2),721–745.
[63] Woolcock, M. (2018): Reasons for using mixed methods in the evaluation of com-
plexprojects.In: M.Nagatsu andA.Ruzzene(Eds): Philosophyandinterdisciplinary
socialscience: Adialogue,149–171.London: BloomsburyAcademic.