Chapter 8
Quantitative Methods
Introduction
• Designing quantitative methods for a research
proposal
• Survey and experimental designs
• Careful measurement, parsimonious variables, theory guided
Creswell, Research Design 5e
SAGE Publishing, 2018 3
Defining Surveys and Experiments
• Survey design
– Quantitative description of trends, attitudes, or
opinions of a population
– Testing association
– Studying a sample of that population
Creswell, Research Design 5e
SAGE Publishing, 2018 4
Defining Surveys and Experiments
• Experimental design
– Systematic manipulation of one or more variables
to evaluate an outcome
– Holds other variables constant to isolate effects
– Generalize to a broader population
Creswell, Research Design 5e
SAGE Publishing, 2018 5
Components of a Survey Method
Plan
• The survey design
• The population and sample
• Instrumentation
• Variables in the study
• Data analysis and interpretation
Creswell, Research Design 5e
SAGE Publishing, 2018 6
Components of a Survey Method
Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 7
Components of a Survey Method
Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 8
Components of a Survey Method
Plan
The survey design:
• Provide a purpose for using survey research
• Indicate why the survey method is preferred
• Indicate the type of survey design
– Cross-sectional (data collected at one point in time)
– Longitudinal (data collected over time)
• Specify the form of data collection – telephone, mail,
Internet personal/group interviews) and rationale
Creswell, Research Design 5e
SAGE Publishing, 2018 9
Components of a Survey Method
Plan
The population and sample:
• Identify the population including size and sampling
frames
• Specify the sampling design
– Single-stage
– Multi-stage (clustering)
• Type of sampling
– Probability
– Nonprobability
Creswell, Research Design 5e
SAGE Publishing, 2018 10
Components of a Survey Method
Plan
The population and sample:
• Indicate if the study involves stratification – ensuring
specific population characteristics (e.g. gender) are
represented
• Indicate number in the sample and procedure to
determine
• Use a power analysis if you plan to detect significant
associations
Creswell, Research Design 5e
SAGE Publishing, 2018 11
Components of a Survey Method
Plan
The population and sample:
• Power analysis involves
– Estimating the size of correlation
– An alpha value (type I error rate)
– A beta value (type II error rate)
– Conducting the power analysis (e.g., G*Power)
• Conduct power analysis during planning
Creswell, Research Design 5e
SAGE Publishing, 2018 12
Components of a Survey Method
Plan
Instrumentation:
• Name the survey instrument used to collect data
• Indicate how instrument was developed
• Describe the established validity scores from past use
– Content validity
– Predictive or concurrent validity
– Construct validity
• Describe reliability of scores from past use
– Internal consistency
– Test-retest
Creswell, Research Design 5e
SAGE Publishing, 2018 13
Components of a Survey Method
Plan
Instrumentation:
• When modifying or combining instruments, the original
validity and reliability may not hold
• Include sample items from the instrument
• Indicate major content sections in the instrument
– Cover letter
– Items – demographics, attitude items, behavior
items, factual items
– Closing instructions
– Type of scale for responses
Creswell, Research Design 5e
SAGE Publishing, 2018 14
Components of a Survey Method
Plan
Instrumentation:
• Discuss pilot testing or field-testing
– Rationale for plans
– Content validity and reliability
– Improve question
• Steps for administering for a mailed survey
Creswell, Research Design 5e
SAGE Publishing, 2018 15
Components of a Survey Method
Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 16
Components of a Survey Method
Plan
Data analysis:
• Computer programs used for analysis
• Data analysis steps
– Step 1. Number who did and did not respond
– Step 2. Method to determine response bias
– Step 3. Plan to provide descriptive analyses
– Step 4. Calculate total scale scores
– Step 5. Statistics and program for inferential
statistical analyses
– Step 6. Present results in figures or tables and
interpret
Creswell, Research Design 5e
SAGE Publishing, 2018 17
Components of a Survey Method
Plan
Interpreting results and writing a discussion section:
• Report how the results answered the research question
or hypothesis
• Practical evidence in terms of effect size and confidence
interval
• Discuss implications
– Consistent with, refute, extent previous studies
Creswell, Research Design 5e
SAGE Publishing, 2018 18
Components of a Survey Method
Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 19
Components of an Experimental
Study Method Plan
• Experimental method plan
– Participants
– Variables
– Instrumentation and materials
– Experimental procedures
– Threats to validity
– Data analysis
– Interpreting results and writing a discussion
Creswell, Research Design 5e
SAGE Publishing, 2018 20
Components of an Experimental
Study Method Plan
Participants:
• Describe procedures for recruiting participants
• Describe the selection of participants as either
– Random
– Nonrandom (convenience)
• True experiment – individuals randomly assigned to
groups
• Quasi-experiment – partial or no control over random
assignment
Creswell, Research Design 5e
SAGE Publishing, 2018 21
Components of an Experimental
Study Method Plan
Participants:
• May measure second predictor variables
• Conduct and report power analysis
• End with formal experimental design statement
– “The experiment consisted of a one-way two groups design comparing burnout symptoms
between full-time and part-time nurses”
Creswell, Research Design 5e
SAGE Publishing, 2018 22
Components of an Experimental
Study Method Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 23
Components of an Experimental
Study Method Plan
Variables:
• Specify the variables and describe in detail
– Identify the independent variables
– Include a manipulation check measure
– Identify dependent variable
– Identify other variables measured
• Participant demographics
• Measure variables that contribute noise
• Potential confounding variables
Creswell, Research Design 5e
SAGE Publishing, 2018 24
Components of an Experimental
Study Method Plan
Instrumentation and materials:
• Describe the instrument(s) participants complete in the
experiment
– Development, items, and scales
– Reliability and validity reports of past uses
• Thoroughly discuss materials used for the treatment
• Cover story to explain procedures if deception is used
Creswell, Research Design 5e
SAGE Publishing, 2018 25
Components of an Experimental
Study Method Plan
Experimental procedures:
• Identify the type of experiment
– Pre-experimental, true experiment, quasi-experiment,
single subject design
• Identify the type of comparisons – within-group or
between-subject
– Provide a visual model to illustrate the research
design used
• X = treatment
• O = observation
• R = random assignment
See Examples 8.2–8.5
Creswell, Research Design 5e
SAGE Publishing, 2018 26
Components of an Experimental
Study Method Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 27
Components of an Experimental
Study Method Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 28
Components of an Experimental
Study Method Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 29
Components of an Experimental
Study Method Plan
Creswell, Research Design 5e
SAGE Publishing, 2018 30
Components of an Experimental
Study Method Plan
Threats to validity:
• Internal validity – procedures, treatments, or
experiences of the participants that threaten inferences
in experiments
• External validity – drawing incorrect inferences from
sample data to other persons, settings, situations
Creswell, Research Design 5e
SAGE Publishing, 2018 31
Components of an Experimental
Study Method Plan
Threats to validity:
• Statistical conclusion validity – inadequate statistical
power or violation of statistical assumptions
• Construct validity – inadequate definitions and
measures of variables
Creswell, Research Design 5e
SAGE Publishing, 2018 32
Components of an Experimental
Study Method Plan
• Threats to internal validity
– History – Compensatory/resentful
– Maturation demoralization
– Regression – Compensatory rivalry
– Selection – Testing
– Mortality – Instrumentation
– Diffusion of treatment
Creswell, Research Design 5e
SAGE Publishing, 2018 33
Components of an Experimental
Study Method Plan
• Threats to external validity
– Interaction of selection and treatment
– Interaction of setting and treatment
– Interaction of history and treatment
Creswell, Research Design 5e
SAGE Publishing, 2018 34
Components of an Experimental
Study Method Plan
The procedure:
• Administer measures of the dependent variable or a
variable closely correlated
• Assign participants to matched pairs
• Randomly assign one member of each pair to the
control and experimental group
• Expose experimental group to the treatment
• Administer measures of dependent variables
• Compare performance of the experimental and control
groups Creswell, Research Design 5e
SAGE Publishing, 2018 35
Components of an Experimental
Study Method Plan
Data analysis:
• Report descriptive statistics (e.g., means, standard
deviations, ranges)
• Indicate inferential statistical tests (e.g., t test, ANOVA,
ANCOVA, or MANOVA)
• Report confidence intervals and effect sizes in addition
to statistical tests
• Use line graphs for single subject designs
Creswell, Research Design 5e
SAGE Publishing, 2018 36
Components of an Experimental
Study Method Plan
Interpreting results and writing a discussion section:
• Interpret findings in light of hypotheses and research
questions
• Whether supported or refuted
• Why results significant or not, literature
• Indicate implications
• Suggest future research
Creswell, Research Design 5e
SAGE Publishing, 2018 37
Summary
• The methodological approach to a survey or
experiment
• Surveys – purpose, population and sample,
instruments, relationship, research questions, items,
analysis
• Experiments – identify participants, variables,
instruments, type of experiment, validly, analysis
Creswell, Research Design 5e
SAGE Publishing, 2018 38
