Preface
1 Random and systematic errors in context
1.1 Research objectives
1.2 Random and systematic errors
1.3 Errors in context
1.3.1 Research questions
1.3.2 Literature review
1.3.3 Sampling
1.3.4 Operationalizations
1.3.5 Designs
1.3.6 Implementation
1.3.7 Data analysis
1.3.8 Reporting
1.4 Recommendations
References
2 Probability sampling
2.1 The elements of probability sampling
2.2 Defining the target population
2.3 Constructing the sampling frame
2.4 Probability sampling
2.4.1 Simple random sampling
2.4.2 Sample size
2.4.3 Stratification
2.4.4 Cluster sampling
2.5 Obtaining participation of sampled persons
2.6 Recommendations
References
3 Nonprobability sampling
3.1 The main elements of nonprobability sampling
3.2 Strategies to control for bias
3.2.1 Representative sampling
3.2.2 Bias reduction by weighting
3.2.3 Generalization across participant characteristics
3.2.4 Comments
3.3 Recommendations
References
4 Random assignment
4.1 Independent and dependent variables
4.2 Association does not mean causation
4.3 Other variable types
4.4 Random assignment to control for selection bias
4.5 Reducing random error variance
4.5.1 Blocking
4.5.2 Covariates
4.6 Cluster randomization
4.7 Missing participants (clusters)
4.8 Random assignment and random selection
4.9 Recommendations
References
5 Propensity scores
5.1 The propensity score
5.2 Estimating the propensity score
5.3 Applying the propensity score
5.4 An example
5.5 Comments
5.6 Recommendations
References
6 Situational bias
6.1 Standardization
6.2 Calibration
6.3 Blinding
6.4 Random assignment
6.5 Manipulation checks and treatment separation
6.6 Pilot studies
6.7 Replications
6.8 Randomization bias
6.9 Pretest effects
6.10 Response shifts
6.11 Recommendations
References
7 Random measurement error
7.1 Tests and test scores
7.2 Measurement precision
7.2.1 Within-person precision
7.2.2 Reliability
7.3 Increasing measurement precision
7.3.1 Item writing
7.3.2 Compiling the test
7.3.3 Classical analysis of test scores
7.3.4 Classical item analysis
7.3.5 Modern item analysis
7.3.6 Test administration
7.3.7 Data processing
7.4 Recommendations
References
8 Systematic measurement error
8.1 Cheating
8.2 Person fit
8.3 Satisficing
8.4 Impression management
8.5 Response styles
8.5.1 'Plodding' and 'fumbling'
8.5.2 The extremity and midpoint style
8.5.3 Acquiescence and dissentience
8.6 Item nonresponse
8.7 Coping with systematic errors
8.8 Recommendations
References
9 Unobtrusive measurements
9.1 Measurement modes
9.2 Examples of unobtrusive measurements
9.3 Random error of unobtrusive measurements
9.4 Systematic errors of unobtrusive measurements
9.5 Comments
9.6 Recommendations
References
10 Test dimensionality
10.1 Types of multidimensionality
10.2 Reliability and test dimensionality
10.3 Detecting test dimensionality
10.3.1 Factor analysis of inter-item product moment correlations
10.3.2 Factor analysis of inter-item tetrachoric and polychoric correlations
10.3.3 Mokken scale analysis
10.3.4 Full-information factor analysis
10.3.5 Comments
10.4 Measurement invariance
10.4.1 Measurement bias with respect to group membership
10.4.2 Measurement invariance and behavioral research
10.5 Recommendations
References
11 Coefficients for bivariate relations
11.1 Bivariate relation types
11.2 Variable types
11.3 Classification of coefficients for bivariate relations
11.4 Examples of coefficients
11.4.1 Dichotomous variables and a symmetrical relation
11.4.2 Dichotomous variables and equality of X- and Y-categories
11.4.3 Dichotomous variables and an asymmetrical relation
11.4.4 Nominal-categorical variables and a symmetrical relation
11.4.5 Nominal-categorical variables and equality of X- and Y-categories
11.4.6 Nominal-categorical variables and an asymmetrical relation
11.4.7 Ordinal-categorical variables and a symmetrical relation
11.4.8 Ordinal-categorical variables and equality of X- and Y-categories
11.4.9 Ordinal-categorical variables and an asymmetrical relation
11.4.10 Ranked variables and a symmetrical relation
11.4.11 Continuous variables and a symmetrical relation
11.4.12 Continuous variables and equality of X- and Y-values
11.4.13 Continuous variables and an asymmetrical relation
11.5 Comments
11.6 Recommendations
References
12 Null hypothesis testing
12.1 The confidence interval approach to null hypothesis testing
12.1.1 Classical confidence intervals of the means of paired scores
12.1.2 Classical confidence intervals of independent DV score means
12.2 Overlapping CIs
12.3 Conditional null hypothesis testing
12.4 Bootstrap methods
12.4.1 The bootstrap t method for paired DV score means
12.4.2 The bootstrap t method for independent DV score means
12.4.3 The modified percentile bootstrap method for the product moment correlation
12.5 Standardized effect sizes
12.6 Power
12.7 Testing multiple null hypothesis
12.8 Null hypothesis testing and data exploration
12.9 Sequential null hypothesis testing
12.10 Equivalence testing
12.11 Recommendations
References
13 Unstandardized effect sizes
13.1 Differences of means
13.2 Probability of superiority
13.3 Linear transformations of observed test scores
13.3.1 The Average Item Score (AIS) transformation
13.3.2 The Proportion of Maximum Possible (POMP) score transformation
13.4 Recommendations
References
14 Pretest-posttest change
14.1 The population/single-person fallacy in pretest-posttest studies
14.2 Group change
14.2.1 Within-group pretest-posttest change
14.2.2 Between-groups change
14.3 Single-person change
14.3.1 Single-person observed test score change
14.3.2 Single-person continuous item response change
14.3.3 Single-person dichotomous item response change
14.4 Comments
14.5 Recommendations
References
15 Reliability
15.1 The classical model of observed test scores
15.2 Measurement precision
15.2.1 Standard error of measurement
15.2.2 Reliability
15.3 Counter-intuitive properties of the reliability of the observed test score
15.3.1 Reliability of the observed test score and unidimensionality
15.3.2 Reliability and true score estimation precision
15.3.3 Reliability and mean test score estimation precision
15.3.4 Reliability and estimating the difference of two independent test score means
15.3.5 Reliability and testing the null hypothesis of equal independent test score means
15.4 Reliability of the difference score
15.4.1 The classical model of the difference score
15.4.2 Unreliable and reliable difference scores
15.4.3 Reliability of the difference score and estimation precision of the true difference score
15.4.4 Reliability of the difference score and estimation precision of the mean difference score
15.4.5 Reliability of the difference score and testing the null hypothesis of equal means of paired test scores
15.5 Reliability of latent variables
15.5.1 Reliability of latent trait estimates
15.5.2 Reliability and discrete latent variables
15.6 Relevance of the reliability concept
15.7 Recommendations
References
16 Missing data
16.1 Missingness types
16.2 Missingness variables
16.3 Data collection methods to reduce missingness
16.4 Sample size maintenance procedures
16.5 Naive missing data methods
16.6 Nonnaive missing variable methods
16.6.1 Statistical methods
16.6.2 Worst-case imputation of missing paired scores
16.6.3 Worst-case imputation of missing independent scores
16.7 Nonnaive missing item methods
16.7.1 Imputing missing maximum performance items
16.7.2 Imputing missing typical response items
16.8 Recommendations
References
17 Outliers
17.1 Outlier detection methods
17.2 Outlier detection and correction
17.3 Coping with coincidental outliers
17.4 Coping with noncoincidental outliers
17.5 Content robustness against outliers
17.6 Robust statistics
17.7 Comparing paired scores
17.8 Comparing independent scores
17.9 Association between two variables
17.10 Recommendations
References
18 Interactions and specific hypotheses
18.1 Factorial designs
18.2 Main and interaction effects
18.3 Testing main and interaction effects
18.3.1 Continuous and ranked DVs
18.3.3 Dichotomous DVs
18.3.3 Nominal-categorical DVs
18.3.4 Ordinal-categorical DVs
18.4 Nonmanipulable factors
18.5 Dichotomization of nonmanipulable independent variables
18.6 Testing specific substantive hypotheses
18.6.1 Planned comparisons of DV-means
18.6.2 Planned comparisons of DV-logits
18.6.3 Testing multiple null hypotheses of contrasts
18.7 Recommendations
References
19 Publishing
19.1 The publication process
19.2 Publication bias
19.3 Replications
19.3.1 Replication hypotheses
19.3.2 Testing a replication hypothesis
19.3.3 Equivalence testing of a linear contrast
19.3.4 A framework for replication research
19.4 Proposals
19.4.1 Attitude towards replication
19.4.2 Editorial policies
19.4.3 Collaboration
References
20 Scientific misconduct
20.1 Plagiarism
20.2 Fabrication and falsification
20.3 Questionable scientific practices
20.3.1 Questionable research practices
20.3.2 Questionable editorial practices
20.4 Policies against misconduct
20.4.1 Educational policies
20.4.2 Editorial policies
20.4.3 Formal policies
References
