Counteracting Methodological Errors in Behavioral Research

Preface

1     Random and systematic errors in context

     1.1  Research objectives

     1.2  Random and systematic errors

     1.3  Errors in context

            1.3.1  Research questions

            1.3.2  Literature review

            1.3.3  Sampling

            1.3.4 Operationalizations

            1.3.5  Designs

            1.3.6  Implementation

             1.3.7  Data analysis

             1.3.8 Reporting

      1.4  Recommendations

      References

2     Probability sampling

       2.1     The elements of probability sampling

       2.2     Defining the target population

       2.3     Constructing the sampling frame

       2.4     Probability sampling

                 2.4.1    Simple random sampling

                 2.4.2    Sample size

                 2.4.3    Stratification

                 2.4.4    Cluster sampling

       2.5     Obtaining participation of sampled persons

       2.6 Recommendations

3     Nonprobability sampling

       3.1     The main elements of nonprobability sampling

       3.2     Strategies to control for bias

                 3.2.1    Representative sampling

                 3.2.2    Bias reduction by weighting

                 3.2.3    Generalization across participant characteristics

                 3.2.4    Comments

       3.3     Recommendations

       References

4     Random assignment

       4.1     Independent and dependent variables

       4.2     Association does not mean causation

       4.3     Other variable types

       4.4     Random assignment to control for selection bias

       4.5     Reducing random error variance

                 4.5.1    Blocking

                 4.5.2    Covariates

       4.6     Cluster randomization

       4.7     Missing participants (clusters)

       4.8     Random assignment and random selection

       4.9     Recommendations

       References

5     Propensity scores

       5.1     The propensity score

       5.2     Estimating the propensity score

       5.3     Applying the propensity score

       5.4     An example

       5.5     Comments

       5.6     Recommendations

       References

6     Situational bias

       6.1     Standardization

       6.3     Blinding

       6.4     Random assignment

       6.5     Manipulation checks and treatment separation

       6.6     Pilot studies

       6.7     Replications

       6.8     Randomization bias

       6.9     Pretest effects

       6.10   Response shifts

       6.11 Recommendations

       References

7     Random measurement error

       7.1     Tests and test scores

       7.2     Measurement precision

                 7.2.1    Within-person precision

                 7.2.2    Reliability

       7.3     Increasing measurement precision

                 7.3.1    Item writing

                 7.3.2    Compiling the test

                 7.3.3    Classical analysis of test scores

                 7.3.4    Classical item analysis

                 7.3.5    Modern item analysis

                 7.3.6    Test administration

                 7.3.7    Data processing

       7.4 Recommendations

       References

8     Systematic measurement error

       8.1     Cheating

       8.2     Person fit

       8.3     Satisficing

       8.4     Impression management

       8.5     Response styles

                 8.5.1    'Plodding' and 'fumbling'

                 8.5.2    The extremity and midpoint style

                 8.5.3    Acquiescence and dissentience

       8.6     Item nonresponse

       8.7     Coping with systematic errors

       8.8    Recommendations

       References

9     Unobtrusive measurements

       9.1     Measurement modes

       9.2     Examples of unobtrusive measurements

       9.3     Random error of unobtrusive measurements

       9.4     Systematic errors of unobtrusive measurements

       9.5     Comments

       9.6    Recommendations

       References

10   Test dimensionality

       10.1   Types of multidimensionality

       10.2   Reliability and test dimensionality

       10.3   Detecting test dimensionality

                 10.3.1    Factor analysis of inter-item product moment correlations

                 10.3.2    Factor analysis of inter-item tetrachoric and polychoric correlations

                 10.3.3    Mokken scale analysis

                 10.3.4    Full-information factor analysis

                 10.3.5    Comments

       10.4   Measurement invariance

                 10.4.1    Measurement bias with respect to group membership

                 10.4.2    Measurement invariance and behavioral research

       10.5   Recommendations

       References

11   Coefficients for bivariate relations

       11.1   Bivariate relation types

       11.2   Variable types

       11.3   Classification of coefficients for bivariate relations

       11.4   Examples of coefficients

                 11.4.1    Dichotomous variables and a symmetrical relation

                 11.4.2    Dichotomous variables and equality of X- and Y-categories

                 11.4.3    Dichotomous variables and an asymmetrical relation

                 11.4.4    Nominal-categorical variables and a symmetrical relation

                 11.4.5    Nominal-categorical variables and equality of X- and Y-categories

                 11.4.6    Nominal-categorical variables and an asymmetrical relation

                 11.4.7    Ordinal-categorical variables and a symmetrical relation

                 11.4.8    Ordinal-categorical variables and equality of X- and Y-categories

                 11.4.10  Ranked variables and a symmetrical relation

                 11.4.11  Continuous variables and a symmetrical relation

                 11.4.12  Continuous variables and equality of X- and Y-values

                 11.4.13  Continuous variables and an asymmetrical relation

       11.5   Comments

       11.6   Recommendations

       References

12   Null hypothesis testing

       12.1   The confidence interval approach to null hypothesis testing

                 12.1.1    Classical confidence intervals of the means of paired scores

                 12.1.2    Classical confidence intervals of independent DV score means

       12.2   Overlapping CIs

       12.3   Conditional null hypothesis testing

       12.4   Bootstrap methods

                 12.4.1    The bootstrap t method for paired DV score means

                 12.4.2    The bootstrap t method for independent DV score means

                 12.4.3    The modified percentile bootstrap method for the product moment correlation

       12.5   Standardized effect sizes

       12.6   Power

       12.7   Testing multiple null hypothesis

       12.8   Null hypothesis testing and data exploration

       12.9   Sequential null hypothesis testing

       12.10 Equivalence testing

       12.11 Recommendations

       References

13   Unstandardized effect sizes

       13.1   Differences of means

       13.2   Probability of superiority

       13.3   Linear transformations of observed test scores

                 13.3.1    The Average Item Score (AIS) transformation

                 13.3.2    The Proportion of Maximum Possible (POMP) score transformation

       13.4   Recommendations

       References

14   Pretest-posttest change

       14.1   The population/single-person fallacy in pretest-posttest studies

       14.2   Group change

                 14.2.1    Within-group pretest-posttest change

                 14.2.2    Between-groups change

       14.3   Single-person change

                 14.3.1    Single-person observed test score change

                 14.3.2    Single-person continuous item response change

                 14.3.3    Single-person dichotomous item response change

       14.4   Comments

       14.5   Recommendations

       References

15   Reliability

       15.1   The classical model of observed test scores

                 15.2.1    Standard error of measurement

                 15.2.2    Reliability

       15.3   Counter-intuitive properties of the reliability of the observed test score

                 15.3.1    Reliability of the observed test score and unidimensionality

                 15.3.2    Reliability and true score estimation precision

                 15.3.3    Reliability and mean test score estimation precision

                 15.3.4    Reliability and estimating the difference of two independent test score means

                 15.3.5    Reliability and testing the null hypothesis of equal independent test score means

       15.4   Reliability of the difference score

                 15.4.1    The classical model of the difference score

                 15.4.2    Unreliable and reliable difference scores

                 15.4.3    Reliability of the difference score and estimation precision of the true difference score

                 15.4.4    Reliability of the difference score and estimation precision of the mean difference score

                 15.4.5    Reliability of the difference score and testing the null hypothesis of equal means of paired test scores

       15.5   Reliability of latent variables

                 15.5.1    Reliability of latent trait estimates

                 15.5.2    Reliability and discrete latent variables

       15.6   Relevance of the reliability concept

       15.7   Recommendations

       References

16   Missing data

       16.1   Missingness types

       16.2   Missingness variables

       16.3   Data collection methods to reduce missingness

       16.4   Sample size maintenance procedures

       16.5   Naive missing data methods

       16.6   Nonnaive missing variable methods

                 16.6.1    Statistical methods

                 16.6.2    Worst-case imputation of missing paired scores

                 16.6.3    Worst-case imputation of missing independent scores

       16.7   Nonnaive missing item methods

                 16.7.1    Imputing missing maximum performance items

                 16.7.2    Imputing missing typical response items

       16.8   Recommendations

       References

17   Outliers

       17.1   Outlier detection methods

       17.2   Outlier detection and correction

       17.3   Coping with coincidental outliers

       17.4   Coping with noncoincidental outliers

       17.5   Content robustness against outliers

       17.6   Robust statistics

       17.7 Comparing paired scores

       17.8   Comparing independent scores

       17.9   Association between two variables

       17.10 Recommendations

       References

18   Interactions and specific hypotheses

       18.1   Factorial designs

       18.2   Main and interaction effects

       18.3   Testing main and interaction effects

                 18.3.1    Continuous and ranked DVs

                 18.3.3    Dichotomous DVs

                 18.3.3    Nominal-categorical DVs

                 18.3.4    Ordinal-categorical DVs

       18.4   Nonmanipulable factors

       18.5   Dichotomization of nonmanipulable independent variables

       18.6   Testing specific substantive hypotheses

                 18.6.1    Planned comparisons of DV-means

                 18.6.2    Planned comparisons of DV-logits

                 18.6.3    Testing multiple null hypotheses of contrasts

       18.7   Recommendations

       References

19   Publishing

       19.1   The publication process

       19.2   Publication bias

       19.3   Replications

                 19.3.1    Replication hypotheses

                 19.3.2    Testing a replication hypothesis

                 19.3.3    Equivalence testing of a linear contrast

                 19.3.4    A framework for replication research

       19.4   Proposals

                 19.4.1    Attitude towards replication

                 19.4.2    Editorial policies

                 19.4.3    Collaboration

       References

20   Scientific misconduct

       20.1   Plagiarism

       20.2   Fabrication and falsification

       20.3   Questionable scientific practices

                 20.3.1    Questionable research practices

                 20.3.2    Questionable editorial practices

       20.4   Policies against misconduct

                 20.4.1    Educational policies

                 20.4.2    Editorial policies

                 20.4.3    Formal policies

       References