Psychological Methods

Sjobeck, Gustav R.; Boker, Steven M.; Scheidt, Carl E.; Tschacher, Wolfgang

doi: 10.1037/met0000341pmid: 38573666

Methods that measure the association between two intensively measured time series are of interest to researchers studying the symmetry of behaviors during social interaction. Such methods have historically focused on aggregating the amount of symmetry across all measurement occasions. However, it is rarely expected that symmetry is present at all measurement occasions. The current method, the pairwise approximate spatiotemporal symmetry (PASS) algorithm, is an approach that may be used to determine which measurement occasions in pairwise time series are indicative of symmetry and which are not. This process thus divides time series into symmetric and nonsymmetric segments. The PASS algorithm is demonstrated here on representative simulated data and naturalistic psychotherapy data. Results suggest that the PASS algorithm has the potential to extract meaningful symmetry segments from human signals.

Liu, Xiao; Liu, Fang; Miller-Graff, Laura; Howell, Kathryn H.; Wang, Lijuan

doi: 10.1037/met0000565pmid: 37053414

Partially nested designs (PNDs) are common in intervention studies in psychology and other social sciences. With this design, participants are assigned to treatment and control groups on an individual basis, but clustering occurs in some but not all groups (e.g., the treatment group). In recent years, there has been substantial development of methods for analyzing data from PNDs. However, little research has been done on causal inference for PNDs, especially for PNDs with nonrandomized treatment assignments. To reduce the research gap, in the current study, we used the expanded potential outcomes framework to define and identify the average causal treatment effects in PNDs. Based on the identification results, we formulated the outcome models that could produce treatment effect estimates with causal interpretation and evaluated how alternative model specifications affect the causal interpretation. We also developed an inverse propensity weighted (IPW) estimation approach and proposed a sandwich-type standard error estimator for the IPW-based estimate. Our simulation studies demonstrated that both the outcome modeling and the IPW methods specified following the identification results can yield satisfactory estimates and inferences of the average causal treatment effects. We applied the proposed approaches to data from a real-life pilot study of the Pregnant Moms’ Empowerment Program for illustration. The current study provides guidance and insights on causal inference for PNDs and adds to researchers’ toolbox of treatment effect estimation with PNDs.

Savalei, Victoria; Brace, Jordan C.; Fouladi, Rachel T.

doi: 10.1037/met0000537pmid: 36622720

Comparison of nested models is common in applications of structural equation modeling (SEM). When two models are nested, model comparison can be done via a chi-square difference test or by comparing indices of approximate fit. The advantage of fit indices is that they permit some amount of misspecification in the additional constraints imposed on the model, which is a more realistic scenario. The most popular index of approximate fit is the root mean square error of approximation (RMSEA). In this article, we argue that the dominant way of comparing RMSEA values for two nested models, which is simply taking their difference, is problematic and will often mask misfit, particularly in model comparisons with large initial degrees of freedom. We instead advocate computing the RMSEA associated with the chi-square difference test, which we call RMSEAD. We are not the first to propose this index, and we review numerous methodological articles that have suggested it. Nonetheless, these articles appear to have had little impact on actual practice. The modification of current practice that we call for may be particularly needed in the context of measurement invariance assessment. We illustrate the difference between the current approach and our advocated approach on three examples, where two involve multiple-group and longitudinal measurement invariance assessment and the third involves comparisons of models with different numbers of factors. We conclude with a discussion of recommendations and future research directions.

Götz, Friedrich M.; Maertens, Rakoen; Loomba, Sahil; van der Linden, Sander

doi: 10.1037/met0000540pmid: 36795435

Measurement is at the heart of scientific research. As many—perhaps most—psychological constructs cannot be directly observed, there is a steady demand for reliable self-report scales to assess latent constructs. However, scale development is a tedious process that requires researchers to produce good items in large quantities. In this tutorial, we introduce, explain, and apply the Psychometric Item Generator (PIG), an open-source, free-to-use, self-sufficient natural language processing algorithm that produces large-scale, human-like, customized text output within a few mouse clicks. The PIG is based on the GPT-2, a powerful generative language model, and runs on Google Colaboratory—an interactive virtual notebook environment that executes code on state-of-the-art virtual machines at no cost. Across two demonstrations and a preregistered five-pronged empirical validation with two Canadian samples (NSample 1 = 501, NSample 2 = 773), we show that the PIG is equally well-suited to generate large pools of face-valid items for novel constructs (i.e., wanderlust) and create parsimonious short scales of existing constructs (i.e., Big Five personality traits) that yield strong performances when tested in the wild and benchmarked against current gold standards for assessment. The PIG does not require any prior coding skills or access to computational resources and can easily be tailored to any desired context by simply switching out short linguistic prompts in a single line of code. In short, we present an effective, novel machine learning solution to an old psychological challenge. As such, the PIG will not require you to learn a new language—but instead, speak yours.

Maier, Maximilian; van Dongen, Noah; Borsboom, Denny

doi: 10.1037/met0000543pmid: 36862460

Theories are among the most important tools of science. Lewin (1943) already noted “There is nothing as practical as a good theory.” Although psychologists discussed problems of theory in their discipline for a long time, weak theories are still widespread in most subfields. One possible reason for this is that psychologists lack the tools to systematically assess the quality of their theories. Thagard (1989) developed a computational model for formal theory evaluation based on the concept of explanatory coherence. However, there are possible improvements to Thagard’s (1989) model and it is not available in software that psychologists typically use. Therefore, we developed a new implementation of explanatory coherence based on the Ising model. We demonstrate the capabilities of this new Ising model of Explanatory Coherence (IMEC) on several examples from psychology and other sciences. In addition, we implemented it in the R-package IMEC to assist scientists in evaluating the quality of their theories in practice.

Chen, Man; Pustejovsky, James E.

doi: 10.1037/met0000510pmid: 35786985

Single-case experimental designs (SCEDs) are used to study the effects of interventions on the behavior of individual cases, by making comparisons between repeated measurements of an outcome under different conditions. In research areas where SCEDs are prevalent, there is a need for methods to synthesize results across multiple studies. One approach to synthesis uses a multilevel meta-analysis (MLMA) model to describe the distribution of effect sizes across studies and across cases within studies. However, MLMA relies on having accurate sampling variances of effect size estimates for each case, which may not be possible due to auto-correlation in the raw data series. One possible solution is to combine MLMA with robust variance estimation (RVE), which provides valid assessments of uncertainty even if the sampling variances of effect size estimates are inaccurate. Another possible solution is to forgo MLMA and use simpler, ordinary least squares (OLS) methods with RVE. This study evaluates the performance of effect size estimators and methods of synthesizing SCEDs in the presence of auto-correlation, for several different effect size metrics, via a Monte Carlo simulation designed to emulate the features of real data series. Results demonstrate that the MLMA model with RVE performs properly in terms of bias, accuracy, and confidence interval coverage for estimating overall average log response ratios. The OLS estimator corrected with RVE performs the best in estimating overall average Tau effect sizes. None of the available methods perform adequately for meta-analysis of within-case standardized mean differences.

Rosseel, Yves; Loh, Wen Wei

doi: 10.1037/met0000503pmid: 36355708

In structural equation modeling (SEM), the measurement and structural parts of the model are usually estimated simultaneously. In this article, we revisit the long-standing idea that we should first estimate the measurement part, and then estimate the structural part. We call this the “structural-after-measurement” (SAM) approach to SEM. We describe a formal framework for the SAM approach under settings where the latent variables and their indicators are continuous. We review earlier SAM methods and establish how they are specific instances of the SAM framework. Decoupled estimation for the measurement and structural parts using SAM possesses three key advantages over simultaneous estimation in standard SEM. First, estimates are more robust against local model misspecifications. Second, estimation routines are less vulnerable to convergence issues in small samples. Third, estimates exhibit smaller finite sample biases under correctly specified models. We propose two variants of the SAM approach. “Local” SAM expresses the mean vector and variance–covariance matrix of the latent variables as a function of the observed summary statistics and the parameters of the measurement model. “Global” SAM holds the parameters of the measurement part fixed while estimating the parameters of the structural part. Our framework includes two-step corrected standard errors, and permits computing both local and global fit measures. Nonetheless, the SAM approach is an estimation strategy, and should not be regarded as a model-building tool.

Brandt, Holger

doi: 10.1037/met0000544pmid: 39311827

In this response to Valente et al. (2022), I am discussing the plausibility and applicability of the proposed mediation model and its causal effects estimation for single case experimental designs (SCEDs). I will focus on the underlying assumptions that the authors use to identify the causal effects. These assumptions include the particularly problematic assumption of sequential ignorability or no-unmeasured confounders. First, I will discuss the plausibility of the assumption in general and then particularly for SCEDs by providing an analytic argument and a reanalysis of the empirical example in Valente et al. (2022). Second, I will provide a simulation that reproduces the design by Valente et al. (2022) with the exception that, for a more realistic depiction of empirical data, an unmeasured confounder affects the mediator and outcome variables. The results of this simulation study indicate that even minor violations will lead to Type I error rates up to 100% and coverage rates as low as 0% for the defined causal direct and indirect effects. Third, using historical data on the effect of birth control on stork population and birth rates, I will show that mediation models like the proposed method can lead to surprising artifacts. These artifacts can hardly be identified with statistically means including methods such as sensitivity analyses.

Steinley, Douglas

doi: 10.1037/met0000705pmid: 39311830

Linde, Maximilian; Tendeiro, Jorge N.; Wagenmakers, Eric-Jan; van Ravenzwaaij, Don

doi: 10.1037/met0000549pmid: 39311828

AbstractLinde et al. (2021) compared the “two one-sided tests” the “highest density interval—region of practical equivalence”, and the “interval Bayes factor” approaches to establishing equivalence in terms of power and Type I error rate using typical decision thresholds. They found that the interval Bayes factor approach exhibited a higher power but also a higher Type I error rate than the other approaches. In response, Campbell and Gustafson (2022) showed that the performances of the three approaches can approximate one another when they are calibrated to have the same Type I error rate. In this article, we argue that these results have little bearing on how these approaches are used in practice; a concrete example is used to highlight this important point.