Reproducible R workflow for evaluating internal-consistency reliability (alpha, beta, omega-total, omega-hierarchical) and ordinal CFA diagnostics for the 66-item DAACS Self-Regulated Learning survey (N = 5,429). Manuscript under review.
Status: Manuscript under peer review (2026). Code and analytic outputs shared for reproducibility.
This repository contains the reproducible R code and analytic outputs for a study evaluating internal-consistency reliability evidence for the 66-item DAACS Self-Regulated Learning (SRL) survey (version 2.0). The analysis covers 14 subscales, 4 broad scales, and the SRL total score, using a pooled multi-institutional sample (N = 5,429) from the IES-funded DAACS project (Grant R305A210269).
The study provides a fully reproducible workflow for choosing and interpreting internal-consistency coefficients (Cronbach's alpha, Revelle's beta, McDonald's omega total and omega hierarchical) under realistic conditions of multidimensionality and model-fit violations. Ordinal confirmatory factor analysis (CFA) diagnostics contextualize the reliability estimates and inform score-interpretation boundaries.
- What are the internal-consistency profiles (alpha, beta, omega total, omega hierarchical) across SRL subscales, scales, and total score?
- How do reliability coefficients behave across score sets that differ in dimensionality and factor-model fit?
- What reporting and score-use guidance follows from the joint pattern of reliability and internal-structure evidence?
- Polychoric correlation matrices with eigenvalue smoothing when needed
- Parallel analysis and 14-factor EFA (oblimin rotation, MINRES) for dimensionality diagnostics
- Coefficient alpha (from polychoric matrices) as the traditional baseline
- Revelle's beta (minimum split-half reliability) as a lower-bound diagnostic
- McDonald's omega total and omega hierarchical via
psych::omega()to estimate total reliability and general-factor saturation - Ordinal CFA (WLSMV in
lavaan) for all subscale, scale, and full 14-factor models - Sensitivity analyses: institution-specific replications (UMGC1, n = 4,144; UA2, n = 1,285) and raw-data omega estimation (MINRES, ML)
├── R/
│ ├── analysis_srl_internal_consistency_workflow.R # Main analysis: reliability and CFA
│ ├── srl_v2_ua2-anSamp2-2022_qa_pipeline.R # QA pipeline for UA2 2022 data
│ └── 01c_srl_v2_ua2_ua22_overlap_audit.R # Data-provenance audit (see below)
├── figures/ # Parallel-analysis and omega diagrams
│ ├── Pooled_PA_*.png # Pooled-sample parallel analyses
│ ├── Pooled_Omega_*.png # Pooled-sample omega path diagrams
│ ├── UMGC1_*.png # UMGC1 sensitivity figures
│ └── UA2_*.png # UA2 sensitivity figures
├── tables/ # Manuscript-ready CSV tables
│ ├── Pooled_Table3_reliability_polychoric.csv # Table 3: reliability coefficients
│ ├── Pooled_SRL_omega_polychoric_all_sets.csv # Full omega results (all score sets)
│ ├── SRL_EFA_total_loadings_oblimin.csv # 14-factor EFA loadings
│ ├── SRL_item_counts.csv # Item counts per score set
│ └── UMGC1_*.csv / UA2_*.csv # Institution-specific sensitivity
├── results/ # CFA fit summaries
│ ├── Pooled_SRL_CFA_fit_summary.csv
│ ├── UMGC1_SRL_CFA_fit_summary.csv
│ ├── UA2_SRL_CFA_fit_summary.csv
│ └── SRL_total_omega_sensitivity_raw_*.csv # Raw-data omega sensitivity checks
└── README.md
The UAlbany 2022 SRL data exists in two independently prepared versions:
- UA2: the original dataset, cleaned and organized by the author from raw institutional data prior to the dissertation and the 2023 NCME poster presentation.
- UA22: the most recent (2026) version of the same data, re-released by the DAACS team in a cleaner, more organized format.
The QA pipeline (R/srl_v2_ua2-anSamp2-2022_qa_pipeline.R) standardizes IDs, recodes demographics, deduplicates records, and runs missingness and item-response diagnostics on the UA2 file. The overlap audit (R/01c_srl_v2_ua2_ua22_overlap_audit.R) then confirms that the UA2 and UA22 datasets are identical at the response-vector level (matched by DAACS_ID, response signatures, and demographic near-matches). This confirmation is important: it validates that the analytic results reported in this study, which were computed on the UA2-era data, are fully consistent with the current institutional data release.
The analytic dataset contains student-level survey responses collected under the DAACS IES grant and is not included in this repository. The R scripts are configured to load the data from a local path; see CONFIG$data_rdata in the main analysis script for details.
All code, output tables, and figures are provided so that the analytic workflow is fully transparent and the results can be verified given access to the data.
- Clone this repository.
- Place the analytic dataset (
SRL_dataClean-UMGC1UA2_1.RData) in the project root or update the path inCONFIG$data_rdata. - Install required packages:
dplyr,tibble,psych,lavaan. - Run
R/analysis_srl_internal_consistency_workflow.R. Outputs are written totables/,results/, andfigures/.
| Package | Purpose |
|---|---|
psych |
Polychoric correlations, parallel analysis, EFA, alpha, beta, omega |
lavaan |
Ordinal CFA (WLSMV estimator) |
dplyr / tibble |
Data wrangling and table construction |
The DAACS SRL survey (66 items, 5-point Likert scale) is organized into 4 scales and 14 subscales:
| Scale | Subscales | Items |
|---|---|---|
| Metacognition | Planning (5), Monitoring (6), Evaluation (3) | 14 |
| Motivation | Mastery Orientation (4), Growth Mindset (4), Managing Test Anxiety (4) | 12 |
| Using Learning Strategies | Learning Environment (4), Managing Time (4), Strategies for Understanding (12), Help Seeking (4) | 24 |
| Self-Efficacy | Mathematics (4), Writing (4), Reading (4), Online Learning (4) | 16 |
An earlier version of this work was presented as an e-board presentation at the 2023 NCME Annual Meeting in Chicago (Rosca, Colvin, Andrade, & Bryer, 2023). That conference paper compared empirical relationships among alpha, beta, omega hierarchical, and omega total to predictions from mathematical and simulation studies, using a 2022 treatment-group subsample (N = 2,190) and a sequential assumption-testing framework in R and Mplus.
The current study substantially expands and reframes the original work. The sample more than doubles (N = 5,429, pooled 2022-2023), all 19 score sets are retained (the 2023 version excluded several), the analytic pipeline moves entirely to R (replacing Mplus), polychoric correlations and WLSMV estimation replace Pearson/MLR, the omega hierarchical-to-total ratio is introduced as a general-factor saturation diagnostic, institution-specific sensitivity replications are added, and the framing shifts from coefficient comparison to score-use-aligned reporting guidance.
This study is part of a broader program of psychometric research on the DAACS formative assessment system. Other repositories in this series cover IRT calibration and DIF analyses for DAACS mathematics and reading assessments.
Oxana Rosca, PhD Psychometrician, Department of Educational & Counseling Psychology, University at Albany (SUNY)
Rosca, O. (2026). Internal consistency and internal structure evidence for interpreting Self-Regulated Learning survey scores [Manuscript submitted for publication]. Department of Educational & Counseling Psychology, University at Albany.
Rosca, O., Colvin, K. F., Andrade, H. L., & Bryer, J. (2023, April 13-15). Reliability coefficients for a single-administration linear assessment [e-Board presentation]. NCME 2023 Annual Meeting, Chicago, IL, United States. DOI: 10.13140/RG.2.2.22756.45447