Learning psychometrics: the statistics you need first

What psychometrics actually is

Psychometrics is the discipline of building and evaluating measurement instruments — scales, tests, questionnaires — for constructs you can't observe directly. When someone answers a five-item depression questionnaire, psychometrics is what tells you whether those five items actually reflect the same underlying thing, whether the score is stable enough to trust, and whether it behaves the same way for different groups of respondents. Every field that runs on survey data (education, HR, marketing, health outcomes, clinical research, UX research) sits on top of psychometric decisions, whether the researchers named them or not.

It's a small field with an outsized footprint: the SAT, the MMPI, the SF-36, the Big Five Inventory, engagement pulse surveys, patient-reported outcome measures — every one of those was built and re-validated by someone doing psychometrics. Getting the measurement layer right is what lets all the downstream analyses mean anything.

A staged learning path

Psychometrics teaches best when you learn the three layers in order: the statistics, then the measurement theory, then the specialist tools. It's tempting to skip ahead to Item Response Theory or CFA because they look impressive, but neither makes sense without the reliability and factor-analysis intuition underneath.

Layer 1 — statistical foundations

Before any measurement model, you need to be fluent in:

• Descriptive statistics — means, medians, standard deviations, and especially the distribution of Likert-scale items (which are ordinal and rarely normal).
• Correlation — the mathematical backbone of almost everything in psychometrics. Pearson for continuous, Spearman for ordinal; if you get comfortable with the correlation matrix, half of factor analysis is already intuitive.
• Choosing the right test — knowing when Likert data merit a non-parametric alternative, and when it's safe to treat the sum score as continuous (a live debate you'll want to read into).
• Regression — used everywhere in validity work: does the scale predict the outcome it should (criterion validity)? Does it explain incremental variance above known predictors (incremental validity)?
• Effect sizes and confidence intervals — reliability and validity are effect-size statistics, not p-values. A reliability of .78 with CI [.72, .83] says something a p-value never can.

Layer 2 — measurement theory

With the stats layer under you, the measurement theory:

• Classical Test Theory (CTT) — the foundational model: observed score = true score + error. Everything else in the layer builds on this single decomposition.
• Reliability — the share of variance in a score that isn't noise. Internal consistency (Cronbach's α, McDonald's ω), test–retest, parallel forms, and inter-rater (Cohen's κ, ICC) each address a different way a measurement can be inconsistent.
• Validity — whether the scale actually measures what it claims to. Content validity (do the items cover the construct?), construct validity (convergent + discriminant with related and unrelated scales), criterion validity (does it predict the real-world outcome?). Validity is a body of evidence, not a single number.
• Factor analysis — the tool that turns a wall of item-correlations into a small number of underlying dimensions. Learn EFA first (exploratory, when you don't know the structure yet), then CFA (confirmatory, when you're testing a theory).
• Item analysis — item-total correlations, item difficulty (mean score), item discrimination (how well the item separates high-scorers from low-scorers). This is where you find the item everyone answers the same way (dead weight) and the one that's negatively correlated with the rest (usually reverse-scored and mis-keyed).

Layer 3 — specialist tools and methods

Once you own layers 1 and 2, the specialist methods and their software are the next step. The most-used methods:

• Item Response Theory (IRT) — a shift from CTT that models each item's difficulty and discrimination directly, and puts respondents and items on the same scale. Includes Rasch (1PL), 2PL, 3PL for dichotomous data; PCM, GRM, GPCM for polytomous. It's what powers modern adaptive testing.
• Confirmatory Factor Analysis (CFA) — the theory-driven counterpart to EFA. Fit indices (CFI, TLI, RMSEA, SRMR), model comparison, and measurement invariance testing across groups (are men and women answering the same latent construct?).
• Structural Equation Modeling (SEM) — CFA plus regression, in one framework: latent variables predicting other latent variables. Mediation, moderation, and full theoretical-model tests live here.
• Differential Item Functioning (DIF) — the tool for finding items that behave differently for different subgroups (test bias). Essential for any high-stakes assessment.

The tools people actually reach for:

• R's psych package (Revelle) — the swiss-army knife: descriptives, reliability, EFA, and item-analysis in one place. Start here.
• lavaan (R) — the de-facto standard for CFA and SEM. Free, well-documented, active community.
• mirt (R) — the most-used IRT package: unidimensional, multidimensional, and mixed models.
• Mplus — commercial; historically the reference for latent-variable modelling with mixed data types. Still common in graduate programs and in published psychometric methods work.
• JASP / jamovi — free GUI stats platforms; both have solid EFA/CFA modules and are the friendliest first landing for people coming from SPSS.

Where thericerca can help — and where it can't

Psychometrics is one of the fields where a general-purpose stats platform can do more of the real work than you'd expect. thericerca handles nearly all of Layer 1, most of the CTT-side reliability work in Layer 2, and exploratory factor analysis end-to-end — enough to complete a first-pass scale validation without leaving the site. What it can't do lies squarely in Layer 3: the latent-variable modelling, IRT, and confirmatory methods.

Resources worth your time

• Books. DeVellis, Scale Development: Theory and Applications, is the standard introduction to building a scale from scratch. Furr, Psychometrics: An Introduction, is the best classroom-style textbook on the concepts. For SEM/CFA specifically, Kline's Principles and Practice of Structural Equation Modeling is the reference nearly every methodologist cites.
• Tutorials. William Revelle's An Introduction to Psychometric Theory with Applications in R is free online and paired with the psych package — the fastest path from zero to a working reliability + EFA workflow. UCLA's OARC statistical consulting notes are excellent for concrete example walkthroughs.
• Practice datasets. The bfi Big Five personality dataset (bundled with the psych package) is the standard teaching dataset for EFA and reliability; every course uses it. The HolzingerSwineford1939 dataset (bundled with lavaan) is the equivalent for CFA.
• Community. The Personality Project (Revelle's site at Northwestern) is the single best free resource on quantitative personality/psychometric analysis. Cross Validated (Stack Exchange) has a deep archive on measurement questions, and the R psychometrics mailing list is where working methodologists solve problems in public.

A concrete first month

If you're starting from a general statistics background, spend Week 1 on correlations and the distribution of Likert data — get comfortable reading a correlation matrix. Week 2, work through CTT and reliability, ideally on a real dataset. Week 3, learn EFA end-to-end: KMO, Bartlett, factor-count selection, rotation, loadings interpretation. Week 4, do a mini-validation project on a real scale — reliability, item analysis, an EFA to check dimensionality, and a correlation with a criterion variable.

thericerca can carry most of that first month. Use the free reliability tool for Cronbach and item-drop; use the guided study pipeline for the EFA and the correlational validity work; use sample size to plan a proper validation study; and use the test-picker when you're not sure whether a group comparison on a subscale should be a t-test or a Mann–Whitney U.

For the layer-3 methods — CFA, IRT, SEM — you'll graduate to lavaan, mirt, or Mplus. That's the way in, and there's no shortcut. But the measurement fluency you build in Layer 2 is what keeps you from misreading model output once you get there.