How to parameterize and identify an IRT 2-PL model? (pymc-bot test) (original) (raw)

@pymc-bot I would like to write an item-response theory 2-parameter logistic (IRT-2PL) model with hierarchical priors for the three effects: student ability, item difficulty, and item discriminativeness. I would like advice on several things around how to parameterize the model so it will sample well in PyMC, including (a) how to deal with the additive and multiplicative non-identifiability (e.g., pinning values, enforcing sum-to-zero constraints), (b) how to deal with the parameterization of the hierarchical model (e.g., centered vs. non-centered), (c) how to deal with constraining discriminativeness to positive values, and (d) whether to introduce constraints (e.g., sum to zero, pinning one value to zero). Please give me Python code to simulate data from the prior, the Python code for the PyMC model, and the Python code to fit and test the fit. If you could explain how I could extend this to simulation-based calibration or even give me code to do that (with either plotting or a simple uniformity test such as one that inverse cdf transforms to normal)
, that would be even better.

pymc-bot April 21, 2025, 10:30pm 2

There are no relevant documents available to address your query directly. However, I can provide guidance based on general knowledge.

To parameterize and identify an IRT 2-PL model in PyMC, you can follow these steps:

  1. Model Specification:
    • Define the latent traits (e.g., student ability) and item parameters (difficulty and discriminativeness).
    • Use hierarchical priors for these parameters to account for variability across students and items.
  2. Dealing with Non-Identifiability:
    • Additive Non-Identifiability: You can enforce sum-to-zero constraints on the ability parameters to address this.
    • Multiplicative Non-Identifiability: Pin one of the discriminativeness parameters to a fixed value (e.g., 1) to resolve this issue.
  3. Parameterization of the Hierarchical Model:
    • Use a non-centered parameterization for hierarchical models to improve sampling efficiency, especially when the data is sparse or the model is complex.
  4. Constraining Discriminativeness:
    • Use a positive prior distribution (e.g., HalfNormal) for the discriminativeness parameters to ensure they remain positive.
  5. Python Code for PyMC Model:
import pymc as pm  
import numpy as np  
# Simulate data  
n_students = 100  
n_items = 20  
true_ability = np.random.normal(0, 1, n_students)  
true_difficulty = np.random.normal(0, 1, n_items)  
true_discriminativeness = np.abs(np.random.normal(1, 0.5, n_items))  
# PyMC model  
with pm.Model() as model:  
    ability = pm.Normal('ability', mu=0, sigma=1, shape=n_students)  
    difficulty = pm.Normal('difficulty', mu=0, sigma=1, shape=n_items)  
    discriminativeness = pm.HalfNormal('discriminativeness', sigma=1, shape=n_items)  
    # IRT 2PL model  
    theta = ability[:, None] - difficulty[None, :]  
    p = pm.math.sigmoid(discriminativeness[None, :] * theta)  
    observed = pm.Bernoulli('observed', p=p, observed=simulated_data)  
    # Sampling  
    trace = pm.sample(1000, return_inferencedata=True)  
# Fit and test the fit  
pm.plot_trace(trace)
  1. Simulation-Based Calibration:
    • You can extend this model to perform simulation-based calibration by generating posterior predictive samples and comparing them to the observed data using plots or statistical tests.

This code provides a basic framework for setting up an IRT 2-PL model in PyMC. You can adjust the priors and constraints based on your specific needs and data characteristics.

This answer was generated by the PyMC Assistant bot in response to your command. Please let us know if it was helpful!

Nope. That answer is terrible in so many ways I’m going to need a list.

  1. It said it would produce a hierarchical model, but didn’t.
  2. I asked it to generate simulated data to fit, but it only generates simulated parameters—it uses a variable simulated_data that doesn’t exist in the model.
  3. It did not solve either of the non-identifiability problems, though it mentioned how to solve them in its initial description.
  4. It did not generate the code I asked to compare the fit to the simulated values.
  5. No extra credit—that explanation of SBC doesn’t say how to do it.

On the plus side, it did code the likelihood correctly even if the simulated data doesn’t exist.

I tried the same prompt in ChatGPT o4-mini-high (they seriously need some rebranding) and the result was equally terrible—it got the manual non-centering backwards. And it also failed to write the SBC code for me or even simple evaluation code. See PyMC example chat.

When I asked the same model in a new chat (don’t know what to make of that now that history is always on) to try it for Stan with a Python driver, it got the non-centering right, the simulation right, and the SBC right, but still couldn’t figure out how to identify the model. Which I guess puts it right about where our mid-level users are at—this kind of how-to-identify a model comes up all the time, which is why I chose it as an example. See Stan example chat.

From experience this onnyx thing with specific sources that the “bot” is using is also pretty myopic. If the answer doesn’t kind of exist already in one the sources (docs, discourse) it refuses to branch out even if the base unconstrained model would figure something better.

FWIW here is Gemini 2.5 Pro’s answer. It does simulate some data, build a hierarchical model (non-centered), applies some constraints, and does some calibration checks at the end. Amusingly calls Neal’s funnel the “funnel from hell”.

(actually, just looks like some parameter recovery plots – the calibration is turned off by default)

That is a lot better. Even though it’s not perfect, the explanations are generally good. And then it has some LLM quirks of inconsistency. After describing the right thing to do (giving the ability parameters a standard normal prior), it then describes the wrong thing to do in the description (affine transform of ability parameters), then turns it around and does the right thing in the code.

It’s right about soft identification, but wrong that it’s usually sufficient for these models. I chose this form of model exactly because the soft identification is problematic with so much non-identifiability in the raw likelihood. These models fit way better with sum-to-zero constraints, though it’s then harder to generalize to new groups (here new students or new questions).

It turned off SBC in the output, but it wrote a function to do it that has the right overall skeleton. I don’t speak Python and PyMC well enough to read this quickly, but the doc comments are on the right track.