8.1 Item-level Absolute Fit measures

Chen proposed three measures for assessing item-level absolute fit.

• Fit a model to a set of data

• Based on the model parameters, simulate item responses of a large number of students

• Denote the responses of students to item \(j\) in the original sample by \(\mathbf{Y}_j\) and in the simulated sample by \(\mathbf{\tilde{Y}}_j\)

• Calculate different item fit statistics:

  • proportion correct \(p_j\). This measure compares the observed proportion correct and the model-predicted for item \(j\). \[ p_{j}=\Bigg\lvert P(\mathbf{Y}_j=1)-{P}(\mathbf{\tilde{Y}}_j=1)\Bigg\rvert \]
  • transformed correlation \(r_{jj'}\): \[ r_{jj'}=\Bigg\lvert Z[Corr(\mathbf{Y}_j,\mathbf{Y}_{j'})]-Z[Corr(\mathbf{\tilde{Y}}_j,\mathbf{\tilde{Y}}_{j'})]\Bigg\rvert \]
  • log odds ratio \(l_{jj'}\): \[ l_{jj'}=\Bigg\lvert \log \frac{N_{11}N_{00}}{N_{01}N_{10}}-\log \frac{\tilde{N}_{11}\tilde{N}_{00}}{\tilde{N}_{01}\tilde{N}_{10}}\Bigg\rvert \]

• Perform hypothesis tests: we can estimate their standard errors and z-scores can be obtained by dividing the statistics by their corresponding standard errors.

\[ z\big[p_j\big]=\frac{p_j}{SE[p_j]}\sim N(0,1) \]

\[ z\big[r_{jj'}\big]=\frac{r_{jj'}}{SE[r_{jj'}]}\sim N(0,1) \]

\[ z\big[l_{jj'}\big]=\frac{l_{jj'}}{SE[l_{jj'}]}\sim N(0,1) \]

Exercise

If a test has 10 items, how many \(z\big[p_j\big]\), \(z\big[r_{jj'}\big]\) and \(z\big[l_{jj'}\big]\) do we have?