8.1 Item-level Absolute Fit measures

Chen proposed three measures for assessing item-level absolute fit.

• Fit a model to a set of data

• Based on the model parameters, simulate item responses of a large number of students

• Denote the responses of students to item $j$ in the original sample by $Y_j$ and in the simulated sample by ${\tilde{Y}}_j$

• Calculate different item fit statistics:

  • proportion correct $p_j$. This measure compares the observed proportion correct and the model-predicted for item $j$. $$p_j=|P(Y_j=1)-P({\tilde{Y}}_j=1)|$$
  • transformed correlation $r_{j{j}^{\prime }}$: $$r_{j{j}^{\prime }}=|Z\left[Corr\right(Y_j,Y_{j^{\prime }}\left)\right]-Z\left[Corr\right({\tilde{Y}}_j,{\tilde{Y}}_{j^{\prime }}\left)\right]|$$
  • log odds ratio $l_{j{j}^{\prime }}$: $$l_{j{j}^{\prime }}=|\log \frac{N_{11}{N}_{00}}{N_{01}{N}_{10}}-\log \frac{{\tilde{N}}_{11}{\tilde{N}}_{00}}{{\tilde{N}}_{01}{\tilde{N}}_{10}}|$$

• Perform hypothesis tests: we can estimate their standard errors and z-scores can be obtained by dividing the statistics by their corresponding standard errors.

$$z[p_j]=\frac{p_j}{SE\left[p_j\right]}\sim N(0,1)$$

$$z[r_{j{j}^{\prime }}]=\frac{r_{j{j}^{\prime }}}{SE\left[r_{j{j}^{\prime }}\right]}\sim N(0,1)$$

$$z[l_{j{j}^{\prime }}]=\frac{l_{j{j}^{\prime }}}{SE\left[l_{j{j}^{\prime }}\right]}\sim N(0,1)$$

Exercise

If a test has 10 items, how many $z[p_j]$, $z[r_{j{j}^{\prime }}]$ and $z[l_{j{j}^{\prime }}]$ do we have?