7.5 Relative Model-Data Fit at Test Level (Cont’d)

To evaluate the model data fit, we need to fit some CDMs to the data first. Let us fit the DINO model to data 1 and Q1.

Code

# Load the GDINA package
library(GDINA)
data1 <- read.table(file = "./data/data1.dat", header = TRUE)
Q1 <- read.table(file = "./data/Q1.txt")
# Fit the DINO model verbose can be used to control what to be printed during the estimation
fit1 <- GDINA(dat = data1, Q = Q1, model = "DINO", verbose = 0)

summary and several other functions can be used to print some test level relative model-data fit statistics:

Code

summary(fit1)

## 
## Test Fit Statistics
## 
## Loglik = -7524.80 
## 
## AIC    = 15123.60  | penalty [2 * p]  = 74.00 
## BIC    = 15298.61  | penalty [log(n) * p]  = 249.00 
## CAIC   = 15335.61  | penalty [(log(n) + 1) * p]  = 286.00 
## SABIC  = 15181.11  | penalty [log((n + 2)/24) * p]  = 131.50 
## 
## No. of parameters (p)  = 37 
##   No. of estimated item parameters =  30 
##   No. of fixed item parameters =  0 
##   No. of distribution parameters =  7 
## 
## Attribute Prevalence
## 
##    Level0 Level1
## A1   0.64   0.36
## A2   0.53   0.47
## A3   0.76   0.24

If we want to compare two nested models, we can use likelihood ratio test. AIC and BIC can also be used when two models are fitted to the same data. To compare whether the DINO model can fit the data as well as the G-DINA model, we use anova function with two model fit objects as the inputs:

Code

# fit1 is based on the DINO model fit2 is based on the G-DINA model
fit2 <- GDINA(dat = data1, Q = Q1, model = "GDINA", verbose = 0)
anova(fit1, fit2)

## 
## Information Criteria and Likelihood Ratio Test
## 
##      #par   logLik Deviance      AIC      BIC     CAIC    SABIC chisq df p-value
## fit1   37 -7524.80 15049.60 15123.60 15298.61 15335.61 15181.11 186.7 30  <0.001
## fit2   67 -7431.45 14862.91 14996.91 15313.81 15380.81 15101.04

We can also compare the fit of various CDM combinations with that of G-DINA model:

Code

# Fit different CDMs to different items
models <- c(rep("GDINA", 4), "LLM", "ACDM", "GDINA", "LLM", "RRUM", "ACDM", "RRUM", "ACDM", "DINA", "RRUM", "RRUM")
fit3 <- GDINA(dat = data1, Q = Q1, model = models, verbose = 0)
anova(fit1, fit2, fit3)

## 
## Information Criteria and Likelihood Ratio Test
## 
##      #par   logLik Deviance      AIC      BIC     CAIC    SABIC chisq df p-value
## fit1   37 -7524.80 15049.60 15123.60 15298.61 15335.61 15181.11 186.7 30  <0.001
## fit2   67 -7431.45 14862.91 14996.91 15313.81 15380.81 15101.04                 
## fit3   50 -7443.62 14887.24 14987.24 15223.73 15273.73 15064.95 24.33 17    0.11
## 
## Notes: In LR tests, models were tested against fit2 
##        LR test(s) do NOT check whether models are nested or not.