Cognitive Diagnosis Modeling
1
Introduction
2
An Introduction to Diagnostic Assessment and Modeling
2.1
Assessments of Learning
2.2
Assessments for Learning
2.3
A comparion
2.4
Educational assessments: reasoning from evidence
2.4.1
IRT as psychometric models
2.4.2
CDM as psychometric models
2.4.3
IRT vs CDM
2.5
An example: A proportional reasoning test
2.5.1
Data
2.5.2
CTT/IRT analysis
2.5.3
CDM analysis
2.6
Some Terms
2.6.1
Attribute
2.6.2
Diagnosis
2.6.3
Latent class
2.7
CDM inputs and outputs
2.7.1
Input: Item responses
2.7.2
Input: Q-matrix
2.7.3
Output: item parameter estimates
2.7.4
Output: Population-level profile estimates
2.7.5
Output: Individual-level profile estimates
3
CDMs for dichotomous data
3.1
MIRT vs CDMs
3.1.1
Multidimensional item response theory (IRT) models
3.1.2
Cognitive Diagnosis models
3.2
CDM Notations
3.3
Attribute Profiles
3.4
Condensation Rules
3.5
The DINA model
3.6
The DINO model
3.7
Reduced RUM
3.8
The LLM
3.9
The Generalized DINA model
3.10
G-DINA model: Link functions and reduced models
3.10.1
G-DINA and DINA
3.10.2
GDINA vs DINO
3.10.3
3.10.4
G-DINA vs LLM and A-CDM
3.11
Joint Attribute Distribution
3.12
The saturated model for joint attribute distribution
3.13
The independent model for joint attribute distribution
3.14
The multivariate normal model for joint attribute distribution
3.15
The higher-order model for joint attribute distribution
3.16
The hierarchical structure model for joint attribute distribution
3.17
The loglinear structure model for joint attribute distribution
4
R Lab I: CDM Analysis
4.1
Resources for Learning the Package
4.2
Features of the GDINA R package
4.3
Data and Q-matrix
4.4
GDINA model estimation
4.5
DINA model estimation
4.6
ACDM estimation
4.7
R-RUM Estimation
4.8
Estimation of A combination of models
4.9
Estimation of a higher-order model
4.10
Estimation of CDMs with attribute hierarchy
4.11
Assignment I
5
Model Estimation
5.1
Model Estimation Approaches
5.2
Likelihood of an individual response vector
5.3
Likelihood of an individual response vector (Cont’d)
5.4
Joint maximum likelihood estimation
5.5
Marginalized likelihood of an individual response vector
5.6
Marginalized maximum likelihood estimation via EM algorithm
5.7
E-step
5.8
M-step
5.9
Joint attribute distribution parameters
5.10
Standard error estimation
5.10.1
Score function
5.10.2
Expected Fisher information
5.10.3
Observed Fisher information
5.10.4
Variance-covariance matrix of model parameters
5.11
Quanitities in GDINA R package
5.12
Bayesian approach for parameter estimation
5.13
MCMC for DINA model in Nimble R package
5.14
MCMC for DINA model in JAGS and R2jags R package
5.15
MCMC for DINA model in STAN
5.16
Optional exercise
6
Model Identifiability
6.1
Global Identifiability
6.1.1
Parameters in CDMs
6.1.2
Global identifiability in CDMs
6.2
Global identifiability of the DINA or DINO model
6.3
Global identifiability of general CDMs
6.4
Generic Identifiability for more general CDMs
6.5
Partial Identifiability of Attribute profiles
6.6
Partial Identifiability: Completeness of the Q-matrix
6.7
Local Identifiability
7
Model-data Fit Evaluation
7.1
Relative Model-Data Fit at Test Level
7.2
Relative Model-Data Fit at Test Level (Cont’d)
7.3
Relative Model-Data Fit at Test Level (Cont’d)
7.4
Relative Model-Data Fit at Test Level (Cont’d)
7.5
Relative Model-Data Fit at Test Level (Cont’d)
7.6
Test-level Absolute Fit measures
7.7
Full information statistics
7.8
Full information statistics (Cont’d)
7.9
Limited information statistics
7.10
Limited information statistics (Cont’d)
7.11
Limited information statistics (Cont’d)
7.12
Limited information statistics (Cont’d)
8
Item FIt Measures
8.1
Item-level Absolute Fit measures
8.2
Item-level Absolute Fit measures (Cont’d)
8.3
Item-level Relative Fit measures
8.4
Item-level Relative Fit measures (Cont’d)
8.5
Item-level Relative Fit measures (Cont’d)
8.6
Item-level Relative Fit measures (Cont’d)
8.7
Item-level Relative Fit measures (Cont’d)
8.8
Item-level Relative Fit measures (Cont’d)
8.9
Item-level Relative Fit measures (Cont’d)
8.10
Item-level Relative Fit measures (Cont’d)
8.11
Item-level Relative Fit measures (Cont’d)
9
ATTRIBUTE PROFILE ESTIMATION
9.1
Maximum Likelihood Estimation (MLE)
9.2
Maximum a Posterior (MAP) Estimation
9.3
Expected a Posterior (EAP) Estimation
9.4
Attribute Estimation in GDINA R package
9.5
Exercises
10
CLASSIFICATION ACCURACY
10.1
The Monte Carlo Approach
10.2
Monte Carlo Approach Using R
10.3
The Analytic Approach
10.4
\(2\times 2\)
Contingency Table in Practice
10.5
Estimating
\(2\times 2\)
Contingency Table
10.6
Pattern Classification Accuracy
10.7
Estimating Classification Accuracy using GDINA R package
10.8
Estimating classification accuracy using multiple imputations
10.9
Exercises
11
CONDUCTING SIMULATION STUDY
11.1
What is a simulation study?
11.2
Some terms
11.2.1
Factor
11.2.2
Level/Condition
11.2.3
Condition combination
11.2.4
Replication
11.3
How to conduct simulations?
11.4
An example: background
11.5
An example: research question and design
11.5.1
research question
11.5.2
factors
11.5.3
conditions
11.5.4
design
11.6
An example: generating data
11.7
An example: estimating parameters
11.8
An example: comparing true and estimated parameters
11.9
An example: replicating the procedure a specified number of times
11.10
Assignment II
12
Q-matrix Validation
12.1
Introduction
12.2
Introduction (Cont’d)
12.3
GDINA discrimination index (GDI)
12.4
Q-matrix validation using the PVAF method
12.5
Q-matrix validation using the PVAF method
12.6
Q-matrix validation using the PVAF method
12.7
Modified PVAF approach and mesa plot
12.8
Stepwise method
12.9
Stepwise method (Cont’d)
12.10
Stepwise method (Cont’d)
12.11
Exercise
References
Published with bookdown
Handout for Cognitive Diagnosis Modeling
7.6
Test-level Absolute Fit measures
If a model fits data, the model predictions and observations should be close to each other.