Saturday, September 08, 2007

cross validation fit index

ECVI (Expected cross-validation index)

  • ECVI is proposed as a means to assess, in a single sample, the likelihood that the model cross-validates similar-size samples from the same population. It measures the discrepancy between the fitted covariance matrix in the analyzed sample, and the expected covariance matrix that would be obtained in another sample of equivalent size. Application of the ECVI assumes a comparison of models whereby an ECVI index is computed for each model and then all ECVI values placed in rank order. The model having the smallest ECVI value exhibits the greatest potential for replication. It is possible to take the precision of the estimated ECVI value into account through the formulation of confidence intervals. By reporting an ECVI value within the bounds of a 95% confidence interval, one can argue that over all possible randomly sampled ECVI, 95% of them will fall within the upper and lower limits of the interval constructed. -- Byrne(1994), Testing for the factorial validity, replication, and invariance of a measuring instrument: a paradigmatic application based on the Maslach Burnout Inventory. Multivariate Behavioral Research, 29 (3), 289-311

CVI (cross-validation index)

  • CVI is developed to measure the extent to which a model cross-validate over independent samples. CVI measures the discrepancy between the fitted covariance matrix in the calibration sample and the sample covariance matrix in the validation sample. The model having the smallest CVI value is the one expected to have the highest degree of stability in repeated samples. -- Byrne (1994) Testing for the factorial validity, replication, and invriance of a measuring instrument: a paradigmatic applicaiton based on the Maslach Burnout Inventory, Multivariate Behavioral Research, 29 (3), 289-311

AIC (Akaike information criterion)

  • AIC is a single sample criteria, thus we don't have to split sample into calibration and validation samples
  • the model that produces the smallest AIC will be selected

CAIC (consistent AIC)

  • CAIC is a single sample criteria
  • the model that produces the smallest CAIC will be selected

Reading list

  • Camstra,A., and Boomsma, A. (1992) Cross-validatin in regression and covariance structure analysis: an overview. Sociological Methods and Research, 21 (1), 89-115
  • Browne, M.W. (2000) Cross-validation methods. Journal of Mathematical Psychology, 44,108-132

No comments: