next_inactive up previous


Cross-Validation in Arc

Sanford Weisberg
School of Statistics, University of Minnesota, St. Paul, MN 55108-6042.
Supported by National Science Foundation Grant DUE 97-52887.

December 7, 1999, revised January 31, 2002

Abstract:

This paper describes an Arc add-in for simple cross-validation.

Introduction

Cross-validation is a common method used for model checking in regression problems. The basic outline is:
  1. Divide the data into two parts, a ``construction" set and a ``validation" set.
  2. Fit a model of interest to the construction set.
  3. Compute a summary statistic, usually a function of the deviance and possibly the number of parameters, based on the validation set only.
  4. Select models that make the statistic chosen small (usually) when applied to the validation data.
This Arc add-in permits selecting the construction and validation set, and (2) automatic computation and printing of a few summary statistics for the validation set.


Getting the Code

Download the file http://www.stat.umn.edu/arc/cv.lsp. Place it in the Extras directory in your Arc directory (if you don't have such a directory, create one). The file will be automatically loaded every time you start Arc.


Using Cross-validation

Loading cv.lsp will add a menu item to the data set menu called Cross validation. Select this item to start using cross validation. You will get a dialog with three options: If you select Cross validation a second time will change the cases in the validation sample.

All models you fit with this data set will exclude the validation set from the fitting. The output will include a few summary statistics based on the validation set only. Here is the output for linear models:

Data set = AIS, Name of Fit = L1
64 cases have been deleted.
Normal Regression
Kernel mean function = Identity
Response      = LBM
Terms         = (Ht Wt RCC Sex)
Coefficient Estimates
Label      Estimate        Std. Error    t-value    p-value
Constant  -0.183342        7.72187        -0.024     0.9811
Ht         0.0920206       0.0397343       2.316     0.0221
Wt         0.646491        0.0266816      24.230     0.0000
RCC        0.920939        0.748719        1.230     0.2209
Sex       -8.63126         0.820372      -10.521     0.0000

R Squared:               0.957295    
Sigma hat:                2.88669    
Number of cases:             202
Number of cases used:        138
Degrees of freedom:          133

Summary Analysis of Variance Table
Source         df       SS            MS           F    p-value
Regression      4   24843.7       6210.93     745.34    0.0000
Residual      133   1108.29       8.33298    

Cross validation summary of cases not used to get estimates:
Sum of squared deviations:      355.439    
Mean squared deviation:         5.55373    
Sqrt(mean squared deviation):   2.35663    
Number of observations:             64
The additional quantites at the end of the regression output are for the validation set only. Users familiar with lisp can modify the method :display-cross-validation to give other statistics of interest.

The output for generalized linear models is based on deviance rather than squared residuals.


next_inactive up previous
S Weisberg
2002-01-31