In regression analysis, variable-selection procedures are aimed at selecting a reduced set of the independent variables – the ones providing the best fit to the model.
The criterion for selecting is usually the following F-statistic:
where n is the total number of data points, SSE is the sum squares due to error – that is, the sum of squares minimized by the least squares method. If adding the variable xp+1 to variables x1,…,xp does not improve (or deletion of the variable xp+1 does not worsen) the fit significantly, this statistic follows an F-distribution; otherwise, the statistic tends to take on larger values.
There are several methods for variable-selection procedures. Some of them are step-wise regression, forward selection, backward elimination.