Non-parametric One-way ANOVA

Regression datasets often have a mix of categorical and continuous predictor variables. When the number of categorical variables is large, how do you pick the ones that are relevant to the regression(i.e. correlated to the response variable)?

The Ames Housing dataset is a property dataset containing 79 features of which 43 are categorical. The task is to predict the saleprice of houses using regression. As part of feature selection step, we would use one-way ANOVA to rank the categorical features so that we can pick the top ones.

There are two ways to do this:

The visual approach

In this approach, we plot boxplots showing the distribution of SalePrice across the values of various categorical variables.

To see the boxplots in context, go to my Kaggle notebook here

From the boxplots, we can easily see that Paved Drive and Sale Type(among others) have a clear influence on the SalePrice.

We can use visual approach when there are a few categorical variables. When the number of categorical variables(also called factors) is more than a handful, we need a quantitative approach to rank their influence on the target variable.

The quantitative approach

Traditional ANOVA has strict requirements in terms of the response variable distributions being Gaussian with equal variances. However, this condition is violated in our dataset.

Hence, we will use the Kruskal-Wallis test which is the non-parametric version of the one-way ANOVA.

We will follow the following steps:

• Group the dataframe based on values of each categorical variable.
• Extract the response variable column from each group.
• Use these columns to do a one-way ANOVA.
• Create a dataframe sorted by p-values then by h-score and pick the top n.

The following is a code fragment from the same notebook mentioned above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

import scipy.stats as stats
response_var = 'SalePrice'  #or 'SalePrice' log
CATVAL, DF = 0,1  #C style enum type. See note below.
arg_list = []  #argument list for f oneway() function.
res_list = []  #result list with tuples: (f-score, p-value)

for feat in cat_feat_names_fin:
    #create cat feature value wise groups of the main dataframe.
    group_list = list(housingdf_dc.groupby(feat))  #get the groups into a list one dataframe per
    #cat value.
    #Structure of this array: Array of tuples - first element is cat value, second element is dataframe.
    #Note: Use the enum values above to traverse through group list.
    
    for grp in group_list:  #group gets tuples: (cat-val, df with selected cat-val)
        arg_list.append(grp[DF][response_var].values)
        #arg_list now has a list of response variable columns, one for each category value.
        
    #Apply the one-way Kruskal-Wallis (non-parametric) ANOVA.
    h, p = stats.kruskal(*arg_list) 
    res_list.append((feat,h,p))
    
fvalue_df = pd.DataFrame(res_list, columns = ['feat', 'h', 'p'])
#We have a dataframe with f/h scores and p-values; we need to pick the best n indicating strongest
#correlation.
fvalue_df.sort_values(['p','h'], ascending = [True,False], inplace = True)
fvalue_df.reset_index(drop = True, inplace = True) #resetting the index gives us an advantage.
#If two categories are related as shown by the chi square test, we can pick the one that has a 
#lower value as that will be more strongly correlated to the response variable.
fvalue_df.iloc[:,:]

From line 7 we iterate over the list of categorical variables, splitting the main dataframe into subframes one for each category value. From line 14, we iterate over the subframes to extract the response variable vectors(one per category value) and append it to an argument list preparatory to calling the ANOVA function. Finally, in the highlighted lines, we calculate the F-score for that particular feature and append it to a list which we later convert to a sorted dataframe from lines 22 onwards.

We would then get a table of categorical features in decreasing order of importance as seen below.

The list throws up some surprising and not-so-surprising insights!