Visualizing the Correlation Matrix

The most straightforward and effective way to identify highly correlated features is through a correlation matrix. This matrix illustrates the correlation between every numeric feature pair in the dataset.

Consider the Melbourne Housing dataset, which consists of 13 numeric features. We can easily calculate the correlation matrix by invoking the .corr() method on the DataFrame, followed by Seaborn's heatmap function to create an appealing visual representation.

However, the default heatmap can be improved. We can create a customized diverging color palette (blue -> white -> red), center the color bar around 0, enable annotations to display each correlation value, and format it to two decimal points.

As shown, the matrix displays each correlation twice, since the correlation between feature A and B is identical to that of feature B and A. Additionally, the diagonal consists of 1s, representing self-correlations. This excess information can be overwhelming; therefore, we can eliminate the upper half of the matrix.

In line 4 of the code snippet, a 2D boolean mask is generated. The np.ones_like function creates a 2D numpy array with True values, which is then transformed into a boolean mask using np.triu. We can then apply this mask to the heatmap's masking argument, producing a clean plot.

Here is a small function to create these ideal correlation matrices:

Even though we could programmatically identify all highly correlated features, visual exploration remains essential. A heatmap can reveal whether correlations are logical. For example, a high correlation between the number of newborns and the number of storks might not be meaningful. Thus, engaging in visual exploration helps ascertain whether feature groups are genuinely related.

Removing Highly Correlated Features

Next, we will explore how to eliminate highly correlated features. We will utilize the Ansur Dataset, which includes extensive body measurements of 6000 US Army personnel and contains 98 numeric features. This dataset is perfect for honing your feature selection skills, as it features numerous correlated attributes.

Focusing solely on numeric features, we will isolate them for analysis. Notably, attempting to visualize the correlation matrix for such a large dataset would be impractical.

Our goal is to eliminate strong correlations, both positive and negative. Therefore, we will construct the matrix using the absolute values of correlations by chaining the .abs() method with .corr().

Subsequently, we will create a boolean mask to subset the matrix. The mask function within Pandas DataFrames allows us to assign NaN values to the upper half and diagonal of the matrix.

Next, we need to determine a threshold to decide whether to remove a feature. This threshold must be chosen with care and a thorough understanding of the dataset. In this case, we will set a threshold of 0.9:

The list comprehension identifies all columns that meet the 0.9 threshold. Using "any" enables us to avoid a nested loop, resulting in a more elegant solution. If we hadn't applied NaNs to the upper half and diagonal, we would have inadvertently discarded all correlated features instead of retaining one.

Here is a function that performs these operations based on a specified threshold and returns a list of columns to be removed:

Now, let’s proceed to drop those columns:

To evaluate whether this decision was justified, we will compare the performance of two RandomForestRegressors: one using the complete dataset and another utilizing the selected features. The target variable will be weight in pounds (Weightlbs):

Impressively, we achieved excellent results even with a significant number of features. Now, let's run the same analysis on the feature-selected DataFrame:

Surprisingly, we observed only a 2% decrease in test score while achieving a twofold reduction in runtime.