Classification problems involve predicting a response variable based on  a set of feature variables for some entity. But there is another problem whose solution is a prerequisite for solving classification problem. We may want to know which among the set of feature variables are most strongly correlated to the response variables. Once we have identified those, we may only want to use that sub set of the feature variables to build the prediction model.

To put this in context, we will use the customer churn prediction problem, specifically for mobile telecom service provider customers. A customer may have attributes like minutes used, data used, number  of customer service calls etc. We are interested in identifying those feature attributes critical to effective prediction of whether a customer will close his or her account. It’s always beneficial to reduce the number of features in building prediction models. It’s known as feature sub set selection.

Correlation

For numerical variables, correlation between two variables is simply the co variance normalized by the products of two standard deviations. For categorical variables, i.e. variables having a finite set of unordered values  it’s more complex. There are several approaches for handling categorical data. The correlation statistic we will be using is called the Cramer Index.

The basic building block for many correlation statistic between categorical variables is the the Contingency Matrix. If a variable a has n possible values and a variable b has m possible values. Then the contingency matrix will me a n X m matrix. Each cell of the matrix will  contain a count of the number of samples that have the corresponding attribute  value pair.

If the we consider the attribute pair minutes used (MU) and account status (AS), we have a 3 x 2 Contingency Matrix as shown below

Just an inspection of the matrix, may provide valuable insight. For example, if we see high value for the bottom left cell i.e. minutes used high and account status open, we know customers are closing account, there is no suitable plan for high minute usage.

This is how   Cramer Index is defined in terms of Contingency Matrix. The index depends on how concentrated the values are across the cells.

CramerIndex = (sum(n(i,j) * n(i,j) / (nr(i) * nc(j))) – 1) / (min(numRow, numCol) – 1)
where
n(i,j) = value of (i,j) cell of contingency matrix
nr(i) = sum of values over all columns for the i th row
nc(j) = sum of values over all rows for the j th column
sum = sum over all i and j

The Cramer index will always be between 0 and 1, 0 indicating weakest correlation and 1 indicating the strongest correlation.

Map Reduce

The attributes to be correlated are specified through the configuration parameters source.attributes and dest.attributes. In the initialize() method of the mapper, for all the possible attribute pairs from the two sets an instance of a contingency matrix is created.

The Cramer Index implementation is part of my open source project avenir, which contains a collection classification and prediction algorithms implemented on Hadoop. This specific  map reduce implementation is available here.

As the mapper  processes each record. for each possible attribute pairs from the two sets, the values are extracted from the record. The value pair is used to locate a cell in the corresponding contingency matrix and it’s value incremented.

In the cleanup() method of the mapper, the contingency matrix for each attribute pair is emitted. The key is the attribute pair and the value is the serialized contingency matrix.

On the reducer side, contingency matrices for a given key i.e. attribute pair are aggregated and the Cramer Index  calculated based on the final aggregated contingency matrix. The reducer emits attribute pair followed by Cramer Index.

Customer Churn Analysis

The mobile service provider customer data has the following feature attributes in the data I am using . The data is over a period  going into the past.

Here are some sample input. The first field is the customer ID and the last field is the response attribute i.e., the account status. The remaining fields are the feature attributes as listed above.

KX9LBZ3ZVLII,med,med,med,poor,4,open
94PMT4ZQU47W,overage,high,low,average,1,closed
DIINUH7HZUUX,low,high,med,good,4,open
H6W0HROO0H2X,high,low,high,average,4,open
31P1TG4RTGQI,overage,med,low,average,1,closed
GTL7W53933LU,high,med,low,good,4,closed
JU39F4BSB70Z,overage,low,low,poor,2,open
2A4RURJLJ5EZ,high,high,low,good,3,open
FS2DZZ2VK063,low,low,low,good,2,open
B3U4OECQ628K,med,med,med,poor,3,closed
5OWQFS2EGIKV,med,low,low,good,2,open
JU2JVU0WL1Y1,overage,med,low,average,2,open

Most of the attributes above are numerical. They have been discretized into categorical values. The response variable is account status which is either open or closed. Each feature variable is paired with the response variable and the corresponding Cramer Index is emitted as output.

Here is the output. Form the output, we can see that minutes used has the strongest correlation to open status.

minUsed,status,0.022663449872222907
dataUsed,status,0.0038947124486503615
CSCalls,status,0.010164336836900434
payment,status,0.00905448707197265
acctAge,status,0.0030426459155057373

Wrapping Up

Not only does correlation calculation helps you identify the critical feature attributes towards the final prediction, it also provides valuable insight. Sometimes that insight is all you need, even if you don’t go all the way to solve the blown prediction problem.

For example,  if high minutes used is found to have the strongest correlation to account closing, a mobile service provider could pro actively seek out such customers, and offer them alternative calling plans before they leave. Here is the tutorial for the example.

Related Blogs:

• Should your company be open to open sourcing its software?
• The 4 Roles Your Chief Data Officer Must Play
• ThirdEye Data Ranked as Top 3 Big Data Analytics Company