One-Hot Encoding a Feature on a Pandas Dataframe: an Example

Last updated:
One-Hot Encoding a Feature on a Pandas Dataframe: an Example
Source
Table of Contents

Dummy encoding is not exactly the same as one-hot encoding. For more information, see Dummy Variable Trap in regression models

When extracting features, from a dataset, it is often useful to transform categorical features into vectors so that you can do vector operations (such as calculating the cosine distance) on them.

Notebook available here

Think about it for a second: how would you naïvely calculate the distance between users using the cosine difference, where their country of origin is the only feature? You need a way that will correctly return zero for users that share the same country and 1 (maximum) for users that don't.

Twitter Linkedin YC Hacker News Reddit

How would you calculate the distance between users in a dataset, where their country of origin is the only feature?

Take this dataset for example:

+---+----------------+
| id| country        |
+---+----------------+
|  0| russia         |
|  1| germany        |
|  2| australia      |
|  3| korea          |
|  4| germany        |
+---+----------------+

One of the ways to do it is to encode the categorical variable as a one-hot vector, i.e. a vector where only one element is non-zero, or hot.

Twitter Linkedin YC Hacker News Reddit

With one-hot encoding, a categorical feature becomes an array whose size is the number of possible choices for that features, i.e.:

+---+---------------+----------------+------------------+---------------+
| id| country=russia| country=germany| country=australia|  country=korea|
+---+---------------+----------------+------------------+---------------+
|  0|              1|               0|                 0|              0|
|  1|              0|               1|                 0|              0|
|  2|              0|               0|                 1|              0|
|  3|              0|               0|                 0|              1|
|  4|              0|               1|                 0|              0|
+---+---------------+----------------+------------------+---------------+

One-hot encoding a column in a Pandas Dataframe

To create a dataset similar to the one used above in Pandas, we could do this:

import pandas as pd

df = pd.DataFrame({'country': ['russia', 'germany', 'australia','korea','germany']})

original-dataframe original-dataframe

Pandas provides the very useful get_dummies method on DataFrame, which does what we want:

import pandas pd

df = pd.DataFrame({'country': ['russia', 'germany', 'australia','korea','germany']})
pd.get_dummies(df,prefix=['country'])

original-dataframe original dataframe
         
one-hot-encoded-features one-hot-encoded-features

One-hot encoding vs Dummy variables

By default, the get_dummies() does not do dummy encoding, but one-hot encoding.

To produce an actual dummy encoding from your data, use drop_first=True (not that 'australia' is missing from the columns)

import pandas as pd

# using the same example as above
df = pd.DataFrame({'country': ['russia', 'germany', 'australia','korea','germany']})

pd.get_dummies(df,prefix=['country'], drop_first=True)

original-dataframe original dataframe
         
one-hot-encoded-with-drop-first Some machine learning techniques require you
to drop one dimension from the representation so as
to avoid dependency among the variables.
Use "drop_first=True" to achieve that.

Add columns for categories that only appear in the test set

You need to inform pandas if you want it to create dummy columns for categories even though never appear (for example, if you one-hot encode a categorical variable that may have unseen values in the test).

import pandas as pd

# say you want a column for "japan" too (it'll be always zero, of course)
df["country"] = train_df["country"].astype('category',categories=["australia","germany","korea","russia","japan"])

# now call .get_dummies() as usual
pd.get_dummies(df,prefix=['country'])
>>
   country_australia  country_germany  country_korea  country_russia  country_japan
0                  0                0              0               1              0 
1                  0                1              0               0              0
2                  1                0              0               0              0
3                  0                0              1               0              0
4                  0                1              0               0              0

Add dummy columns to dataframe

All in one line: df = pd.concat([df,pd.get_dummies(df['mycol'], prefix='mycol',dummy_na=True)],axis=1).drop(['mycol'],axis=1)

For example, if you have other columns (in addition to the column you want to one-hot encode) this is how you replace the country column with all 3 derived columns, and keep the other one:

Use pd.concat() to join the columns and then drop() the original country column:

import pandas as pd

# df now has two columns: name and country
df = pd.DataFrame({
        'name': ['josef','michael','john','bawool','klaus'],
        'country': ['russia', 'germany', 'australia','korea','germany']
    })

# use pd.concat to join the new columns with your original dataframe
df = pd.concat([df,pd.get_dummies(df['country'], prefix='country')],axis=1)

# now drop the original 'country' column (you don't need it anymore)
df.drop(['country'],axis=1, inplace=True)

original dataframe BEFORE: original columns 'name' and 'country'
dataframe-after-column-dropping AFTER: 'country' column has been replaced
by the derived one-hot columns

Treat Nulls/NaNs as a separate category

Use dummy_na=True:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'country': ['germany',np.nan,'germany','united kingdom','america','united kingdom']
})

pd.get_dummies(df,dummy_na=True)

In the original dataframe one of the values is NaN In the original dataframe one
of the values is NaN
ohe-dataset If you pass dummy_na=True, you get an extra column for NaNs


References

Dialogue & Discussion