一 前言
對于一個類別特征，如果這個特征的取值非常多，則稱它為高基數(shù)（high-cardinality）類別特征。在深度學(xué)習(xí)場景中，對于類別特征我們一般采用Embedding的方式，通過預(yù)訓(xùn)練或直接訓(xùn)練的方式將類別特征值編碼成向量。在經(jīng)典機器學(xué)習(xí)場景中，對于有序類別特征，我們可以使用LabelEncoder進行編碼處理，對于低基數(shù)無序類別特征（在lightgbm中，默認(rèn)取值個數(shù)小于等于4的類別特征），可以采用OneHotEncoder的方式進行編碼，但是對于高基數(shù)無序類別特征，若直接采用OneHotEncoder的方式編碼，在目前效果比較好的GBDT、Xgboost、lightgbm等樹模型中，會出現(xiàn)特征稀疏性的問題，造成維度災(zāi)難， 若先對類別取值進行聚類分組，然后再進行OneHot編碼，雖然可以降低特征的維度，但是聚類分組過程需要借助較強的業(yè)務(wù)經(jīng)驗知識。本文介紹一種針對高基數(shù)無序類別特征非常有效的預(yù)處理方法：平均數(shù)編碼（Mean Encoding）。在很多數(shù)據(jù)挖掘類競賽中，有許多人使用這種方法取得了非常優(yōu)異的成績。
二 原理
平均數(shù)編碼，有些地方也稱之為目標(biāo)編碼（Target Encoding），是一種基于目標(biāo)變量統(tǒng)計（Target Statistics）的有監(jiān)督編碼方式。該方法基于貝葉斯思想，用先驗概率和后驗概率的加權(quán)平均值作為類別特征值的編碼值，適用于分類和回歸場景。平均數(shù)編碼的公式如下所示：
其中：
1. prior為先驗概率，在分類場景中表示樣本屬于某一個_y__i_的概率
?其中_n__y__i_??表示y =_y__i_?時的樣本數(shù)量，_n__y_?表示y的總數(shù)量；在回歸場景下，先驗概率為目標(biāo)變量均值：
2. posterior為后驗概率，在分類場景中表示類別特征為k時樣本屬于某一個_y__i_?的概率
在回歸場景下表示 類別特征為k時對應(yīng)目標(biāo)變量的均值。
3. _λ_為權(quán)重函數(shù)，本文中的權(quán)重函數(shù)公式相較于原論文做了變換，是一個單調(diào)遞減函數(shù)，函數(shù)公式：
其中 輸入是特征類別在訓(xùn)練集中出現(xiàn)的次數(shù)n，權(quán)重函數(shù)有兩個參數(shù)：
① k：最小閾值，當(dāng)n = k時，λ= 0.5，先驗概率和后驗概率的權(quán)重相同；當(dāng)n < k時，λ> 0.5, 先驗概率所占的權(quán)重更大。
② f：平滑因子，控制權(quán)重函數(shù)在拐點處的斜率，f越大，曲線坡度越緩。下面是k=1時，不同f對于權(quán)重函數(shù)的影響：
由圖可知，f越大，權(quán)重函數(shù)S型曲線越緩，正則效應(yīng)越強。
對于分類問題，在計算后驗概率時，目標(biāo)變量有C個類別，就有C個后驗概率，且滿足
一個 _y__i_? 的概率值必然和其他 _y__i_? 的概率值線性相關(guān)，因此為了避免多重共線性問題，采用平均數(shù)編碼后數(shù)據(jù)集將增加C-1列特征。對于回歸問題，采用平均數(shù)編碼后數(shù)據(jù)集將增加1列特征。
三 實踐
平均數(shù)編碼不僅可以對單個類別特征編碼，也可以對具有層次結(jié)構(gòu)的類別特征進行編碼。比如地區(qū)特征，國家包含了省，省包含了市，市包含了街區(qū)，對于街區(qū)特征，每個街區(qū)特征對應(yīng)的樣本數(shù)量很少，以至于每個街區(qū)特征的編碼值接近于先驗概率。平均數(shù)編碼通過加入不同層次的先驗概率信息解決該問題。下面將以分類問題對這兩個場景進行展開：
1. 單個類別特征編碼：
在具體實踐時可以借助category_encoders包，代碼如下：
import pandas as pd
from category_encoders import TargetEncoder
df = pd.DataFrame({'cat': ['a', 'b', 'a', 'b', 'a', 'a', 'b', 'c', 'c', 'd'],
'target': [1, 0, 0, 1, 0, 0, 1, 1, 0, 1]})
te = TargetEncoder(cols=["cat"], min_samples_leaf=2, smoothing=1)
df["cat_encode"] = te.transform(df)["cat"]
print(df)
# 結(jié)果如下：
cat target cat_encode
0 a 1 0.279801
1 b 0 0.621843
2 a 0 0.279801
3 b 1 0.621843
4 a 0 0.279801
5 a 0 0.279801
6 b 1 0.621843
7 c 1 0.500000
8 c 0 0.500000
9 d 1 0.634471
2. 層次結(jié)構(gòu)類別特征編碼：
對以下數(shù)據(jù)集，方位類別特征具有{‘N’: (‘N’, ‘NE’), ‘S’: (‘S’, ‘SE’), ‘W’: ‘W’}層級關(guān)系，以compass中類別NE為例計算_y__i_?=1，k = 2 f = 2時編碼值，計算公式如下：
其中_p_1為HIER_compass_1中類別N的編碼值，計算可以參考單個類別特征編碼: 0.74527，posterior=3/3=1，λ= 0.37754 ，則類別NE的編碼值：0.37754 * 0.74527 + （1 - 0.37754）* 1 = 0.90383。
代碼如下：
from category_encoders import TargetEncoder
from category_encoders.datasets import load_compass
X, y = load_compass()
# 層次參數(shù)hierarchy可以為字典或者dataframe
# 字典形式
hierarchical_map = {'compass': {'N': ('N', 'NE'), 'S': ('S', 'SE'), 'W': 'W'}}
te = TargetEncoder(verbose=2, hierarchy=hierarchical_map, cols=['compass'], smoothing=2, min_samples_leaf=2)
# dataframe形式，HIER_cols的層級順序由頂向下
HIER_cols = ['HIER_compass_1']
te = TargetEncoder(verbose=2, hierarchy=X[HIER_cols], cols=['compass'], smoothing=2, min_samples_leaf=2)
te.fit(X.loc[:,['compass']], y)
X["compass_encode"] = te.transform(X.loc[:,['compass']])
X["label"] = y
print(X)
# 結(jié)果如下，compass_encode列為結(jié)果列：
index compass HIER_compass_1 compass_encode label
0 1 N N 0.622636 1
1 2 N N 0.622636 0
2 3 NE N 0.903830 1
3 4 NE N 0.903830 1
4 5 NE N 0.903830 1
5 6 SE S 0.176600 0
6 7 SE S 0.176600 0
7 8 S S 0.460520 1
8 9 S S 0.460520 0
9 10 S S 0.460520 1
10 11 S S 0.460520 0
11 12 W W 0.403328 1
12 13 W W 0.403328 0
13 14 W W 0.403328 0
14 15 W W 0.403328 0
15 16 W W 0.403328 1
注意事項：
采用平均數(shù)編碼，容易引起過擬合，可以采用以下方法防止過擬合：
- 增大正則項f
- k折交叉驗證
以下為自行實現(xiàn)的基于k折交叉驗證版本的平均數(shù)編碼，可以應(yīng)用于二分類、多分類、回歸場景中對單一類別特征或具有層次結(jié)構(gòu)類別特征進行編碼，該版本中用prior對unknown類別和缺失值編碼。
from itertools import product
from category_encoders import TargetEncoder
from sklearn.model_selection import StratifiedKFold, KFold
class MeanEncoder:
def __init__(self, categorical_features, n_splits=5, target_type='classification',
min_samples_leaf=2, smoothing=1, hierarchy=None, verbose=0, shuffle=False,
random_state=None):
"""
Parameters
----------
categorical_features: list of str
the name of the categorical columns to encode.
n_splits: int
the number of splits used in mean encoding.
target_type: str,
'regression' or 'classification'.
min_samples_leaf: int
For regularization the weighted average between category mean and global mean is taken. The weight is
an S-shaped curve between 0 and 1 with the number of samples for a category on the x-axis.
The curve reaches 0.5 at min_samples_leaf. (parameter k in the original paper)
smoothing: float
smoothing effect to balance categorical average vs prior. Higher value means stronger regularization.
The value must be strictly bigger than 0. Higher values mean a flatter S-curve (see min_samples_leaf).
hierarchy: dict or dataframe
A dictionary or a dataframe to define the hierarchy for mapping.
If a dictionary, this contains a dict of columns to map into hierarchies. Dictionary key(s) should be the column name from X
which requires mapping. For multiple hierarchical maps, this should be a dictionary of dictionaries.
If dataframe: a dataframe defining columns to be used for the hierarchies. Column names must take the form:
HIER_colA_1, ... HIER_colA_N, HIER_colB_1, ... HIER_colB_M, ...
where [colA, colB, ...] are given columns in cols list.
1:N and 1:M define the hierarchy for each column where 1 is the highest hierarchy (top of the tree). A single column or multiple
can be used, as relevant.
verbose: int
integer indicating verbosity of the output. 0 for none.
shuffle : bool, default=False
random_state : int or RandomState instance, default=None
When `shuffle` is True, `random_state` affects the ordering of the
indices, which controls the randomness of each fold for each class.
Otherwise, leave `random_state` as `None`.
Pass an int for reproducible output across multiple function calls.
"""
self.categorical_features = categorical_features
self.n_splits = n_splits
self.learned_stats = {}
self.min_samples_leaf = min_samples_leaf
self.smoothing = smoothing
self.hierarchy = hierarchy
self.verbose = verbose
self.shuffle = shuffle
self.random_state = random_state
if target_type == 'classification':
self.target_type = target_type
self.target_values = []
else:
self.target_type = 'regression'
self.target_values = None
def mean_encode_subroutine(self, X_train, y_train, X_test, variable, target):
X_train = X_train[[variable]].copy()
X_test = X_test[[variable]].copy()
if target is not None:
nf_name = '{}_pred_{}'.format(variable, target)
X_train['pred_temp'] = (y_train == target).astype(int) # classification
else:
nf_name = '{}_pred'.format(variable)
X_train['pred_temp'] = y_train # regression
prior = X_train['pred_temp'].mean()
te = TargetEncoder(verbose=self.verbose, hierarchy=self.hierarchy,
cols=[variable], smoothing=self.smoothing,
min_samples_leaf=self.min_samples_leaf)
te.fit(X_train[[variable]], X_train['pred_temp'])
tmp_l = te.ordinal_encoder.mapping[0]["mapping"].reset_index()
tmp_l.rename(columns={"index":variable, 0:"encode"}, inplace=True)
tmp_l.dropna(inplace=True)
tmp_r = te.mapping[variable].reset_index()
if self.hierarchy is None:
tmp_r.rename(columns={variable: "encode", 0:nf_name}, inplace=True)
else:
tmp_r.rename(columns={"index": "encode", 0:nf_name}, inplace=True)
col_avg_y = pd.merge(tmp_l, tmp_r, how="left",on=["encode"])
col_avg_y.drop(columns=["encode"], inplace=True)
col_avg_y.set_index(variable, inplace=True)
nf_train = X_train.join(col_avg_y, on=variable)[nf_name].values
nf_test = X_test.join(col_avg_y, on=variable).fillna(prior, inplace=False)[nf_name].values
return nf_train, nf_test, prior, col_avg_y
def fit(self, X, y):
"""
:param X: pandas DataFrame, n_samples * n_features
:param y: pandas Series or numpy array, n_samples
:return X_new: the transformed pandas DataFrame containing mean-encoded categorical features
"""
X_new = X.copy()
if self.target_type == 'classification':
skf = StratifiedKFold(self.n_splits, shuffle=self.shuffle, random_state=self.random_state)
else:
skf = KFold(self.n_splits, shuffle=self.shuffle, random_state=self.random_state)
if self.target_type == 'classification':
self.target_values = sorted(set(y))
self.learned_stats = {'{}_pred_{}'.format(variable, target): [] for variable, target in
product(self.categorical_features, self.target_values)}
for variable, target in product(self.categorical_features, self.target_values):
nf_name = '{}_pred_{}'.format(variable, target)
X_new.loc[:, nf_name] = np.nan
for large_ind, small_ind in skf.split(y, y):
nf_large, nf_small, prior, col_avg_y = self.mean_encode_subroutine(
X_new.iloc[large_ind], y.iloc[large_ind], X_new.iloc[small_ind], variable, target)
X_new.iloc[small_ind, -1] = nf_small
self.learned_stats[nf_name].append((prior, col_avg_y))
else:
self.learned_stats = {'{}_pred'.format(variable): [] for variable in self.categorical_features}
for variable in self.categorical_features:
nf_name = '{}_pred'.format(variable)
X_new.loc[:, nf_name] = np.nan
for large_ind, small_ind in skf.split(y, y):
nf_large, nf_small, prior, col_avg_y = self.mean_encode_subroutine(
X_new.iloc[large_ind], y.iloc[large_ind], X_new.iloc[small_ind], variable, None)
X_new.iloc[small_ind, -1] = nf_small
self.learned_stats[nf_name].append((prior, col_avg_y))
return X_new
def transform(self, X):
"""
:param X: pandas DataFrame, n_samples * n_features
:return X_new: the transformed pandas DataFrame containing mean-encoded categorical features
"""
X_new = X.copy()
if self.target_type == 'classification':
for variable, target in product(self.categorical_features, self.target_values):
nf_name = '{}_pred_{}'.format(variable, target)
X_new[nf_name] = 0
for prior, col_avg_y in self.learned_stats[nf_name]:
X_new[nf_name] += X_new[[variable]].join(col_avg_y, on=variable).fillna(prior, inplace=False)[
nf_name]
X_new[nf_name] /= self.n_splits
else:
for variable in self.categorical_features:
nf_name = '{}_pred'.format(variable)
X_new[nf_name] = 0
for prior, col_avg_y in self.learned_stats[nf_name]:
X_new[nf_name] += X_new[[variable]].join(col_avg_y, on=variable).fillna(prior, inplace=False)[
nf_name]
X_new[nf_name] /= self.n_splits
return X_new
四 總結(jié)
本文介紹了一種對高基數(shù)類別特征非常有效的編碼方式：平均數(shù)編碼。詳細的講述了該種編碼方式的原理，在實際工程應(yīng)用中有效避免過擬合的方法，并且提供了一個直接上手的代碼版本。
作者：京東保險 趙風(fēng)龍文章來源：http://www.zghlxwxcb.cn/news/detail-694579.html
來源：京東云開發(fā)者社區(qū) 轉(zhuǎn)載請注明來源文章來源地址http://www.zghlxwxcb.cn/news/detail-694579.html
到了這里，關(guān)于高基數(shù)類別特征預(yù)處理：平均數(shù)編碼 | 京東云技術(shù)團隊的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
