hands-on-data-analysis 第二单元 - 数据清洗及特征处理

1.缺失值观察与处理

首先当然是导入相应的模块

 #加载所需的库
 import numpy as np
 import pandas as pd

1.1 缺失值观察

接下来就是观察缺失值：

df.info()

 df.info()

 RangeIndex: 891 entries, 0 to 890
 Data columns (total 12 columns):
  #   Column       Non-Null Count  Dtype  
 ---  ------       --------------  -----  
  0   PassengerId  891 non-null    int64  
  1   Survived     891 non-null    int64  
  2   Pclass       891 non-null    int64  
  3   Name         891 non-null    object 
  4   Sex          891 non-null    object 
  5   Age          714 non-null    float64
  6   SibSp        891 non-null    int64  
  7   Parch        891 non-null    int64  
  8   Ticket       891 non-null    object 
  9   Fare         891 non-null    float64
  10  Cabin        204 non-null    object 
  11  Embarked     889 non-null    object 
 dtypes: float64(2), int64(5), object(5)
 memory usage: 83.7+ KB

df.isnull().sum()

 PassengerId      0
 Survived         0
 Pclass           0
 Name             0
 Sex              0
 Age            177
 SibSp            0
 Parch            0
 Ticket           0
 Fare             0
 Cabin          687
 Embarked         2
 dtype: int64

1.2 缺失值处理

数值列读取数据后，空缺值的NaN为浮点型，最好用np.nan判断是否是NaN。

isnull()可以筛选出缺失的值

 df[df['Age'].isnull()]
 df.tail(5)

np.isnan()也可以筛选出缺失的值

 df[np.isnan(df['Age'])]
 df.tail(5)

但是，np.isnan不可以用来与任何数值进行>,==，!=之类的比较

 np.nan != np.nan

 True

df.dropna(inplace=True)可以用来丢弃掉有NaN数据的那一行，其中inplace=True表示修改原数据。

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

df.fillna(0,inplace=True) 可以用来将NaN数据用0填充，其中inplace=True表示修改原数据。

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

2.重复值

重复值可以使用df.duplicated()来查询

 df[df.duplicated()]

drop_duplicates()可以用来删除重复值

 df = df.drop_duplicates()

3.分箱（离散化）处理

3.1.平均分箱

 # 将连续变量Age平均分箱成5个年龄段，并分别用类别变量12345表示
 df['AgeBand'] = pd.cut(df['Age'], 5,labels = [1,2,3,4,5])

3.2.划分分箱

 df['AgeBand'] = pd.cut(df['Age'],[0,5,15,30,50,80],labels = [1,2,3,4,5])

### 3.3.概率分箱

 df['AgeBand'] = pd.qcut(df['Age'],[0,0.1,0.3,0.5,0.7,0.9],labels = [1,2,3,4,5])

4.文本变量进行转换

4.1. 查看文本变量名和种类

 #方法一: value_counts
 df['Sex'].value_counts()

 #方法二: unique
 df['Sex'].unique()
 df['Sex'].nunique()

4.2 文本转换

 #将类别文本转换为12345
 #方法一: replace
 df['Sex_num'] = df['Sex'].replace(['male','female'],[1,2])

 #方法二: map
 df['Sex_num'] = df['Sex'].map({'male': 1, 'female': 2})

 #方法三: 使用sklearn.preprocessing的LabelEncoder
 from sklearn.preprocessing import LabelEncoder
 for feat in ['Cabin', 'Ticket']:
     lbl = LabelEncoder()
     print(f"feat is {feat}") 
     print("end")
     label_dict = dict(zip(df[feat].unique(), range(df[feat].nunique())))
     print(f"label_dict is {label_dict}")
     print("end label_dict")
     df[feat + "_labelEncode"] = df[feat].map(label_dict)
     df[feat + "_labelEncode"] = lbl.fit_transform(df[feat].astype(str))

5. 独热编码

 #OneHotEncoder
 for feat in ["Age", "Embarked"]:
     x = pd.get_dummies(df[feat], prefix=feat)
     df = pd.concat([df, x], axis=1)
 df.head()

参考资料

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html

本项目地址：

hands-on-data-analysis 第二单元 - 飞桨AI Studio (baidu.com)