重复值处理

• 数据清洗一般先从重复值和缺失值开始处理
• 重复值一般采取删除法来处理
• 但有些重复值不能删除，例如订单明细数据或交易明细数据等

import pandas as pd
import numpy as np
import os

  

 

  
1
  
2
  
3
 

os.getcwd()

  

 

  
1
 

'D:\\Jupyter\\notebook\\Python数据清洗实战\\数据清洗之数据预处理'

  

 

  
1
 

os.chdir('D:\\Jupyter\\notebook\\Python数据清洗实战\\数据')

  

 

  
1
 

df = pd.read_csv('MotorcycleData.csv', encoding='gbk', na_values='Na')

  

 

  
1
 

df.head(5)

  

 

  
1
 

5 rows × 22 columns

def f(x): if '$' in str(x): x = str(x).strip('$') x = str(x).replace(',', '') else: x = str(x).replace(',', '') return float(x)

  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
 

df['Price'] = df['Price'].apply(f)

  

 

  
1
 

df['Mileage'] = df['Mileage'].apply(f)

  

 

  
1
 

df.head(5)

  

 

  
1
 

5 rows × 22 columns

df.info()

  

 

  
1
 

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7493 entries, 0 to 7492
Data columns (total 22 columns):
Condition 7493 non-null object
Condition_Desc 1656 non-null object
Price 7493 non-null float64
Location 7491 non-null object
Model_Year 7489 non-null float64
Mileage 7467 non-null float64
Exterior_Color 6778 non-null object
Make 7489 non-null object
Warranty 5108 non-null object
Model 7370 non-null object
Sub_Model 2426 non-null object
Type 6011 non-null object
Vehicle_Title 268 non-null object
OBO 7427 non-null object
Feedback_Perc 6611 non-null object
Watch_Count 3517 non-null object
N_Reviews 7487 non-null object
Seller_Status 6868 non-null object
Vehicle_Tile 7439 non-null object
Auction 7476 non-null object
Buy_Now 7256 non-null object
Bid_Count 2190 non-null float64
dtypes: float64(4), object(18)
memory usage: 1.3+ MB

  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
  
8
  
9
  
10
  
11
  
12
  
13
  
14
  
15
  
16
  
17
  
18
  
19
  
20
  
21
  
22
  
23
  
24
  
25
  
26
  
27
 

any(df.duplicated())

  

 

  
1
 

True

  

 

  
1
 

# 显示重复数据
# df[df.duplicated()] 

  

 

  
1
  
2
 

# 统计重复数据
np.sum(df.duplicated())

  

 

  
1
  
2
 

1221

  

 

  
1
 

# 删除重复值
df.drop_duplicates(inplace=True)

  

 

  
1
  
2
 

df.columns

  

 

  
1
 

Index(['Condition', 'Condition_Desc', 'Price', 'Location', 'Model_Year', 'Mileage', 'Exterior_Color', 'Make', 'Warranty', 'Model', 'Sub_Model', 'Type', 'Vehicle_Title', 'OBO', 'Feedback_Perc', 'Watch_Count', 'N_Reviews', 'Seller_Status', 'Vehicle_Tile', 'Auction', 'Buy_Now', 'Bid_Count'], dtype='object')

  

 

  
1
  
2
  
3
  
4
  
5
  
6
 

# 根据指定变量判断重复值
df.drop_duplicates(subset=['Condition', 'Condition_Desc', 'Price', 'Location'], inplace=True)

  

 

  
1
  
2
 

# 重复已经被删除
np.sum(df.duplicated())

  

 

  
1
  
2
 

0

  

 

  
1
 

文章来源: ruochen.blog.csdn.net，作者：若尘，版权归原作者所有，如需转载，请联系作者。

原文链接：ruochen.blog.csdn.net/article/details/105611012