python3-pandas 缺失数据的处理

1、缺失数据的处理

1.1、判断是否为nan

isnull：是nan为true
notnull：不是nan为true

import pandas as pd
import numpy as npt3 = pd.DataFrame(np.arange(12).reshape(3,4), index=list("abc"), columns=list("wxyz"))
print(t3)
"""w  x   y   z
a  0  1   2   3
b  4  5   6   7
c  8  9  10  11
"""
t3.iloc[1:,:2] = np.nan
print(t3)
"""w    x   y   z
a  0.0  1.0   2   3
b  NaN  NaN   6   7
c  NaN  NaN  10  11
"""
print(pd.isnull(t3))
"""w      x      y      z
a  False  False  False  False
b   True   True  False  False
c   True   True  False  False
"""
print(pd.notnull(t3))
"""w      x     y     z
a   True   True  True  True
b  False  False  True  True
c  False  False  True  True
"""
print(t3[pd.notnull(t3["w"])])
"""w    x  y  z
a  0.0  1.0  2  3
"""

1.2、删除为nan的行或列

https://www.runoob.com/pandas/pandas-cleaning.html
dropna(axis=0,how=“any”, thresh=None,subset=None，inplace=False)

dropna 删除为nan 的行或者列 ;
axis=0 删除行 axis=1 删除列
how=“all” 全为nan 的删除， how=“any” 有一个为nan的就删除
inplace=True 就地修改，修改后直接会修改原数据，默认为 False，类似于 t3 = t3.dropna(axis=0, how=“any”)

print(t3.dropna(axis=0, how="all"))
"""w    x   y   z
a  0.0  1.0   2   3
b  NaN  NaN   6   7
c  NaN  NaN  10  11
"""
print(t3.dropna(axis=0, how="any"))
"""w    x  y  z
a  0.0  1.0  2  3
"""

1.3、为nan 填充数据

print(t3)
"""w    x   y   z
a  0.0  1.0   2   3
b  NaN  NaN   6   7
c  NaN  NaN  10  11
"""
# 填充固定数字
print(t3.fillna(10))
"""w     x   y   z
a   0.0   1.0   2   3
b  10.0  10.0   6   7
c  10.0  10.0  10  11
"""
# 填充均值
print(t3.fillna(t3.mean()))
"""w    x   y   z
a  0.0  1.0   2   3
b  0.0  1.0   6   7
c  0.0  1.0  10  11
"""
# 只对某一列 填充均值
print(t3["x"].fillna(t3["x"].mean()))
"""
a    1.0
b    1.0
c    1.0
Name: x, dtype: float64
"""
t3["x"] = t3["x"].fillna(t3["x"].mean())
print(t3)
"""w    x   y   z
a  0.0  1.0   2   3
b  NaN  1.0   6   7
c  NaN  1.0  10  11
"""
"""
算均值时 pandas 不会把 nan 的算进去
但是 nunpy 会把nan 的算进去
"""
print(np.array([[1,6,5,np.nan],[1,6,5,np.nan]]).mean())  # nan
print(np.array([[1,1,1,1],[2,2,2,2]]).mean())  # 1.5

不要随便把nan赋值为0，不然计算均值等值时，pandas会把它计算在内
算均值时 pandas 不会把 nan 的算进去，但是 nunpy 会把nan 的算进去

1.4、清洗重复数据

https://www.runoob.com/pandas/pandas-cleaning.html

import pandas as pdperson = {"name": ['Google', 'Runoob', 'Runoob', 'Taobao'],"age": [50, 40, 40, 23]
}
df = pd.DataFrame(person)print(df.duplicated())
"""
0    False
1    False
2     True
3    False
dtype: bool
"""

删除重复数据，可以直接使用drop_duplicates() 方法。

import pandas as pdpersons = {"name": ['Google', 'Runoob', 'Runoob', 'Taobao'],"age": [50, 40, 40, 23]
}df = pd.DataFrame(persons)df.drop_duplicates(inplace = True)
print(df)
"""name  age
0  Google   50
1  Runoob   40
3  Taobao   23
"""

视频顺序错位 30–》29–》42
https://www.kaggle.com/damianpanek/sunday-eda/data

https://www.bilibili.com/video/BV1hx411d7jb?p=32

https://www.bilibili.com/video/BV1hx411d7jb?p=28
https://www.bilibili.com/video/BV1hx411d7jb?p=30
https://www.bilibili.com/video/BV1hx411d7jb?p=29
https://www.bilibili.com/video/BV1hx411d7jb?p=42
https://www.bilibili.com/video/BV1hx411d7jb?p=31