赛题介绍

在分布式系统中某个节点发生故障时，故障会沿着分布式系统的拓扑结构进行传播，造成自身节点及其邻接节点相关的KPI指标和发生大量日志异常。本次比赛提供分布式数据库的故障特征数据和标签数据，其中特征数据是系统发生故障时的KPI指标数据，KPI指标包括由feature0、feature1 …feature106共107个指标，标签数据为故障类别数据，共6个类别，用0、1、2、3、4、5分别表示6个故障，参赛人员可根据这些数据，借助机器学习、深度学习、web等技术搭建故障诊断系统，该系统支持用户上传训练集对模型进行训练和模型下载，同时支持用户上传单条或多条测试语句进行测试并可视化测试结果，支持测试结果下载。

baseline: RandomForest

数据分析

读取数据

df = pd.read_csv('data/train/train.csv', index_col=None)

判断是否有缺失值

df.isnull().any()
'''
output: True即为存在缺失值
sample_id     False
feature0       True
feature1       True
feature2       True
feature3       True...
feature103     True
feature104     True
feature105    False
feature106     True
label         False
Length: 109, dtype: bool
'''

数据标准化及缺失值填充

# 数据标准化
features = df.iloc[:, 1:-1]
numeric_features = features.dtypes[features.dtypes != 'object'].index
features[numeric_features] = features[numeric_features].apply(lambda x: (x - x.mean()) / (x.std())
)
# 在标准化数据之后，所有均值消失，因此我们可以将缺失值设置为0
features[numeric_features] = features[numeric_features].fillna(0)
features_labels = pd.concat([features, df[['label']]], axis=1)
train_features = pd.concat([df[['sample_id']], features], axis=1)
train_label = df[['sample_id', 'label']]
df = pd.concat([train_features, train_label[['label']]], axis=1)

观察数据基本信息

# 观察前五行数据
df.head()

# 数据大小
df.shape
'''
output:
(6296, 109)
'''

df.dtypes
'''
output:
sample_id       int64
feature0      float64
feature1      float64
feature2      float64
feature3      float64...
feature103    float64
feature104    float64
feature105    float64
feature106    float64
label           int64
Length: 109, dtype: object
'''

# 类别分布
df['label'].value_counts().sort_index().plot(kind='bar')
plt.show()df['label'].value_counts().sort_index().plot(kind='pie')
plt.show()

features.describe()

# 分组的平均数据统计
label_Summary = features_labels.groupby('label')
label_Summary.mean()

# 相关性矩阵
corr = features_labels.corr()
sns.set_context({'figure.figsize':[100, 100]})
fig = sns.heatmap(corr, xticklabels=corr.columns.values, yticklabels=corr.columns.values)
heatmap = fig.get_figure()
heatmap.savefig('work/heatmap.png', dpi=300)
corr

# 各个特征的概率密度函数
feature_names = features.columns.values.tolist()
for name in feature_names:fig = plt.figure(figsize=(15, 4), )ax = sns.kdeplot(df.loc[(df['label'] == 0), name], color='b', shade=True, label='0')ax = sns.kdeplot(df.loc[(df['label'] == 1), name], color='r', shade=True, label='1')ax = sns.kdeplot(df.loc[(df['label'] == 2), name], color='g', shade=True, label='2')ax = sns.kdeplot(df.loc[(df['label'] == 3), name], color='y', shade=True, label='3')ax = sns.kdeplot(df.loc[(df['label'] == 4), name], color='m', shade=True, label='4')ax = sns.kdeplot(df.loc[(df['label'] == 5), name], color='c', shade=True, label='5')ax.set(xlabel=name, ylabel='频率')plt.title('{} Probabilitydensity function'.format(name))plt.savefig('work/{}的概率密度函数图.png'.format(name))

划分数据集

from sklearn.model_selection import train_test_splittarget_name = 'label'
x = df.drop(['sample_id', 'label'], axis=1)
y = df[['label']]x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.15, random_state=123, stratify=y)

评价指标

# 指标计算 参数：array
def metrics_calculate(pred, y_test, txt_path):TP = [0, 0, 0, 0, 0, 0]FP = [0, 0, 0, 0, 0, 0]FN = [0, 0, 0, 0, 0, 0]for i in range(len(y_test)):if pred[i] == 0 and y_test[i] == 0:TP[0] += 1if pred[i] != 0 and y_test[i] == 0:FN[0] += 1if pred[i] == 0 and y_test[i] != 0:FP[0] += 1if pred[i] == 1 and y_test[i] == 1:TP[1] += 1if pred[i] != 1 and y_test[i] == 1:FN[1] += 1if pred[i] == 1 and y_test[i] != 1:FP[1] += 1if pred[i] == 2 and y_test[i] == 2:TP[2] += 1if pred[i] != 2 and y_test[i] == 2:FN[2] += 1if pred[i] == 2 and y_test[i] != 2:FP[2] += 1if pred[i] == 3 and y_test[i] == 3:TP[3] += 1if pred[i] != 3 and y_test[i] == 3:FN[3] += 1if pred[i] == 3 and y_test[i] != 3:FP[3] += 1if pred[i] == 4 and y_test[i] == 4:TP[4] += 1if pred[i] != 4 and y_test[i] == 4:FN[4] += 1if pred[i] == 4 and y_test[i] != 4:FP[4] += 1if pred[i] == 5 and y_test[i] == 5:TP[5] += 1if pred[i] != 5 and y_test[i] == 5:FN[5] += 1if pred[i] == 5 and y_test[i] != 5:FP[5] += 1Precision = [0, 0, 0, 0, 0, 0]Recall = [0, 0, 0, 0, 0, 0]F1 = [0, 0, 0, 0, 0, 0]if (TP[0] + FP[0]) == 0:Precision[0] = 0.0else:Precision[0] = TP[0] / (TP[0] + FP[0])if (TP[1] + FP[1]) == 0:Precision[1] = 0.0else:Precision[1] = TP[1] / (TP[1] + FP[1])if (TP[2] + FP[2]) == 0:Precision[2] = 0.0else:Precision[2] = TP[2] / (TP[2] + FP[2])if (TP[3] + FP[3]) == 0:Precision[3] = 0.0else:Precision[3] = TP[3] / (TP[3] + FP[3])if (TP[4] + FP[4]) == 0:Precision[4] = 0.0else:Precision[4] = TP[4] / (TP[4] + FP[4])if (TP[5] + FP[5]) == 0:Precision[5] = 0.0else:Precision[5] = TP[5] / (TP[5] + FP[5])# for i in range(6):#     print('Precision: {}\n'.format(Precision[i]))if (TP[0] + FN[0]) == 0:Recall[0] = 0.0else:Recall[0] = TP[0] / (TP[0] + FN[0])if (TP[1] + FN[1]) == 0:Recall[1] = 0.0else:Recall[1] = TP[1] / (TP[1] + FN[1])if (TP[2] + FN[2]) == 0:Recall[2] = 0.0else:Recall[2] = TP[2] / (TP[2] + FN[2])if (TP[3] + FN[3]) == 0:Recall[3] = 0.0else:Recall[3] = TP[3] / (TP[3] + FN[3])if (TP[4] + FN[4]) == 0:Recall[4] = 0.0else:Recall[4] = TP[4] / (TP[4] + FN[4])if (TP[5] + FN[5]) == 0:Recall[5] = 0.0else:Recall[5] = TP[5] / (TP[5] + FN[5])# for i in range(6):#     print('Recall: {}\n'.format(Recall[i]))if (Precision[0] + Recall[0]) == 0:F1[0] = 0.0else:F1[0] = (2 * Precision[0] * Recall[0]) / (Precision[0] + Recall[0])if (Precision[1] + Recall[1]) == 0:F1[1] = 0.0else:F1[1] = (2 * Precision[1] * Recall[1]) / (Precision[1] + Recall[1])if (Precision[2] + Recall[2]) == 0:F1[2] = 0.0else:F1[2] = (2 * Precision[2] * Recall[2]) / (Precision[2] + Recall[2])if (Precision[3] + Recall[3]) == 0:F1[3] = 0.0else:F1[3] = (2 * Precision[3] * Recall[3]) / (Precision[3] + Recall[3])if (Precision[4] + Recall[4]) == 0:F1[4] = 0.0else:F1[4] = (2 * Precision[4] * Recall[4]) / (Precision[4] + Recall[4])if (Precision[5] + Recall[5]) == 0:F1[5] = 0.0else:F1[5] = (2 * Precision[5] * Recall[5]) / (Precision[5] + Recall[5])# for i in range(6):#     print('F1: {}\n'.format(F1[i]))Macro_Precision = sum([Precision[0], Precision[1], Precision[2],Precision[3], Precision[4], Precision[5]]) / 6Macro_Recall = sum([Recall[0], Recall[1], Recall[2],Recall[3], Recall[4], Recall[5]]) / 6Macro_F1 = sum([F1[0], F1[1], F1[2], F1[3], F1[4], F1[5]]) / 6l_sum = sum([TP[0], TP[1], TP[2], TP[3], TP[4], TP[5]])m_sum = l_sum + sum([FP[0], FP[1], FP[2], FP[3], FP[4], FP[5]])n_sum = l_sum + sum([FN[0], FN[1], FN[2], FN[3], FN[4], FN[5]])if m_sum == 0:Micro_Precision = 0.0else:Micro_Precision = l_sum / m_sum# print('Micro_Precision: {}\n'.format(Micro_Precision))if n_sum == 0:Micro_Recall = 0.0else:Micro_Recall = l_sum / n_sum# print('Micro_Recall: {}\n'.format(Micro_Recall))if (Micro_Precision + Micro_Recall) == 0:Micro_F1 = 0.0else:Micro_F1 = (2 * Micro_Precision * Micro_Recall) / (Micro_Precision + Micro_Recall)# print('Micro_F1: {}\n'.format(Micro_F1))if txt_path == None:return '%.4f' % Micro_F1else:f = open(txt_path, 'a', encoding='utf-8')for i in range(6):f.write('类别{}: '.format(i))f.write('\n')f.write('Precision: {:.2f}%'.format(Precision[i] * 100))f.write('\n')f.write('Recall: {:.2f}%'.format(Recall[i] * 100))f.write('\n')f.write('F1: {:.2f}'.format(F1[i]))f.write('\n')f.write('Macro_Precision: {:.2f}%'.format(Macro_Precision * 100))f.write('\n')f.write('Macro_Recall: {:.2f}%'.format(Macro_Recall * 100))f.write('\n')f.write('Macro_F1: {:.2f}'.format(Macro_F1))f.write('\n')f.write('Micro_Precision: {:.2f}%'.format(Micro_Precision * 100))f.write('\n')f.write('Micro_Recall: {:.2f}%'.format(Micro_Recall * 100))f.write('\n')f.write('Micro_F1: {:.2f}'.format(Micro_F1))f.write('\n')f.close()

模型训练

# 实例化随机森林
rf = RandomForestClassifier(criterion='entropy',n_estimators=3,max_depth=None,  # 定义树的深度，可以用来防止过拟合min_samples_split=10,  # 定义至少多少个样本的情况下才继续分叉min_samples_leaf=0.02  # 定义叶子节点最少需要包含多少个样本（百分比表达），防止过拟合
)
# 模型训练
rf.fit(x_train, y_train)

指标计算

# 先对t_test（DataFrame）进行reshape
y_test = y_test.values.reshape(-1, )
rf_pred = rf.predict(x_test)
txt_path = 'work/result_RandomForest.txt'
metrics_calculate(rf_pred, y_test, txt_path)'''
output:
Precision: 0.7547169811320755Precision: 0.7230769230769231Precision: 0.7387387387387387Precision: 1.0Precision: 0.8297872340425532Precision: 1.0Recall: 0.9856262833675564Recall: 0.47474747474747475Recall: 0.5394736842105263Recall: 0.3132530120481928Recall: 0.7358490566037735Recall: 0.8450704225352113F1: 0.854853072128228F1: 0.573170731707317F1: 0.623574144486692F1: 0.47706422018348627F1: 0.78F1: 0.916030534351145Micro_Precision: 0.7767195767195767Micro_Recall: 0.7767195767195767Micro_F1: 0.7767195767195768
'''

随机森林可视化

# 随机森林可视化
Estimators = rf.estimators_
class_names = ['0', '1', '2', '3', '4', '5']
feature_names = df.columns[1:-1]
for index, model in enumerate(Estimators):dot_data = StringIO()export_graphviz(model, out_file=dot_data,feature_names=feature_names,class_names=class_names,filled=True, rounded=True, special_characters=True)graph = pydotplus.graph_from_dot_data(dot_data.getvalue())graph.write_png('work/Rf{}.png'.format(index))plt.figure(figsize=(20, 20))plt.imshow(plt.imread('work/Rf{}.png'.format(index)))plt.axis('off')

第十二届“中国软件杯”大赛：A10-基于机器学习的分布式系统故障诊断系统——baseline（二）相关推荐

第十二届“中国软件杯”大学生软件设计大赛线下集训营·武汉站活动圆满结束...
青春五月,活力四射."中国软件杯"大学生软件设计大赛(简称"大赛")组委会携手中国移动云能力中心走进湖北工业大学开展武汉站线下集训活动. 深入高校推动校企合作 ...
院士联合指导+超强专家阵容+丰厚奖金机会，第十二届“麒麟杯”大赛报名正式开启！
当前,开放.协作.共享的开源模式已成为全球软件技术和产业创新的主导,也为信息技术国产自主化提供了强大助力.高校师生作为国产开源建设的主要技术群体之一,是国产开源未来发展的中坚力量. 2023年第十二届 ...
第十二届蓝桥杯大赛软件赛省赛 C/C++ 大学 B 组解析
第十二届蓝桥杯大赛软件赛省赛 C/C++ 大学 B 组试题 A: 空间试题 B: 卡片试题 C: 直线试题 D: 货物摆放试题 E: 路径试题 F: 时间显示试题 G: 砝码称重试题 ...
第十二届蓝桥杯大赛软件赛省赛 Java 大学 B 组（2021年4月18日）
第十二届蓝桥杯大赛软件赛省赛 Java 大学 B 组第一场下载原题PDF 欢迎评论区留下答案讨论!!! 试题 A: ASC 本题总分:5 分 [问题描述] 已知大写字母 A 的 ASCII 码为 6 ...
2021第十二届蓝桥杯大赛软件赛省赛C++ C组真题题解
============================== 2019-2021蓝桥杯C++ C组真题题解: 2019第十届蓝桥杯大赛软件类省赛C++ C组真题题解 2020第十一届蓝桥杯大赛软件类省 ...
第十二届蓝桥杯大赛软件赛省赛Java 大学 C 组
第十二届蓝桥杯大赛软件赛省赛Java 大学 C 组试题 A: ASC 试题 B: 空间试题 C: 卡片试题 D: 相乘试题 E: 路径试题 F: 时间显示试题 G: 最少砝码试题 H: ...
第十二届蓝桥杯大赛软件赛省赛 Python 大学 A 组试题
第十二届蓝桥杯大赛软件赛省赛 Python 大学 A 组试题大家好,我叫亓官劼(qí guān jié ),在GitHub & CSDN中记录学习的点滴历程,时光荏苒,未来可期,加油~ ...
第十二届蓝桥杯大赛软件类省赛第一场 Java 大学 B 组题目蓝桥杯JavaB组大赛软件类省赛第十二届第一场
第十二届蓝桥杯大赛软件类省赛第一场 Java 大学 B 组题目在线看题题目PDF下载链接百度云链接:https://pan.baidu.com/s/1LSZvUV5dFwNtSbOshORU1 ...
报！！第十二届蓝桥杯大赛报名启动！！
蓝桥杯全国软件和信息技术专业人才大赛是由工业和信息化部人才交流中心举办的全国性IT学科赛事.共有北京大学.清华大学.上海交通大学等全国1200余所高校参赛,累计参赛人数超过40万人. 2020年,蓝桥 ...

第十二届“中国软件杯”大赛：A10-基于机器学习的分布式系统故障诊断系统——baseline（二）