【蓝蜗牛】骨龄检测（一）

新手。。请多多指教~

数据集rsna-bone-age来自kaggle
云盘链接提取码: n24q
因为上传限制，训练集分成了三个压缩包。解压后照着boneage-test-dataset，把图片放在boneage-training-dataset文件夹里。

参考代码：
Bone-Age-Detection-From-X-Ray

数据集解压后，boneage-training-dataset文件夹中是：

boneage-training-dataset.csv中是：

第一步，读取数据集

import os
import pandas as pdprint("Reading data...")
base_data_dir = ’/input/rsna-bone-age‘
img_dir = os.path.join(base_data_dir , 'boneage-training-dataset/')
csv_dir = os.path.join(base_data_dir , 'boneage-training-dataset.csv')boneage_df = pd.read_csv(csv_dir)
print (boneage_df)column_headers = list(boneage_df.columns.values)
print(column_headers)  #列标签
print(boneage_df.sample(3))  #csv中随机抽取3行

输出：

(12611, 3)
['id', 'boneage', 'male']  #列标签id  boneage   male
11314  11314       96   True
11583  11583      228   True
4583    4583       94  False

参考：
1.pandas文档 —— pandas.read_csv
2.python pandas读取csv后，获取列标签
——————————————————————————————————

获得需要的标签和数据，比如性别（male或female），img文件夹中图片的路径等

boneage_df['path'] = boneage_df['id'].map(lambda x: img_dir+"{}.png".format(x))
boneage_df['gender'] = boneage_df['male'].map(lambda x: "male" if x else "female")
boneage_df['exists'] = boneage_df['path'].map(os.path.exists) #判断img_dir中的图片和csv中的数据是否对应#标准化骨龄数据
mu = boneage_df['boneage'].mean()
sigma = boneage_df['boneage'].std()
boneage_df['zscore'] = boneage_df['boneage'].map(lambda x: (x-mu)/sigma)boneage_df.dropna(inplace=True) column_headers = list(boneage_df.columns.values)
print("column_hander = ",column_headers)  #列标签
#print(boneage_df.sample(3))  #csv中随机抽取3行print("{} images found out of total {} images".format(boneage_df['exists'].sum(),boneage_df.shape[0]))
print("Reading complete !!!\n")boneage_df [['boneage','zscore']].hist()

输出：

column_hander = ['id', 'boneage', 'male', 'path', 'exists', 'gender', 'zscore']
12611 images found out of total 12611 images
Reading complete !!!

横轴是骨龄（按月计算），纵轴是样本数量，标准化后，样本的骨龄数据转化为-3~3的分值

参考：
3.pandas中强大的绘制图表功能——DataFrame.hist()
4.3.4.2数据标准化（一） - Z-Score标准化

第二步，准备数据集（训练集/测试集/验证集）

1 划分数据集

from sklearn.model_selection import train_test_splitprint("Preparing training, testing and validation datasets ...")#按骨龄的梯度分为10个量级
boneage_df['boneage_category'] = pd.cut(boneage_df['boneage'], 10) #划分数据集
raw_train_df, test_df = train_test_split(boneage_df, test_size = 0.2, random_state = 2018,stratify = boneage_df['boneage_category'])
raw_train_df, valid_df = train_test_split(raw_train_df,test_size = 0.25,random_state = 2018,stratify = raw_train_df['boneage_category'])
raw_train_df[['boneage']].hist(figsize = (10, 5)) #绘图（均衡前）
# Training images:   7566 | Validation images: 2522 | Test images:  2523#Balance the distribution in the training set
train_df = raw_train_df.groupby(['boneage_category', 'male']).apply(lambda x: x.sample(500, replace = True)).reset_index(drop=True)
train_df[['boneage']].hist(figsize = (10, 5)) #绘图 (均衡后)
# Training images:   10000 | Validation images: 2522 | Test images:   2523train_size = train_df.shape[0]
valid_size = valid_df.shape[0]
test_size = test_df.shape[0]
print("# Training images:   {}".format(train_size))
print("# Validation images: {}".format(valid_size))
print("# Test images:   {}".format(test_size))

输出：

# Training images:   10000
# Validation images: 2522
# Test images:   2523

注：

数据均衡过程：

train_df = raw_train_df.groupby(['boneage_category', 'male']).apply(lambda x: x.sample(500, replace = True)).reset_index(drop=True)

raw_train_df中，boneage_category有10类，male有两类，排列组合共20种
每类重复采样500次，共得到 20 * 500 = 10000 个样本

参考：
5.pandas文档 ——pandas.cut
6. .sample method
7.Sample Pandas dataframe based on values in column

2 数据预处理

from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.applications.inception_v3 import InceptionV3, preprocess_input
import tensorflow as tf
import numpy as np
IMG_SIZE = (224,224)
BATCH_SIZE_TRAIN = 10
BATCH_SIZE_VAL = 16
SEED = 1234# 将图像和年龄作为输入，构建数据生成器
def gen_2inputs(imgDatGen, df, batch_size, seed, img_size):gen_img = imgDatGen.flow_from_dataframe(dataframe=df,x_col='path', y_col='zscore',batch_size=batch_size, seed=seed, shuffle=True, class_mode='other',target_size=img_size, color_mode='rgb',drop_duplicates=False)gen_gender = imgDatGen.flow_from_dataframe(dataframe=df,x_col='path', y_col='gender',batch_size=batch_size, seed=seed, shuffle=True, class_mode='other',target_size=img_size, color_mode='rgb',drop_duplicates=False)while True:X1i = gen_img.next()X2i = gen_gender.next()yield [X1i[0], X2i[1]], X1i[1]def test_gen_2inputs(imgDatGen, df, batch_size, img_size):gen_img = imgDatGen.flow_from_dataframe(dataframe=df,x_col='path', y_col='zscore',batch_size=batch_size, shuffle=False, class_mode='other',target_size=img_size, color_mode='rgb',drop_duplicates=False)gen_gender = imgDatGen.flow_from_dataframe(dataframe=df,x_col='path', y_col='gender',batch_size=batch_size, shuffle=False, class_mode='other',target_size=img_size, color_mode='rgb',drop_duplicates=False)while True:X1i = gen_img.next()X2i = gen_gender.next()yield [X1i[0], X2i[1]], X1i[1]

train_idg = ImageDataGenerator(zoom_range=0.2,fill_mode='nearest',rotation_range=25,  width_shift_range=0.25,  height_shift_range=0.25,  vertical_flip=False, horizontal_flip=True,shear_range = 0.2,samplewise_center=False, samplewise_std_normalization=False)val_idg = ImageDataGenerator(width_shift_range=0.25, height_shift_range=0.25, horizontal_flip=True)test_idg = ImageDataGenerator()train_flow = gen_2inputs(train_idg, train_df, BATCH_SIZE_TRAIN, SEED, IMG_SIZE)
valid_flow = gen_2inputs(val_idg, valid_df, BATCH_SIZE_VAL, SEED, IMG_SIZE)
test_flow = test_gen_2inputs(test_idg, test_df, 500, IMG_SIZE)# 计算月份平均绝对误差
def mae_months(in_gt, in_pred):return mean_absolute_error(boneage_div * in_gt, boneage_div * in_pred)

参考：
8.Tutorial on Keras flow_from_dataframe
9.师兄的代码