前言

数据是深度学习的核心，大部分论文里都会提到data-driven这个词，也就是数据驱动的意思。基本的模型搭建完成后，如何处理数据，如何将数据送给网络，如何做数据增强等等，对于提高网络的性能都十分重要，本篇文章会简单讲述下数据处理过程，后续有时间会持续更新这方面的内容，互相学习，共勉！

数据集的均值和标准差

def compute_mean_and_std(dataset):    # 输入为PyTorch的dataset，即数据集，输出为对应数据集均值和标准差        # 均值    mean_r = 0    mean_g = 0    mean_b = 0    for img, _ in dataset:        img = np.asarray(img) # 将 PIL Image 改变成numpy的数组类型        mean_b += np.mean(img[:, :, 0])        mean_g += np.mean(img[:, :, 1])        mean_r += np.mean(img[:, :, 2])    mean_b /= len(dataset)    mean_g /= len(dataset)    mean_r /= len(dataset)    diff_r = 0    diff_g = 0    diff_b = 0    N = 0    for img, _ in dataset:        img = np.asarray(img)        diff_b += np.sum(np.power(img[:, :, 0] - mean_b, 2))        diff_g += np.sum(np.power(img[:, :, 1] - mean_g, 2))        diff_r += np.sum(np.power(img[:, :, 2] - mean_r, 2))        N += np.prod(img[:, :, 0].shape)    std_b = np.sqrt(diff_b / N)    std_g = np.sqrt(diff_g / N)    std_r = np.sqrt(diff_r / N)    mean = (mean_b.item() / 255.0, mean_g.item() / 255.0, mean_r.item() / 255.0)    std = (std_b.item() / 255.0, std_g.item() / 255.0, std_r.item() / 255.0)   return mean, std

常用训练和验证数据预处理

ToTensor 会将 PIL.Image 或形状为 H×W×D，数值范围为 [0, 255] 的numpy数组转换为形状为 D×H×W，数值范围为 [0.0, 1.0] 的 torch.Tensor。

train_transform = torchvision.transforms.Compose([    torchvision.transforms.RandomResizedCrop(size=224, scale=(0.08, 1.0)),    torchvision.transforms.RandomHorizontalFlip(),    torchvision.transforms.ToTensor(),    torchvision.transforms.Normalize(mean=(0.485, 0.456, 0.406)std=(0.229, 0.224, 0.225)), ]) val_transform = torchvision.transforms.Compose([    torchvision.transforms.Resize(256),    torchvision.transforms.CenterCrop(224),    torchvision.transforms.ToTensor(),    torchvision.transforms.Normalize(mean=(0.485, 0.456, 0.406),                                     std=(0.229, 0.224, 0.225)),])

视频数据

import cv2video = cv2.VideoCapture(mp4_path)height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH))num_frames = int(video.get(cv2.CAP_PROP_FRAME_COUNT))fps = int(video.get(cv2.CAP_PROP_FPS))video.release()

未完待续，持续更新！