mask_rcnn keras源码跟读2）数据部分

数据生成部分主要用了keras_model.fit_generator接口，具体函数在model.py的1626行，其函数说明如下：

def data_generator(dataset, config, shuffle=True, augment=False, augmentation=None,random_rois=0, batch_size=1, detection_targets=False):"""A generator that returns images and corresponding target class ids,bounding box deltas, and masks.dataset: The Dataset object to pick data fromconfig: The model config objectshuffle: If True, shuffles the samples before every epochaugment: (Depricated. Use augmentation instead). If true, apply randomimage augmentation. Currently, only horizontal flipping is offered.augmentation: Optional. An imgaug (https://github.com/aleju/imgaug) augmentation.For example, passing imgaug.augmenters.Fliplr(0.5) flips imagesright/left 50% of the time.random_rois: If > 0 then generate proposals to be used to train thenetwork classifier and mask heads. Useful if trainingthe Mask RCNN part without the RPN.batch_size: How many images to return in each calldetection_targets: If True, generate detection targets (class IDs, bboxdeltas, and masks). Typically for debugging or visualizations becausein trainig detection targets are generated by DetectionTargetLayer.Returns a Python generator. Upon calling next() on it, thegenerator returns two lists, inputs and outputs. The containtesof the lists differs depending on the received arguments:inputs list:- images: [batch, H, W, C]- image_meta: [batch, (meta data)] Image details. See compose_image_meta()- rpn_match: [batch, N] Integer (1=positive anchor, -1=negative, 0=neutral)- rpn_bbox: [batch, N, (dy, dx, log(dh), log(dw))] Anchor bbox deltas.- gt_class_ids: [batch, MAX_GT_INSTANCES] Integer class IDs- gt_boxes: [batch, MAX_GT_INSTANCES, (y1, x1, y2, x2)]- gt_masks: [batch, height, width, MAX_GT_INSTANCES]. The height and widthare those of the image unless use_mini_mask is True, in whichcase they are defined in MINI_MASK_SHAPE.outputs list: Usually empty in regular training. But if detection_targetsis True then the outputs list contains target class_ids, bbox deltas,and masks."""

1.Anchor生成

# Anchors
# [anchor_count, (y1, x1, y2, x2)]
#compute_backbone_shapes：获得P2-P5对应特征图大小
backbone_shapes = compute_backbone_shapes(config, config.IMAGE_SHAPE)
anchors = utils.generate_pyramid_anchors(config.RPN_ANCHOR_SCALES,config.RPN_ANCHOR_RATIOS,backbone_shapes,config.BACKBONE_STRIDES,config.RPN_ANCHOR_STRIDE)

接下来看generate_pyramid_anchors函数，这里依次生成P2-P5对应的anchor

def generate_pyramid_anchors(scales, ratios, feature_shapes, feature_strides,anchor_stride):"""Generate anchors at different levels of a feature pyramid. Each scaleis associated with a level of the pyramid, but each ratio is used inall levels of the pyramid.Returns:anchors: [N, (y1, x1, y2, x2)]. All generated anchors in one array. Sortedwith the same order of the given scales. So, anchors of scale[0] comefirst, then anchors of scale[1], and so on."""# Anchors# [anchor_count, (y1, x1, y2, x2)]anchors = []for i in range(len(scales)):anchors.append(generate_anchors(scales[i], ratios, feature_shapes[i],feature_strides[i], anchor_stride))return np.concatenate(anchors, axis=0)

然后进入generate_anchors函数，这里使用np.meshgrid来生成对应的网格，把scales和ratios组合在一起，注意这里的scales(anchor默认大小)其实是标量。假如输入的图片大小为512*512，对于P2来说，其anchor_number=feature_map_size*feature_map_size*3=512/4*512/4*3=49152
（3为横纵比，4为stride，anchor_stride默认为1）

def generate_anchors(scales, ratios, shape, feature_stride, anchor_stride):"""scales: 1D array of anchor sizes in pixels. Example: [32, 64, 128]ratios: 1D array of anchor ratios of width/height. Example: [0.5, 1, 2]shape: [height, width] spatial shape of the feature map over whichto generate anchors.feature_stride: Stride of the feature map relative to the image in pixels.anchor_stride: Stride of anchors on the feature map. For example, if thevalue is 2 then generate anchors for every other feature map pixel."""# Get all combinations of scales and ratiosscales, ratios = np.meshgrid(np.array(scales), np.array(ratios))scales = scales.flatten()ratios = ratios.flatten()# Enumerate heights and widths from scales and ratiosheights = scales / np.sqrt(ratios)widths = scales * np.sqrt(ratios)# Enumerate shifts in feature spaceshifts_y = np.arange(0, shape[0], anchor_stride) * feature_strideshifts_x = np.arange(0, shape[1], anchor_stride) * feature_strideshifts_x, shifts_y = np.meshgrid(shifts_x, shifts_y)# Enumerate combinations of shifts, widths, and heightsbox_widths, box_centers_x = np.meshgrid(widths, shifts_x)box_heights, box_centers_y = np.meshgrid(heights, shifts_y)# Reshape to get a list of (y, x) and a list of (h, w)box_centers = np.stack([box_centers_y, box_centers_x], axis=2).reshape([-1, 2])box_sizes = np.stack([box_heights, box_widths], axis=2).reshape([-1, 2])# Convert to corner coordinates (y1, x1, y2, x2)boxes = np.concatenate([box_centers - 0.5 * box_sizes,box_centers + 0.5 * box_sizes], axis=1)return boxes

2.获得gt_box和mask
load_image_gt函数会返回当前图像，图像原数据，标签类别，gt_boxs和gt_max

# Get GT bounding boxes and masks for image.
image_id = image_ids[image_index]
image, image_meta, gt_class_ids, gt_boxes, gt_masks = \
load_image_gt(dataset, config, image_id, augment=augment,augmentation=augmentation,use_mini_mask=config.USE_MINI_MASK)

比较核心的函数load_image_gt

def load_image_gt(dataset, config, image_id, augment=False, augmentation=None,use_mini_mask=False):"""Load and return ground truth data for an image (image, mask, bounding boxes).augment: (Depricated. Use augmentation instead). If true, apply randomimage augmentation. Currently, only horizontal flipping is offered.augmentation: Optional. An imgaug (https://github.com/aleju/imgaug) augmentation.For example, passing imgaug.augmenters.Fliplr(0.5) flips imagesright/left 50% of the time.use_mini_mask: If False, returns full-size masks that are the same heightand width as the original image. These can be big, for example1024x1024x100 (for 100 instances). Mini masks are smaller, typically,224x224 and are generated by extracting the bounding box of theobject and resizing it to MINI_MASK_SHAPE.Returns:image: [height, width, 3]shape: the original shape of the image before resizing and cropping.class_ids: [instance_count] Integer class IDsbbox: [instance_count, (y1, x1, y2, x2)]mask: [height, width, instance_count]. The height and width are thoseof the image unless use_mini_mask is True, in which case they aredefined in MINI_MASK_SHAPE."""# Load image and mask#load_image就是继承超类utils.Dataset需要实现的函数image = dataset.load_image(image_id)#load_mask同样是需要重写的超类函数mask, class_ids = dataset.load_mask(image_id)original_shape = image.shape#根据配置文件中的参数,对图片进行预处理 *这里可以了解到配置文件中参数的具体含义image, window, scale, padding, crop = utils.resize_image(image,min_dim=config.IMAGE_MIN_DIM,min_scale=config.IMAGE_MIN_SCALE,max_dim=config.IMAGE_MAX_DIM,mode=config.IMAGE_RESIZE_MODE)#若对原始图片进行了预处理，该处理同步于mask上mask = utils.resize_mask(mask, scale, padding, crop)# Random horizontal flips.# TODO: will be removed in a future update in favor of augmentationif augment:logging.warning("'augment' is depricated. Use 'augmentation' instead.")if random.randint(0, 1):image = np.fliplr(image)mask = np.fliplr(mask)# Augmentation   数据增强部分# This requires the imgaug lib (https://github.com/aleju/imgaug)if augmentation:import imgaug# Augmentors that are safe to apply to masks# Some, such as Affine, have settings that make them unsafe, so always# test your augmentation on masksMASK_AUGMENTERS = ["Sequential", "SomeOf", "OneOf", "Sometimes","Fliplr", "Flipud", "CropAndPad","Affine", "PiecewiseAffine"]def hook(images, augmenter, parents, default):"""Determines which augmenters to apply to masks."""return (augmenter.__class__.__name__ in MASK_AUGMENTERS)# Store shapes before augmentation to compareimage_shape = image.shapemask_shape = mask.shape# Make augmenters deterministic to apply similarly to images and masksdet = augmentation.to_deterministic()image = det.augment_image(image)# Change mask to np.uint8 because imgaug doesn't support np.boolmask = det.augment_image(mask.astype(np.uint8),hooks=imgaug.HooksImages(activator=hook))# Verify that shapes didn't changeassert image.shape == image_shape, "Augmentation shouldn't change image size"assert mask.shape == mask_shape, "Augmentation shouldn't change mask size"# Change mask back to boolmask = mask.astype(np.bool)# Note that some boxes might be all zeros if the corresponding mask got cropped out.# and here is to filter them out_idx = np.sum(mask, axis=(0, 1)) > 0mask = mask[:, :, _idx]class_ids = class_ids[_idx]# Bounding boxes. Note that some boxes might be all zeros# if the corresponding mask got cropped out.# bbox: [num_instances, (y1, x1, y2, x2)]#mask->bbox 边缘宽体,mask_rcnn->faster_rncc 可以修改这里bbox = utils.extract_bboxes(mask)# Active classes# Different datasets have different classes, so track the# classes supported in the dataset of this image.active_class_ids = np.zeros([dataset.num_classes], dtype=np.int32)source_class_ids = dataset.source_class_ids[dataset.image_info[image_id]["source"]]active_class_ids[source_class_ids] = 1# Resize masks to smaller size to reduce memory usage#若在指定了MINI_MASK_SHAPE，则把包含mask的box，resize成MINI_MASK_SHAPE*MINI_MASK_SHAPEif use_mini_mask:mask = utils.minimize_mask(bbox, mask, config.MINI_MASK_SHAPE)# Image meta dataimage_meta = compose_image_meta(image_id, original_shape, image.shape,window, scale, active_class_ids)return image, image_meta, class_ids, bbox, mask

在看看里面的函数，便于理解配置文件中参数的含义
utils.resize_image()
若指定了min_dim,则会让原始图片的较小边image_min缩放到min_dim，并获得缩放比例scale。
若指定了max_dim(square模式下),先判断通过min_dim获得的scale得到的较长边image_max*scale是否大于了max_dim,若大于了，则重新获得scale=max_dim / image_max

通过scale通比例缩放图片后,width和height可能不一致，则使用mode来调整
若mode为square：
对width和height同时进行padding_0操作，直至等于max_dim
若mode为pad64:
对width和height向上就近padding_0至64的倍数，比如width=112，则padding_0至128
若mode为crop:
则先按min_dim进行缩放，然后随机裁剪获得min_dim*min_dim的新图

def resize_image(image, min_dim=None, max_dim=None, min_scale=None, mode="square"):"""Resizes an image keeping the aspect ratio unchanged.min_dim: if provided, resizes the image such that it's smallerdimension == min_dimmax_dim: if provided, ensures that the image longest side doesn'texceed this value.min_scale: if provided, ensure that the image is scaled up by at leastthis percent even if min_dim doesn't require it.mode: Resizing mode.none: No resizing. Return the image unchanged.square: Resize and pad with zeros to get a square imageof size [max_dim, max_dim].pad64: Pads width and height with zeros to make them multiples of 64.If min_dim or min_scale are provided, it scales the image upbefore padding. max_dim is ignored in this mode.The multiple of 64 is needed to ensure smooth scaling of featuremaps up and down the 6 levels of the FPN pyramid (2**6=64).crop: Picks random crops from the image. First, scales the image basedon min_dim and min_scale, then picks a random crop ofsize min_dim x min_dim. Can be used in training only.max_dim is not used in this mode.Returns:image: the resized imagewindow: (y1, x1, y2, x2). If max_dim is provided, padding mightbe inserted in the returned image. If so, this window is thecoordinates of the image part of the full image (excludingthe padding). The x2, y2 pixels are not included.scale: The scale factor used to resize the imagepadding: Padding added to the image [(top, bottom), (left, right), (0, 0)]"""

minimize_mask()
把包含mask的box插值缩放到mini_shape*mini_shape，首先由bbox坐标先把包含mask的box扣出来，然后在进行插值缩放，expand_masks则是它的逆操作。（这里和最后的fcn有一定的联系）

def minimize_mask(bbox, mask, mini_shape):"""Resize masks to a smaller version to reduce memory load.Mini-masks can be resized back to image scale using expand_masks()See inspect_data.ipynb notebook for more details."""mini_mask = np.zeros(mini_shape + (mask.shape[-1],), dtype=bool)for i in range(mask.shape[-1]):# Pick slice and cast to bool in case load_mask() returned wrong dtypem = mask[:, :, i].astype(bool)y1, x1, y2, x2 = bbox[i][:4]m = m[y1:y2, x1:x2]if m.size == 0:raise Exception("Invalid bounding box with area of zero")# Resize with bilinear interpolationm = skimage.transform.resize(m, mini_shape, order=1, mode="constant")mini_mask[:, :, i] = np.around(m).astype(np.bool)return mini_mask

3.build_rpn_targets
通过 anchors and GT boxes获得rpn_match(anchor的类型)和rpn_bbox(positive_anchor和gt_box的偏移)

def build_rpn_targets(image_shape, anchors, gt_class_ids, gt_boxes, config):"""Given the anchors and GT boxes, compute overlaps and identify positiveanchors and deltas to refine them to match their corresponding GT boxes.anchors: [num_anchors, (y1, x1, y2, x2)]gt_class_ids: [num_gt_boxes] Integer class IDs.gt_boxes: [num_gt_boxes, (y1, x1, y2, x2)]Returns:rpn_match: [N] (int32) matches between anchors and GT boxes.1 = positive anchor, -1 = negative anchor, 0 = neutralrpn_bbox: [N, (dy, dx, log(dh), log(dw))] Anchor bbox deltas."""

区分 1 = positive anchor, -1 = negative anchor, 0 = neutral的规则如下：

    # Match anchors to GT Boxes# If an anchor overlaps a GT box with IoU >= 0.7 then it's positive.# If an anchor overlaps a GT box with IoU < 0.3 then it's negative.# Neutral anchors are those that don't match the conditions above,# and they don't influence the loss function.# However, don't keep any GT box unmatched (rare, but happens). Instead,# match it to the closest anchor (even if its max IoU is < 0.3).

mask_rcnn keras源码跟读2）数据部分相关推荐

mask_rcnn keras源码跟读3）配置文件
config.py文件,参数配置一个一个地看 # NUMBER OF GPUs to use. For CPU training, use 1 GPU_COUNT = 1 # Number of im ...
YOLO3 数据处理与数据加载 Keras源码分析
YOLO3 Keras 源码:https://github.com/qqwweee/keras-yolo3 前言本文从主要是从源码层面对 YOLO3 的数据处理相关内容进行分析与讲解.通常,一个功能 ...
Tensorflow 2.x(keras)源码详解之第四章：DatasetTFRecord
大家好,我是爱编程的喵喵.双985硕士毕业,现担任全栈工程师一职,热衷于将数据思维应用到工作与生活中.从事机器学习以及相关的前后端开发工作.曾在阿里云.科大讯飞.CCF等比赛获得多次Top名次.现 ...
Linux驱动开发 / fbtft源码速读
哈喽,老吴又来分享学习心得啦~ 一.目标与体系目标是关于你想要达到的结果,而体系是涉及导致这些结果的过程; 目标的意义在于确定大方向,但体系才能促进进步.完全忽略目标,只关注体系,仍然会成功. 结果 ...
Tensorflow 2.x(keras)源码详解之第十二章：keras中的损失函数之BinaryCrossentropy详解
大家好,我是爱编程的喵喵.双985硕士毕业,现担任全栈工程师一职,热衷于将数据思维应用到工作与生活中.从事机器学习以及相关的前后端开发工作.曾在阿里云.科大讯飞.CCF等比赛获得多次Top名次.现 ...
Tensorflow 2.x(keras)源码详解之第九章：模型训练和预测的三种方法(fittf.GradientTapetrain_steptf.data)
大家好,我是爱编程的喵喵.双985硕士毕业,现担任全栈工程师一职,热衷于将数据思维应用到工作与生活中.从事机器学习以及相关的前后端开发工作.曾在阿里云.科大讯飞.CCF等比赛获得多次Top名次.现 ...
Tensorflow 2.x(keras)源码详解之第十一章：keras损失函数及自定义损失函数
大家好,我是爱编程的喵喵.双985硕士毕业,现担任全栈工程师一职,热衷于将数据思维应用到工作与生活中.从事机器学习以及相关的前后端开发工作.曾在阿里云.科大讯飞.CCF等比赛获得多次Top名次.现 ...
Keras源码学习笔记
文章目录 Keras源码结构 keras搭建网络和运行的一般过程 Keras源码文件结构 Keras核心类卷积层优化函数训练过程中的回调函数损失函数权重初始化方法 Keras源码结构 ker ...
Tensorflow 2.x(keras)源码详解之第十五章：迁移学习与微调
大家好,我是爱编程的喵喵.双985硕士毕业,现担任全栈工程师一职,热衷于将数据思维应用到工作与生活中.从事机器学习以及相关的前后端开发工作.曾在阿里云.科大讯飞.CCF等比赛获得多次Top名次.现 ...

mask_rcnn keras源码跟读2）数据部分

mask_rcnn keras源码跟读2）数据部分相关推荐

最新文章

热门文章