37、在OAK摄像头上部署tensorflow deeplabv3+进行实例分割

基本思想：手中有个OAK摄像头，一直想移植一下官方的deeplabv3的模型，逐记录一下训练过程和模型转换，从pb转模型到openvino，然后在移植oak摄像头，tensorflow/model的版本为2022-09-11之前的版本（含）建议看我的博客https://sxj731533730.blog.csdn.net/article/details/127964980

链接：https://pan.baidu.com/s/118DdBuk6kNeUEfRUQfQ4DQ?pwd=wuje
提取码：wuje
--来自百度网盘超级会员V1的分享

实验的结果和数据集

链接：https://pan.baidu.com/s/1nDgJ-1SP6hcSweKPH3chlw
提取码：mn5e

第一步：首先创建conda环境，配置model

ubuntu@ubuntu:~$ conda create -n tf python=3.6
ubuntu@ubuntu:~$ conda activate tf
(tf) ubuntu@ubuntu:~$ git clone https://github.com/tensorflow/models.git

配置tensorflow/models参考31、TensorFlow训练模型转成tfilte，进行Android端进行车辆检测、跟踪、部署_sxj731533730的博客-CSDN博客

刷新tensorflow的版本

(tf) ubuntu@ubuntu:~$ pip install -i https://pypi.tuna.tsinghua.edu.cn/simple tensorflow-gpu==1.15.0 tensorflow==1.15.0
(tf) ubuntu@ubuntu:~$ python3
Python 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
True(tf) ubuntu@ubuntu:~$ conda install cudatoolkit=10.0 -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/linux-64/
(tf) ubuntu@ubuntu:~$ conda install cudnn=7.6.5 -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64/(tf) ubuntu@ubuntu:~$ conda update -n base -c defaults conda

但是官方给的训练方法是cpu版本，我使用gpu训练偶尔会报错，建议使用cpu训练

第二步：下载lableme的源码，以备生成数据集使用

(tf) ubuntu@ubuntu:~$ git clone https://github.com/wkentaro/labelme.git

第三步：下载coco数据集，提取了里面的单类人的数据集且图片大小筛选了480×640的图片

45、实例分割的labelme数据集转coco数据集以及coco数据集转labelme数据集_sxj731533730的博客-CSDN博客_转coco数据集

提供一下本菜的代码

# -*- coding: utf-8 -*-
import glob
import os
import cv2
import json
import iococo = ["person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light","fire hydrant", "", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow","elephant", "bear", "zebra", "giraffe", "", "backpack", "umbrella", "", "", "handbag", "tie", "suitcase","frisbee","skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard","tennis racket", "bottle", "", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple","sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch","potted plant", "bed", "", "dining table", "", "", "toilet", "", "tv", "laptop", "mouse", "remote", "keyboard","cell phone","microwave", "oven", "toaster", "sink", "refrigerator", "book", "", "clock", "vase", "scissors", "teddy bear","hair drier", "toothbrush"]label = dict()
for idx, item in enumerate(coco):label.update({idx: item})
labelme_path = r'F:\CP210x_USB_TO_UART\val2017\labelme'
coco_json_path = r'F:\CP210x_USB_TO_UART\val2017'
jpg_path = r'F:\CP210x_USB_TO_UART\val2017\val2017'
coco_json = glob.glob(os.path.join(coco_json_path, "*.json"))[0]
file_json = io.open(coco_json, 'r', encoding='utf-8')
m_json_data = file_json.read()
m_data = json.loads(m_json_data)
# m_type=m_data['type']for item in m_data['images']:flag = Falsem_images_file_name = item['file_name'](filename_path, m_filename) = os.path.split(m_images_file_name)(m_name, extension) = os.path.splitext(m_filename)m_image = cv2.imread(os.path.join(jpg_path, m_name + ".jpg"))m_images_height = item['height']m_images_width = item['width']m_images_id = item['id']data = {}data['imagePath'] = m_filenamedata['flags'] = {}data['imageWidth'] = m_images_widthdata['imageHeight'] = m_images_heightdata['imageData'] = Nonedata['version'] = "5.0.1"data["shapes"] = []for annit in m_data['annotations']:m_image_id = annit['image_id']m_category_id = annit['category_id']if m_image_id == m_images_id and label[m_category_id - 1] == 'person' and m_images_width==640 and m_images_height==480:flag = Truefor segitem in annit['segmentation']:points = []for idx in range(0, len(segitem), 2):x, y = segitem[idx], segitem[idx + 1]if str(x).isalpha() or str(y).isalpha():flag = Falsebreakpoints.append([x, y])itemData = {'points': []}if len(points) == 0:flag = FalsebreakitemData['points'].extend(points)itemData["flag"] = {}itemData["group_id"] = NoneitemData["shape_type"] = "polygon"itemData["label"] = label[m_category_id - 1]data["shapes"].append(itemData)if flag:jsonName = ".".join([m_name, "json"])jpgName = ".".join([m_name, "jpg"])print(labelme_path, jsonName)jsonPath = os.path.join(labelme_path, jsonName)jpgPath = os.path.join(labelme_path, jpgName)with open(jsonPath, "w") as f:json.dump(data, f)cv2.imwrite(jpgPath, m_image)print("加载入文件完成...")

将上述的if判断条件稍微修改就能从coco数据集转成labelme数据集，然后提取出关心的类别和严格要求的图片大小图片

ubuntu@ubuntu:~/Downloads/dataset$ tree -L 1
.
├── train
├── trainval
└── val3 directories, 0 files
图片的宽度是640 高度480

train里面存放的只含人的实例分割的人的图片和lableme标注格式的json文件，其它文件夹类似

第四步：生成tensroflow的deeplabv3+数据集

1）生成数据集转换格式

(tf) ubuntu@ubuntu:~/labelme/examples/semantic_segmentation$ python3 labelme2voc.py /home/ubuntu/Downloads/dataset/train /home/ubuntu/Downloads/dataset/train_voc --labels labels.txt
(tf) ubuntu@ubuntu:~/labelme/examples/semantic_segmentation$ python3 labelme2voc.py /home/ubuntu/Downloads/dataset/trainval /home/ubuntu/Downloads/dataset/trainval_voc --labels labels.txt
(tf) ubuntu@ubuntu:~/labelme/examples/semantic_segmentation$ python3 labelme2voc.py /home/ubuntu/Downloads/dataset/val /home/ubuntu/Downloads/dataset/val_voc --labels labels.txt

其中labels.txt格式

__ignore__
_background_
person

目录格式

(tf1.15) ubuntu@ubuntu:~/Downloads$ tree -L 2
.
└── dataset├── total├── train├── trainval├── trainval_voc├── train_voc├── val└── val_voc8 directories, 0 files

目录格式

(tf1.15) ubuntu@ubuntu:~/Downloads/dataset/train_voc$ tree -L 1
.
├── class_names.txt
├── JPEGImages
├── SegmentationClass
├── SegmentationClassPNG
├── SegmentationClassRaw
├── SegmentationClassVisualization6 directories, 1 file

2）生成单通道的数据集

(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ python3 remove_gt_colormap.py --original_gt_folder=/home/ubuntu/Downloads/dataset/train_voc/SegmentationClassPNG --output_dir=/home/ubuntu/Downloads/dataset/train_voc/SegmentationClassRaw
(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ python3 remove_gt_colormap.py --original_gt_folder=/home/ubuntu/Downloads/dataset/val_voc/SegmentationClassPNG --output_dir=/home/ubuntu/Downloads/dataset/val_voc/SegmentationClassRaw
(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ python3 remove_gt_colormap.py --original_gt_folder=/home/ubuntu/Downloads/dataset/trainval_voc/SegmentationClassPNG --output_dir=/home/ubuntu/Downloads/dataset/trainval_voc/SegmentationClassRaw

全黑很正常，因为本菜只用了一个类，像素值只是1,而背景是0,很难看到目标信息，（测试中不该这么做）如果想看到可以更改~/models/research/deeplab/datasets/remove_gt_colormap.py 中的51行增加

old_raw_pic=np.array(Image.open(filename))
raw_pic=old_raw_pic*50
return raw_pic

这样就能看到生成单通道png存在画面黑白轮廓信息，但是这样搞就不对了，虽然直观上能看见，但是语义分割是按照像素级别进行分割，你只有一个目标那除了背景，另个像素目标就是1,你改成50,那这个目标就是第50个目标，所以还是全黑就全黑吧，视觉看不到就看不到吧，不影响训练，只是为了求证这个事情

3）在文件夹下创建三个文件夹，分别是在train_voc/trainlist val_voc/vallist trainval_voc/trainvallist

目录格式

(tf1.15) ubuntu@ubuntu:~/Downloads/dataset/train_voc$ tree -L 1
.
├── class_names.txt
├── JPEGImages
├── SegmentationClass
├── SegmentationClassPNG
├── SegmentationClassRaw
├── SegmentationClassVisualization
└── trainlist6 directories, 1 file

在各个train_voc/JPEGImages中执行

find . -name "*.jpg" > ../trainlist/train.txt
find . -name "*.jpg" > ../vallist/val.txt
find . -name "*.jpg" > ../trainvallist/trainval.txt
使用编辑文本替换的功能修正为txt只有文件名字列表，没有后缀名和文件夹路径

目录的结构是这样train.txt

000000002153
000000015335
000000079588
000000140203
000000119641
000000087144
000000018837
000000118405
000000032887

4）生成tfrecord数据集，预先在~/models/research/deeplab/datasets创建目录datasetData、 log 、result目录

(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ python3 build_voc2012_data.py --image_folder=/home/ubuntu/Downloads/dataset/train_voc/JPEGImages --semantic_segmentation_folder=/home/ubuntu/Downloads/dataset/train_voc/SegmentationClassRaw --list_folder=/home/ubuntu/Downloads/dataset/train_voc/trainlist --image_format="jpg"  --output_dir=/home/ubuntu/models/research/deeplab/datasets/datasetData
(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ python3 build_voc2012_data.py --image_folder=/home/ubuntu/Downloads/dataset/trainval_voc/JPEGImages --semantic_segmentation_folder=/home/ubuntu/Downloads/dataset/trainval_voc/SegmentationClassRaw --list_folder=/home/ubuntu/Downloads/dataset/trainval_voc/trainvallist --image_format="jpg"  --output_dir=/home/ubuntu/models/research/deeplab/datasets/datasetData
(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ python3 build_voc2012_data.py --image_folder=/home/ubuntu/Downloads/dataset/val_voc/JPEGImages --semantic_segmentation_folder=/home/ubuntu/Downloads/dataset/val_voc/SegmentationClassRaw --list_folder=/home/ubuntu/Downloads/dataset/val_voc/vallist --image_format="jpg"  --output_dir=/home/ubuntu/models/research/deeplab/datasets/datasetData

第五步：修改代码一些逻辑

1）修改路径/home/ubuntu/models/research/deeplab/datasets/data_generator.py

_MYDATA_INFORMATION = DatasetDescriptor(splits_to_sizes={'train': 271,  # 训练集数量'trainval': 136,  # 训练集数量'val': 70,  # 测试集数量},num_classes=3,#__ignore__+_background_+Arrow =3ignore_label=255,
)112行_DATASETS_INFORMATION = {'cityscapes': _CITYSCAPES_INFORMATION,'pascal_voc_seg': _PASCAL_VOC_SEG_INFORMATION,'ade20k': _ADE20K_INFORMATION,'mydata':_MYDATA_INFORMATION, # 添加自己的数据集}

2）修改路径/home/ubuntu/models/research/deeplab/utils/train_utils.py

  # Variables that will not be restored.#exclude_list = ['global_step']exclude_list = ['global_step','logits']if not initialize_last_layer:exclude_list.extend(last_layers)

3）修改路径/home/ubuntu/models/research/deeplab/utils/get_dataset_colormap.py

41行
_DATASET_NAME='mydata'   # 添加在这里，和注册的名字相同_DATASET_NAME: 3,   # 在这里添加 colormap 的颜色数
51行
def create_dataset_name_label_colormap(): return np.asarray([ [165, 42, 42],[0, 192, 0],[196, 196, 196],])
390行
elif dataset == _DATASET_NAME:             # 添加在这里return create_dataset_name_label_colormap()

4)修改/home/ubuntu/models/research/deeplab/vis.py 增加mydata

flags.DEFINE_enum('colormap_type', 'pascal', ['mydata','pascal', 'cityscapes', 'ade20k'],'Visualization colormap type.')

5)修改/home/ubuntu/models/research/deeplab/utils/train_utils.py 153行下添加

ignore_weight = 0label0_weight = 1  # 对应background，mask中灰度值0label1_weight = 10  # 对应a，mask中灰度值1not_ignore_mask = tf.to_float(tf.equal(scaled_labels, 0)) * label0_weight + \tf.to_float(tf.equal(scaled_labels, 1)) * label1_weight + \tf.to_float(tf.equal(scaled_labels, ignore_label)) * ignore_weighttf.losses.softmax_cross_entropy(train_labels,tf.reshape(logits, shape=[-1,num_classes]),weights=not_ignore_mask,scope=loss_scope)

但是好像tensorflow新版本的训练参数是多了一个 --label_weights={0,0.1,10}可以改变权重比例，这个权重应该怎么设置，需要matlab支持查看png的各目标像素占比，请自行搜索，我设置了个1:10而已，我随机设置的

第六步：下载预训练权重

(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ wget -nd -c http://download.tensorflow.org/models/deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz
(tf) ubuntu@ubuntu:~/models/research/deeplab/datasets$ tar -zxvf deeplabv3_mnv2_pascal_train_aug_2018_01_29.tar.gz

第七步训练，训练参数也是参考oak官网设置

(tf) ubuntu@ubuntu:~/models/research$ CUDA_VISIBLE_DEVICES=0 python3 deeplab/train.py --logtostderr --training_number_of_steps=3000 --train_split="train" --model_variant="mobilenet_v2" --output_stride=8 --fine_tune_batch_norm=true --label_weights={0,0.1,10} --train_batch_size=2 --train_crop_size="481,641" --dataset="mydata" --tf_initial_checkpoint='/home/ubuntu/models/research/deeplab/datasets/deeplabv3_mnv2_pascal_train_aug/model.ckpt-30000' --train_logdir='/home/ubuntu/models/research/deeplab/datasets/result' --dataset_dir='/home/ubuntu/models/research/deeplab/datasets/datasetData'

训练过程，如果中间遇到错误，不是配置错误的话，先用cpu训练试试是否可以，gpu训练偶尔有错误，感觉是bug


INFO:tensorflow:Recording summary at step 2932.
I0911 13:25:50.207441 140671436445440 supervisor.py:1050] Recording summary at step 2932.
INFO:tensorflow:global step 2940: loss = 0.0116 (3.233 sec/step)
I0911 13:26:13.105489 140674671449920 learning.py:507] global step 2940: loss = 0.0116 (3.233 sec/step)
INFO:tensorflow:global step 2950: loss = 0.0139 (3.186 sec/step)
I0911 13:26:44.115453 140674671449920 learning.py:507] global step 2950: loss = 0.0139 (3.186 sec/step)
INFO:tensorflow:global step 2960: loss = 0.0166 (3.240 sec/step)
I0911 13:27:14.827654 140674671449920 learning.py:507] global step 2960: loss = 0.0166 (3.240 sec/step)
INFO:tensorflow:global step 2970: loss = 0.0158 (3.210 sec/step)
I0911 13:27:46.387307 140674671449920 learning.py:507] global step 2970: loss = 0.0158 (3.210 sec/step)
INFO:tensorflow:global step 2980: loss = 0.0152 (3.233 sec/step)
I0911 13:28:17.760236 140674671449920 learning.py:507] global step 2980: loss = 0.0152 (3.233 sec/step)
INFO:tensorflow:global step 2990: loss = 0.0165 (3.007 sec/step)
I0911 13:28:49.145774 140674671449920 learning.py:507] global step 2990: loss = 0.0165 (3.007 sec/step)
INFO:tensorflow:global step 3000: loss = 0.0178 (3.238 sec/step)
I0911 13:29:20.179072 140674671449920 learning.py:507] global step 3000: loss = 0.0178 (3.238 sec/step)
INFO:tensorflow:Stopping Training.
I0911 13:29:20.179394 140674671449920 learning.py:777] Stopping Training.
INFO:tensorflow:Finished training! Saving model to disk.
I0911 13:29:20.179480 140674671449920 learning.py:785] Finished training! Saving model to disk.
/home/ubuntu/miniconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/summary/writer/writer.py:386: UserWarning: Attempting to use a closed FileWriter. The operation will be a noop unless the FileWriter is explicitly reopened.warnings.warn("Attempting to use a closed FileWriter.

第八步：测试

(tf) ubuntu@ubuntu:~/models/research$ python deeplab/eval.py --logtostderr --eval_split="val" --model_variant="mobilenet_v2" --eval_crop_size="481,641" --dataset="mydata" --output_stride=8 --checkpoint_dir=/home/ubuntu/models/research/deeplab/datasets/result --eval_logdir=/home/ubuntu/models/research/deeplab/datasets/log --dataset_dir=/home/ubuntu/models/research/deeplab/datasets/datasetData --max_number_of_evaluations=1

测试结果

eval/miou_1.0_class_1[0.552526116]
eval/miou_1.0_class_2[nan]
eval/miou_1.0_class_0[0.901511848]
eval/miou_1.0_overall[0.727019072]

第九步：可视化

(tf) ubuntu@ubuntu:~/models/research$ python deeplab/vis.py --logtostderr --vis_split="val" --model_variant="mobilenet_v2" --vis_crop_size="481,641" --dataset="mydata" --colormap_type="mydata" --output_stride=8 --checkpoint_dir=/home/ubuntu/models/research/deeplab/datasets/result --vis_logdir=/home/ubuntu/models/research/deeplab/datasets/log --dataset_dir=/home/ubuntu/models/research/deeplab/datasets/datasetData --max_number_of_iterations=1  --also_save_raw_predictions=True

可视化结果

随机找两张图

第十步：转pb模型测试

(tf) ubuntu@ubuntu:~/models/research$ python3 deeplab/export_model.py --logtostderr --checkpoint_path=/home/ubuntu/models/research/deeplab/datasets/result/model.ckpt-3000 --model_variant="mobilenet_v2" --crop_size=481 --crop_size=641  --inference_scales=1.0 --num_classes=3 --export_path=/home/ubuntu/models/research/deeplab/datasets/log/export/frozen_inference_graph.pb

测试代码

#!/usr/bin/env python
# coding: utf-8import os
from io import BytesIO
import tarfile
import tempfile
from six.moves import urllib
from matplotlib import gridspec
from matplotlib import pyplot as plt
import numpy as np
from PIL import Image
import tensorflow as tf
import scipyLABEL_NAMES = np.asarray(["background", "class1", "class2"])class DeepLabModel(object):"""Class to load deeplab model and run inference."""INPUT_TENSOR_NAME = "ImageTensor:0"OUTPUT_TENSOR_NAME = "SemanticPredictions:0"INPUT_SIZE = 321FROZEN_GRAPH_NAME = "frozen_inference_graph"def __init__(self, modelname):"""Creates and loads pretrained deeplab model."""self.graph = tf.Graph()graph_def = Nonewith open(modelname, "rb") as fd:graph_def = tf.GraphDef.FromString(fd.read())if graph_def is None:raise RuntimeError("Cannot find inference graph in tar archive.")with self.graph.as_default():tf.import_graph_def(graph_def, name="")self.sess = tf.Session(graph=self.graph)def run(self, image):"""Runs inference on a single image.Args:image: A PIL.Image object, raw input image.Returns:resized_image: RGB image resized from original input image.seg_map: Segmentation map of `resized_image`."""width, height = image.sizeresize_ratio = 1.0 * self.INPUT_SIZE / max(width, height)target_size = (int(resize_ratio * width), int(resize_ratio * height))resized_image = image.convert("RGB").resize(target_size, Image.ANTIALIAS)batch_seg_map = self.sess.run(self.OUTPUT_TENSOR_NAME,feed_dict={self.INPUT_TENSOR_NAME: [np.asarray(resized_image)]},)seg_map = batch_seg_map[0]return resized_image, seg_mapdef create_pascal_label_colormap():"""Creates a label colormap used in PASCAL VOC segmentation benchmark.Returns:A Colormap for visualizing segmentation results."""colormap = np.zeros((256, 3), dtype=int)ind = np.arange(256, dtype=int)for shift in reversed(range(8)):for channel in range(3):colormap[:, channel] |= ((ind >> channel) & 1) << shiftind >>= 3return colormap# 从 label 到 color_image
def label_to_color_image(label):"""Adds color defined by the dataset colormap to the label.Args:label: A 2D array with integer type, storing the segmentation label.Returns:result: A 2D array with floating type. The element of the arrayis the color indexed by the corresponding element in the input labelto the PASCAL color map.Raises:ValueError: If label is not of rank 2 or its value is larger than colormap maximum entry."""if label.ndim != 2:raise ValueError("Expect 2-D input label")colormap = create_pascal_label_colormap()if np.max(label) >= len(colormap):raise ValueError("label value too large.")return colormap[label]# 分割结果可视化
def vis_segmentation(image, seg_map, name):"""Visualizes input image, segmentation map and overlay view."""plt.figure(figsize=(15, 5))grid_spec = gridspec.GridSpec(1, 4, width_ratios=[6, 6, 6, 1])plt.subplot(grid_spec[0])plt.imshow(image)plt.axis("off")plt.title("input image")plt.subplot(grid_spec[1])seg_image = label_to_color_image(seg_map).astype(np.uint8)plt.imshow(seg_image)plt.axis("off")plt.title("segmentation map")plt.subplot(grid_spec[2])plt.imshow(image)plt.imshow(seg_image, alpha=0.7)plt.axis("off")plt.title("segmentation overlay")unique_labels = np.unique(seg_map)ax = plt.subplot(grid_spec[3])plt.imshow(FULL_COLOR_MAP[unique_labels].astype(np.uint8), interpolation="nearest")ax.yaxis.tick_right()plt.yticks(range(len(unique_labels)), LABEL_NAMES[unique_labels])plt.xticks([], [])ax.tick_params(width=0.0)plt.grid("off")if not os.path.exists("./seg_map_result/"):os.mkdir("./seg_map_result/")plt.savefig("./seg_map_result/" + name + ".png")# plt.show()FULL_LABEL_MAP = np.arange(len(LABEL_NAMES)).reshape(len(LABEL_NAMES), 1)
FULL_COLOR_MAP = label_to_color_image(FULL_LABEL_MAP)def main_test(filepath):# 加载模型modelname = "/home/ubuntu/models/research/deeplab/datasets/log/export/frozen_inference_graph.pb"MODEL = DeepLabModel(modelname)print("model loaded successfully!")filelist = os.listdir(filepath)for item in filelist:print("process image of ", item)name = item.split(".jpg", 1)[0]original_im = Image.open(filepath + item)resized_im, seg_map = MODEL.run(original_im)# 分割结果拼接vis_segmentation(resized_im, seg_map, name)# 单独保存分割结果# seg_map_name = name + '_seg.png'# resized_im_name = name + '_in.png'# path = './seg_map_result/'# scipy.misc.imsave(path + resized_im_name,resized_im)# scipy.misc.imsave(path + seg_map_name,seg_map)if __name__ == "__main__":filepath = "/home/ubuntu/Downloads/dataset/val_voc/JPEGImages/"main_test(filepath)

测试结果

第十一步：转openvino模型,环境配置openvino :31、OAK使用Yolov4-tiny进行训练、部署、测距功能使用_sxj731533730的博客-CSDN博客

ubuntu@ubuntu:~$ python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo_tf.py --input_model /home/ubuntu/models/research/deeplab/datasets/log/export/frozen_inference_graph.pb --model_name deeplab_v3_plus_mnv2_decoder_256 --data_type FP16 --input_shape [1,256,256,3] --reverse_input_channel --output_dir ./....
While validating node 'v0::Concat Concat_1003 (strided_slice_10/stack_1/Unsqueeze[0]:i32{1}, strided_slice_10/stack_1/Unsqueeze1082[0]:i32{1}, strided_slice_10/stack_1/Unsqueeze1084[0]:i32{1}, strided_slice_10/extend_end_const1243231247[0]:i64{1}) -> ()' with friendly_name 'Concat_1003':
Argument element types are inconsistent.[ WARNING ]  Using fallback to produce IR.
[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: /home/ubuntu/yy/deeplab_v3_plus_mnv2_decoder_256.xml
[ SUCCESS ] BIN file: /home/ubuntu/yy/deeplab_v3_plus_mnv2_decoder_256.bin
[ SUCCESS ] Total execution time: 18.66 seconds.
[ SUCCESS ] Memory consumed: 384 MB.
It's been a while, check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html?cid=other&source=prod&campid=ww_2021_bu_IOTG_OpenVINO-2021-4-LTS&content=upg_all&medium=organic or on the GitHub*

然后开始在转换模型BlobConverter | Convert model to MyriadX blob

参考上述转换最后一行提示修改i64改成i32 ，

At this point, since OpenVINO version 2020.3, the obtained .xml is broken and will not let you directly create a .blob. To fix this, we need to change the following code located in .xml:<layer id="490" name="strided_slice_10/extend_end_const1243231247" type="Const" version="opset1"><data element_type="i64" offset="924018" shape="1" size="8"/><output><port id="0" precision="I64"><dim>1</dim></port></output>
</layer>
We need to change element_type to i32 instead of i64. The edited line should look like this:<data element_type="i32" offset="924018" shape="1" size="8"/>

网页转换

第十二步：测试OAK深度相机

这里也可以转onnx模型在转blob也可以

ubuntu@ubuntu:~$ python3 -m tf2onnx.convert --graphdef frozen_inference_graph.pb --output model.onnx --inputs ImageTensor:0 --outputs SemanticPredictions:0

测试代码

#!/usr/bin/env python3import cv2
import depthai as dai
import numpy as np
import argparse
import time'''
Deeplabv3 multiclass running on selected camera.
Run as:
python3 -m pip install -r requirements.txt
python3 main.py -cam rgb
Possible input choices (-cam):
'rgb', 'left', 'right'Blob is taken from ML training examples:
https://github.com/luxonis/depthai-ml-training/tree/master/colab-notebooksYou can clone the DeepLabV3plus_MNV2.ipynb notebook and try training the model yourself.'''num_of_classes = 1 # define the number of classes in the dataset
cam_options = ['rgb', 'left', 'right']parser = argparse.ArgumentParser()
parser.add_argument("-cam", "--cam_input", help="select camera input source for inference", default='rgb', choices=cam_options)
parser.add_argument("-nn", "--nn_model", help="select model path for inference", default='/home/ubuntu/yy/deeplab_v3_plus_mnv2_decoder_256_3.blob', type=str)args = parser.parse_args()cam_source = args.cam_input
nn_path = args.nn_modelnn_shape = 256def decode_deeplabv3p(output_tensor):output = output_tensor.reshape(nn_shape,nn_shape)# scale to [0 ... 2555] and apply colormapoutput = np.array(output) * (255/num_of_classes)output = output.astype(np.uint8)output_colors = cv2.applyColorMap(output, cv2.COLORMAP_JET)# reset the color of 0 classoutput_colors[output == 0] = [0,0,0]return output_colorsdef show_deeplabv3p(output_colors, frame):return cv2.addWeighted(frame,1, output_colors,0.4,0)# Start defining a pipeline
pipeline = dai.Pipeline()pipeline.setOpenVINOVersion(version = dai.OpenVINO.VERSION_2021_4)# Define a neural network that will make predictions based on the source frames
detection_nn = pipeline.create(dai.node.NeuralNetwork)
detection_nn.setBlobPath(nn_path)detection_nn.setNumPoolFrames(4)
detection_nn.input.setBlocking(False)
detection_nn.setNumInferenceThreads(2)cam=None
# Define a source - color camera
if cam_source == 'rgb':cam = pipeline.create(dai.node.ColorCamera)cam.setPreviewSize(nn_shape,nn_shape)cam.setInterleaved(False)cam.preview.link(detection_nn.input)
elif cam_source == 'left':cam = pipeline.create(dai.node.MonoCamera)cam.setBoardSocket(dai.CameraBoardSocket.LEFT)
elif cam_source == 'right':cam = pipeline.create(dai.node.MonoCamera)cam.setBoardSocket(dai.CameraBoardSocket.RIGHT)if cam_source != 'rgb':manip = pipeline.create(dai.node.ImageManip)manip.setResize(nn_shape,nn_shape)manip.setKeepAspectRatio(True)manip.setFrameType(dai.RawImgFrame.Type.BGR888p)cam.out.link(manip.inputImage)manip.out.link(detection_nn.input)cam.setFps(40)# Create outputs
xout_rgb = pipeline.create(dai.node.XLinkOut)
xout_rgb.setStreamName("nn_input")
xout_rgb.input.setBlocking(False)detection_nn.passthrough.link(xout_rgb.input)xout_nn = pipeline.create(dai.node.XLinkOut)
xout_nn.setStreamName("nn")
xout_nn.input.setBlocking(False)detection_nn.out.link(xout_nn.input)# Pipeline defined, now the device is assigned and pipeline is started
with dai.Device() as device:cams = device.getConnectedCameras()depth_enabled = dai.CameraBoardSocket.LEFT in cams and dai.CameraBoardSocket.RIGHT in camsif cam_source != "rgb" and not depth_enabled:raise RuntimeError("Unable to run the experiment on {} camera! Available cameras: {}".format(cam_source, cams))device.startPipeline(pipeline)# Output queues will be used to get the rgb frames and nn data from the outputs defined aboveq_nn_input = device.getOutputQueue(name="nn_input", maxSize=4, blocking=False)q_nn = device.getOutputQueue(name="nn", maxSize=4, blocking=False)start_time = time.time()counter = 0fps = 0layer_info_printed = Falsewhile True:# instead of get (blocking) used tryGet (nonblocking) which will return the available data or None otherwisein_nn_input = q_nn_input.get()in_nn = q_nn.get()frame = in_nn_input.getCvFrame()layers = in_nn.getAllLayers()# get layer1 datalay1 = np.array(in_nn.getFirstLayerInt32()).reshape(nn_shape,nn_shape)found_classes = np.unique(lay1)output_colors = decode_deeplabv3p(lay1)frame = show_deeplabv3p(output_colors, frame)cv2.putText(frame, "NN fps: {:.2f}".format(fps), (2, frame.shape[0] - 4), cv2.FONT_HERSHEY_TRIPLEX, 0.4, (255, 0, 0))cv2.putText(frame, "Found classes {}".format(found_classes), (2, 10), cv2.FONT_HERSHEY_TRIPLEX, 0.4, (255, 0, 0))cv2.imshow("nn_input", frame)counter+=1if (time.time() - start_time) > 1 :fps = counter / (time.time() - start_time)counter = 0start_time = time.time()if cv2.waitKey(1) == ord('q'):break

测试结果

官方的那个人体分割的例子是256*256*1*1通道，速度在27fps 多类别的速度和我的差不多是3fps的研究一下为啥单类那么快

参考

https://github.com/luxonis/depthai-ml-training/blob/master/colab-notebooks/DeepLabV3plus_MNV2.ipynb

使用deeplabv3+训练自己数据集（迁移学习） - vcjmhg - 博客园

37、在OAK摄像头上部署tensorflow deeplabv3+进行实例分割相关推荐

cloud 部署_使用Google Cloud AI平台开发，训练和部署TensorFlow模型
cloud 部署实用指南 (A Practical Guide) The TensorFlow ecosystem has become very popular for developing ap ...
基于人脸识别的课堂签到管理系统（四）---摄像头上传实时数据，百度AI读取并返回信息以及多线程操作
基于人脸识别的课堂签到管理系统(四)---摄像头上传实时数据,百度AI读取并返回以及多线程操作一. 前言概述二.摄像头上传数据,读取百度AI返回信息三.多线程操作四.程序展示五.相关下载一 ...
Android q索尼手机相机算法,手机厂商套路太多！竟在手机摄像头上玩“掩眼法”？...
原标题:手机厂商套路太多!竟在手机摄像头上玩"掩眼法"? 不知道从什么时候开始,国内消费者判断智能手机高低水准的第一印象,就是看硬件参数.甚至于到了拍照这些项目上,也是下意识地先看 ...
【java当中摄像头调用保姆级别教程和在摄像头上实现滤镜效果】
JAVA当中摄像头调用(并实现摄像头的滤镜效果) 摄像头的调用 1.包的导入 1.1首先进入webcam官网 1.2然后点击Download下的下载的webcam-capture-0.3.10-dis ...
RealSense技术在SR300摄像头上的应用
RealSense技术在SR300摄像头上的应用一. 实感摄像头 1. RealSense技术在计算机的发展过程中我们始终没有抛弃键盘和鼠标,前几年win8和触屏的配合形成了一 ...
Anaconda 虚拟环境安装部署Tensorflow 2.x版本
Anaconda 虚拟环境安装部署Tensorflow 2.x版本目录卸载Tensorflow1.x版本查询Anaconda 的Tensorflow版本安装Tensorflow 2.x版本安 ...
摄像头上（ONVIF/RTSP）介绍
简介: 摄像头上面的两个标记"ONVIF/RTSP"就是摄像头符合"网络视频标准规范"和"实时流传输协议"容的标准生产出来的,符合质量要求的 ...
vlc视频转发注意事项海康摄像头上云注意事项
<<<用vlc做视频转发服务器>>> 一.vlc做视频转发服务器分三段: 1.局域网一台电脑上,vlc读取媒体流通过udp往公网服务器上推流. 之后提到这个阶段都简 ...
斐迅N1 刷armbian部署tensorflow ，开启人工智能之旅
电梯直达 1# 发表于 2018-10-9 21:45 | 只看该作者 | 只看大图 | 倒序浏览 | 阅读模式本帖最后由陆不败于 2018-11-15 17:14 编辑斐迅N1 刷armbi ...

37、在OAK摄像头上部署tensorflow deeplabv3+进行实例分割

37、在OAK摄像头上部署tensorflow deeplabv3+进行实例分割相关推荐

最新文章

热门文章