主要展示keras 的使用方法，搭建网络模型的步骤。

c++部署模型，利用深度学习做视觉里程计：
项目
Github地址：https://github.com/zouyuelin/SLAM_Learning_notes/tree/main/PoseEstimation
网盘地址
链接：https://pan.baidu.com/s/1x6-FwlYOfAC0WCQUL_WMWA
提取码：ixy3

概要

这个网络是利用两帧图像，预测相机的位姿变换矩阵，总共6个自由度
网络的输入： [image_ref , image_cur]
网络的输出: [tx , ty , tz , roll , pitch , yaw]

一、数据集的处理

数据集的格式：

# image_ref  image_cur  tx  ty  tz  roll(x) pitch(y) yaw(z)
0 images/0.jpg images/1.jpg 0.000999509 -0.00102794 0.00987293 0.00473228 -0.0160252 -0.0222079
1 images/1.jpg images/2.jpg -0.00544488 -0.00282174 0.00828871 -0.00271557 -0.00770117 -0.0195182
2 images/2.jpg images/3.jpg -0.0074375 -0.00368121 0.0114751 -0.00721246 -0.0103843 -0.0171883
3 images/3.jpg images/4.jpg -0.00238111 -0.00371362 0.0120466 -0.0081171 -0.0149111 -0.0198595
4 images/4.jpg images/5.jpg 0.000965841 -0.00520437 0.0135452 -0.0141721 -0.0126401 -0.0182697
5 images/5.jpg images/6.jpg -0.00295753 -0.00340146 0.0144557 -0.013633 -0.00463747 -0.0143332

通过load_image映射数据集到 TF tensor

class datasets:def __init__(self, datasetsPath:str):self.dataPath = datasetsPathself.dim = 512self.epochs = 40self.batch_size = 8self.train_percent = 0.92self.learning_rate = 2e-4self.model_path = 'kerasTempModel/'self.posetxt = os.path.join(self.dataPath,'pose.txt') self.GetTheImagesAndPose()self.buildTrainData()def GetTheImagesAndPose(self):self.poselist = []with open(self.posetxt,'r') as f:for line in f.readlines():line = line.strip()line = line.split(' ')line.remove(line[0])self.poselist.append(line)# im1 im2 tx,ty,tz,roll,pitch,yaw#打乱数据集length = np.shape(self.poselist)[0]train_num =int(length * self.train_percent) test_num = length - train_numrandomPoses = np.array(random.sample(self.poselist,length)) #取出所有数据集self.train_pose_list = randomPoses[0:train_num,:]self.test_pose_list = randomPoses[train_num:length+1,:]print(f"The size of train pose list is : {np.shape(self.train_pose_list)[0]}")print(f"The size of test pose list is : {np.shape(self.test_pose_list)[0]}")def load_image(self,index:tf.Tensor):img_ref= img_ref = tf.io.read_file(index[0])img_ref = tf.image.decode_jpeg(img_ref) #此处为jpeg格式img_ref = tf.image.resize(img_ref,(self.dim,self.dim))/255.0#img = tf.reshape(img,[self.dim,self.dim,3])img_ref = tf.cast(img_ref,tf.float32)img_cur = img_cur = tf.io.read_file(index[1])img_cur = tf.image.decode_jpeg(img_cur) #此处为jpeg格式img_cur = tf.image.resize(img_cur,(self.dim,self.dim))/255.0#img = tf.reshape(img,[self.dim,self.dim,3])img_cur = tf.cast(img_cur,tf.float32)pose = tf.strings.to_number(index[2:8],tf.float32)return (img_ref,img_cur),(pose)def buildTrainData(self):'''for example:\\>>> poses = dataset.y_train.take(20)\\>>> imgs = dataset.x1_train.take(40)\\>>> print(np.array(list(imgs.as_numpy_iterator()))[39]) \\>>> imgs = dataset.x2_train.take(40)\\>>> print(np.array(list(imgs.as_numpy_iterator()))[39]) \\>>> print(np.array(list(poses.as_numpy_iterator()))[19]) \\'''self.traindata = tf.data.Dataset.from_tensor_slices(self.train_pose_list) \.map(self.load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE) \.shuffle(500)\.repeat(10)\.batch(self.batch_size) \.prefetch(tf.data.experimental.AUTOTUNE)#.cache() self.testdata = tf.data.Dataset.from_tensor_slices(self.test_pose_list) \.map(self.load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE) \.shuffle(500)\.repeat(10)\.batch(self.batch_size) \.prefetch(tf.data.experimental.AUTOTUNE)

二、网络模型的搭建

# 模型
def model(dim):First = K.layers.Input(shape=(dim,dim,3),name="input1")Second = K.layers.Input(shape=(dim,dim,3),name="input2")x1 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(First)x1 = K.layers.Conv2D(512,kernel_size=(3,3), strides=2,padding='same')(x1)x1 = K.layers.BatchNormalization()(x1)x1 = K.layers.ReLU()(x1)x1 = K.layers.Conv2D(256,kernel_size=(3,3), strides=2,padding='same')(x1)x1 = K.layers.BatchNormalization()(x1)x1 = K.layers.ReLU()(x1)x1 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(x1)x2 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(Second)x2 = K.layers.Conv2D(512,kernel_size=(3,3), strides=2,padding='same')(x2)x2 = K.layers.BatchNormalization()(x2)x2 = K.layers.ReLU()(x2)x2 = K.layers.Conv2D(256,kernel_size=(3,3), strides=2,padding='same')(x2)x2 = K.layers.BatchNormalization()(x2)x2 = K.layers.ReLU()(x2)x2 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(x2)x = K.layers.concatenate([x1,x2])x = K.layers.Conv2D(256,kernel_size=(3,3), strides=1,padding='same',activation='relu')(x)x = K.layers.BatchNormalization()(x)x = K.layers.ReLU()(x)x = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(x)x = K.layers.Conv2D(128,kernel_size=(3,3), strides=1,padding='same',activation='relu')(x)x = K.layers.BatchNormalization()(x)x = K.layers.ReLU()(x)x = K.layers.Flatten()(x)x = K.layers.Dense(1024)(x)x = K.layers.Dense(6,name='Output')(x)poseModel = K.Model([First,Second],x)return poseModel
#损失函数
def loss_fn(y_true,y_pre):loss_value = K.backend.mean(K.backend.square(y_true-y_pre))return loss_value# 学习率下降回调函数
class learningDecay(K.callbacks.Callback):def __init__(self,schedule=None,alpha=1,verbose=0):super().__init__()self.schedule = scheduleself.verbose = verboseself.alpha = alphadef on_epoch_begin(self, epoch, logs=None):lr = float(K.backend.get_value(self.model.optimizer.lr))if self.schedule != None:lr = self.schedule(epoch,lr)else:if epoch != 0:lr = lr*self.alphaK.backend.set_value(self.model.optimizer.lr,K.backend.get_value(lr))if self.verbose > 0:print(f"Current learning rate is {lr}")
# 学习率计划
def scheduler(epoch, lr):if epoch < 10:return lrelse:return lr * tf.math.exp(-0.1)

三、模型的训练

build()函数用来编译模型，建立回调函数；
train_fit() 函数是利用keras 的 fit函数来训练；
train_gradient() 函数是利用 apply_gradients函数训练，可以实时对loss和梯度进行监控，灵活性强；
save_model() 函数是保存模型的函数，可以有多种保存的方式，利用model.save() 可以保存为h5文件和tf格式的模型。

class Posenet:def __init__(self,dataset:datasets):self.dataset = datasetself.build()def build(self):self.poseModel = model(self.dataset.dim)self.poseModel.summary()self.optm = K.optimizers.RMSprop(1e-4,momentum=0.9) #,decay=1e-5/self.dataset.epochsself.decayCallback = learningDecay(schedule = None,alpha = 0.99,verbose = 1)decayCallbackScheduler = K.callbacks.LearningRateScheduler(scheduler)self.callbacks = [decayCallbackScheduler]try:print("************************loading the model weights***********************************")self.poseModel.load_weights("model.h5")except:passdef train_fit(self):self.poseModel.compile(optimizer=self.optm,loss=loss_fn,metrics=['accuracy'])self.poseModel.fit(self.dataset.traindata,validation_data=self.dataset.testdata,epochs=self.dataset.epochs,callbacks=[self.decayCallback],verbose=1)def train_gradient(self):for step in range(self.dataset.epochs):loss = 0val_loss = 0lr = float(self.optm.lr)tf.print(">>> [Epoch is %s/%s]"%(step,self.dataset.epochs))for (x1,x2),y in self.dataset.traindata:with tf.GradientTape() as tape:prediction = self.poseModel([x1,x2])# y = tf.cast(y,dtype=prediction.dtype)loss = loss_fn(y,prediction)gradients = tape.gradient(loss,self.poseModel.trainable_variables)self.optm.apply_gradients(zip(gradients,self.poseModel.trainable_variables))# 测试for (x1,x2),y in self.dataset.testdata:prediction = self.poseModel([x1,x2])val_loss = loss_fn(y,prediction)tf.print("The loss is %s,the learning rate is : %s, test loss is %s]"%(np.array(loss),lr,val_loss))K.backend.set_value(self.optm.lr,K.backend.get_value(lr*0.99))def save_model(self):'''利用 save 函数来保存，可以保存为h5文件，也可以保存为文件夹的形式，推荐保存第二种，再使用tf2onnx转onnx>>> python -m tf2onnx.convert --saved-model kerasTempModel --output "model.onnx" --opset 14'''self.poseModel.save("model.h5")self.poseModel.save(self.dataset.model_path)# self.poseModel.save_weights("model.h5") #只保存权重，没有保存结构# tf.saved_model.save(self.poseModel,'tf2TempModel') #这种保存方式不再使用了

主函数：

if __name__ == "__main__":dataset = datasets("images")posenet = Posenet(dataset)posenet.train_fit()# posenet.train_gradient() #利用 apply_gradient的方式训练posenet.save_model()

四、加载保存的模型

保存的模型文件夹为： kerasTempModel\

model = K.models.load_model(dataset.model_path,compile=False)

测试模型：

output = model([img_ref,img_cur])

五、完整代码

import argparse
import tensorflow as tf
import tensorflow.keras as K
import numpy as np
import cv2 as cv
import os
import time
import random
from tensorflow.keras import optimizers
from tensorflow.keras import callbacksclass datasets:def __init__(self, datasetsPath:str):self.dataPath = datasetsPathself.dim = 512self.epochs = 40self.batch_size = 8self.train_percent = 0.92self.learning_rate = 2e-4self.model_path = 'kerasTempModel/'self.posetxt = os.path.join(self.dataPath,'pose.txt') self.GetTheImagesAndPose()self.buildTrainData()def GetTheImagesAndPose(self):self.poselist = []with open(self.posetxt,'r') as f:for line in f.readlines():line = line.strip()line = line.split(' ')line.remove(line[0])self.poselist.append(line)# im1 im2 tx,ty,tz,roll,pitch,yaw#打乱数据集length = np.shape(self.poselist)[0]train_num =int(length * self.train_percent) test_num = length - train_numrandomPoses = np.array(random.sample(self.poselist,length)) #取出所有数据集self.train_pose_list = randomPoses[0:train_num,:]self.test_pose_list = randomPoses[train_num:length+1,:]print(f"The size of train pose list is : {np.shape(self.train_pose_list)[0]}")print(f"The size of test pose list is : {np.shape(self.test_pose_list)[0]}")def load_image(self,index:tf.Tensor):img_ref= img_ref = tf.io.read_file(index[0])img_ref = tf.image.decode_jpeg(img_ref) #此处为jpeg格式img_ref = tf.image.resize(img_ref,(self.dim,self.dim))/255.0#img = tf.reshape(img,[self.dim,self.dim,3])img_ref = tf.cast(img_ref,tf.float32)img_cur = img_cur = tf.io.read_file(index[1])img_cur = tf.image.decode_jpeg(img_cur) #此处为jpeg格式img_cur = tf.image.resize(img_cur,(self.dim,self.dim))/255.0#img = tf.reshape(img,[self.dim,self.dim,3])img_cur = tf.cast(img_cur,tf.float32)pose = tf.strings.to_number(index[2:8],tf.float32)return (img_ref,img_cur),(pose)def buildTrainData(self):'''for example:\\>>> poses = dataset.y_train.take(20)\\>>> imgs = dataset.x1_train.take(40)\\>>> print(np.array(list(imgs.as_numpy_iterator()))[39]) \\>>> imgs = dataset.x2_train.take(40)\\>>> print(np.array(list(imgs.as_numpy_iterator()))[39]) \\>>> print(np.array(list(poses.as_numpy_iterator()))[19]) \\'''self.traindata = tf.data.Dataset.from_tensor_slices(self.train_pose_list) \.map(self.load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE) \.shuffle(500)\.repeat(10)\.batch(self.batch_size) \.prefetch(tf.data.experimental.AUTOTUNE)#.cache() self.testdata = tf.data.Dataset.from_tensor_slices(self.test_pose_list) \.map(self.load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE) \.shuffle(500)\.repeat(10)\.batch(self.batch_size) \.prefetch(tf.data.experimental.AUTOTUNE)def model(dim):First = K.layers.Input(shape=(dim,dim,3),name="input1")Second = K.layers.Input(shape=(dim,dim,3),name="input2")x1 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(First)x1 = K.layers.Conv2D(512,kernel_size=(3,3), strides=2,padding='same')(x1)x1 = K.layers.BatchNormalization()(x1)x1 = K.layers.ReLU()(x1)x1 = K.layers.Conv2D(256,kernel_size=(3,3), strides=2,padding='same')(x1)x1 = K.layers.BatchNormalization()(x1)x1 = K.layers.ReLU()(x1)x1 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(x1)x2 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(Second)x2 = K.layers.Conv2D(512,kernel_size=(3,3), strides=2,padding='same')(x2)x2 = K.layers.BatchNormalization()(x2)x2 = K.layers.ReLU()(x2)x2 = K.layers.Conv2D(256,kernel_size=(3,3), strides=2,padding='same')(x2)x2 = K.layers.BatchNormalization()(x2)x2 = K.layers.ReLU()(x2)x2 = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(x2)x = K.layers.concatenate([x1,x2])x = K.layers.Conv2D(256,kernel_size=(3,3), strides=1,padding='same',activation='relu')(x)x = K.layers.BatchNormalization()(x)x = K.layers.ReLU()(x)x = K.layers.MaxPool2D(pool_size=(2,2),strides=2)(x)x = K.layers.Conv2D(128,kernel_size=(3,3), strides=1,padding='same',activation='relu')(x)x = K.layers.BatchNormalization()(x)x = K.layers.ReLU()(x)x = K.layers.Flatten()(x)x = K.layers.Dense(1024)(x)x = K.layers.Dense(6,name='Output')(x)poseModel = K.Model([First,Second],x)return poseModeldef loss_fn(y_true,y_pre):loss_value = K.backend.mean(K.backend.square(y_true-y_pre))return loss_valueclass learningDecay(K.callbacks.Callback):def __init__(self,schedule=None,alpha=1,verbose=0):super().__init__()self.schedule = scheduleself.verbose = verboseself.alpha = alphadef on_epoch_begin(self, epoch, logs=None):lr = float(K.backend.get_value(self.model.optimizer.lr))if self.schedule != None:lr = self.schedule(epoch,lr)else:if epoch != 0:lr = lr*self.alphaK.backend.set_value(self.model.optimizer.lr,K.backend.get_value(lr))if self.verbose > 0:print(f"Current learning rate is {lr}")def scheduler(epoch, lr):if epoch < 10:return lrelse:return lr * tf.math.exp(-0.1) class Posenet:def __init__(self,dataset:datasets):self.dataset = datasetself.build()def build(self):self.poseModel = model(self.dataset.dim)self.poseModel.summary()self.optm = K.optimizers.RMSprop(1e-4,momentum=0.9) #,decay=1e-5/self.dataset.epochsself.decayCallback = learningDecay(schedule = None,alpha = 0.99,verbose = 1)decayCallbackScheduler = K.callbacks.LearningRateScheduler(scheduler)self.callbacks = [decayCallbackScheduler]try:print("************************loading the model weights***********************************")self.poseModel.load_weights("model.h5")except:passdef train_fit(self):self.poseModel.compile(optimizer=self.optm,loss=loss_fn,metrics=['accuracy'])self.poseModel.fit(self.dataset.traindata,validation_data=self.dataset.testdata,epochs=self.dataset.epochs,callbacks=[self.decayCallback],verbose=1)def train_gradient(self):for step in range(self.dataset.epochs):loss = 0val_loss = 0lr = float(self.optm.lr)tf.print(">>> [Epoch is %s/%s]"%(step,self.dataset.epochs))for (x1,x2),y in self.dataset.traindata:with tf.GradientTape() as tape:prediction = self.poseModel([x1,x2])# y = tf.cast(y,dtype=prediction.dtype)loss = loss_fn(y,prediction)gradients = tape.gradient(loss,self.poseModel.trainable_variables)self.optm.apply_gradients(zip(gradients,self.poseModel.trainable_variables))# 测试for (x1,x2),y in self.dataset.testdata:prediction = self.poseModel([x1,x2])val_loss = loss_fn(y,prediction)tf.print("The loss is %s,the learning rate is : %s, test loss is %s]"%(np.array(loss),lr,val_loss))K.backend.set_value(self.optm.lr,K.backend.get_value(lr*0.99))def save_model(self):'''利用 save 函数来保存，可以保存为h5文件，也可以保存为文件夹的形式，推荐保存第二种，再使用tf2onnx转onnx>>> python -m tf2onnx.convert --saved-model kerasTempModel --output "model.onnx" --opset 14'''self.poseModel.save("model.h5")self.poseModel.save(self.dataset.model_path)# self.poseModel.save_weights("model.h5") #只保存权重，没有保存结构# tf.saved_model.save(self.poseModel,'tf2TempModel') #这种保存方式不再使用了if __name__ == "__main__":dataset = datasets("images")posenet = Posenet(dataset)posenet.train_fit()# posenet.train_gradient() #利用 apply_gradient的方式训练posenet.save_model()

六、运行结果

将训练的模型配置到C++中运行：
C++ 上用 ONNXruntime 部署自己的模型

利用pnp的方式运行的结果

处理速度：43ms

利用深度学习位姿估计的结果

处理速度：11ms

tensorflow keras 搭建相机位姿估计网络--例相关推荐

深度学习系列笔记——贰（基于Tensorflow Keras搭建的猫狗大战模型一）
猫狗大战是著名的竞赛网站kaggle几年前的一个比赛,参赛者得到猫狗各12500张图片,作为训练集,另外还会得到12500张猫和狗的图片,作为验证.最后提交结果至kaggle平台,获得评测分数. 本篇 ...
【SLAM文献】2017-2018 CVPR ICCV ECCV 相机位姿估计、视觉定位、SLAM相关论文综述
作者:变胖是梦想2014 来源链接:https://www.jianshu.com/p/22151f39b50c 目录 CVPR-2018 references CVPR-2017 reference ...
相机位姿估计2：[应用]实时位姿估计与三维重建相机姿态
关键词:相机位姿估计 OpenCV::solvePnP labview三维图片文章类型:应用展示+Demo演示 @Author:VShawn(singlex@foxmail.com) @Date:2 ...
相机计算坐标公式_相机位姿估计3：根据两幅图像的位姿估计结果求某点的世界坐标...
关键词:相机位姿估计,单目尺寸测量,环境探知用途:基于相机的环境测量,SLAM,单目尺寸测量文章类型:原理说明.Demo展示 @Author:VShawn @Date:2016-11-28 @La ...
《增强现实：原理、算法与应用》读书笔记（5）运动恢复结构（上）初始化、相机位姿估计、集束调整
<增强现实:原理.算法与应用>读书笔记(5)运动恢复结构(上)初始化.相机位姿估计.集束调整运动恢复结构(SfM)是一种从运动的相机拍摄的图像或视频序列中自动地恢复出相机运动轨迹以及场景 ...
30行代码就可以实现看图识字！python使用tensorflow.keras搭建简单神经网络
文章目录搭建过程 1. 引入必需的库 2. 引入数据集 3. 搭建神经网络层 4. 编译神经网络模型 5. 训练模型效果测试大概几个月前,神经网络.人工智能等概念在我心里仍高不可攀,直到自己亲身 ...
c++/opencv利用相机位姿估计实现2D图像像素坐标到3D世界坐标的转换
最近在做自动泊车项目中的车位线检测,用到了将图像像素坐标转换为真实世界坐标的过程,该过程可以通过世界坐标到图像像素坐标之间的关系进行求解,在我的一篇博文中已经详细讲解了它们之间的数学关系,不清楚的童鞋 ...
【记录】本科毕设：基于树莓派的智能小车设计（使用Tensorflow + Keras 搭建CNN卷积神经网络使用端到端的学习方法训练CNN）
0 申明这是本人2020年的本科毕业设计,内容多为毕设论文和答辩内容中挑选.最初的灵感来自于早前看过的一些项目(抱歉时间久远,只记录了这一个,见下),才让我萌生了做个机电(小车动力与驱动)和控制(树 ...
ubuntu16.04+七彩虹GTX1060的NVIDIA驱动+Cuda8.0+cudnn5.1+tensorflow+keras搭建深度学习环境【学习笔记】【原创】
平台信息: PC:ubuntu16.04.i5.七彩虹GTX1060显卡作者:庄泽彬(欢迎转载,请注明作者) 说明:参考了网上的一堆的资料搭建了深度学习的开发环境,下班在宿舍折腾了好几个晚上才搞定, ...
CVPR 2021 | 国防科大：基于几何稳定性分析的物体位姿估计方法
作者|机器之心编辑部来源|机器之心物体 6D 姿态估计是机器人抓取.虚拟现实等任务中的核心研究问题.近些年来,随着深度学习技术和图像卷积神经网络的快速发展,在提取物体的几何特征方面出现了许多需要改 ...

tensorflow keras 搭建相机位姿估计网络--例

目录

概要