复现KM3D:Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training

时间：2022年5月27日

1.下载官方代码

到github中，搜索KM3D找到官方代码，下载。网址如下：
https://github.com/Banconxuan/RTM3D

2. 安装环境（install）

查看install.md，按照里面的内容进行环境的安装。
2.1 安装pytorch环境
官方要求安装的pytorch环境太老了。我没有按照官方的。我的环境如下：
Ubuntu 18.04.3 LTS
conda + python 3.7 + cuda10.1
pytorch1.6.0
torchvision0.7.0
显卡：Quadro P4000 8GB

conda create -n km3d python=3.7 -y
conda activate km3d
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch -y

既然选择了较新的环境，意味着后面要修改代码bug，以适应环境。
2.2 Install the requirements
进入km3d路径下，执行如下命令：

pip install -r requirements.txt

2.3 安装DCN v2.
参考知乎《单目3D目标检测RTM3D》作者：Vision 连接：https://zhuanlan.zhihu.com/p/415084925
没有用官方github提供的方法，提供的DCNv2版本太老了，不适用。所以，在github上搜索DCN v2找到适合pytorch的版本并安装。
我的环境pytorch1.6，所以安装的DCNv2要适配pytorch1.6
github搜索DCNv2，找到DCNv2_latest，作者jinfagang，使用pytorch1.6分支，按照说明进行安装即可，连接：https://github.com/jinfagang/DCNv2_latest/tree/pytorch1.6

2.4 安装 iou3d
参考知乎《单目3D目标检测RTM3D》作者：Vision 连接：https://zhuanlan.zhihu.com/p/415084925
进入src\lib\utils\iou3d\src，编辑iou3d.cpp
在所有#include<>后面添加3行代码即可：

#ifndef AT_CHECK
#define AT_CHECK TORCH_CHECK
#endif

整体就是：

#include <torch/serialize/tensor.h>
#include <torch/extension.h>
#include <vector>
#include <cuda.h>
#include <cuda_runtime_api.h>#ifndef AT_CHECK
#define AT_CHECK TORCH_CHECK
#endif#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x," must be a CUDAtensor ")
#define CHECK_CONTIGUOUS(x) AT_CHECK(x.is_contiguous(), #x," must be contiguous ")
#define CHECK_INPUT(x) CHECK_CUDA(x);CHECK_CONTIGUOUS(x)#define DIVUP(m,n) ((m) / (n) + ((m) % (n) > 0))#define CHECK_ERROR(ans) { gpuAssert((ans), __FILE__, __LINE__); }
inline void gpuAssert(cudaError_t code, const char *file, int line, bool abort=true)
{if (code != cudaSuccess){fprintf(stderr,"GPUassert: %s %s %d\n", cudaGetErrorString(code), file, line);if (abort) exit(code);}
}const int THREADS_PER_BLOCK_NMS = sizeof(unsigned long long) * 8;
后面的代码省略，要不然太长了
后面的代码省略，要不然太长了
后面的代码省略，要不然太长了
后面的代码省略，要不然太长了

修改好iou3d.cpp后，执行如下命令，即可成功安装：

cd $KM3D_ROOT/src/lib/utiles/iou3d
python setup.py install

3.处理kitti数据集

kitti数据集按照km3d要求来处理

KM3DNet
├── kitti_format
│   ├── data
│   │   ├── kitti
│   │   |   ├── annotations /kitti_val.json kitti_train.json
│   │   │   ├── calib /000000.txt .....
│   │   │   ├── image(left[0-7480] right[7481-14961] input augmentatiom)
│   │   │   ├── label /000000.txt .....
|   |   |   ├── train.txt val.txt trainval.txt

calib是left image training集的。kitti自带。
image是left image training和right image training在一起的。kitti自带。其中，right image training集图片的名称要修改一下，修改脚本如下：

import os
import cv2
original_path = './image_3/'
process_path = './process_right_image/'
original_name = os.listdir(original_path)
for i, ori_name in enumerate(original_name):ori_image = cv2.imread(original_path + ori_name)ori_num = ori_name.replace('.png', '')ori_num = int(ori_num)new_num = ori_num + 7481if new_num > 9999:new_name = '0' + str(new_num) + '.png'else:new_name = '00' + str(new_num) + '.png'cv2.imwrite(process_path + new_name, ori_image)print(i+1)print(ori_num)print(new_num)print(new_name)

label是left image training 的标签。kitti自带。
train.txt val.txt trainval.txt 是KM3D自带。
annotations文件夹要自己新建，然后里面的kitti_val.json和kitti_train.json需要运行代码生成！！！生成方法如下：

python src/tools/kitti.py

如果没有annotations和kitti_val.json文件的话，后面运行程序会报错：

No such file or directory: '../kitti_format/data/kitti/annotations/kitti_val.json'

4.训练

参考官方文档Getting Started部分。
根据自己的显卡调整训练的参数。
ResNet-18骨干网络：

python ./src/main.py --data_dir ./kitti_format --exp_id KM3D_res18 --arch res_18 --batch_size 8 --master_batch_size 16 --lr 1.5625e-5 --gpus 0 --num_epochs 200

执行上述命令，会出现如下错误：
报错1

Traceback (most recent call last):File "./src/main.py", line 111, in <module>main(opt)File "./src/main.py", line 73, in mainlog_dict_train, _ = trainer.train(epoch, train_loader)File "/root/RTM3D/src/lib/trains/base_trainer.py", line 162, in trainreturn self.run_epoch('train', epoch, data_loader,unlabel_loader1,unlabel_loader2,unlabel_set,iter_num,uncert)File "/root/RTM3D/src/lib/trains/base_trainer.py", line 97, in run_epochoutput, loss, loss_stats = model_with_loss(batch,phase=phase)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_implresult = self.forward(*input, **kwargs)File "/root/RTM3D/src/lib/trains/base_trainer.py", line 33, in forwardloss, loss_stats = self.loss(outputs, batch,phase)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_implresult = self.forward(*input, **kwargs)File "/root/RTM3D/src/lib/trains/car_pose.py", line 52, in forwardcoor_loss, prob_loss, box_score = self.position_loss(output, batch,phase)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_implresult = self.forward(*input, **kwargs)File "/root/RTM3D/src/lib/models/losses.py", line 332, in forwardcys = (batch['ind'] / self.opt.output_w).int().float()
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

版本问题，tensor和int之间的除法不能直接用’/‘，用torch.true_divide()代替。修改’/src/lib/models/losses.py’中的第332行，修改后：

cys = torch.true_divide(batch['ind'], self.opt.output_w).int().float()

报错2

Traceback (most recent call last):File "./src/main.py", line 111, in <module>main(opt)File "./src/main.py", line 73, in mainlog_dict_train, _ = trainer.train(epoch, train_loader)File "/root/RTM3D/src/lib/trains/base_trainer.py", line 162, in trainreturn self.run_epoch('train', epoch, data_loader,unlabel_loader1,unlabel_loader2,unlabel_set,iter_num,uncert)File "/root/RTM3D/src/lib/trains/base_trainer.py", line 97, in run_epochoutput, loss, loss_stats = model_with_loss(batch,phase=phase)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_implresult = self.forward(*input, **kwargs)File "/root/RTM3D/src/lib/trains/base_trainer.py", line 33, in forwardloss, loss_stats = self.loss(outputs, batch,phase)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_implresult = self.forward(*input, **kwargs)File "/root/RTM3D/src/lib/trains/car_pose.py", line 52, in forwardcoor_loss, prob_loss, box_score = self.position_loss(output, batch,phase)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_implresult = self.forward(*input, **kwargs)File "/root/RTM3D/src/lib/models/losses.py", line 440, in forwarddim_mask_score_mask = 1 - (dim_mask_score_mask > 0)File "/root/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/tensor.py", line 396, in __rsub__return _C._VariableFunctions.rsub(self, other)
RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported. If you are trying to invert a mask, use the `~` or `logical_not()` operator instead.

原因是pytorch最新版本中，不支持bool型与int型减法，如果需要对bool变量进行反转，则使用“~”操作。修改’/src/lib/models/losses.py‘中第440行，修改后

dim_mask_score_mask = ~ (dim_mask_score_mask > 0)

5 Results generation

python ./src/faster.py --demo ./kitti_format/data/kitti/val.txt --data_dir ./kitti_format --calib_dir ./kitti_format/data/kitti/calib/ --load_model ./kitti_format/exp/KM3D_res18/model_last.pth --gpus 0 --arch res_18

报错1：

Traceback (most recent call last):File "./src/faster.py", line 55, in <module>demo(opt)File "./src/faster.py", line 46, in demoret = detector.run(image_name)File "/root/RTM3D/src/lib/detectors/base_detector.py", line 139, in runoutput, dets, forward_time = self.process(images,meta,return_time=True)File "/root/RTM3D/src/lib/detectors/car_pose.py", line 57, in processoutput['hm'], output['hps'], output['dim'], output['rot'], prob=output['prob'],K=self.opt.K, meta=meta, const=self.const)File "/root/RTM3D/src/lib/models/decode.py", line 788, in car_pose_decode_fasterscores, inds, clses, ys, xs = _topk(heat, K=K)File "/root/RTM3D/src/lib/models/decode.py", line 109, in _topktopk_ys   = (topk_inds / width).int().float()
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

版本问题，tensor和int之间的除法不能直接用’/'，用torch.true_divide()代替。

topk_ys   = torch.true_divide(topk_inds, width).int().float()

后面还有类似的bug，这里只列举一个。
报错2：
参考https://blog.csdn.net/long_songs/article/details/124736338?ops_request_misc=&request_id=&biz_id=102&utm_term=Overload%20resolution%20failed:%20>%20&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduweb~default-0-124736338.nonecase&spm=1018.2226.3001.4187

具体的报错信息忘了截图了，类似于下面的错误

cv2.error: OpenCV(4.5.2) :-1: error: (-5:Bad argument) in function 'pointPolygonTest'
> Overload resolution failed:
>  - Can't parse 'pt'. Sequence item with index 0 has a wrong type
>  - Can't parse 'pt'. Sequence item with index 0 has a wrong type

这种就是因为不是int，在能加int的地方全给 int()上就完事了！。
/src/lib/utils/ddd_utils.py中的draw_bos_3d()函数中cv2.line()函数的参数修改成int()类型。
修改前：

def draw_box_3d(image, corners, c=(0, 0, 255)):face_idx = [[0,1,5,4],[1,2,6, 5],[2,3,7,6],[3,0,4,7]]for ind_f in range(3, -1, -1):f = face_idx[ind_f]for j in range(4):cv2.line(image, (corners[f[j], 0], corners[f[j], 1]),(corners[f[(j+1)%4], 0], corners[f[(j+1)%4], 1]), c, 2, lineType=cv2.LINE_AA)if ind_f == 0:cv2.line(image, (corners[f[0], 0], corners[f[0], 1]),(corners[f[2], 0]),corners[f[2], 1]), c, 1, lineType=cv2.LINE_AA)cv2.line(image, (corners[f[1], 0], corners[f[1], 1]),(corners[f[3], 0], corners[f[3], 1]), c, 1, lineType=cv2.LINE_AA)return image

修改后：

def draw_box_3d(image, corners, c=(0, 0, 255)):face_idx = [[0,1,5,4],[1,2,6, 5],[2,3,7,6],[3,0,4,7]]for ind_f in range(3, -1, -1):f = face_idx[ind_f]for j in range(4):cv2.line(image, (int(corners[f[j], 0]),int(corners[f[j], 1])),(int(corners[f[(j+1)%4], 0]), int(corners[f[(j+1)%4], 1])), c, 2, lineType=cv2.LINE_AA)if ind_f == 0:cv2.line(image, (int(corners[f[0], 0]), int(corners[f[0], 1])),(int(corners[f[2], 0]),int(corners[f[2], 1])), c, 1, lineType=cv2.LINE_AA)cv2.line(image, (int(corners[f[1], 0]), int(corners[f[1], 1])),(int(corners[f[3], 0]), int(corners[f[3], 1])), c, 1, lineType=cv2.LINE_AA)return image

6 Visualization

python ./src/faster.py --vis --demo ./kitti_format/data/kitti/val.txt --data_dir ./kitti_format --calib_dir ./kitti_format/data/kitti/calib/ --load_model ./kitti_format/exp/KM3D_res18/model_last.pth --gpus 0 --arch res_18

报错1：

Traceback (most recent call last):File "./src/faster.py", line 55, in <module>demo(opt)File "./src/faster.py", line 46, in demoret = detector.run(image_name)File "/root/RTM3D/src/lib/detectors/base_detector.py", line 163, in runself.show_results(debugger, image, results, calib_numpy)File "/root/RTM3D/src/lib/detectors/car_pose.py", line 113, in show_resultsdebugger.show_all_imgs(pause=self.pause)File "/root/RTM3D/src/lib/utils/debugger.py", line 364, in show_all_imgscv2.imshow('{}'.format(i), v)
cv2.error: OpenCV(4.5.5) /io/opencv/modules/highgui/src/window.cpp:1268: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support.If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvShowImage'

解决方法参考：
https://blog.csdn.net/tudou2013goodluck/article/details/108402055?ops_request_misc=&request_id=&biz_id=102&utm_term=If%20you%20are%20on%20Ubuntu%20or%20Debian&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduweb~default-1-108402055.nonecase

安装open-contrib-python

pip install opencv-contrib-python

我又试了试，发现这个项目只需要opencv-contrib-python这个包。可以把opencv其他的包卸载掉，只安装opencv-contrib-python。

效果图：

按键盘上的上下左右键可以切换图片。
注意这张图片中的黑色部分是终端黑窗口，截图的时候截下来了，不是图片显示错误。

7. All-in-one fashion

Generating results in all-in-one fashion. It simultaneously generates main center, 2D bounding box, regressed keypoints, heatmap keypoint, dim, orientation, confidence.
官方给的命令如下：

python ./src/faster.py --demo ./kitti_format/data/kitti/val.txt --calib_dir ./kitti_format/data/kitti/calib/ --load_model ./kitti_format/exp/KM3D_res18/model_last.pth --gpus 0 --arch res_18

执行命令会报错：

[ WARN:0@4.303] global /io/opencv/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('data/kitti/image/000001.png'): can't open/read file: check file path/integrity
Traceback (most recent call last):File "./src/faster.py", line 57, in <module>demo(opt)File "./src/faster.py", line 48, in demoret = detector.run(image_name)File "/root/RTM3D/src/lib/detectors/base_detector.py", line 127, in runimages, meta = self.pre_process(image, scale, meta)File "/root/RTM3D/src/lib/detectors/base_detector.py", line 44, in pre_processheight, width = image.shape[0:2]
AttributeError: 'NoneType' object has no attribute 'shape'

注意报错信息的第一行，无法读取’data/kitti/image/000001.png’，显然路径是有问题的，无法读入图片，这才导致‘None Type’类型。
仔细观察这条命令，会发现这条命令和上文中Results generation部分的命令只少了data_dir参数：

--data_dir ./kitti_format

个人感觉作者的命令是漏掉了这个参数，补上这个参数就能正常执行了，补上这个参数后路径就变成了./kitti_format/data/kitti/image/000001.png，这样路径才是正确的啊。
正确命令如下，同Results generation部分的命令：

python ./src/faster.py --demo ./kitti_format/data/kitti/val.txt --data_dir ./kitti_format --calib_dir ./kitti_format/data/kitti/calib/ --load_model ./kitti_format/exp/KM3D_res18/model_last.pth --gpus 0 --arch res_18

个人的理解就是这样的，愿意听听大家的理解。

8.Evaluation

 python ./src/tools/kitti-object-eval-python/evaluate.py evaluate --label_path=./kitti_format/data/kitti/label/ --label_split_file ./kitti_format/data/kitti/val.txt --current_class=0,1,2 --coco=False --result_path=./kitti_format/exp/results/data/

结果如下：

Car AP@0.70, 0.70, 0.70:
bbox AP:84.86, 67.33, 57.84
bev  AP:22.03, 15.89, 16.11
3d   AP:11.93, 8.26, 6.63
aos  AP:84.84, 66.52, 57.12Car AP_R40@0.70, 0.70, 0.70:
bbox AP:86.85, 70.19, 60.34
bev  AP:15.91, 9.66, 8.50
3d   AP:11.40, 5.57, 4.38
aos  AP:86.82, 69.23, 59.24Car AP@0.70, 0.50, 0.50:
bbox AP:84.86, 67.33, 57.84
bev  AP:48.56, 33.41, 31.73
3d   AP:43.06, 27.55, 26.34
aos  AP:84.84, 66.52, 57.12Car AP_R40@0.70, 0.50, 0.50:
bbox AP:86.85, 70.19, 60.34
bev  AP:45.02, 30.01, 26.71
3d   AP:42.28, 25.16, 22.66
aos  AP:86.82, 69.23, 59.24Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:24.48, 27.27, 27.27
bev  AP:3.03, 4.55, 4.55
3d   AP:3.03, 4.55, 4.55
aos  AP:19.60, 22.03, 22.03Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:19.23, 24.33, 24.33
bev  AP:0.00, 1.93, 1.93
3d   AP:0.00, 1.25, 1.25
aos  AP:14.75, 18.41, 18.41Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:24.48, 27.27, 27.27
bev  AP:9.09, 13.64, 13.64
3d   AP:9.09, 13.64, 13.64
aos  AP:19.60, 22.03, 22.03Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:19.23, 24.33, 24.33
bev  AP:2.60, 6.25, 6.25
3d   AP:2.60, 6.25, 6.25
aos  AP:14.75, 18.41, 18.41Cyclist AP@0.50, 0.50, 0.50:
bbox AP:3.03, 3.03, 3.03
bev  AP:2.27, 2.27, 2.27
3d   AP:2.27, 2.27, 2.27
aos  AP:3.00, 3.00, 3.00Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:0.83, 0.83, 0.83
bev  AP:0.00, 0.00, 0.00
3d   AP:0.00, 0.00, 0.00
aos  AP:0.82, 0.82, 0.82Cyclist AP@0.50, 0.25, 0.25:
bbox AP:3.03, 3.03, 3.03
bev  AP:2.27, 2.27, 2.27
3d   AP:2.27, 2.27, 2.27
aos  AP:3.00, 3.00, 3.00Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:0.83, 0.83, 0.83
bev  AP:0.00, 0.00, 0.00
3d   AP:0.00, 0.00, 0.00
aos  AP:0.82, 0.82, 0.82

复现KM3D:Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training相关推荐

【论文翻译】FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
文章目录 PaperInfo Abstract 1 Introduction 2 Related Work 2D Object Detection Monocular 3D Object Detect ...
论文精读《CaDNN: Categorical Depth Distribution Network for Monocular 3D Object Detection》
CaDNN: Categorical Depth Distribution Network for Monocular 3D Object Detection 文章目录 CaDNN: Categori ...
论文阅读 [TPAMI-2022] Disentangling Monocular 3D Object Detection: From Single to Multi-Class Recognitio
论文阅读 [TPAMI-2022] Disentangling Monocular 3D Object Detection: From Single to Multi-Class Recognitio ...
复现 MonoEF：Monocular 3D Object Detection: An Extrinsic Parameter Free Approach
复现 MonoEF:Monocular 3D Object Detection: An Extrinsic Parameter Free Approach 时间:2022年7月13日代码连接: ht ...
论文阅读笔记：(2021.06, cvpr) Monocular 3D Object Detection: An Extrinsic Parameter Free Approach
这是一篇发表在了cvpr 2021上,能够在线估计外参的单目3D目标检测算法,借鉴了visual odometry和style transfer的方法,效果好, 速度快(~30ms), 意料之外,情理 ...
Delving into Localization Errors for Monocular 3D Object Detection 论文学习
论文地址:Delving into Localization Errors for Monocular 3D Object Detection Github地址:Delving into Locali ...
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection
Paper name FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection Paper Reading Note UR ...
论文精读《OFT: Orthographic Feature Transform for Monocular 3D Object Detection》
OFT: Orthographic Feature Transform for Monocular 3D Object Detection 文章目录 OFT: Orthographic Feature ...
【3D目标检测】SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation
目录前提概述细节网络结构特征提取网络检测头1:关键点检测检测头2:3D参数回归损失函数前提需要先了解下CenterNet [目标检测]Objects as Points 概述本文 ...

复现KM3D:Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training

复现KM3D:Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training

1.下载官方代码

2. 安装环境（install）

3.处理kitti数据集

4.训练

5 Results generation

6 Visualization

7. All-in-one fashion

8.Evaluation

复现KM3D:Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training相关推荐

最新文章

热门文章