将Cityscape和Foggy_Cityscape转换为PASACAL VOC格式的目标检测数据集

文章目录

将Cityscape和Foggy_Cityscape转换为PASACAL VOC格式的目标检测数据集
一、Cityscape转换为PASACAL VOC格式的目标检测数据集
- 1 .下载Cityscape和Foggy_Cityscape数据集：
- 2 .将gtFine中的json格式的文件转化为xml格式的文件：
二、将Foggy_Cityscape转换为PASACAL VOC格式的目标检测数据集
三、制作ImageSets/Main/.txt

需要将其转换成VOC格式的数据集如下所示：
（根据自己模型中加载数据来决定，有些模型的路径没有城市名称）
├── Cityscape
│ ├── ImageSets
│ │ ├── Main
│ │ │ ├── .txt
│ ├── Annotations
│ │ ├──train
│ │ │├──aachen
│ │ ││├── .xml
│ ├── leftImg8bit
│ │ ├──train
│ │ │├──aachen
│ │ ││├── .png

一、Cityscape转换为PASACAL VOC格式的目标检测数据集

1 .下载Cityscape和Foggy_Cityscape数据集：

数据集官网
（1）解压后的数据集如下所示：

文件夹leftImg8bit_trainvaltest/leftImg8bit：先分train/val/test三个文件夹，每个文件夹下按城市分子文件夹，每个城市文件夹下就是png图像。

文件夹leftImg8bit_trainvaltest_foggy/leftImg8bit_foggy：foggy_cityscapes没有标签，可以直接使用cityscapes的标签；

其对应关系如下：
cityscapes的一张原图对应到foggy_cityscapes中有3张图，
beta=[0.005,0.01,0.02]，分别进行转换可以得到3个foggy_cityscapes数据集。

（2）gtFine数据集格式如下所示：
文件夹gtFine_trainvaltes/gtFine：先分train/val/test三个文件夹，每个文件夹下又按城市分子文件夹，每个城市文件夹下针对每张源图png文件对应了4张标注文件：

在json文件中，包含所需的标注文件：

2 .将gtFine中的json格式的文件转化为xml格式的文件：

这里参考别人的代码，并对其进行修改，

将json转为xml的代码如下所示：（由于我在训练模型时，仅需要’person’, ‘rider’, ‘car’, ‘truck’,
‘bus’, ‘train’, ‘motorcycle’, 'bicycle’这8个类别，所以这里只选取了这8类）

import json
import os
import os.path
from PIL import Imagedef position(pos):# 该函数用来找出xmin,ymin,xmax,ymax即bbox包围框x = []y = []nums = len(pos)for i in range(nums):x.append(pos[i][0])y.append(pos[i][1])x_max = max(x)x_min = min(x)y_max = max(y)y_min = min(y)b = (float(x_min), float(y_min), float(x_max), float(y_max))return bdef convert_annotation(image_id):load_f = open(rootdir + '/' + image_id + "_gtFine_polygons.json", 'r')  # 导入json标签的路径load_dict = json.load(load_f)out_file = open(rootdir + '/' + '%s_leftImg8bit.txt' % (image_id), 'w')  # 输出标签的路径objects = load_dict['objects']nums = len(objects)cls_id = ''for i in range(0, nums):labels = objects[i]['label']if (labels in ['person', 'rider', 'car', 'truck', 'bus', 'train', 'motorcycle', 'bicycle']): #这里我需要用到的类别是这8类print(labels)pos = objects[i]['polygon']bb = position(pos)cls_id = labelsout_file.write(cls_id + " " + " ".join([str(a) for a in bb]) + '\n')if cls_id == '':print('no label json:', "%s_gtFine_polygons.json" % (image_id))def images_id(orgin_picture_dir):  # 获取训练集每个图像的名称  （orgin_picture_dir训练集图像的路径）a = []for parent, dirnames, filenames in os.walk(orgin_picture_dir):for filename in filenames:print(filename)filename = filename.split('_leftImg8bit.png')[0]a.append(filename)return adef xml_generator(orgin_picture_dir, rootdir): #生成xml文件img_basenames = os.listdir(orgin_picture_dir)img_names = []for item in img_basenames:temp1, temp2 = os.path.splitext(item)  # 分别提取图片名称和图片后缀名称（具有_leftImg8bit）img_names.append(temp1)print(img_names)for img in img_names:  # img是没有后缀的pic名称im = Image.open((orgin_picture_dir + img + '.png'))width, height = im.size# open the crospronding txt filegt = open(rootdir + '/' + img + '.txt').read().splitlines()# write in xml filexml_file = open((rootdir + '/' + img + '.xml'), 'w')xml_file.write('<annotation>\n')xml_file.write('    <folder>CITYSCAPE</folder>\n')xml_file.write('    <filename>' + str(img) + '.png' + '</filename>\n')xml_file.write('    <size>\n')xml_file.write('        <width>' + str(width) + '</width>\n')xml_file.write('        <height>' + str(height) + '</height>\n')xml_file.write('        <depth>3</depth>\n')xml_file.write('    </size>\n')# write the region of image on xml filefor img_each_label in gt:spt = img_each_label.split(' ')  # 这里如果txt里面是以逗号‘，’隔开的，那么就改为spt = img_each_label.split(',')。xml_file.write('    <object>\n')xml_file.write('        <name>' + str(spt[0]) + '</name>\n')xml_file.write('        <pose>Unspecified</pose>\n')xml_file.write('        <truncated>0</truncated>\n')xml_file.write('        <difficult>0</difficult>\n')xml_file.write('        <bndbox>\n')xml_file.write('            <xmin>' + str(spt[1]) + '</xmin>\n')xml_file.write('            <ymin>' + str(spt[2]) + '</ymin>\n')xml_file.write('            <xmax>' + str(spt[3]) + '</xmax>\n')xml_file.write('            <ymax>' + str(spt[4]) + '</ymax>\n')xml_file.write('        </bndbox>\n')xml_file.write('    </object>\n')xml_file.write('</annotation>')if __name__ == '__main__':city_name = os.listdir('D:/ENTERAINMENT/datasets/gtFine_trainvaltest/gtFine/train/')print(city_name)for city in city_name:rootdir = 'D:/ENTERAINMENT/datasets/gtFine_trainvaltest/gtFine/train/' + city + '/'  # json所在路径orgin_picture_dir = 'D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest/leftImg8bit/train/' + city + '/'   #cityscape的train所在路径names = images_id(orgin_picture_dir)for image_id in names:print(image_id)convert_annotation(image_id)xml_generator(orgin_picture_dir, rootdir)

xml与json在同一个路径下：

xml文件打开如下：

如果不想将xml与json放在同一路径下，将xml单独放在一个文件夹中，
将xml_generator替换成xml_newdir函数即可，代码如下：

def xml_newdir(orgin_picture_dir, rootdir, new_rootdir):img_basenames = os.listdir(orgin_picture_dir)img_names = []for item in img_basenames:temp1, temp2 = os.path.splitext(item)  # 分别提取图片名称和图片后缀名称（具有_leftImg8bit）img_names.append(temp1)print(img_names)for img in img_names:  # img是没有后缀的pic名称im = Image.open((orgin_picture_dir + img + '.png'))width, height = im.size# open the crospronding txt filegt = open(rootdir + '/' + img + '.txt').read().splitlines()# write in xml filexml_file = open((new_rootdir + '/' + img + '.xml'), 'w')xml_file.write('<annotation>\n')xml_file.write('    <folder>CITYSCAPE</folder>\n')xml_file.write('    <filename>' + str(img) + '.png' + '</filename>\n')xml_file.write('    <size>\n')xml_file.write('        <width>' + str(width) + '</width>\n')xml_file.write('        <height>' + str(height) + '</height>\n')xml_file.write('        <depth>3</depth>\n')xml_file.write('    </size>\n')# write the region of image on xml filefor img_each_label in gt:spt = img_each_label.split(' ')  # 这里如果txt里面是以逗号‘，’隔开的，那么就改为spt = img_each_label.split(',')。xml_file.write('    <object>\n')xml_file.write('        <name>' + str(spt[0]) + '</name>\n')xml_file.write('        <pose>Unspecified</pose>\n')xml_file.write('        <truncated>0</truncated>\n')xml_file.write('        <difficult>0</difficult>\n')xml_file.write('        <bndbox>\n')xml_file.write('            <xmin>' + str(spt[1]) + '</xmin>\n')xml_file.write('            <ymin>' + str(spt[2]) + '</ymin>\n')xml_file.write('            <xmax>' + str(spt[3]) + '</xmax>\n')xml_file.write('            <ymax>' + str(spt[4]) + '</ymax>\n')xml_file.write('        </bndbox>\n')xml_file.write('    </object>\n')xml_file.write('</annotation>')if __name__ == '__main__':city_name = os.listdir('D:/ENTERAINMENT/datasets/gtFine_trainvaltest/gtFine/train/')print(city_name)for city in city_name:rootdir = 'D:/ENTERAINMENT/datasets/gtFine_trainvaltest/gtFine/train/' + city + '/'  # json所在路径orgin_picture_dir = 'D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest/leftImg8bit/train/' + city + '/'   #cityscape的train所在路径names = images_id(orgin_picture_dir)for image_id in names:print(image_id)convert_annotation(image_id)#xml_generator(orgin_picture_dir, rootdir)new_rootdir = 'D:/ENTERAINMENT/datasets/gtFine_trainvaltest/xml/'  # 新路径if os.path.exists(new_rootdir + city):passelse:os.makedirs(new_rootdir + city)xml_newdir(orgin_picture_dir, rootdir, new_rootdir + city)

用labelimg打开的结果：

二、将Foggy_Cityscape转换为PASACAL VOC格式的目标检测数据集

转换的代码如下：

# 将cityscpe_xml转化为fog_cityscapes_xml:
import os
import xml.etree.ElementTree as ET
source_path = "D:/ENTERAINMENT/datasets/gtFine_trainvaltest/gtFine/train/"
city_names =os.listdir(source_path)
for city in city_names:cityscape_xml_path = "D:/ENTERAINMENT/datasets/gtFine_trainvaltest/gtFine/train/" + city + '/'foggy_xml_path = "D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest_foggy/leftImg8bit_foggy/train/" + city + '/'betas = [0.01, 0.02, 0.005]city_xml_list = os.listdir(cityscape_xml_path)for file in city_xml_list:if file.endswith('.xml') or file.endswith('XML'):city_xml = fileprint(city_xml)tree = ET.parse(cityscape_xml_path + city_xml)root = tree.getroot()filename = root.find('filename').text  # 获取xml中的filenamefor beta in betas:filename = filename.split(".png")[0] + "_foggy_beta_" + str(beta) + ".png"root.find('filename').text = filenamefoggy_city_xml = city_xml.split(".xml")[0] + "_foggy_beta_" + str(beta) + ".xml"tree.write(foggy_xml_path + foggy_city_xml)

结果如下：

用labelimg打开的结果：

三、制作ImageSets/Main/.txt

1.cityscape数据集如下所示（这里就以train来为例）

# 制作cityscape数据集的：/ImageSets/Main/train.txt
# 批量读取文件名（不带后缀）(加入前缀train/city_name/pic_name)import os
file_path = "D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest/leftImg8bit/train/"
citys = os.listdir(file_path)
for city in citys:pic_path = file_path + city + "/"pic_path_list = os.listdir(pic_path)for file_list in pic_path_list:if file_list.endswith('png') or file_list.endswith('PNG'):with open("D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest/ImageSets/Main/train.txt", "a") as f:f.write("train/" + city + "/" + file_list.split("_leftImg8bit.png")[0] + "\n")

结果如下：

2.foggy_cityscape数据集与cityscape数据集制作方式一样。
如果需要加入后缀beta=[0.01, 0.02, 0.005]时，代码如下所示（这里就以train来为例，是否加入beta的数值需要根据模型中对数据集的要求来决定）：

# 制作foggy_cityscapes_beta数据集的：/ImageSets/Main/train.txt
# P02 批量读取文件名（不带后缀）(加入前缀train/city_name/pic_name)
import os
xml_path = "D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest_foggy/leftImg8bit_foggy/train/"
citys = os.listdir(xml_path)
for city in citys:pic_path = xml_path + city + "/"pic_path_list = os.listdir(pic_path)for file_list in pic_path_list:if file_list.endswith('xml') or file_list.endswith('XML'):print("train/" + city + "/" + file_list.split(".xml")[0])with open("D:/ENTERAINMENT/datasets/leftImg8bit_trainvaltest_foggy/ImageSets/Main/train.txt", "a") as f:f.write("train/" + city + "/" + file_list.split(".xml")[0] + "\n")

参考资料：
[1] 将Cityscape转换为PASACAL VOC格式的目标检测数据集
[2] cityscapes和foggy_cityscapes数据集使用记录