我的问题是如何从多个(或分片)tfrecords获得批量输入.我已经阅读了示例https://github.com/tensorflow/models/blob/master/inception/inception/image_processing.py#L410.基本管道是,以训练集为例,(1)首先生成一系列tfrecords(例如,000-train-of-005,train-001-of-005 ,. ..),(2)从这些文件名中,生成一个列表并将它们输入到tf.train.string_input_producer中以获取队列,(3)同时生成一个tf.RandomShuffleQueue来做其他的事情,(4)使用tf.train .batch_join生成批输入.

我认为这很复杂,我不确定这个程序的逻辑.在我的情况下,我有一个.npy文件列表,我想生成分片的tfrecords(多个分离的tfrecords,而不只是一个单个大文件).这些.npy文件中的每一个都包含不同数量的正样本和负样本(2个类).一种基本方法是生成一个单个大型tfrecord文件.但文件太大(~20Gb).所以我采用分片的tfrecords.有没有更简单的方法来做到这一点？谢谢.

解决方法:

使用Dataset API简化整个过程.以下是两个部分：(1)：将numpy数组转换为tfrecords和(2,3,4)：读取tfrecords以生成批次.

1.从numpy数组创建tfrecords：

def npy_to_tfrecords(...):

# write records to a tfrecords file

writer = tf.python_io.TFRecordWriter(output_file)

# Loop through all the features you want to write

for ... :

let say X is of np.array([[...][...]])

let say y is of np.array[[0/1]]

# Feature contains a map of string to feature proto objects

feature = {}

feature['X'] = tf.train.Feature(float_list=tf.train.FloatList(value=X.flatten()))

feature['y'] = tf.train.Feature(int64_list=tf.train.Int64List(value=y))

# Construct the Example proto object

example = tf.train.Example(features=tf.train.Features(feature=feature))

# Serialize the example to a string

serialized = example.SerializeToString()

# write the serialized objec to the disk

writer.write(serialized)

writer.close()

2.使用Dataset API读取tfrecords(tensorflow> = 1.2)：

# Creates a dataset that reads all of the examples from filenames.

filenames = ["file1.tfrecord", "file2.tfrecord", ..."fileN.tfrecord"]

dataset = tf.contrib.data.TFRecordDataset(filenames)

# for version 1.5 and above use tf.data.TFRecordDataset

# example proto decode

def _parse_function(example_proto):

keys_to_features = {'X':tf.FixedLenFeature((shape_of_npy_array), tf.float32),

'y': tf.FixedLenFeature((), tf.int64, default_value=0)}

parsed_features = tf.parse_single_example(example_proto, keys_to_features)

return parsed_features['X'], parsed_features['y']

# Parse the record into tensors.

dataset = dataset.map(_parse_function)

# Shuffle the dataset

dataset = dataset.shuffle(buffer_size=10000)

# Repeat the input indefinitly

dataset = dataset.repeat()

# Generate batches

dataset = dataset.batch(batch_size)

# Create a one-shot iterator

iterator = dataset.make_one_shot_iterator()

# Get batch X and y

X, y = iterator.get_next()

标签：python,tensorflow,tensorflow-datasets,tfrecord

来源： https://codeday.me/bug/20190917/1809981.html