python unpack原理_Python transformers.Unpack方法代码示例
本文整理汇总了Python中fuel.transformers.Unpack方法的典型用法代码示例。如果您正苦于以下问题:Python transformers.Unpack方法的具体用法?Python transformers.Unpack怎么用?Python transformers.Unpack使用的例子?那么恭喜您, 这里精选的方法代码示例或许可以为您提供帮助。您也可以进一步了解该方法所在模块fuel.transformers的用法示例。
在下文中一共展示了transformers.Unpack方法的7个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Python代码示例。
示例1: setup_datastream
点赞 3
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def setup_datastream(path, vocab_file, config):
ds = QADataset(path, vocab_file, config.n_entities, need_sep_token=config.concat_ctx_and_question)
it = QAIterator(path, shuffle=config.shuffle_questions)
stream = DataStream(ds, iteration_scheme=it)
if config.concat_ctx_and_question:
stream = ConcatCtxAndQuestion(stream, config.concat_question_before, ds.reverse_vocab[''])
# Sort sets of multiple batches to make batches of similar sizes
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size * config.sort_batch_count))
comparison = _balanced_batch_helper(stream.sources.index('question' if config.concat_ctx_and_question else 'context'))
stream = Mapping(stream, SortMapping(comparison))
stream = Unpack(stream)
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size))
stream = Padding(stream, mask_sources=['context', 'question', 'candidates'], mask_dtype='int32')
return ds, stream
开发者ID:thomasmesnard,项目名称:DeepMind-Teaching-Machines-to-Read-and-Comprehend,代码行数:21,
示例2: setup_cnnsquad_datastream
点赞 3
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def setup_cnnsquad_datastream(sq_path, cnn_path, vocab_file, config):
ds = CNNSQDataset(sq_path, cnn_path, vocab_file)
it = CNNSQIterator(sq_path, cnn_path, cnn_ratio=config.add_cnn_data)
stream = DataStream(ds, iteration_scheme=it)
# Sort sets of multiple batches to make batches of similar sizes
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size * config.sort_batch_count))
comparison = _balanced_batch_helper(stream.sources.index('context'))
stream = Mapping(stream, SortMapping(comparison))
stream = Unpack(stream)
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size))
stream = Padding(stream, mask_sources=['context', 'question', 'answer'], mask_dtype='int32')
return ds, stream
开发者ID:arianhosseini,项目名称:Question-Answering,代码行数:18,
示例3: setup_squad_datastream
点赞 3
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def setup_squad_datastream(path, vocab_file, config):
ds = SQuADDataset(path, vocab_file)
it = SQuADIterator(path)
stream = DataStream(ds, iteration_scheme=it)
if config.concat_ctx_and_question:
stream = ConcatCtxAndQuestion(stream, config.concat_question_before, ds.reverse_vocab[''])
# Sort sets of multiple batches to make batches of similar sizes
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size * config.sort_batch_count))
comparison = _balanced_batch_helper(stream.sources.index('context'))
stream = Mapping(stream, SortMapping(comparison))
stream = Unpack(stream)
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size))
stream = Padding(stream, mask_sources=['context', 'question', 'answer', 'ans_indices','ans_boundaries'], mask_dtype='int32')
return ds, stream
#train examples count 1836975
#dev examples count 221697
开发者ID:arianhosseini,项目名称:Question-Answering,代码行数:24,
示例4: setup_squad_ranker_datastream
点赞 3
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def setup_squad_ranker_datastream(path, vocab_file, config, example_count=1836975):
ds = SQuADRankerDataset(path, vocab_file)
it = ShuffledExampleScheme(examples=example_count)
stream = DataStream(ds, iteration_scheme=it)
# Sort sets of multiple batches to make batches of similar sizes
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size * config.sort_batch_count))
comparison = _balanced_batch_helper(stream.sources.index('question'))
stream = Mapping(stream, SortMapping(comparison))
stream = Unpack(stream)
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size))
stream = Padding(stream, mask_sources=['question', 'answer', 'better', 'worse', 'b_left', 'b_right','w_left','w_right'], mask_dtype='int32')
return ds, stream
开发者ID:arianhosseini,项目名称:Question-Answering,代码行数:18,
示例5: setup_datastream
点赞 3
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def setup_datastream(path, batch_size, sort_batch_count, valid=False):
A = numpy.load(os.path.join(path, ('valid_x_raw.npy' if valid else 'train_x_raw.npy')))
B = numpy.load(os.path.join(path, ('valid_phn.npy' if valid else 'train_phn.npy')))
C = numpy.load(os.path.join(path, ('valid_seq_to_phn.npy' if valid else 'train_seq_to_phn.npy')))
D = [B[x[0]:x[1], 2] for x in C]
ds = IndexableDataset({'input': A, 'output': D})
stream = DataStream(ds, iteration_scheme=ShuffledExampleScheme(len(A)))
stream = Batch(stream, iteration_scheme=ConstantScheme(batch_size * sort_batch_count))
comparison = _balanced_batch_helper(stream.sources.index('input'))
stream = Mapping(stream, SortMapping(comparison))
stream = Unpack(stream)
stream = Batch(stream, iteration_scheme=ConstantScheme(batch_size, num_examples=len(A)))
stream = Padding(stream, mask_sources=['input', 'output'])
return ds, stream
开发者ID:thomasmesnard,项目名称:CTC-LSTM,代码行数:21,
示例6: _get_sgnmt_tr_stream
点赞 2
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def _get_sgnmt_tr_stream(data_stream,
src_vocab_size=30000,
trg_vocab_size=30000,
seq_len=50,
batch_size=80,
sort_k_batches=12,
src_sparse_feat_map='',
trg_sparse_feat_map='',
**kwargs):
"""Prepares the raw text file stream ``data_stream`` for the Blocks
main loop. This includes handling UNKs, splitting ino batches, sort
locally by sequence length, and masking. This roughly corresponds
to ``get_sgnmt_tr_stream`` in ``machine_translation/stream`` in the
blocks examples.
The arguments to this method are given by the configuration dict.
"""
# Filter sequences that are too long
s = Filter(data_stream, predicate=stream._too_long(seq_len=seq_len))
# Replacing out of vocabulary tokens with unk token already
# handled in the `DataSet`s
# Build a batched version of stream to read k batches ahead
s = Batch(s, iteration_scheme=ConstantScheme(batch_size*sort_k_batches))
# Sort all samples in the read-ahead batch
s = Mapping(s, SortMapping(stream._length))
# Convert it into a stream again
s = Unpack(s)
# Construct batches from the stream with specified batch size
s = Batch(s, iteration_scheme=ConstantScheme(batch_size))
# Pad sequences that are short
masked_stream = stream.PaddingWithEOS(s, [utils.EOS_ID, utils.EOS_ID])
return masked_stream
开发者ID:ucam-smt,项目名称:sgnmt,代码行数:42,
示例7: setup_toy_datastream
点赞 2
# 需要导入模块: from fuel import transformers [as 别名]
# 或者: from fuel.transformers import Unpack [as 别名]
def setup_toy_datastream(config):
ds = ToyDataset()
it = ToyIterator()
stream = DataStream(ds, iteration_scheme=it)
# Sort sets of multiple batches to make batches of similar sizes
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size * config.sort_batch_count))
comparison = _balanced_batch_helper(stream.sources.index('context'))
stream = Mapping(stream, SortMapping(comparison))
stream = Unpack(stream)
stream = Batch(stream, iteration_scheme=ConstantScheme(config.batch_size))
stream = Padding(stream, mask_sources=['context', 'question', 'answer','ans_indices'], mask_dtype='int32')
return ds, stream
开发者ID:arianhosseini,项目名称:Question-Answering,代码行数:17,
注:本文中的fuel.transformers.Unpack方法示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。
python unpack原理_Python transformers.Unpack方法代码示例相关推荐
- python iteritems函数_Python six.iteritems方法代码示例
本文整理汇总了Python中sklearn.externals.six.iteritems方法的典型用法代码示例.如果您正苦于以下问题:Python six.iteritems方法的具体用法?Pyth ...
- python html模板_Python html.format_html方法代码示例
本文整理汇总了Python中django.utils.html.format_html方法的典型用法代码示例.如果您正苦于以下问题:Python html.format_html方法的具体用法?Pyt ...
- python session模块_Python backend.set_session方法代码示例
本文整理汇总了Python中keras.backend.set_session方法的典型用法代码示例.如果您正苦于以下问题:Python backend.set_session方法的具体用法?Pyth ...
- python color属性_Python turtle.color方法代码示例
本文整理汇总了Python中turtle.color方法的典型用法代码示例.如果您正苦于以下问题:Python turtle.color方法的具体用法?Python turtle.color怎么用?P ...
- python end用法_Python turtle.end_fill方法代码示例
本文整理汇总了Python中turtle.end_fill方法的典型用法代码示例.如果您正苦于以下问题:Python turtle.end_fill方法的具体用法?Python turtle.end_ ...
- python qt 按钮_Python QtWidgets.QPushButton方法代码示例
本文整理汇总了Python中PySide2.QtWidgets.QPushButton方法的典型用法代码示例.如果您正苦于以下问题:Python QtWidgets.QPushButton方法的具体用 ...
- python geometry用法_Python geometry.MultiPolygon方法代码示例
本文整理汇总了Python中shapely.geometry.MultiPolygon方法的典型用法代码示例.如果您正苦于以下问题:Python geometry.MultiPolygon方法的具体用 ...
- python logistic步骤_Python api.Logit方法代码示例
本文整理汇总了Python中statsmodels.api.Logit方法的典型用法代码示例.如果您正苦于以下问题:Python api.Logit方法的具体用法?Python api.Logit怎么 ...
- python gc模块_Python gc.collect方法代码示例
本文整理汇总了Python中gc.collect方法的典型用法代码示例.如果您正苦于以下问题:Python gc.collect方法的具体用法?Python gc.collect怎么用?Python ...
- python fmod函数_Python numpy.fmod方法代码示例
本文整理汇总了Python中numpy.fmod方法的典型用法代码示例.如果您正苦于以下问题:Python numpy.fmod方法的具体用法?Python numpy.fmod怎么用?Python ...
最新文章
- 【pmcaff】12月5日凯文·凯利上海演讲实录
- android6.0源码分析之Zygote进程分析
- 浅析SQL Server数据库中的伪列以及伪列的含义
- inner join 和join的区别_left join、right join和join ???
- ue4集合类型_UE4项目问题集合
- HTML窗口与对话框
- zookeeper 进入客户端_探究—Zookeeper的启动过程
- axios直传阿里云,获取上传进度已以及取消上传
- linux后台执行命令与putty打开程序界面 screen
- wps office2019PC版和Mac版_来试试这款政府版WPS吧
- hibernate5.x下载地址
- Linux meset
- 想知道电脑上怎么压缩图片?用这3个方法实现快速压缩
- HTML中1mm等于多少像素,像素英寸和像素厘米-像素与毫米的关系1毫米等于多少像素 – 手机爱问...
- 常见文件文件头和隐写术总结 CTF中Misc必备
- 生产者消费者模型【新版】
- SonarQube在Windows环境下下载安装,中文包下载安装,mysql配置,maven配置,idea配置,项目配置
- 详谈京东的商品搜索系统架构设计
- 怎么样给小孩取名字?给自己的孩子起名就应该注意这些问题
- 易语言用多文件对话框写MP3播放器
热门文章
- java集成easyExcel动态生成表头并在浏览器下载excel
- 七甲川荧光染料IR820 NHS ester,新吲哚菁绿-活化酯,New Indocyanine Green-nhs ester
- matlab 图片黑白图片,MATLAB读取黑白图像显示却是黑色,24位深转8位深黑白图像解决方法(示例代码)...
- NOIP 2017 列队 (线段树动态开点)
- 传播智客风清扬JAVA 10G
- 如何解决vue项目本地ip地址无法访问项目问题?【亲测有效】
- 【水哥RPA】配置水哥RPA环境变量报错的解决方法
- CocosCreater 发布apk接穿山甲广告SDK(一)
- myexcel初步使用
- Android简易实战教程--第三十八话《自定义通知NotifiCation》