pytorch gather_【Pytorch】Pytorch-1.1.0 版本新特性
2019年05月01日,Pytorch 1.1.0 版本正式发布啦~https://github.com/pytorch/pytorch/releases/tag/v1.1.0
主要的几个功能:
1. TensorBoard (currently experimental)
2. JIT 的升级
· [JIT] Attributes in ScriptModules
· [JIT] Dictionary and List Support in TorchScript
· [JIT] User-defined classes in TorchScript (experimental)
3. DistributedDataParallel new functionality and tutorials
TensorBoard (currently experimental)
- PyTorch now supports TensorBoard logging with a simple
from torch.utils.tensorboard import SummaryWriter
command. - Histograms, embeddings, scalars, images, text, graphs, and more can be visualized across training runs.
- TensorBoard support is currently experimental. You can browse the docs here.
JIT
- Attributes in ScriptModules
- Attributes can be assigned on a
ScriptModule
by wrapping them withtorch.jit.Attribute
and specifying the type. - They will be serialized along with any paramters/buffers when you call
torch.jit.save()
, so they are a great way to store arbitrary state in your model. - See the docs for more info.
- Example:
class Foo(torch.jit.ScriptModule):def __init__(self, a_dict):super(Foo, self).__init__(False)self.words = torch.jit.Attribute([], List[str])self.some_dict = torch.jit.Attribute(a_dict, Dict[str, int])@torch.jit.script_methoddef forward(self, input: str) -> int:self.words.append(input)return self.some_dict[input]
- Dictionary and List Support in TorchScript
- TorchScript now has robust support for list and dictionary types. They behave much like Python lists and dictionaries, supporting most built-in methods, as well as simple comprehensions and
for…in
constructs.
- User-defined classes in TorchScript (experimental)
- For more complex stateful operations, TorchScript now supports annotating a class with
@torch.jit.script
. Classes used this way can be JIT-compiled and loaded in C++ like other TorchScript modules. - See the docs for more info.
- Example:
@torch.jit.script
class Pair:def __init__(self, first, second)self.first = firstself.second = seconddef sum(self):return self.first + self.second
DistributedDataParallel new functionality and tutorials
nn.parallel.DistributedDataParallel
: can now wrap multi-GPU modules, which enables use cases such as model parallel (tutorial) on one server and data parallel (tutorial) across servers. (19271).
Breaking Changes
Tensor.set_
: thedevice
of a Tensor can no longer be changed viaTensor.set_
. This would most commonly happen when setting up a Tensor with the default CUDA device and later swapping in aStorage
on a different CUDA device. Instead, set up the Tensor on the correct device from the beginning. (18832).- Pay attention to the order change of
lr_scheduler.step()
. (7889). torch.unique
: changed the default value ofsorted
toTrue
. (15379).- [JIT] Rename isTensor api -> isCompleteTensor. #18437
- [JIT] Remove GraphExecutor's python bindings. #19141
- [C++]: many methods on
Type
no longer exist; use the functional or Tensor method equivalent. (17991). - [C++]: the
Backend
constructor ofTensorOptions
no longer exists. (18137). - [C++, Distributed]: Remove c10d
ProcessGroup::getGroupRank
has been removed. (19147).
【New Features】
这次的版本更新添加了很多可调用的方法。
Operators
torch.tril_indices
,torch.triu_indices
: added operator with same behavior as NumPy. (14904, 15203).torch.combinations
,torch.cartesian_prod
: added newitertools
-like operators. (9393).torch.repeat_interleave
: new operator similar tonumpy.repeat
. (18395).torch.from_file
: new operator similar toStorage.from_file
, but returning a tensor. (18688).torch.unique_consecutive
: new operator with semantics similar tostd::unique
in C++. (19060).torch.tril
,torch.triu
,torch.trtrs
: now support batching. (15257, 18025).torch.gather
: add support forsparse_grad
option. (17182).torch.std
,torch.max_values
,torch.min_values
,torch.logsumexp
can now operate over multiple dimensions at once. (14535, 15892, 16475).torch.cdist
: added operator equivalent toscipy.spatial.distance.cdist
. (16168, 17173).torch.__config__.show()
: reports detailed version of all libraries. (18579).
NN
nn.MultiheadedAttention
: new module implementing MultiheadedAttention fromAttention Is All You Need
. (18334).nn.functional.interpolate
: added support forbicubic
. (9849).nn.SyncBatchNorm
: support synchronous Batch Normalization. (14267).nn.Conv
: added support for Circular Padding viamode='circular'
. (17240).nn.EmbeddingBag
: now supports trainableper_sample_weights
. (18799).nn.EmbeddingBag
: add support forfrom_pretrained
method, as innn.Embedding
. (15273).RNNs
: automatically handle unsorted variable-length sequences viaenforce_sorted
. (15225).nn.Identity
: new module for easier model surgery. (19249).
Tensors / dtypes
torch.bool
: added support fortorch.bool
dtype and Tensors with that dtype (1-byte storage). NumPy conversion is supported, but operations are currently limited. (16810).
Optim
optim.lr_scheduler.CyclicLR
: Support for Cyclical Learning Rate and Momentum. (18001).optim.lr_scheduler.CosineAnnealingWarmRestarts
: new scheduler: Stochastic Gradient Descent with Warm Restarts). (17226).- Support multiple simultaneous LR schedulers. (14010)
Distributions
torch.distributions
: now support multiple inheritance. (16772).
Samplers
quasirandom.SobolEngine
: new sampler. (10505).
DistributedDataParallel
nn.parallel.DistributedDataParallel
: now supports modules with unused parameters (e.g. control flow, like adaptive softmax, etc). (18251, 18953).
TorchScript and Tracer
- Allow early returns from if-statements. (#154463)
- Add an
@ignore
annotation, which statically tells the TorchScript compiler to ignore the Python function. (#16055) - Simple
for...in
loops on lists. (#16726) - Ellipses (
...
) in Tensor indexing. (#17763) None
in Tensor indexing. (#18615)- Support for basic list comprehensions. (#17267)
- Add implicit unwrapping of optionals on
if foo is not None
. (#15587) - Tensors, ints, and floats will once again be implicitly cast to bool if used in a conditional. (#18755).
- Implement
to()
,cpu()
, andcuda()
on ScriptModules. (#15340 , #15904) - Add support for various methods on lists: (
clear()
,pop()
,reverse()
,copy()
,extend()
,index()
,count()
,insert()
,remove()
). - Add support for
sort()
on lists of specialized type (Tensors
,int
,float
,bool
). (#19572) - Add support for various methods on strings: (
index()
,slice()
,len()
) - Support
Tensor.to()
in TorchScript. ( #15976 ) - Support for
Torch.tensor()
in TorchScript. (#14913, #19445) - Support for
torch.manual_seed()
in TorchScript. (#19510) - Support for
nn.LSTM
in TorchScript. (#15744) - Support for
nn.init
in TorchScript. (#19640) - Add
hash()
builtin. (#18258) - Add
min()
andmax()
builtins for numerical types. (#15680) - Add
isinstance()
builtin, which performs a static type check. (#15076) - Add
train()
/eval()
/is_training()
to C++ ScriptModule API. (#16044) - Allow List arguments to Python functions called from TorchScript. (#15721)
- Allow using
std::vector
andstd::unordered_map
as arguments to custom operators. (#17587) - Tracer: now allows passing static dicts and lists as trace inputs. (#18092, #19580)
- Allow generic containers as ScriptModule inputs. (#16482)
- Allow
nn.Sequential
in ModuleList. (#16882)
Experimental Features
- [Quantization] (API unstable): added limited support for quantized datatypes via
torch.qint8
dtype,torch.quantize_linear
conversion function. (18230). - [MKLDNN tensor] (API unstable): Added limited (opaque) support for
MKLDNN
tensors viaTensor.to_mkldnn()
; operators are currently limited to ResNext101 operators. (17748).
另外,日志里还有关于【Improvements】【Bug Fixes】【Deprecations】【Performance】【Documentation】【ONNX】的说明。
下边是已经修复了的比较严重的一些Bug。
torch.prod
: correct erroneous calculation on large tensors. (15653).torch.mean
(and other reductions): fix incorrect calculation on CUDA on large inputs. (16023).nn.Conv
: correctly handle non-contiguous inputs on MKLDNN convolution codepath. (16300).Tensor.eq_
: Fix erroneous calculation. (15475).torch.mean
: Fix fp16 output calculation. (14878).nn.PoissonNLLLoss
: Properly handlereduction=None
. (17358).- [JIT] Fix bug where custom ops could get optimized out if their outputs weren't used. (#18711).
- [JIT] Fix bug where the model serializer would accidentally reorder statements. (#17557).
下边是挑出的几个比较显著的【Performance】。
nn.BatchNorm
CPU inference speed increased up to ~19x.(19152).nn.AdaptiveAvgPool
: speed up common-case of size=1 output by ~30x. (17011).nn.EmbeddingBag
CPU performance increased by ~4x. (19329).Tensor.copy_
: sped up larger tensor copy ~2-3x, small regression in small tensor copy. (18618).torch.nonzero
: is now ~2x faster than numpy on CPU. (15190)- Improve caching allocator for Pascal and newer GPUs; 10-20% better memory utilization on Mask-RCNN. (17120).
reduction functions
: Speed up some large Tensor cases by 50-80%. (17428).- [JIT] Graph fuser: better fusion for backwards graphs in the presence of broadcasting. (#14957)
- [JIT] Graph fuser:
batch_norm
fusion for inference. (#15146) - [JIT] Graph fuser:
layer_norm
fusion for inference. (#18266)
pytorch gather_【Pytorch】Pytorch-1.1.0 版本新特性相关推荐
- Spark 3.2.0 版本新特性 push-based shuffle 论文详解(一)概要和介绍
前言 本文隶属于专栏<大数据技术体系>,该专栏为笔者原创,引用请注明来源,不足和错误之处请在评论区帮忙指出,谢谢! 本专栏目录结构和参考文献请见大数据技术体系 目录 Spark 3.2.0 ...
- Spark 3.2.0 版本新特性 push-based shuffle 论文详解(二)背景和动机
前言 本文隶属于专栏<大数据技术体系>,该专栏为笔者原创,引用请注明来源,不足和错误之处请在评论区帮忙指出,谢谢! 本专栏目录结构和参考文献请见大数据技术体系 目录 Spark 3.2.0 ...
- Android Q(10.0)版本新特性以及兼容性适配
北京时间2019年3月14日Google正式对外发布Android Q Beta 1及预览版SDK,这意味着安卓开发者们又即将迎来一年一度的新版本适配工作了.Android Q 为开发者们带来了许多新 ...
- Atitit..jdk java 各版本新特性 1.0 1.1 1.2 1.3 1.4 1.5(5.0) 1.6(6.0) 7.0 8.0 9.0 attilax 大总结...
Atitit..jdk java 各版本新特性 1.0 1.1 1.2 1.3 1.4 1.5(5.0) 1.6(6.0) 7.0 8.0 9.0 attilax 大总结 1.1. Java的编年史2 ...
- Android Q(10.0 API29)版本新特性和兼容性适配
摘要 1.本文档基于谷歌AndroidQ官方文档和一加Q版本应用兼容性整改指导 2.本文档主要对影响比较大的部分进行简单总结,内容并不全面: 3.版本号对应关系: Android-Q = Androi ...
- Android Q(10.0)版本新特性和兼容性适配
北京时间2019年3月14日Google正式对外发布Android Q Beta 1及预览版SDK,这意味着安卓开发者们又即将迎来一年一度的新版本适配工作了.Android Q 为开发者们带来了许多新 ...
- Neo4j图数据科学及2.0版本新功能介绍
本文转载自DataFunTalk,作者刘洋,Neo4j亚太区高级技术顾问. 导读:本文将探讨Neo4j的图数据科学平台,以及2.0版本的新功能.主要内容包括: Neo4j图数据科学(GDS)的前世今生 ...
- Tensorflow 2.0的新特性
Tensorflow 2.0的新特性 几天前,Tensorflow刚度过自己的3岁生日,作为当前最受欢迎的机器学习框架,Tensorflow在这个宝座上已经盘踞了近三年.无论是成熟的Keras,还是风 ...
- mysql 5.0 php_PHP 5.0的新特性
PHP 5.0的新特性 最近,读者可以从PHP 4.x版本转移到PHP 5.0版本.正如读者期望的那样,在一个新的主要版本中,它做出了一些重要变更.在这个版本中,PHP后台的Zend引擎经过了完全的重 ...
最新文章
- requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('',))
- linux java内存分析_Java内存分析利器MAT使用详解
- 计算机中什么是适配器及作用,适配器是什么?适配器的作用主要有哪些
- iOS - UISearchController
- python写接口自动化需要rsa加密_RSA加密,请问如何用Python实现该加密过程
- 指令篇: 查看系统版本信息___uname
- windows7旗舰版系统自带组件IIS搭建ftp
- Spark记录-Scala数据类型
- layer弹出层扩展自定义样式
- 【引用】使用CommonDialog的ShowSave后如何判断是保存还是取消?
- python获取城市区域边界坐标
- Python3实现两个Excel文件内容比对
- 登录注册判断+Mysql
- 盘古搜索22日开通 欲打造一流搜索引擎
- hu沪江计算机词汇,拼音带hu的字大全150个拼音含hu的字组词 - 小孩子点读
- Python练习task2:条件与循环
- 无线蜂窝通信模组是什么?
- 误删除与误格式化的挽回(图)
- 网页禁止粘贴的解决方法(以学习通网页为例)
- 发帖添加作者水印插件无法发帖问题-缺少GD库支持,php如何安装gd库-一颗优雅草科技伊凡