如何理解在 PyTorch 中创建叶张量?

How to understand creating leaf tensors in PyTorch?(如何理解在 PyTorch 中创建叶张量?)

问题描述

来自 PyTorch 文档:

b = torch.rand(10, requires_grad=True).cuda()
b.is_leaf
False
# b was created by the operation that cast a cpu Tensor into a cuda Tensor

e = torch.rand(10).cuda().requires_grad_()
e.is_leaf
True
# e requires gradients and has no operations creating it

f = torch.rand(10, requires_grad=True, device="cuda")
f.is_leaf
True
# f requires grad, has no operation creating it

是不是因为在就地操作 requires_grad_() 之前将 Tensor e 投射到 Cuda 中?

Is it because Tensor e was cast into Cuda before the in-place operation requires_grad_()?

并且因为 f 是通过赋值 device="cuda" 而不是通过方法 .cuda() 进行转换的?

And because f was cast by assignment device="cuda" rather than by method .cuda()?

推荐答案

当张量第一次被创建时，它变成了一个叶子节点.

When a tensor is first created, it becomes a leaf node.

基本上，神经网络的所有输入和权重都是计算图的叶节点.

Basically, all inputs and weights of a neural network are leaf nodes of the computational graph.

当对张量执行any操作时，它不再是叶节点.

When any operation is performed on a tensor, it is not a leaf node anymore.

b = torch.rand(10, requires_grad=True) # create a leaf node
b.is_leaf # True
b = b.cuda() # perform a casting operation
b.is_leaf # False

requires_grad_() 与 cuda() 或其他操作方式不同.
它创建了一个新的张量，因为需要梯度(可训练权重)的张量不能依赖于其他任何东西.

requires_grad_() is not an operation in the same way as cuda() or others are.
It creates a new tensor, because tensor which requires gradient (trainable weight) cannot depend on anything else.

e = torch.rand(10) # create a leaf node
e.is_leaf # True
e = e.cuda() # perform a casting operation
e.is_leaf # False
e = e.requires_grad_() # this creates a NEW tensor
e.is_leaf # True

此外，detach() 操作会创建一个不需要梯度的新张量:

Also, detach() operation creates a new tensor which does not require gradient:

b = torch.rand(10, requires_grad=True)
b.is_leaf # True
b = b.detach()
b.is_leaf # True

在最后一个例子中，我们创建了一个已经在 cuda 设备上的新张量.
我们不需要任何操作来转换它.

In the last example we create a new tensor which is already on a cuda device.
We do not need any operation to cast it.

f = torch.rand(10, requires_grad=True, device="cuda") # create a leaf node on cuda

这篇关于如何理解在 PyTorch 中创建叶张量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！