Layers

Convolutions

torchelie.nn.Conv2d(in_ch, out_ch, ks, stride=1, bias=True)

A Conv2d with ‘same’ padding

torchelie.nn.Conv3x3(in_ch, out_ch, stride=1, bias=True)

A 3x3 Conv2d with ‘same’ padding

torchelie.nn.Conv1x1(in_ch, out_ch, stride=1, bias=True)

A 1x1 Conv2d

class torchelie.nn.MaskedConv2d(in_chan, out_chan, ks, center, stride=1, bias=(1, 1))

A masked 2D convolution for PixelCNN

Parameters:

• in_chan (int) – number of input channels
• out_chan (int) – number of output channels
• ks (int) – kernel size
• center (bool) – whereas central pixel is masked or not
• stride (int) – stride, defaults to 1
• bias (2-tuple of ints) – A spatial bias. Either the spatial dimensions of the input for a different bias at each location, or (1, 1) for the same bias everywhere (default)

forward(x)

class torchelie.nn.TopLeftConv2d(in_chan, out_chan, ks, center, stride=1, bias=(1, 1))

A 2D convolution for PixelCNN made of a convolution above the current pixel and another on the left.

Parameters:

• in_chan (int) – number of input channels
• out_chan (int) – number of output channels
• ks (int) – kernel size
• center (bool) – whereas central pixel is masked or not
• stride (int) – stride, defaults to 1
• bias (2-tuple of ints) – A spatial bias. Either the spatial dimensions of the input for a different bias at each location, or (1, 1) for the same bias everywhere (default)

forward(x)

Normalization

class torchelie.nn.AdaIN2d(channels, cond_channels)

Adaptive InstanceNormalization from _Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization (Huang et al, 2017)

Parameters:

• channels (int) – number of input channels
• cond_channels (int) – number of conditioning channels from which bias and scale will be derived

condition(z)

Conditions the layer before the forward pass if z will not be present when calling forward

Parameters:z (2D tensor, optional) – conditioning vector

forward(x, z: Optional[<sphinx.ext.autodoc.importer._MockObject object at 0x7fd9a359a748>] = None)

Forward pass

Parameters:

• x (4D tensor) – input tensor
• z (2D tensor, optional) – conditioning vector. If not present,
• `condition –

Returns:

x, renormalized

class torchelie.nn.FiLM2d(channels, cond_channels)

Feature-wise Linear Modulation from https://distill.pub/2018/feature-wise-transformations/ The difference with AdaIN is that FiLM does not uses the input’s mean and std in its calculations

Parameters:

• channels (int) – number of input channels
• cond_channels (int) – number of conditioning channels from which bias and scale will be derived

condition(z)

Conditions the layer before the forward pass if z will not be present when calling forward

Parameters:z (2D tensor, optional) – conditioning vector

forward(x, z: Optional[<sphinx.ext.autodoc.importer._MockObject object at 0x7fd9a359ab38>] = None)

Forward pass

Parameters:

• x (4D tensor) – input tensor
• z (2D tensor, optional) – conditioning vector. If not present,
• `condition –

Returns:

x, conditioned

class torchelie.nn.PixelNorm(*args, **kwargs)

PixelNorm from ProgressiveGAN

forward(x)

FIXME: Uuughhh, this will be a pain to document .. automodule:: torchelie.nn.batchnom

members: undoc-members:

class torchelie.nn.ImageNetInputNorm

Normalize images channels as torchvision models expects, in a differentiable way

forward(input)

Misc

class torchelie.nn.VQ(latent_dim, num_tokens, dim=1, commitment=0.25, mode='nearest', init_mode='normal', return_indices=True)

Quantization layer from _Neural Discrete Representation Learning_

Parameters:

• latent_dim (int) – number of features along which to quantize
• num_tokens (int) – number of tokens in the codebook
• dim (int) – dimension along which to quantize
• mode ('angular' or 'nearest') – whether the distance between the input vectors and the codebook vectors is computed with a L2 distance or an angular distance
• return_indices (bool) – whether to return the indices of the quantized code points

forward(x)

Forward pass

Parameters:x (tensor) – input tensor Returns:quantized tensor, or (quantized tensor, indices) if self.return_indices

class torchelie.nn.Noise(ch)

Add gaussian noise to the input, with a per channel or global learnable std.

Parameters:ch (int) – number of input channels for a different std on each channel, or 1

forward(x)

class torchelie.nn.Debug(name)

An pass-through layer that prints some debug info during forward pass. It prints its name, the input’s shape, mean of channels means, mean, mean of channels std, and std.

Parameters:name (str) – this layer’s name

forward(x)

class torchelie.nn.Dummy(*args, **kwargs)

A pure pass-through layer

forward(x)

class torchelie.nn.Lambda(lam)

Applies a lambda function on forward()

Parameters:lamb (fn) – the lambda function

forward(x)

class torchelie.nn.Reshape(*shape)

Reshape the input volume

Parameters:

• *shape (ints) – new shape, WITHOUT specifying batch size as first
• as it will remain unchanged. (dimension,) –

forward(x)

Blocks

FIXME: Document it in torchelie.nn namespace

class torchelie.nn.blocks.AutoGANGenBlock(in_ch, out_ch, skips_ch, ks=3, mode='nearest')

A block of the generator discovered by AutoGAN.

Parameters:

• in_ch (int) – number of input channels
• out_ch (int) – number of output channels
• skips_ch (list of int) – a list with one element per incoming skip connection. Each element is the number of channels of the incoming skip connection.
• ks (int) – kernel size of the convolutions
• mode (str) – usampling mode, ‘nearest’ or ‘bilinear’

forward(x, skips=[])

Forward pass

Parameters:

• x (tensor) – input tensor
• skips (list of tensor) – a tensor per incoming skip connection

Returns:

output tensor, intermediate value to use in the next block’s skip

connections

torchelie.nn.blocks.Conv2dBNReLU(in_ch, out_ch, ks, stride=1, leak=0)

A packed block with Conv-BN-ReLU

Parameters:

• in_ch (int) – input channels
• out_ch (int) – output channels
• ks (int) – kernel size
• stride (int) – stride of the conv
• leak (float) – negative slope of the LeakyReLU or 0 for ReLU

Returns:

A packed block with Conv-BN-ReLU as a CondSeq

class torchelie.nn.blocks.Conv2dCondBNReLU(in_ch, out_ch, cond_ch, ks, leak=0)

condition(z)

forward(x, z=None)

torchelie.nn.blocks.Conv2dNormReLU(in_ch, out_ch, ks, norm, stride=1, leak=0)

A packed block with Conv-Norm-ReLU

Parameters:

• in_ch (int) – input channels
• out_ch (int) – output channels
• ks (int) – kernel size
• norm (ctor) – A normalization layer constructor in the form (num_layers) -> norm layer or None
• stride (int) – stride of the conv
• leak (float) – negative slope of the LeakyReLU or 0 for ReLU

Returns:

A packed block with Conv-Norm-ReLU as a CondSeq

class torchelie.nn.blocks.HardSigmoid(*args, **kwargs)

Hard Sigmoid

forward(x)

class torchelie.nn.blocks.HardSwish(*args, **kwargs)

Hard Swish

forward(x)

torchelie.nn.blocks.MConvBNrelu(in_ch, out_ch, ks, center=True)

A packed block with Masked Conv-BN-ReLU

Parameters:

• in_ch (int) – input channels
• out_ch (int) – output channels
• ks (int) – kernel size
• center (bool) – whether the masked conv has access to the central pixel or not

Returns:

A packed block with MaskedConv-BN-ReLU as a CondSeq

torchelie.nn.blocks.MConvNormReLU(in_ch, out_ch, ks, norm, center=True)

A packed block with Masked Conv-Norm-ReLU

Parameters:

• in_ch (int) – input channels
• out_ch (int) – output channels
• ks (int) – kernel size
• norm (ctor) – A normalization layer constructor in the form (num_layers) -> norm layer or None
• center (bool) – whether the masked conv has access to the central pixel or not

Returns:

A packed block with MaskedConv-Norm-ReLU as a CondSeq

class torchelie.nn.blocks.PreactResBlock(in_ch, out_ch, stride=1, norm=<sphinx.ext.autodoc.importer._MockObject object>, use_se=False, bottleneck=False)

A Preactivated Residual Block. Skip connection will be added if the number of input and output channels don’t match or stride > 1 is used.

Parameters:

• in_ch (int) – input channels
• out_ch (int) – output channels
• stride (int) – stride
• norm (callable or None) – a norm layer constructor in the form (num channels) -> norm layer or None.
• use_se (bool) – whether to use Squeeze-And-Excite after the convolutions for a SE-ResBlock
• bottleneck (bool) – whether to use the standard block with two 3x3 convs or the bottleneck block with 1x1 -> 3x3 -> 1x1 convs.

condition(z)

forward(x, z=None)

class torchelie.nn.blocks.ResBlock(in_ch, out_ch, stride=1, norm=<sphinx.ext.autodoc.importer._MockObject object>, use_se=False, bottleneck=False)

A Residual Block. Skip connection will be added if the number of input and output channels don’t match or stride > 1 is used.

Parameters:

• in_ch (int) – input channels
• out_ch (int) – output channels
• stride (int) – stride
• norm (callable or None) – a norm layer constructor in the form (num channels) -> norm layer or None.
• use_se (bool) – whether to use Squeeze-And-Excite after the convolutions for a SE-ResBlock
• bottleneck (bool) – whether to use the standard block with two 3x3 convs or the bottleneck block with 1x1 -> 3x3 -> 1x1 convs.

condition(z)

forward(x, z=None)

class torchelie.nn.blocks.SEBlock(in_ch, reduction=16)

A Squeeze-And-Excite block

Parameters:

• in_ch (int) – input channels
• reduction (int) – channels reduction factor for the hidden number of channels

forward(x)

class torchelie.nn.blocks.SNResidualDiscrBlock(in_ch, out_ch, downsample=False)

A residual block with downsampling and spectral normalization.

Parameters:

• in_ch (int) – number of input channels
• out_ch (int) – number of output channels
• downsample (bool) – whether to downsample

forward(x)

Forward pass

Parameters:x (tensor) – input tensor Returns:output tensor

class torchelie.nn.blocks.SpadeResBlock(in_ch, out_ch, cond_channels, hidden, stride=1)

A Spade ResBlock from Semantic Image Synthesis with Spatially-Adaptive Normalization

https://arxiv.org/abs/1903.07291

torchelie.nn.blocks.make_resnet_shortcut(in_ch, out_ch, stride, norm=<sphinx.ext.autodoc.importer._MockObject object>)

Sequential

class torchelie.nn.WithSavedActivations(model, types=(<sphinx.ext.autodoc.importer._MockObject object>, <sphinx.ext.autodoc.importer._MockObject object>), names=None)

FIXME: PLZ DOCUMENT ME

forward(input, detach)

set_keep_layers(types=(<sphinx.ext.autodoc.importer._MockObject object>, <sphinx.ext.autodoc.importer._MockObject object>), names=None)

class torchelie.nn.CondSeq(*args, **kwargs)

An extension to torch’s Sequential that allows conditioning either as a second forward argument or condition()

condition(z)

Conditions all the layers on z

Parameters:z – conditioning

forward(x, z=None)

Forward pass

Parameters:

• x – input
• z (optional) – conditioning. condition() must be called first if left None