Figure 1: The result of using a spatial transformer as the  first layer of a fully-connected network trained for distorted  MNIST digit classification. (a) The input to the spatial transformer  network is an image of an MNIST digit that is distorted  with random translation, scale, rotation, and clutter. (b)  The localisation network of the spatial transformer predicts a  transformation to apply to the input image. (c) The output  of the spatial transformer, after applying the transformation.  (d) The classification prediction produced by the subsequent  fully-connected network on the output of the spatial transformer.  The spatial transformer network (a CNN including a  spatial transformer module) is trained end-to-end with only  class labels – no knowledge of the groundtruth transformations  is given to the system.

Figure 2: The architecture of a spatial transformer module. The input feature map U is passed to a localisation  network which regresses the transformation parameters θ. The regular spatial grid G over V is transformed to  the sampling grid Tθ(G), which is applied to U as described in Sect. 3.3, producing the warped output feature  map V . The combination of the localisation network and sampling mechanism defines a spatial transformer.

where (x  t  i  , yt  i  ) are the target coordinates of the regular grid in the output feature map, (x  s  i  , ys  i  ) are  the source coordinates in the input feature map that define the sample points, and Aθ is the affine  transformation matrix. We use height and width normalised coordinates, such that −1 ≤ x  t  i  , yt  i ≤ 1  when within the spatial bounds of the output, and −1 ≤ x  s  i  , ys  i ≤ 1 when within the spatial bounds  of the input (and similarly for the y coordinates). The source/target transformation and sampling is  equivalent to the standard texture mapping and coordinates used in graphics [8].

The class of transformations Tθ may be more constrained, such as that used for attention  Aθ =    s 0 tx  0 s ty    (2)  allowing cropping, translation, and isotropic scaling by varying s, tx, and ty. The transformation  Tθ can also be more general, such as a plane projective transformation with 8 parameters, piecewise  affine, or a thin plate spline. Indeed, the transformation can have any parameterised form,  provided that it is differentiable with respect to the parameters – this crucially allows gradients to be  backpropagated through from the sample points Tθ(Gi) to the localisation network output θ. If the  transformation is parameterised in a structured, low-dimensional way, this reduces the complexity  of the task assigned to the localisation network. For instance, a generic class of structured and differentiable  transformations, which is a superset of attention, affine, projective, and thin plate spline  transformations, is Tθ = MθB, where B is a target grid representation (e.g. in (10), B is the regular  grid G in homogeneous coordinates), and Mθ is a matrix parameterised by θ. In this case it is  possible to not only learn how to predict θ for a sample, but also to learn B for the task at hand.

To perform a spatial transformation of the input feature map, a sampler must take the set of sampling  points Tθ(G), along with the input feature map U and produce the sampled output feature map V .  Each (x  s  i  , ys  i  ) coordinate in Tθ(G) defines the spatial location in the input where a sampling kernel  is applied to get the value at a particular pixel in the output V . This can be written as

where Φx and Φy are the parameters of a generic sampling kernel k() which defines the image  interpolation (e.g. bilinear), U  c  nm is the value at location (n, m) in channel c of the input, and V  c  i  is the output value for pixel i at location (x  t  i  , yt  i  ) in channel c. Note that the sampling is done  identically for each channel of the input, so every channel is transformed in an identical way (this  preserves spatial consistency between channels).  In theory, any sampling kernel can be used, as long as (sub-)gradients can be defined with respect to  x  s  i  and y  s  i  . For example, using the integer sampling kernel reduces (3) to  V  c  i =  X  H  n  X  W  m  U  c  nmδ(bx  s  i + 0.5c − m)δ(by  s  i + 0.5c − n) (4)  where bx + 0.5c rounds x to the nearest integer and δ() is the Kronecker delta function. This  sampling kernel equates to just copying the value at the nearest pixel to (x  s  i  , ys  i  ) to the output location  (x  t  i  , yt  i  ). Alternatively, a bilinear sampling kernel can be used, giving  V  c  i =  X  H  n  X  W  m  U  c  nm max(0, 1 − |x  s  i − m|) max(0, 1 − |y  s  i − n|) (5)  To allow backpropagation of the loss through this sampling mechanism we can define the gradients  with respect to U and G. For bilinear sampling (5) the partial derivatives are  ∂V c  i  ∂Uc  nm  =  X  H  n  X  W  m  max(0, 1 − |x  s  i − m|) max(0, 1 − |y  s  i − n|) (6)  ∂V c  i  ∂xs  i  =  X  H  n  X  W  m  U  c  nm max(0, 1 − |y  s  i − n|)        0 if |m − x  s  i  | ≥ 1  1 if m ≥ x  s  i  −1 if m < xs  i  (7)  and similarly to (7) for ∂V c  i  ∂ys  i  .

Table 1: Left: The percentage errors for different models on different distorted MNIST datasets. The different  distorted MNIST datasets we test are TC: translated and cluttered, R: rotated, RTS: rotated, translated, and  scaled, P: projective distortion, E: elastic distortion. All the models used for each experiment have the same  number of parameters, and same base structure for all experiments. Right: Some example test images where  a spatial transformer network correctly classifies the digit but a CNN fails. (a) The inputs to the networks. (b)  The transformations predicted by the spatial transformers, visualised by the grid Tθ(G). (c) The outputs of the  spatial transformers. E and RTS examples use thin plate spline spatial transformers (ST-CNN TPS), while R  examples use affine spatial transformers (ST-CNN Aff) with the angles of the affine transformations given. For  videos showing animations of these experiments and more see https://goo.gl/qdEhUu.

In this section we use the MNIST handwriting dataset as a testbed for exploring the range of transformations  to which a network can learn invariance to by using a spatial transformer.

We begin with experiments where we train different neural network models to classify MNIST data  that has been distorted in various ways: rotation (R), rotation, scale and translation (RTS), projective  transformation (P), and elastic warping (E) – note that elastic warping is destructive and can not be  inverted in some cases. The full details of the distortions used to generate this data are given in  Appendix A. We train baseline fully-connected (FCN) and convolutional (CNN) neural networks,  as well as networks with spatial transformers acting on the input before the classification network  (ST-FCN and ST-CNN). The spatial transformer networks all use bilinear sampling, but variants use  different transformation functions: an affine transformation (Aff), projective transformation (Proj),  and a 16-point thin plate spline transformation (TPS) [2]. The CNN models include two max-pooling  layers. All networks have approximately the same number of parameters, are trained with identical  optimisation schemes (backpropagation, SGD, scheduled learning rate decrease, with a multinomial  cross entropy loss), and all with three weight layers in the classification network.

Table 2: Left: The sequence error for SVHN multi-digit recognition on crops of 64 × 64 pixels (64px), and  inflated crops of 128 × 128 (128px) which include more background. *The best reported result from [1] uses  model averaging and Monte Carlo averaging, whereas the results from other models are from a single forward  pass of a single model. Right: (a) The schematic of the ST-CNN Multi model. The transformations applied by  each spatial transformer (ST) is applied to the convolutional feature map produced by the previous layer. (b)  The result of multiplying out the affine transformations predicted by the four spatial transformers in ST-CNN  Multi, visualised on the input image.

Table 3: Left: The accuracy on CUB-200-2011 bird classification dataset. Spatial transformer networks with  two spatial transformers (2×ST-CNN) and four spatial transformers (4×ST-CNN) in parallel achieve higher  accuracy. 448px resolution images can be used with the ST-CNN without an increase in computational cost  due to downsampling to 224px after the transformers. Right: The transformation predicted by the spatial  transformers of 2×ST-CNN (top row) and 4×ST-CNN (bottom row) on the input image. Notably for the  2×ST-CNN, one of the transformers (shown in red) learns to detect heads, while the other (shown in green)  detects the body, and similarly for the 4×ST-CNN.