OCNet: Object Context for Semantic Segmentation

Abstract

In this paper, we address the semantic segmentation task with a new context aggregation scheme named object context, which focuses on enhancing the role of object information. Motivated by the fact that the category of each pixel is inherited from the object it belongs to, we define the object context for each pixel as the set of pixels that belong to the same category as the given pixel in the image. We use a binary relation matrix to represent the relationship between all pixels, where the value one indicates the two selected pixels belong to the same category and zero otherwise. We propose to use a dense relation matrix to serve as a surrogate for the binary relation matrix. The dense relation matrix is capable to emphasize the contribution of object information as the relation scores tend to be larger on the object pixels than the other pixels. Considering that the dense relation matrix estimation requires quadratic computation overhead and memory consumption w.r.t. the input size, we propose an efficient interlaced sparse self-attention scheme to model the dense relations between any two of all pixels via the combination of two sparse relation matrices. To capture richer context information, we further combine our interlaced sparse self-attention scheme with the conventional multi-scale context schemes including pyramid pooling (Zhao et al. 2017) and atrous spatial pyramid pooling (Chen et al. 2018). We empirically show the advantages of our approach with competitive performances on five challenging benchmarks including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff.

Appendix

1.1 A. More Discussions on the Benefits of object context

We give a further explanation on why we believe OC is superior to the previous two representation methods including PPM (Zhao et al. 2017) and ASPP (Chen et al. 2017) as following:

Fig. 10

Effect of background context on semantic segmentation and the context explanation provided by grid saliency for an erroneous prediction, the image is taken from MS COCO Lin et al. (2014). The grid saliency (a) shows the responsible context for misclassifying the cow (green) as horse (purple) in the semantic segmentation (b). It shows the training bias that horses are more likely on road than cows. Removing the background context context (c) yields a correctly classified cow (d)

• In theory, enhancing the object information in the context can decrease the variance of the context information, in other words, the context of PPM and ASPP suffers from larger variance than the OC context. Because the OC context only contains the variance of the {object information} while the context of PPM/ASPP further contains the variance of {object information, useful background information, irrelevant background information }. The recent study (Hoyer et al. 2019; Shetty et al. 2019) has verified that the overuse of the noisy context information based on PPM suffers from poor generalization ability. For example, the “cow” pixels might be mis-classified as “horse” pixels when the “cow” appears on the road. We directly use the Fig. 10 from Hoyer et al. (2019) to support our point. In summary, explicitly enhancing the object information might decrease the variance of the context information, thus, increases the generalization ability of model.

• In experiments, according to the results in the Table 2 (in the paper), we have verified that the OCNet outperforms both PSPNet and DeepLabv3 under the fair comparison settings.

1.2 B. Formulation of PPM Context

We illustrate the definition of the context ${\mathcal {I}}_i$ based on PPM (Zhao et al. 2017) scheme:

$$\begin{aligned} {\mathcal {I}}_i = \left\{ j\in {\mathcal {I}}~|~\left\lfloor \frac{(j - 1)k}{N} \right\rfloor = \left\lfloor \frac{(i - 1)k}{N} \right\rfloor \right\} , \end{aligned}$$

(22)

where $k \in \{2, 3, 6\}$ represents different pyramid region partitions. Such context is a aggregation of the pixels the the same quotient.

1.3 C. Formulation of Permutation Matrix

We illustrate the definition of each value $p_{i,j}$ in the permutation matrix ${\mathbf {P}}$:

$$\begin{aligned} p_{i,j} = {\left\{ \begin{array}{ll} 1, &{} j = ((i-1) \bmod P) \times P + \lfloor \frac{i-1}{P}\rfloor + 1; \\ 0, &{} \mathrm {otherwise}, \end{array}\right. } \end{aligned}$$

(23)

where, according to ${\mathbf {W}} = {\mathbf {W}}^l {\mathbf {P}}^{\top } {\mathbf {W}}^g {\mathbf {P}}$, we permute the i-th column of ${\mathbf {W}}^g$ / ${\mathbf {W}}^l$ to the j-th column if $p_{i,j}=1$ / $p_{j,i}=1$ when multiplying permutation matrix ${\mathbf {P}}$ / ${\mathbf {P}}^{\top }$ on the right side of ${\mathbf {W}}^g$ / ${\mathbf {W}}^l$ respectively.

1.4 D. Why the Sparse Relation is More Efficient?

For the convenience of analysis, we rewrite the mathematical formulation of computing the context representations (w/o considering the transform functions $\delta (\cdot )$ and $\rho (\cdot )$) based on dense relation scheme and sparse relation scheme as following. The formulation of dense relation scheme is ${\mathbf {Z}} = {\mathbf {W}} {\mathbf {X}}$ and the formulation of sparse relation scheme is ${\mathbf {Z}} =({\mathbf {W}}^{l}{\mathbf {P}}^{\top }{\mathbf {W}}^{g}{\mathbf {P}}){\mathbf {X}}$. We can see that the formulation of the sparse relation scheme still requires ${\mathcal {O}}(N^2)$ GPU memory to store the reconstructed dense relation matrix ${\mathbf {W}}^{l}{\mathbf {P}}^{\top }{\mathbf {W}}^{g}{\mathbf {P}}$. To avoid such expensive GPU memory consumption, we rewrite the formulation of the sparse relation scheme as ${\mathbf {Z}} ={\mathbf {W}}^{l}({\mathbf {P}}^{\top }({\mathbf {W}}^{g}({\mathbf {P}}{\mathbf {X}})))$ according to the associative laws. Because both ${\mathbf {W}}^{l}$ and ${\mathbf {W}}^{g}$ are sparse block matrices and each block is independent from the other blocks, we compute the multiple block matrices concurrently via transforming these block matrices to align on the batch dimension. Besides, we also implement the permutation matrix via the combination of permute and reshape operation provided in PyTorch. More details are illustrated in the discussion in Sect. 3.3 and Algorithm 1

Fig. 11

Illustrating how the sparse relation approximates the dense relation. We use $\mathrm {A}_1,\mathrm {B}_1,...,\mathrm {B}_3$ to represent the different input positions. The gray arrows represent the information propagation path from one input position to one output position. In (a) Dense Relation, each output position connects with all input positions directly, thus, the relation matrix is fully dense. We use the dense relation matrix ${\mathbf {W}}$ to record the weights on all connections. In (b) Sparse Relation, we have two relation matrices and each relation matrix only contains the sparse connections to a small set of selected pixels, and the combination of the two sparse connections ensures that each output position has direct or indirect relations with all input positions. We use the two sparse relation matrices ${\mathbf {W}}^g$ and ${\mathbf {W}}^l$ to record the weights on all sparse connections

Fig. 12

Illustrating the Interlaced Sparse Self-Attention with Indices Permutation. We mark the positions in the input feature map with the indices from a to p, e.g., the index a represents the spatial position (1, 1) and the index p represents the spatial position (4, 4). We illustrate how we permute the indices in all stages as following: (i) For the Permute in the global relation stage, we permute the positions according to the remainder of the indices divided by the group numbers, e.g., 2 for both height and width dimension. For example, all positions including $\{(1, 1), (1, 3), (3, 1), (2, 2)\}$ share the same remainder (1, 1) when we divide the indices by 2 for both dimensions, thus, we group the positions $\{\mathrm{a}, \mathrm{c}, \mathrm{i}, \mathrm{k}\}$ together. Similarly, we get the other 3 groups of positions: $\{\mathrm{b}, \mathrm{d}, \mathrm{j}, \mathrm{l}\}$ (share the same remainder (1, 0)), $\{\mathrm{e}, \mathrm{g}, \mathrm{m}, \mathrm{o}\}$ (share the same remainder (0, 1)) and $\{\mathrm{f}, \mathrm{h}, \mathrm{n}, \mathrm{p}\}$ (share the same remainder (0, 0)). (ii) For the Permute in the local relation stage, we permute the positions according to the quotient of the indices divided by the group numbers (2, 2). Similarly, we get 4 groups of positions: $\{\mathrm{a}, \mathrm{b}, \mathrm{e}, \mathrm{f}\}$ (share the same quotient (0, 0)), $\{\mathrm{c}, \mathrm{d}, \mathrm{g}, \mathrm{h}\}$ (share the same quotient (0, 1)), $\{\mathrm{i}, \mathrm{j}, \mathrm{m}, \mathrm{n}\}$ (share the same quotient (1, 0)) and $\{\mathrm{k}, \mathrm{l}, \mathrm{o}, \mathrm{p}\}$ (share the same quotient (1, 1))

1.5 E. Intuitive Example of the Sparse Relation Scheme

We use an one-dimensional example in Fig. 11 to explain why the combination of two sparse relation matrices are capable to approximate the dense relation matrix. In other words, both dense relation and sparse relation ensure that each output position is connected with all input positions. Specifically, in Fig. 11 (b), the output position ${\mathrm{A}}_1$ has direct relations with $\{{\mathrm{A}}_2, {\mathrm{A}}_3, {\mathrm{B}}_1\}$ and indirect relations with $\{{\mathrm{B}}_2, {\mathrm{B}}_3\}$ via ${\mathrm{B}}_1$.

1.6 F. Complexity Analysis

We illustrate the proof of the complexity of interlaced sparse self-attention scheme:

Proof

. The shapes of the input & output are: ${\mathbf {X}}$ is of shape $HW\times C$, $\theta ({\mathbf {X}}), \phi ({\mathbf {X}}), \delta ({\mathbf {X}}) \in {\mathbb {R}}^{HW\times \frac{C}{2}}$, $\rho ({\mathbf {X}}) \in {\mathbb {R}}^{HW\times C}$.

In the formulation of the ISA’s global relation stage, the overall complexity of $\theta (\cdot )$, $\phi (\cdot )$, $\delta (\cdot )$, and $\rho (\cdot )$ are ${\mathcal {O}}(HWC^2)$. The overall complexity of $\theta ({\mathbf {X}}^{g}_p)\phi ({\mathbf {X}}^{g}_p)^{\top }$ and ${\mathbf {W}}_p^{g} \delta ({\mathbf {X}}^{g}_p)$ are ${\mathcal {O}}((\frac{HW}{P_{h} P_{w}})^{2}C)$ within each group of positions. There exist $P_{h}P_{w}$ groups in the global relation stage and $P_{h}P_{w}$. Thus, the overall complexity of the global relation stage of ISA is:

$$\begin{aligned} \begin{aligned} T({\mathrm{ISA/global}}) =&T(\theta (\cdot )) + T(\phi (\cdot )) + T(\delta (\cdot )) + T(\rho (\cdot )) \\&+ P_{h}P_{w} T(\theta ({\mathbf {X}}^{g}_p)\phi ({\mathbf {X}}^{g}_p)^{\top }) \\ =&{\mathcal {O}}\left( HWC^2+(HW)^2{C}\frac{1}{{P}_{h}{P}_{w}}\right) , \end{aligned} \end{aligned}$$

(24)

Similarly, we can get the complexity of the lobal relation stage in ISA:

$$\begin{aligned} \begin{aligned} T({\mathrm{ISA/local}}) = {\mathcal {O}}\left( HWC^2+(HW)^2{C}\frac{1}{{Q}_{h}{Q}_{w}}\right) , \end{aligned} \end{aligned}$$

(25)

In summary, we can compute the final complexity of ISA via adding $T({\mathrm{ISA/global}})$ and $T({\mathrm{ISA/local}})$,

$$\begin{aligned} \begin{aligned} T({\mathrm{ISA}})&= {\mathcal {O}}\left( HWC^2+(HW)^2{C}\left( \frac{1}{{P}_{h}{P}_{w}}+\frac{1}{{Q}_{h}{Q}_{w}}\right) \right) \end{aligned} \end{aligned}$$

(26)

where we can achieve the minimized computation complexity of ${\mathcal {O}}(HWC^2+(HW)^{\frac{3}{2}}{C})$ when ${P_{h}P_{w}} = {Q_{h}Q_{w}}$ is satisfied (according to arithmetic mean $\ge$ geometric mean). $\square$

1.7 G. Illustrating the Permutation Scheme of ISA

To help the readers to understand how we select and permute the indices within Interlaced Sparse Self-Attention, we use an example in Fig. 12 to explain the details.

1.8 H. More Details of Pyramid-OC

We explain the details of Pyramid-OC as following: Given an input feature map ${\mathbf {X}}$ of shape ${H\times W\times C}$, we first divide it into $k\times k$ groups ($k\in \{1,2,3,6\}$) following the pyramid partitions of PPM (Zhao et al. 2017):

$$\begin{aligned} {\mathbf {X}} \rightarrow \begin{bmatrix} {\mathbf {X}}_{1,1} &{} {\mathbf {X}}_{1,2} &{} \cdots &{} {\mathbf {X}}_{1,k} \\ {\mathbf {X}}_{2,1} &{} {\mathbf {X}}_{2,2} &{} \cdots &{} {\mathbf {X}}_{2,k} \\ \vdots &{}\vdots &{} \ddots &{} \vdots \\ {\mathbf {X}}_{k,1} &{} {\mathbf {X}}_{k,2} &{} \cdots &{} {\mathbf {X}}_{k,k} \\ \end{bmatrix}, \end{aligned}$$

(27)

where each ${\mathbf {X}}_{i,j}$ is of shape ${\frac{H}{k}\times \frac{W}{k}\times C}, \forall ~i, j\in \{1,2,..,k\}$. We apply the object context pooling (OCP) on each group ${\mathbf {X}}_{i,j}$ (note that the parameters of OCP are shared across the groups within the same partition) to compute the context representations, then, we concatenate the context representations to obtain the output feature ${\mathbf {Z}}^{k}$ of shape ${H\times W\times C}$:

$$\begin{aligned} \begin{bmatrix} \mathrm {OCP}({\mathbf {X}}_{1,1}) &{} \mathrm {OCP}({\mathbf {X}}_{1,2}) &{} \cdots &{} \mathrm {OCP}({\mathbf {X}}_{1,k}) \\ \mathrm {OCP}({\mathbf {X}}_{2,1}) &{} \mathrm {OCP}({\mathbf {X}}_{2,2}) &{} \cdots &{} \mathrm {OCP}({\mathbf {X}}_{2,k}) \\ \vdots &{}\vdots &{} \ddots &{} \vdots \\ \mathrm {OCP}({\mathbf {X}}_{k,1}) &{} \mathrm {OCP}({\mathbf {X}}_{k,2}) &{} \cdots &{} \mathrm {OCP}({\mathbf {X}}_{k,k}) \\ \end{bmatrix} \rightarrow {\mathbf {Z}}^{k}. \end{aligned}$$

(28)

We compute four different context feature maps $\{{\mathbf {Z}}^{1}, {\mathbf {Z}}^{2}, {\mathbf {Z}}^{3},$${\mathbf {Z}}^{6}\}$ based on four different pyramid partitions. Last, we concatenate these context feature maps:

$$\begin{aligned} {\mathbf {Z}} = \mathrm{concate}({\mathbf {Z}}^{1}, {\mathbf {Z}}^{2}, {\mathbf {Z}}^{3}, {\mathbf {Z}}^{6}). \end{aligned}$$

(29)

Fig. 13

Illustrating that the shape of the checkerboard in dense relation (based on interlaced sparse self-attention) is the same as the shape of the global relation map. The left / right two columns present the example from the 2-rd / 1-st row of Fig. 6 (b) / (c) separately

1.9 I. Checkered Artefact with ISA

We can observe checkered artefact in the Figs. 6 and  7, which is caused by our implementation on the visualization of the global relation, local relation and dense relation. We illustrate the related pseudo-code in Algorithm 2. Specifically speaking, for a selected pixel, we multiply a set of global relation matrices (associates with the pixels that belong to the same group as the selected pixel in the local relation stage) with the its local relation matrix. Therefore, the shape of the checkerboard is exactly the same as the shape of the global relation map. For example, we zoom in the dense relation and global relation of some examples in the Fig. 13.