Understanding Padding in TensorFlow's tf.nn.max_pool: 'SAME' vs. 'VALID'

When implementing pooling operations like tf.nn.max_pool in TensorFlow, a crucial decision is how to handle padding. The two primary padding strategies, 'SAME' and 'VALID', significantly influence the output shape and the movement of the pooling window across the input feature map.

What is tf.nn.max_pool?

tf.nn.max_pool is a fundamental TensorFlow function used to perform max pooling. This is a downsampling technique commonly employed in Convolutional Neural Networks (CNNs) to reduce spatial dimensions (width and height) while retaining the most salient features.

Basic Syntax:

tf.nn.max_pool(
    input,
    ksize,
    strides,
    padding,
    data_format='NHWC',
    name=None
)

• input: The input tensor (e.g., a feature map).
• ksize: The size of the pooling window (kernel). It's typically a list or tuple of integers specifying the height and width (and potentially depth and batch dimensions).
• strides: The steps the pooling window moves across the input. Similar to ksize, it specifies movement in height and width.
• padding: The padding strategy, either 'SAME' or 'VALID'.
• data_format: Specifies the data format of the input tensor. Defaults to 'NHWC' (Batch, Height, Width, Channels).
• name: An optional name for the operation.

1. 'SAME' Padding

Definition:

With 'SAME' padding, TensorFlow adds zero-padding to the borders of the input tensor. This is done strategically so that the output tensor's spatial dimensions (height and width) are as close as possible to the input tensor's dimensions, given the specified strides.

Key Features:

• Output Size Preservation: When strides=1, the output spatial dimensions generally match the input spatial dimensions. If strides > 1, the output size is calculated to be approximately ceil(input_dim / stride).
• Symmetrical Padding: Padding is typically added symmetrically around the borders of the input.
• Use Case: 'SAME' padding is particularly useful when you want to maintain the spatial dimensions of your feature maps throughout the network, often seen in deeper CNN architectures or when the output of a pooling layer needs to match the input for subsequent operations.

Output Shape Formula (Approximate):

For a given dimension (e.g., height or width), the output dimension is calculated as:

output_dim = ceil(input_dim / stride)

Example:

Consider an input feature map of shape [1, 3, 3, 1] and a pooling window of ksize=2 with strides=1, using 'SAME' padding.

import tensorflow as tf

# Input shape: [batch_size, height, width, channels]
input_tensor = tf.constant([[[[1.0], [2.0], [3.0]],
                             [[4.0], [5.0], [6.0]],
                             [[7.0], [8.0], [9.0]]]], dtype=tf.float32)

# Apply max pooling with 'SAME' padding
output_tensor_same = tf.nn.max_pool(
    input_tensor,
    ksize=[1, 2, 2, 1],  # Kernel size (batch, height, width, channels)
    strides=[1, 1, 1, 1],  # Strides (batch, height, width, channels)
    padding='SAME'
)

print(f"Input shape: {input_tensor.shape}")
print(f"Output shape with 'SAME' padding: {output_tensor_same.shape}")

Expected Output Shape: [1, 3, 3, 1]

Notice how the output shape remains largely the same as the input shape, demonstrating the effect of 'SAME' padding in preserving spatial dimensions.

2. 'VALID' Padding

Definition:

With 'VALID' padding, no padding is added to the input tensor. The pooling window only slides over locations where the entire window fits within the bounds of the input. This means that parts of the input near the borders might not be fully covered by the pooling operation.

Key Features:

• Output Reduction: The output tensor will always be smaller than the input tensor in terms of spatial dimensions.
• No Artificial Padding: 'VALID' padding strictly adheres to the original input dimensions, avoiding any artificially introduced values.
• Use Case: This padding strategy is ideal when you explicitly want to reduce the spatial dimensions of your feature maps, leading to a more compact representation and potentially faster computation. It's useful in scenarios where exact spatial preservation isn't critical.

Output Shape Formula:

For a given dimension (e.g., height or width), the output dimension is calculated as:

output_dim = floor((input_dim - filter_dim) / stride) + 1

Where:

• input_dim: The dimension of the input tensor.
• filter_dim: The size of the pooling window in that dimension.
• stride: The stride in that dimension.

Example:

Using the same input tensor and pooling window as the 'SAME' example, but with 'VALID' padding:

# Apply max pooling with 'VALID' padding
output_tensor_valid = tf.nn.max_pool(
    input_tensor,
    ksize=[1, 2, 2, 1],  # Kernel size
    strides=[1, 1, 1, 1],  # Strides
    padding='VALID'
)

print(f"Output shape with 'VALID' padding: {output_tensor_valid.shape}")

Expected Output Shape: [1, 2, 2, 1]

As you can see, the output has been shrunk because the pooling window only considered valid positions where it could fully overlap with the input data.

Comparison Table

Conclusion

The choice between 'SAME' and 'VALID' padding in tf.nn.max_pool is a fundamental design decision that directly impacts how your CNN processes spatial information.

• Choose 'SAME' padding when you need to preserve the spatial dimensions of your feature maps, maintain spatial alignment across layers, or ensure that border information is not lost during pooling. This is common in deeper networks or when specific output sizes are required.

• Choose 'VALID' padding when you aim to reduce spatial dimensions, create more compact representations, or when the exact preservation of border features is not a primary concern. This can lead to computational efficiency and potentially faster training or inference.

A thorough understanding of these padding mechanisms is crucial for effectively designing and optimizing CNN architectures in TensorFlow.