Section 1.1: Convolutional Neural Networks (CNNs)

CNNs are the most recognized architecture in image processing, warranting a brief overview before delving into other models. They learn features through a sequence of convolutions, pooling, and activation layers. The convolutional layers utilize a “kernel,” which mimics a sliding window that traverses the matrix, applying simple operations like multiplication to distill each segment into a feature. Numerous parameters can be adjusted in these kernels, such as size and stride (the distance the window shifts).

The pooling layer aims to minimize the spatial dimensions of the image, employing various techniques like max pooling or min pooling. Ultimately, a CNN processes the image through multiple convolutional, pooling, and activation layers, resulting in a compact matrix of extracted features.

While this overview may seem basic, most readers are likely familiar with CNNs.

Section 1.2: Vision Transformers (ViTs)

Photo by Daniil Kuželev on Unsplash

Vision Transformers have replaced traditional convolutional and pooling layers with self-attention mechanisms, utilizing Multi-head Self Attention layers. These layers operate based on an attention mechanism that incorporates queries, keys, and values to focus on information from different representations across various positions.

A standard transformer block for image processing begins with a Feed Forward Network, followed by a Multi-head Self Attention layer. Notably, the feed-forward network employs a Gaussian Error Linear Unit activation function to regularize the model by randomly nullifying certain activations.

Self-attention creates connections between pieces of information bidirectionally, meaning the sequence of inputs is irrelevant. Operations primarily consist of dot products using keys, queries, and values, akin to searching through a dictionary.

ViTs also leverage positional embeddings, segmenting images into small patches (typically 16 x 16) and calculating the “distances” between them to determine their similarity. This approach addresses a common limitation of CNNs, where kernels capture localized features rather than global ones.

Despite CNNs avoiding hand-crafted feature extraction, their architecture is specifically tailored for images, often leading to higher computational demands. As we anticipate the next generation of scalable vision models, a pertinent question arises: is the domain-specific design of CNNs essential, or can more generalized and efficient architectures be utilized to achieve top-tier outcomes?

Another noteworthy aspect is that ViTs generally consume less memory than CNNs. While self-attention layers are compute-intensive, CNNs often require numerous convolutional layers, ultimately leading to greater memory usage. However, ViTs necessitate extensive datasets for pre-training to match the performance levels of CNNs. Their potential shines when trained on larger datasets like JFT-300M, surpassing the more conventional ImageNet.

For instance, ViTs trained on ImageNet have achieved a top-1 accuracy score of 77.9%. Although commendable for an initial attempt, this still lags behind the best CNN, which can reach 85.8% with no additional data.

Section 1.3: The MLP Mixer

The MLP Mixer presents a unique architecture. Contrary to expectations for an upgrade over self-attention, it opts for traditional multi-layer perceptrons (MLPs). Despite this simplicity, it achieves state-of-the-art performance. So, how does the MLP Mixer extract features from images?

Similar to ViTs, the MLP Mixer analyzes images in patches but also considers the channels across those patches. It employs two types of layers: the first, a channel-mixing layer, facilitates interaction among channels within independent patches. The second, a patch-mixing layer, allows communication between different patches.

Essentially, the MLP Mixer seeks the optimal method to blend channels and patches, starting with encoding those patches through a "mixing table." This process can be likened to solving a jigsaw puzzle, where pieces are adjusted until they form a coherent whole.

The essence of contemporary image processing networks lies in mixing features either at specific locations or between various locations. While CNNs achieve this through convolutions and pooling, Vision Transformers do so via self-attention. The MLP Mixer, however, adopts a more distinct approach, relying solely on MLPs. The primary advantage of this simplicity is enhanced computational speed.

Although the MLP Mixer does not outperform previous architectures, it is up to three times faster, which is logical given its straightforward design.

Final Thoughts

The evolution of computer vision is a vast topic. This article aimed to provide a snapshot of recent breakthroughs and reflect on how these networks decompose images into features—essentially the fundamental goal of computer vision. If any details were overlooked or inaccuracies noted, I welcome feedback in the comments as I am eager to learn!

For those interested in receiving regular reviews of the latest AI and machine learning papers, subscribe with your email here:

This lecture from MIT explores the role of convolutions in image processing, providing foundational insights into CNNs.

This video discusses the evolution of Convolutional Neural Networks since the 1990s, highlighting key advancements and challenges.