SIMD-accelerated computer vision on a $2 microcontroller

I wrote an accelerated FAST feature detector for the ESP32-S3 that leverages its 128-bit SIMD instructions. It achieves roughly double the performance of the reference implementation, and takes around ~6ms to process a QVGA (320x240) frame. You can find the source code here. For its price, the ESP32-S3 is a powerhouse of a microcontroller. Within its unassuming plastic package lies a dual-core CPU running at a maximum of 240MHz with a slew of peripherals, including WiFi and Bluetooth Low Energy radios. While digging through its technical reference manual I discovered that the chip supports a limited set of SIMD instructions. For silicon that’s cheaper than the average coffee, that’s pretty cool. Let’s see what exciting things we can do with it! In the process of writing a SIMD-accelerated FAST corner detector, I learned the basics of assembly on the ESP32-S3, worked around its eccentric limitations, and ended up writing my own basic register allocator, basm. After some more web browsing, I realized that these instructions weren’t exactly secret, but rather relatively unknown. I wanted to experiment with them myself and write more advanced code than what I could find online. As I’ve been really interested in computer vision lately, I decided on writing a SIMD-accelerated implementation of the FAST feature detector for the ESP32-S3. I started by looking for example code that uses SIMD instructions. At the time of writing, I could only find three sources: Larry Bank’s blog, where he explores the feature and provides a minimal example Espressif’s esp-dsp library, used for DSP-esque features (convolutions, FFT, etc.) Espressif’s esp-dl library, used for on-chip AI acceleration. After perusing through them, I was relieved to find that assembly on the ESP32-S3 isn’t that bad. I settled on this general structure:     .section .rodata

read-only data here

    .section .text     .align 4     .global add     .type   add,@function

a simple function that adds two numbers

int32_t add(int32_t a, int32_t b)

add: entry           sp, 48 # sp (stack pointer) is an alias for a1     #define r_a a2     #define r_b a3

code here

add a2, r_a, r_b # a2 = a + b     retw.n # return, with a2 containing the return value. From there, I started writing a SIMD-accelerated version of the FAST corner pre-test. Essentially, it samples one pixel in each cardinal direction and checks if there are at least three “extreme” pixels. In this case, an extreme pixel is one whose absolute …