Researchers Introduce HyenaND, a Subquadratic Operator for Native Multi-Dimensional Data
A new paper introduces HyenaND, a subquadratic, global, input-dependent operator that processes multi-dimensional data natively. Across genomics, vision, medical imaging, and PDE modeling, it matches or beats attention baselines.
Intelligence analysis by Llama
Subquadratic attention alternatives have struggled with multi-dimensional data, forcing awkward 1D rasterization. HyenaND uses input-dependent multi-dimensional convolutions to preserve native geometry, with a fused CUDA implementation called nSubQ that turns O(L log L) scaling into real wall-clock speedups.
Imagine reading a picture row by row, top to bottom. You'd lose the shape of the dog because you only see strips. Most fast AI tools do that. HyenaND looks at the whole picture, top, side, and depth, all at once, while still being cheaper to run than the old slow version.
Analysis
The Geometry Problem Behind Subquadratic Models
Most attempts to replace quadratic attention with faster operators stumble on a fundamental mismatch: the data they are meant to process is rarely one-dimensional. The paper notes that standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing images, volumes, and PDE grids into an ad hoc 1D scan order that violates their spatial structure. That scan-order hack has long been the dirty secret of subquadratic architectures, and it caps their accuracy on tasks where two- and three-dimensional structure carries the signal.
HyenaND Keeps the Native Shape
HyenaND's contribution is to drop the rasterization step entirely. The operator acts directly on the native geometry of multi-dimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional kernels, according to the paper. Because the kernels are input-dependent, the operator can adapt to each sample, recovering one of the key inductive biases that has made attention so effective. The authors also ship nSubQ, a CUDA implementation that fuses the FFT-convolution path, so the theoretical O(L log L) scaling translates into measured wall-clock speedups rather than a paper-only win.
Beating Attention Where It Should Be Hardest
The empirical case is built on the domains where multi-dimensional structure matters most. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, the paper reports. Hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids, suggesting the operator is complementary rather than merely competitive. With eleven co-authors spanning institutions in the Netherlands and the United States, the work is positioned as a general-purpose primitive for scientific and medical ML pipelines where attention's quadratic cost has historically been a real bottleneck.
Key points
- HyenaND is a subquadratic, global, input-dependent operator that processes multi-dimensional data in its native geometry without 1D rasterization.
- The nSubQ CUDA implementation fuses the FFT-convolution path, turning the theoretical O(L log L) scaling into measured wall-clock speedups.
- Pure HyenaND stacks matched strong attention baselines across genomics, computer vision, medical imaging, and PDE modeling, according to the paper.
- Hybrid models that interleave HyenaND and attention layers reportedly outperformed both pure attention and strong recurrence-based hybrids.
- The work targets scientific and medical domains where quadratic attention's compute cost is a real bottleneck, rather than language modeling directly.
If HyenaND holds up in independent benchmarks, the nSubQ implementation could become a default building block for long-context scientific models, cutting training and inference costs in genomics, medical imaging, and physics simulation. Hybrid stacks that mix HyenaND with attention may offer a practical near-term path for labs that cannot afford full quadratic attention on large multi-dimensional inputs.
Subquadratic alternatives have a track record of underperforming attention once workloads scale beyond what the original paper measures, and FFT-based convolutions can be memory-hungry on very large volumes. Adoption also depends on whether the nSubQ kernels are released in a form that integrates with mainstream frameworks; without that, the wall-clock advantage may stay confined to the authors' own pipelines.
