- What changed
- Cohere introduced a serving engine for North Mini Code using a decode megakernel that runs the entire forward pass as a single persistent kernel.
- Why you should care
- Megakernels reduce memory bandwidth bottlenecks during small batch autoregressive decoding by eliminating kernel launch wait times.
- Your move
- Test. Evaluate the open source megakernel implementation against current serving stacks to verify performance gains for memory bound decoding workloads.
- What to watch next
- Independent community benchmarks and integration into broader serving frameworks like vLLM.
- Event
- release
- Event date
- Sep 8, 2026
- Relevant to
- General AI readers