There's some interesting work in NeurIPS this year on fused kernels for MoE too:...

		emaadm 87 days ago \| parent \| context \| favorite \| on: We reverse-engineered Flash Attention 4 There's some interesting work in NeurIPS this year on fused kernels for MoE too: https://flash-moe.github.io/