Mixture of experts
also: MoEA model whose feed-forward layers are many experts, of which only a few run per token. Large on disk, cheaper per token.
Reference ↗Related
1 article
A model whose feed-forward layers are many experts, of which only a few run per token. Large on disk, cheaper per token.
Reference ↗