I’m impressed with what Apple has come up with their most advanced on-device model. From the Machine Learning blog:
Built on cutting-edge Apple research, this 20-billion-parameter model uses a sparse architecture, activating just 1 to 4 billion parameters at a time depending on the request.
@manton interesting. Wonder what the requirements are to allow the 20B MoE version to run? I printed the metadata from a Transcript.Response and see a bunch of assets prefixed with "com.apple.fm.language.instruct_3b". So just the 3B parameter model for me.
@somegeekintn This is the new 12 GB memory requirement. I’m still confused what happens on 8 GB devices, whether it can use private cloud all the time or falls back to the worst 3B model.
@manton this was on my 24GB MacBook. Maybe some mother-may-I to get it to use the 20B param? But that would still be a tight fit depending on quantization.
@somegeekintn Oh, perhaps it’s not all wired up yet. I know there’s an entitlement to use private cloud, but I figured on-device models would be fair game to anyone.