When I get a questionable notification summary in the iOS 18.1 beta, I run the original text through OpenAI to compare. It’s usually better. Certainly not a deal-breaker, but the thing about AI is it needs to actually be good or you lose the illusion.

@manton It’s telling that Apple’s model does worse, even with their expert-tuned prompt, than OpenAI’s generic chat input does. World knowledge is surprisingly helpful for summarization

@manton I have an LLM enhancement feature in my in-progress Micropub client called Smart Summary, and it's basically bring-your-own-model (uses Ollama that you have to host somewhere). The prompt is also customizable.

Maybe Apple needs to consider letting users bring their own models (and prompts?), too.

@apike I worry about this a little for Apple. The seamless balance of on-device models and the cloud is a nice strategy, but the small models won't get better for years because of hardware limits.

@samradford To be clear, I like the Apple summaries too. I think a cloud-based solution like OpenAI would be fast enough, though. A bigger problem is making sure it's private enough since you're sharing other people's messages.

@manton @apike Small models have been getting better as training and architecture improves .. there’s a limit, but I don't think we're quite stuck where we are.

@stevex @manton Yeah for sure – although Apple is currently (AFAIK) trying to use one small model for a wide set of tasks, there seems to be especially a lot of headroom left for distilling large models into small task-specific models.