I think in the next couple years OS-shipped #LocalAI will replace the use of heavy cloud based #AI.
-
I think in the next couple years OS-shipped #LocalAI will replace the use of heavy cloud based #AI. Microsoft, Google, and soon Apple will be shipping devices with local LLMs and it'll be cheaper for applications to target those APIs rather than pay OpenAI or the such. This will also mean that we'll get into a sort of "browser wars" of model functionality gated by hardware vendors.
-
Mauve ๐๐replied to Mauve ๐๐ last edited by
I don't think cloud AI will fully go away but I think it'll make less and less sense for consumer facing use cases as the small models become more viable via better training and better hardware acceleration.
-
the harbinger of eternal septreplied to Mauve ๐๐ last edited by
@mauve iโve played around a little with local llamafiles
my current takeaway is local is the future, but only when it doesnโt chew through batteries.
that was the alarming thingโ โdata centers are draining power this hard 24/7โ?
-
Mauve ๐๐replied to the harbinger of eternal sept last edited by
@tychi I tried Qwen2:0.5B and it's almost able to do stuff with very little usage. That's like several hundred orders of magnitude less power consumption whole being ablw to do some small tasks. I think these things could be put to work for specificallt crafted prompts with multi shot for a lot of use cases
-
Mauve ๐๐replied to Mauve ๐๐ last edited by
For example, Chrome is working on shipping web APIs for LLM access. I'm planning to release something similar in @agregore in the next week or two.
prompt-api/chrome-implementation-differences.md at main ยท explainers-by-googlers/prompt-api
A proposal for a web API for prompting browser-provided language models - prompt-api/chrome-implementation-differences.md at main ยท explainers-by-googlers/prompt-api
GitHub (github.com)
-
ฮตฯฮผฮทฯฮนฮบฯฯreplied to Mauve ๐๐ last edited by
-
Mauve ๐๐replied to ฮตฯฮผฮทฯฮนฮบฯฯ last edited by
@hermeticvm @agregore Oh snap. Is their api stable? Have you tried it out?
-
ฮตฯฮผฮทฯฮนฮบฯฯreplied to Mauve ๐๐ last edited by
@mauve all you need is a local ollama instance which is pretty much compatible to ChatGPT.
-
Mauve ๐๐replied to ฮตฯฮผฮทฯฮนฮบฯฯ last edited by
@hermeticvm @agregore Ohhh I see. This is for the built in LLM UI they have. I am working on JavaScript APIs for web apps to have access to.