68k Ollama LLM

[iantm]

I'm making no pretense that I'm going to use an LLM to help me learn to code a Classic Mac App. I have been coding on some level for my whole career, but not for bonafide application development, so as such my knowledge is not particularly deep into CS. But I enjoy it, all the same.

I heard about running a LLM offline, like Ollama. Anyone tried this? Once I give it documentation (e.g. InsideMacintosh) would it really be able to assist?

Keep in mind I want to mostly type and hack through it, but having an LLM to bounce questions off of is really valuable to me. For time being I'll use Claude, but the idea of training a local LLM is intriguing.

Thinking about running the 14B on an M4 Max (36GB RAM).

 

[PL212]

One approach would be retrieval-augmented generation, where a local model queries embeddings from PDFs or texts you provide. There are a lot of projects on GitHub for this kind of thing, although I don't know the one specific one I would try.

There is a plugin for LMStudio (a cross-platform local AI app) that does RAG: https://lmstudio.ai/mindstudio/big-rag . The reason LMStudio (or other tools such as Olama) doesn't do this out of the box is I think the need for a local vector database to store the textual embeddings.

Google's NotebookLM service offers a very nice turnkey solution, albeit one that is not in any way local. But it will directly cite relevant parts of PDFs you upload.

There's a user on Macintosh Garden who has, somewhat idiosyncratically, collected as much System 7-era stuff as they could find and packaged it into an odd tarball, complete with python scripts and sql datastores, to enable this kind of work. I think they met with a mixed reaction on that site to their work.

 

[blturner]

I have dual 16GB 5060 Ti's I use to run local LLMs with ollama. They can do stuff, yes. I've mostly used them for Python web development and had best success with Qwen 3.6 27b and Gemini 4 models.

 

[SitesOnFire]

From my personal experience learning how to use Think C and ANSI C, Claude produced the best code, and had the best understanding. I used it as a 'smarter Google'. I can already program, so I'd ask it questions to help me understand the differences from what I know (Java/TS/php). At the time, ChatGPT would hallucinate lots of answers, esp regarding the Macintosh Toolbox & MacTCP

There's lots of good PDF programming books from the era, it'd be great to train a model on those. I'm less interested in AI writing the code for me, than I am having a quick way to find answers.