Run Your First Local LLM with Ollama, Step by Step

Running a language model on your own machine takes about fifteen minutes and requires no account, no API key and no cloud. This guide uses Ollama because it is the shortest path from nothing to a working local model on Windows, macOS or Linux.

The first four steps get you talking to a model. Everything after that is what makes it usable a week later, which is the part most guides skip.

What you need

• Any reasonably modern computer. 8 GB of RAM runs small models; 16 GB runs 7 to 8B models comfortably.
• A GPU helps a lot but is not required. Ollama falls back to the CPU, more slowly.
• A few gigabytes of disk per model, and they accumulate faster than you expect.

Step 1: Install Ollama

Download the installer from ollama.com and run it. On Linux it is one command:

curl -fsSL https://ollama.com/install.sh | sh

Piping a script from the internet into a shell is worth a moment’s thought. If you would rather read it first, fetch it to a file, look at it, then run it.

Step 2: Pull and run a model

Open a terminal:

ollama run llama3.2

The first run downloads the model, a couple of gigabytes, then drops you into a chat prompt. Type a question. /bye exits, and /? lists the other commands available inside the chat.

Nothing you type leaves the machine. That is the whole point of the exercise.

Step 3: Pick the right size for your hardware

Rules of thumb until you check how much VRAM you need for every model size:

• 3B models: run on nearly anything, good for quick tasks
• 7 to 8B models: the sweet spot for 16 GB machines
• Larger models: need serious VRAM or a large pool of unified memory

Size is only half of it. The other half is how fast the model prints, and a 70B model that fits your machine at 2 tokens per second is a completely different experience from an 8B that runs at 30. The tokens per second simulator streams text at whatever rate you give it, so you can feel the difference before buying anything for it.

Ollama’s default tag for a model is a 4-bit quantization, which is the right default and worth understanding rather than accepting blindly. You can ask for a specific one:

ollama run llama3.1:8b-instruct-q5_K_M

What those suffixes mean, and why a file labelled 4-bit is never quite 4 bits, is its own subject.

Step 4: Check it is actually using your GPU

This is the step that explains most “why is it so slow” questions. With a model loaded, in a second terminal:

ollama ps

That lists what is currently in memory and reports whether it is running on the GPU or the CPU. A model that spilled to the CPU because it did not fit will still answer you, just far more slowly, and nothing about the chat prompt tells you that happened.

If it is on the CPU and you expected otherwise, the model is too large for your video memory once its context is included. Drop a size class, drop to a smaller quantization, or shorten the context. The model fit calculator will tell you which of those buys you the most.

The commands worth knowing

Six of them cover almost everything:

ollama list          # what you have downloaded
ollama ps            # what is loaded right now, and on what
ollama pull <model>  # download without starting a chat
ollama rm <model>    # delete one, and reclaim the disk
ollama run <model>   # chat
ollama serve         # run the server in the foreground

ollama list is the one to run every few weeks. Models are gigabytes each and ollama rm is the only thing that ever removes them.

Where the models live

By default they go in ~/.ollama/models on macOS and Linux, and in .ollama\models inside your user folder on Windows. If that is on a small system drive, point Ollama somewhere else with the OLLAMA_MODELS environment variable before pulling anything large.

This is worth setting up early. Moving a model collection later is easy and remembering that you needed to is not, and a full system drive announces itself at an inconvenient moment.

There is an API, and it is already running

Ollama runs a local server on port 11434 whether or not you are using the chat prompt, which is what makes it useful as a component rather than a toy:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

It also exposes an OpenAI-compatible endpoint, so a lot of software written against that API can be pointed at your machine by changing a base URL. That is the mechanism behind most “put a web interface in front of it” guides, including the Open WebUI one queued for the Self-Hosted Software pillar.

Two environment variables matter once you go this far. OLLAMA_HOST controls what the server binds to, and the default of localhost is a deliberate one: binding it to your whole network exposes an unauthenticated API to everything on that network. OLLAMA_KEEP_ALIVE controls how long a model stays in memory after its last request, which is the trade between a fast second question and not holding gigabytes of memory all day.

Where to go next

• A web interface: point one at the API above, rather than living in a terminal. The Open WebUI walkthrough is queued.
• Understand the speed you are getting: what tokens per second feels like, and what it costs to run the machine that produces it, in homelab electricity cost.
• Decide whether this should be your machine at all: the honest arithmetic is in cloud GPU vs owning hardware, and for a lot of usage patterns it does not favour buying.
• Measured results for specific hardware go in the benchmark database as runs are published.

FAQ

Is my data private with a local model?

Yes. Inference happens entirely on your machine, and nothing you type is sent anywhere. That is a core reason to run models locally at all. The caveat is the one you introduce yourself: if you bind the API to your network or put a web interface in front of it, whatever can reach that interface can use the model.

Does Ollama cost anything?

No. Ollama is free and open source, and the openly licensed models it runs are free to download. The costs that do exist are disk, which is a few gigabytes per model, and electricity, which is the machine rather than the software.

Why is my model so slow?

Almost always because it did not fit in video memory and is running on the CPU, which ollama ps will tell you in one line. After that, the usual causes are a model larger than the machine wants, a context long enough that its cache has pushed the total over your memory, or genuinely modest hardware. The first two are fixable and the model fit calculator says which lever to pull.

Can I run several models at once?

Yes, memory permitting, and that is the constraint rather than any limit in Ollama. Each loaded model holds its weights plus its cache, so two 8B models want roughly twice the memory of one. ollama ps shows what is resident, and OLLAMA_KEEP_ALIVE decides how long an idle one stays there.

How do I get rid of a model I no longer want?

ollama rm <model>, which is the only thing that reclaims the disk. Run ollama list first to see the exact name and its size, because pulling variants of the same model is easy and noticing that you did is not.

Does Ollama update models automatically?

No. ollama pull <model> fetches the current version of a tag, and running a model you already have uses the copy on disk. That is usually what you want, since a model changing underneath you would make any comparison meaningless, but it does mean staying current is a thing you do rather than a thing that happens.

Sources

• Installation, commands, environment variables and the API surface described here are Ollama’s own documented behaviour. Its repository and documentation are the reference, and both move quickly enough to be worth checking against the version you install.
• Memory sizing advice on this page is the arithmetic worked through in how much VRAM you need for every model size, computed by the same code that runs the model fit calculator.
• No speed figure on this page was measured on this machine. Measured speeds for specific hardware live in the benchmark database.