# Using a mac mini (2012) for LLM inference with KoboldCpp 

I have an old by now 2012 Intel Mac Mini. Its one of these amazing machines that still work really well if you dont try to put any 2020-era software on it.

So I have been making it a music server to access all my music library and have installed all kinds of dockerised services like [PiHole](https://pi-hole.net/) , a way that i can filter ads on all machines in our home network and a [pgvector](https://github.com/pgvector/pgvector) Postgres based vector db because these are always useful to have around. But I thought that it would be neat to also have it run a small(ish) LLMmodel on it using its CPU (because of course it doesnt have any semblance of a GPU in it).

![](https://cdn.hashnode.com/uploads/covers/6a89f690783e91c80e79ceab/e2661c81-a2b9-4fd4-84b4-9e6a699b5b81.png align="center")

[KoboldCpp](https://github.com/lostruins/koboldcpp) can run language models and serves them through a browser interface and an HTTP API. It builds on [llama.cpp](https://github.com/ggml-org/llama.cpp) and loads models in GGUF format, so you can use it for chat or connect a script to it.

**One of the biggest advantages of KoboldCPP** is its ability to run on these forgotten machines. Popular LLM environments like **LM Studio** or **Ollama** don't support this generation of **Mac Mini** or its final operating system, **macOS 10.15 (Catalina)**. However, KoboldCPP works perfectly as long as you **compile it from source**. After that, all you need are some models from **Hugging Face** [🤗](https://huggingface.co/) to start experimenting.

![](https://cdn.hashnode.com/uploads/covers/6a89f690783e91c80e79ceab/e9860ceb-3ced-4e28-8a98-773510aa8362.jpg align="center")

But there is a problem.The Mini’s Ivy Bridge processor supports AVX, but it doesn't support [AVX2](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions) instructions

So... lets try to build KoboldCPP from source so it works here.

We start by installing Apple’s command-line tools if they are not already available:

```bash
xcode-select --install
```

Wait for the installation to finish, then clone KoboldCpp:

```bash
mkdir -p ~/src
cd ~/src
git clone https://github.com/LostRuins/koboldcpp.git
cd koboldcpp
```

Now the instructions on the koboldcpp repo tell you to just run `make` on this directory... which is fine and dandy if you have a newer machine but not on this dinosaur.

It will fail after 5-6 mins of compiling 🤦🏻‍♂️ with some inscrutable error messages as some header files are missing. Anyway this is how to deal with those if/when they come up.

First edit the first file `gguf.cpp`:

```bash
nano ggml/src/gguf.cpp
```

Near the top, add `<cerrno>` alongside the existing includes. That part of the file should look like this:

```cpp
#include "ggml-impl.h"
#include "gguf.h"
#include <cerrno>
#include <cinttypes>
#include <cstddef>
```

Save with **Ctrl+O**, press **Enter** to confirm the filename, then exit with **Ctrl+X**.

Open the second file `mtmd-helper.cpp` :

```bash
nano tools/mtmd/mtmd-helper.cpp
```

Add the same header after `llama.h`:

```cpp
#include "mtmd-helper-common.h"
#include "llama.h"
#include <cerrno>
#include "vendor/hash/hash.h"
```

Save and exit with **Ctrl+O**, **Enter**, then **Ctrl+X**. If either file already includes `<cerrno>`, leave that include as is.

These edits have now explicitly included the standard C++ header for `errno` and related error constants. Without these additions the build will not complete.

Check the changes before compiling:

```bash
git diff --check
git diff -- ggml/src/gguf.cpp tools/mtmd/mtmd-helper.cpp
```

The first command should produce no output. The second should show the two added includes.

Now clear any previous build products if they failed and compile:

```bash
make clean
make -j"$(sysctl -n hw.physicalcpu)" LLAMA_PORTABLE=1 LLAMA_NOAVX2=1
```

The `-j` argument runs one compilation job per physical CPU core. `LLAMA_PORTABLE=1` selects explicit instruction-set flags, and `LLAMA_NOAVX2=1` selects the <mark class="bg-yellow-200 dark:bg-yellow-500/30">AVX-compatible configuration without enabling AVX2 or FMA</mark> as these technologies came after and Mac Mini 2012 predates them . Those options are defined in KoboldCpp’s [Makefile](https://github.com/LostRuins/koboldcpp/blob/concedo/Makefile).

Once the build finishes (after 10 minutes!!), you can finally load a GGUF model. This assumes Python 3 is installed and the model file has already been downloaded:

```bash
cd ~/src/koboldcpp # if your code lives there

# Install the modern UI package KoboldCpp uses
pip3 install customtkinter

# Run
python3 koboldcpp.py \
  --model "/absolute/path/to/your-model.gguf" \
  --noavx2 \
  --gpulayers 0 \
  --threads "$(sysctl -n hw.physicalcpu)" \
  --contextsize 4096 \
  --port 5001 
```

**Replace the placeholder model path** with the actual location of your **GGUF file**. The provided launch command automatically selects the **non-AVX2 backend** to keep the model running on the **CPU**. The context size specified here is just a baseline setting to help you get started; **once you are comfortable** with KoboldCPP, you can begin **optimising these flags and options** to suit your hardware available [here](https://github.com/LostRuins/koboldcpp/wiki/Home/210c1883de3bc1945ad7a00bd88b157d6a853963) and [here](https://github.com/LostRuins/koboldcpp/blob/concedo/koboldcpp.py) if you want to go deeper on the settings.

After the model loads, open [localhost:5001](http://localhost:5001).

And tada 🎉 you are faced with this screen

![](https://cdn.hashnode.com/uploads/covers/6a89f690783e91c80e79ceab/6df7c78a-276e-4c4f-9498-f593befe6fb1.png align="center")

On this 2012 miniMac , I used `Qwen2.5-Coder-7B-Instruct-Q5_K_M`

*   In general the quality (from worst to best) and filesize (from smallest to biggest) follows this order: \[Q2K, Q3\_K\_S, Q3\_K\_M, Q3\_K\_L, Q4\_0, Q4\_K\_S, Q4\_1, Q4\_K\_M, Q5\_0, Q5\_1, Q5\_K\_S, **Q5\_K\_M**, Q6\_K, Q8\_0, F16\]
    

I've asked it to create a Python function to check if a string is a palindrome.

```python
def is_palindrome(s): 
    # Convert string to lowercase to ensure case insensitivity 
    s =  s.lower()

    # Remove all non-alphanumeric characters from the string
    s = ''.join(char for char in s if char.isalnum())

    # Initialize pointers for the start and end of the string
    left, right = 0, len(s) - 1

    # Loop until the two pointers meet in the middle
    while left < right:
        # If two pointers are not the same, it's not a palindrome
        if s[left] != s[right]:
            return False
        # Move the pointers towards the center
        left += 1
        right -= 1
    
    # If all characters matched correctly, it's a palindrome
    return True
```

It works really well. These are the stats i got for this request

| Metric | Performance | Description |
| --- | --- | --- |
| **Generation Speed** | **3.30 t/s** | The speed at which the model actively types out new text. |
| **Prompt Processing** | **4.16 t/s** | How fast the model reads and understands your initial prompt. |
| **Time to First Token (TTFT)** | **9.38s** | The initial delay before the model starts typing its response. |

For interactive use its slow-ish but still usable. But the model is relatively good giving accurate results (up to a point) and its running on a machine that was out in 2012. In that sense getting this even to work is great.

But we dont want it to be interactive as its main mode of use. Something like work to be left to be done during nighttime seems more suitable. This needs to work as an OpenAI compatible endpoint to be used by scripts or orchestrators etc . In order to do this we need to create a background service or daemon on the mac that is running automagically even if the system reboots

To set up a macOS Launch Daemon for koboldcpp, you need to format the commands and flags into a valid XML property list (.plist) file where each flag and value must be its own element. Here is the complete guide to creating, formatting, and activating your koboldcpp background service.

### a. Create and Open the Plist File

Open your terminal and run the following command to create the file in the system-wide daemon directory:

`sudo nano /Library/LaunchDaemons/com.koboldcpp.ai.plist`

### b. Paste the Correct XML Structure

Copy and paste the exact block below into the file, replacing the paths with your own.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.koboldcpp.ai</string>

    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/python3</string>
        <string>/path/to/your/koboldcpp.py</string>
        <string>--model</string>
        <string>/path/to/your/gguf/qwen2.5-coder-7b-instruct-q5_k_m.gguf</string>
        <string>--usecpu</string>
        <string>--gpulayers</string>
        <string>0</string>
        <string>--host</string>
        <string>0.0.0.0</string>
        <string>--port</string>
        <string>5001</string>
        <string>--contextsize</string>
        <string>4096</string>
        <string>--threads</string>
        <string>3</string>
        <string>--blasthreads</string>
        <string>4</string>
        <string>--batchsize</string>
        <string>512</string>
        <string>--quantkv</string>
        <string>f16</string>
        <string>--noflashattention</string>
        <string>--usemlock</string>
        <string>--gendefaults</string>
        <string>{"temperature":0.15,"top_p":0.95,"top_k":20,"min_p":0.05,"rep_pen":1.0,"repetition_penalty":1.0,"repeat_penalty":1.0,"rep_pen_range":0,"presence_penalty":0}</string>
        <string>--gendefaultsoverwrite</string>
        <string>--highpriority</string>
        <string>--skiplauncher</string>
    </array>

    <key>WorkingDirectory</key>
    <string>/Volumes/Macintosh HDD/ai/koboldcpp/</string>

    <key>RunAtLoad</key>
    <true/>

    <key>KeepAlive</key>
    <true/>

    <key>StandardOutPath</key>
    <string>/path/ to / your /kobold_output.log</string>

    <key>StandardErrorPath</key>
    <string>/path/ to/ your / kobold_output.log </string>
</dict>
</plist>
```

Press Ctrl + O then Enter to save, and Ctrl + X to exit nano.

In order to come to this set of flags I benchmarked each flag with koboldcpp's own `--benchmark`, changing one variable at a time, with the daemon stopped and nothing else running. This way I found ways to improve four settings I had recommended in the first version of this post. Taking them in order of how wrong I was:

![](https://cdn.hashnode.com/uploads/covers/6a89f690783e91c80e79ceab/85c73442-8a06-463e-b0de-aeeac6445b9f.png align="center")

`--quantkv q4_0` **was costing me 54% of my generation speed.** I had described it as saving "a massive amount of system RAM with an almost unnoticeable drop in response quality." The quality part may be true. The speed part is not: compressing the KV cache to 4-bit means the model has to *decompress* it on every single token, and this CPU has no AVX2 and no FMA to spare for that. It was spending 162 ms per token to save 18 MB of RAM — an effective 113 MB/s on a machine that streams weights at 22 GB/s. On a box with 16 GB, saving half a gigabyte of KV cache is not worth paying for in arithmetic. `--quantkv f16` is both faster and higher quality.

`--batchsize 256` **was costing 68% of my prompt processing speed.** I called 256 "a solid middle ground for CPU processing." It isn't; 512 is koboldcpp's default for a reason, and bigger batches amortise the matrix setup better. This one only affects how fast it reads your prompt, not how fast it writes — but reading a 1600-token prompt is five minutes on this machine, so it matters.

`--smartcontext` **is deprecated.** I called it "a massive performance booster." Its own help text in the current source reads *"Outdated. Not recommended."* ContextShift, which is on by default, does the same job without reserving half your context window for the privilege.

### Other Flags of Interest

**Logging and always be running flags:**

*   `RunAtLoad`: Forces KoboldCpp to start automatically every time your Mac boots up.
    
*   `KeepAlive`: Automatically restarts the process if it crashes or gets killed.
    
*   `Logging`: Output and errors will save directly to files , allowing you to troubleshoot easily.
    

**Hardware & Processing :**

*   `--usecpu` & `--gpulayers 0`: Forces the model to run entirely on your CPU rather than looking for a GPU. This is ideal if you are running this on an older Mac or want to leave your GPU free for other tasks.
    
*   `--threads 3`: Dedicates 3 CPU threads to the AI. You can tweak this based on your Mac’s core count (leaving at least one free for your operating system).
    
*   `--highpriority`: Tells macOS to treat this background process as a high-priority task, reducing the chance of the system throttling its performance.
    

And then the one which turned out to be the biggest.

### d. Flash attention is a GPU optimisation, and this is not a GPU

koboldcpp turns flash attention on by default now. Disabling it made generation **30% faster** at 2048 tokens of context, and the advantage grows as the window fills.

![](https://cdn.hashnode.com/uploads/covers/6a89f690783e91c80e79ceab/4c06d4f6-379c-4b12-8fd8-9145d4b1ab96.png align="center")

This chart is the biggest takeaway and rather than the findings on Flash Attention.

Generation speed on a CPU obeys a straight line: a fixed cost to read the entire model once per token, plus a cost proportional to **how much context is already in the window**. Fitted to my measurements:

$$t_{\text{token}} = 0.247\text{ s} + 94\text{ ms} \times \left(\frac{\text{context}}{1000}\right)$$

The first term is the whole 5.44 GB model streaming through memory at 22 GB/s — which is 86% of what dual-channel DDR3-1600 can theoretically deliver, so that part genuinely is at the hardware ceiling and no flag will move it.

The second term is attention, which represents the primary computational bottleneck in this process. At 512 tokens of context it's 16% of each token. At 2048 it's 44%. At the **10,000+ token context I originally recommended in this post, it is over 80%** — the model spends most of its time re-reading what it already said, and the DDR3 bandwidth I'd been thinking about barely matters.

Which explains the flash attention result. Flash attention trades extra arithmetic for drastically less memory traffic. On a GPU that is a brilliant deal, because bandwidth is the scarce resource and arithmetic is nearly free. This Mac mini has the opposite problem:

| Resource | Mac mini 2012 | A modern GPU |
| --- | --- | --- |
| Arithmetic | ~67 GFLOPS, no FMA, no AVX2 | tens of TFLOPS |
| Bandwidth | 22 GB/s, already at 86% of peak | hundreds of GB/s |
| Scarce resource | **arithmetic** | **bandwidth** |

FlashAttention expends arithmetic cycles—the most constrained resource on this hardware—to economise on memory bandwidth, which is already being utilised optimally. While the default implementation is optimised for the GPU architectures used by most . Not in my case tho 😭 on this machine.

### e. Why the context window is now 4096

This is the flag I'd have told you to leave alone, and it turns out to be the biggest lever on the page. Going from 12000 to 4096 is roughly a **2.2× speedup**, for free, because every token you allow into the window gets paid for on every subsequent token.

Ten thousand tokens sounded generous when I wrote it. What it actually bought me was a model that generates at half a token per second once a conversation gets going. I set it to 4096 and adjusted what I send instead — for a code assistant answering discrete questions, that is plenty.

### f. For quality of output repetition penalty matters more

> Everything above is throughput. None of it makes the code *correct*. The setting that does is repetition penalty, and the default will quietly wreck your output.

KoboldAI Lite's web UI defaults `rep_pen` to 1.1 over a 320-token window. **Code is supposed to repeat.** Indentation, closing braces, `self.`, `return`, the same variable name eleven times in a loop — a repetition penalty actively pushes the model away from the correct next token. You see it as drifting variable names and dropped brackets that get worse the longer the file gets.

That's what the `--gendefaults` block in the plist is doing:

```json
{"temperature":0.15,"top_p":0.95,"top_k":20,"min_p":0.05,
 "rep_pen":1.0,"repetition_penalty":1.0,"repeat_penalty":1.0,
 "rep_pen_range":0,"presence_penalty":0}
```

Low temperature for deterministic code, and repetition penalty pinned firmly off. Note that **all three spellings of repetition penalty are listed**, and that is not redundancy. koboldcpp normalises the three aliases by taking their *maximum*:

```python
genparams["rep_pen"] = max(repeat_penalty, repetition_penalty, rep_pen)
```

### g. Set File Permissions

macOS is incredibly strict about Launch Daemon security. The file must be owned by root and cannot be writable by anyone else. Run these commands to fix permissions:

`sudo chown root:wheel /Library/LaunchDaemons/com.koboldcpp.ai.plist`

`sudo chmod 644 /Library/LaunchDaemons/com.koboldcpp.ai.plist`

### h. Launch and Manage the Daemon

Load and start your new background service using launchctl:

*   to start the daemon:
    

`sudo launchctl load -w /Library/LaunchDaemons/com.koboldcpp.ai.plist`

*   to stop the daemon (if you need to change settings):
    

`sudo launchctl unload /Library/LaunchDaemons/com.koboldcpp.ai.plist`

*   to check the live logs:
    

`tail -f /var/log/koboldcpp.log`

So now we have the endpoint at <mini\_mac\_internal\_IP>:5001 ready for business.

### j. How about using llama.cpp instead?

Compiling `llama.cpp` on a 2012 Intel Mac Mini (Ivy Bridge) has its own traps. the default build scripts will fail due to the hardware and software limitations specific to this generation as discussed before: (No AVX2 , No Metal compute). Allowing the compiler to auto-detect the CPU (`-march=native`) enables AVX2 by default, resulting in an immediate `illegal hardware instruction` crash when launching models. So these are the steps to compile llama.cpp successfully:

1.  Ensure you are in the root folder of the project.
    

```bash
cd "/your/path/to/llama.cpp"

```

2.  **Apply the C++17 workaround:** Hot-patch. These two commands scan all C and C++ files in the repository and replace the missing `std::` cache line variables with `64` (the standard cache line byte size for Intel processors). This bypasses the compiler error.
    

```bash
find . -type f -name "*.[ch]*" -exec sed -i '' 's/std::hardware_destructive_interference_size/64/g' {} +
find . -type f -name "*.[ch]*" -exec sed -i '' 's/std::hardware_constructive_interference_size/64/g' {} +

```

3.  **Create a clean build environment:** Directory management. Wipe any previous failed compilation attempts and create a fresh build directory.
    

```bash
rm -rf build
mkdir build
cd build

```

4.  **Configure CMake for legacy hardware:** Critical Step. Generate the build files with AVX2, FMA, and Metal explicitly turned off.
    

```bash
cmake .. -DGGML_AVX2=OFF -DGGML_FMA=OFF -DGGML_METAL=OFF

```

5.  **Compile the binaries:** Takes a few minutes. Build the project utilizing all 4 physical cores of the processor to speed up compilation.
    

```bash
cmake --build . --config Release -j 4

```

6.  **Install globally:** System-wide access. Install the finished, customized binaries into your Mac's global PATH so they overwrite any generic versions and can be called from any folder.
    

```bash
sudo cmake --install . --prefix /usr/local

```

So now llama.cpp is also available if needed 👍. Once installed, you can verify the compilation succeeded by running the CLI from your home directory:

```bash
cd ~
llama-cli -m "/path/to/yot/gguf/qwen2.5-1.5b-instruct-q4_k_m.gguf" -p "Hello! Are you working right now?" -n 50 --threads 3

```

If the compiler flags were applied correctly, the model will load into RAM and stream a text response. If AVX2 was accidentally left enabled, it will instantly crash with an `illegal hardware instruction` or `SIGILL` error.

### j. Setting up a Vector database for RAG

One last thing before putting this little box of wonders in a corner headless without a screen as a server:

> this mac mini 2012 can also be hosting a vector database for this and any other LLM needing this service

So lets get [pgvector](https://github.com/pgvector/pgvector) up and running as a dockerised service:

create somewhere appropriate a file called `docker-compose.yml` and in there add this (use your preferred username and password of course)

```yaml
services:
  postgres-vector:
    image: pgvector/pgvector:pg16
    container_name: postgres_rag
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mysecretpassword
      POSTGRES_DB: rag_database
    volumes:
      - postgres_vector_data:/var/lib/postgresql/data
    restart: always
```

and run

`docker-compose up -d`

Because PostgreSQL requires vector tracking extensions to be manually activated inside target databases, you need to hop hopped into the active container shell using psql to trigger the environment:

`docker exec -it postgres_rag psql -U myuser -d rag_database`

and instantiate the extension globally in psql :

`CREATE EXTENSION vector;`

`/q` to get out of the shell

So now ill be able to expose pgvec from anywhere on my local network using this as a connection string

`postgresql://myuser:mysecretpassword@<HOST_IP_ADDRESS>:5432/rag_database`

or

`postgresql://myuser:mysecretpassword@postgres_rag:5432/rag_database`

from the macmini

**The next step** is to orchestrate all of these moving parts using [**Prime-Agent**](https://github.com/PrimeIntellect-ai/prime-agent). This will allow any model (including this one and one running on my other [local LLM server](https://mamonu.hashnode.dev/running-a-a-27b-model-in-3-5-gb-vram)) to to [perform long-running work](https://www.primeintellect.ai/blog/prime-agent) using `/loop` and `/goal` but that's a subject for another blogpost!

**🤖 Meta Note:** This blog post was peer-reviewed by [gpt-oss-20b](https://openai.com/index/introducing-gpt-oss/) running on my [Optiplex](https://mamonu.hashnode.dev/turning-a-dell-optiplex-7060-sff-into-a-cheap-local-ai-inference-box) because who better to review a post about self hosted LLMs than a self hosted LLM itself?