# Trusting Local LLMs with Code: Building little-llms-bench

When it comes to code generation, looks can be deceiving. As developers, we need to know whether the small, self-hosted LLMs running on our local setups can actually produce functional code, or if they're just hallucinating confident-looking syntax.

To answer this for the workflows I have in mind I built [`little-llms-bench`](https://github.com/mamonu/little-llms-bench) , a small, self-contained benchmark designed specifically for grading local LLMs on Python and Bash code generation.

## Under the Hood of `little-llms-bench`

The philosophy behind this benchmark is simplicity and execution-based verification. Instead of relying on another "judge" AI to grade the output, the harness tests the code by actually running it.

*   **Zero Dependencies:** The benchmark operates using only Python 3.10+ and the standard library, requiring absolutely no `pip install` to get started.
    
*   **20 Python Tasks:** Models must reply with exactly one Python code block. The harness extracts the code, appends hidden unit tests, and executes it in an isolated subprocess (`python -I`) with a strict timeout.
    
*   **10 Bash Tasks:** These are output-prediction tasks graded by normalized string matching rather than shell execution, meaning the benchmark runs safely and consistently even on Windows machines.
    
*   **Streaming & Reasoning:** By using the `--stream` flag, developers can watch live tokens and reasoning (such as `<think>` blocks) as the server emits them. This is especially useful for debugging reasoning models, allowing you to see their progress before the final answer arrives. (Is a model looping endlessly? is it moving towards the right solution? etc)
    

### From Sanity Checks to Algorithmic Traps: How Tasks Are Tiered

When evaluating small, self-hosted LLMs for coding, simple pass/fail metrics don't tell the full story. A model might generate immaculate boilerplate code for basic tasks yet completely fall apart when hit with subtle edge cases or strict runtime requirements.

To truly measure reliability, `little-llms-bench` breaks its 30 tasks (20 Python, 10 Bash) into three distinct difficulty tiers: **Easy**, **Medium**, and **Hard**. Here is how these tiers test the limits of small models, complete with real-world examples.

* * *

#### 1\. Easy: The "Can You Follow Instructions?" Phase

The **Easy** tier acts as a baseline filter. It evaluates whether a model understands fundamental programming concepts, syntax, and prompt constraints—such as returning *only* valid code without conversational markdown chatter.

*   **Python Example (**`py01_reverse_words`**):** Write a function that reverses whitespace-separated words in a string, collapses extra spaces, and strips padding.
    

> *Example:* `" hello big world "` -> `"world big hello"`

*   **Bash Example (**`sh04_brace`**):** Predict the exact output of `echo {1..5}`.
    

> *Expected:* `1 2 3 4 5`

**Where Models Fail:** Smaller models often pass the core logic but fail on execution formatting. For instance, on `sh04_brace`, `qwen2.5-1.5b` incorrectly predicted a multi-line output (`1\n2\n3...`) instead of space-delimited text.

* * *

#### 2\. Medium: Hidden Edge Cases & Defensive Coding

The **Medium** tier moves beyond simple logic and introduces state management, string parsing, and subtle requirements like input immutability.

*   **Python Example (**`py09_merge_intervals`**):** Merge overlapping or touching interval pairs. The hidden catch? **The input list must not be mutated.**
    
*   **Python Example (**`py15_parse_config`**):** Parse INI-like text into a nested dictionary. Keys appearing *before* any section header must go into an empty string key `""`.
    
*   **Bash Example (**`sh07_param_expansion`**):** Predict parameter expansion results for paths:
    

```bash
f=/tmp/logs/app.2024.log
echo "${f##*/}"  # app.2024.log
echo "${f%.*}"   # /tmp/logs/app.2024
echo "${f%%.*}"  # /tmp/logs/app

```

**Where Models Fail:** This tier exposes "copy-paste memory" issues. In my testing, `qwen2.5-coder-7b` failed `py09_merge_intervals` because it ran `intervals.sort()` directly on the input list instead of working on a shallow copy. Similarly, multiple models threw a `KeyError: ''` on `py15_parse_config` because they assumed every config file always starts with a `[section]` header.

* * *

#### 3\. Hard: Performance, Meta-Programming & Anti-Cheat

| Task ID | Topic | Crucial Challenge |
| --- | --- | --- |
| `py17_retry_decorator` | Meta-programming | Must build a decorator factory handling retries, preserving metadata via `functools.wraps`, and avoiding execution blocks. |
| `py18_sliding_window_max` | Algorithm Efficiency | Must find maximums in sliding windows in $O(n)$ time. Naive $O(n \\cdot k)$ solutions fail a hidden timing check on a 200,000-element array. |
| `py19_tokenizer` | Parsing & Anti-Cheat | Must evaluate math expressions like `"2 * (3 + 4)"` \*\*without using `eval()`, `exec()`, or `compile()**`. Source code is inspected to enforce this. |
| `sh10_chmod_numeric` | Unix Permissions | Must translate a permissions string like `-rwxr-xr--` into its exact octal equivalent, `754`. |

* * *

**The Meta-Programming Trap (**`py17`**)** Writing a decorator is one thing, but writing a decorator *factory* (a decorator that takes arguments, like `@retry(attempts=3)`) requires a model to successfully nest three layers of functions without losing track of scope. Smaller models frequently get tangled in the `*args` and `**kwargs`, or completely forget to apply `functools.wraps`. When that happens, the wrapped function loses its original name and docstring—which is a nightmare for real-world debugging.

**The Brute-Force Filter (**`py18`**)** This task is a classic algorithmic bottleneck. When tasked with finding the maximum value in a sliding window, a model's first instinct is usually to write a loop that repeatedly calls `max()` on every sub-array. It passes the baseline tests beautifully, but when `little-llms-bench` quietly hands it an array of 200,000 integers, the naive O(n `*` k) approach hits a wall. Only models that understand how to implement a double-ended queue (deque) to achieve true O(n) efficiency will survive the timeout.

**The "No Cheating" Parser (**`py19`**)** When asked to evaluate a math string like `"2 * (3 + 4)"`, almost every model reaches for Python's built-in `eval()`. It is the fastest, easiest path to a correct answer! But since `eval()` is a massive security risk in production environments, the benchmark strictly forbids it. The model is forced to step up and write a recursive descent parser or an abstract syntax tree evaluator entirely from scratch. Watching a 7B parameter model attempt this is the ultimate stress test of its context limits and structural logic.

**The Unix Mindbender (**`sh10`**)** String manipulation is generally easy for LLMs, but translating a symbolic permission string (`-rwxr-xr--`) into a numeric octal requires mapping semantic logic to positional values. The model has to know that `r=4`, `w=2`, and `x=1`, correctly chunk the string into user, group, and other blocks, and sum them up perfectly. If the model hallucinated the meaning of a single dash, it fails.

* * *

#### Why This Progression Matters

By organizing benchmarks into these distinct tiers, you can quickly see where a self-hosted model's reliability boundaries lie:

*   **Easy pass:** Good for basic autocomplete.
    
*   **Medium pass:** Safe for refactoring and standard web/utility scripts.
    
*   **Hard pass:** Capable of writing complex logic, state machines, and high-performance code independently.
    

* * *

#### Model Benchmark Results

To test the harness, I ran three self-hosted local models I have had going here at mamonulabs across the full 30-task suite (12 Easy, 13 Medium, 5 Hard). The raw execution results reveal clear tier boundaries between parameter sizes and model capabilities:

#### Score Breakdown by Tier

| Model ID | Easy (12) | Medium (13) | Hard (5) | Total Score (%) |
| --- | --- | --- | --- | --- |
| `gpt-oss-20b` | **12 / 12** (100.0%) | **12 / 13** (92.3%) | **5 / 5** (100.0%) | **29 / 30 (96.7%)** |
| `qwen2.5-coder-7b-instruct` *(Q5\_K\_M)* | **10 / 12** (83.3%) | **8 / 13** (61.5%) | **4 / 5** (80.0%) | **22 / 30 (73.3%)** |
| `qwen2.5-1.5b-instruct` *(Q4\_K\_M)* | **8 / 12** (66.7%) | **3 / 13** (23.1%) | **0 / 5** (0.0%) | **11 / 30 (36.7%)** |

* * *

### Key Takeaways from the current Data

*   `gpt-oss-20b` **:** Demonstrated near-flawless code generation across both Python and Bash. It swept the entire Hard tier, successfully navigating algorithmic efficiency, complex parsing, and meta-programming. Its only failure was a single medium-difficulty Bash parameter expansion task (`sh07_param_expansion`).
    
*   `qwen2.5-coder-7b` **:** Showed impressive capability on hard algorithmic tasks like sliding window max and decorator wrappers (scoring 80% on Hard). However, it unexpectedly stumbled on medium-tier Python edge cases—specifically mutating input lists when explicit non-mutation was required (`py09`) and throwing key errors on unsectioned config parsing (`py15`).
    
*   `qwen2.5-1.5b` **:** Highlights the strict limits of sub-2B parameter models. While it handles straightforward syntax functions reasonably well (66.7% on Easy), it drops off sharply at the Medium tier and completely fails the Hard tier due to missing standard library imports (e.g., forgetting `import itertools`) and invalid variable scoping. Not suitable at all for my uses.
    
*   `Gemini 3.5 Flash-Lite` looks useful for ordinary, testable coding and shell questions, and it offers a better time-to-results balance than the 7B setup**.** I would still favour `gpt-oss-20b` for correctness on this suite.Note that `Gemini 3.5 Flash-Lite` is the "free" model that is used in the Google Search "AI mode".
    

### What's Next?

Running LLM-generated code in a real interpreter gives an essential reality check. Pinpointing where a model handles complex logic versus where it outputs confident gibberish helps me figure out which small models are ready for prime time in my projects. As I learn the ropes of configuring and hosting more models, I'm planning to put them through their paces and keep building out the test suite.

[Contributions welcome](https://github.com/mamonu/little-llms-bench) !

\*\* an update that `Gemini 3.5 Flash-Lite` failed the `py19` test. Will update soon the post with results from the full battery of tests

### Updated Score Breakdown by Tier

`Qwen2.5-1.5b` model completed the thirty questions in 6 minutes 8 seconds, scoring 12/30. `Qwen2.5-coder-7b` took 28 minutes for 23/30, while `gemini-3.5-flash-lite` took 12 minutes 12 seconds for 26/30. `GPT-oss-20b` scored 29/30 and accounted for 24 minutes 56 seconds.

| Model ID | Easy (12) | Medium (13) | Hard (5) | Total Score (%) |
| --- | --- | --- | --- | --- |
| gpt-oss-20b | 12 / 12 (100.0%) | 12 / 13 (92.3%) | 5 / 5 (100.0%) | 29 / 30 (96.7%) |
| gemini-3.5-flash-lite | 11 / 12 (91.7%) | 12 / 13 (92.3%) | 3 / 5 (60.0%) | 26 / 30 (86.7%) |
| qwen2.5-coder-7b-instruct (Q5\_K\_M) | 10 / 12 (83.3%) | 8 / 13 (61.5%) | 4 / 5 (80.0%) | 22 / 30 (73.3%) |
| qwen2.5-1.5b-instruct (Q4\_K\_M) | 8 / 12 (66.7%) | 3 / 13 (23.1%) | 0 / 5 (0.0%) | 11 / 30 (36.7%) |