Optimizing Self-Hosted LLM Inference
Private AI Stack
Running a Large Language Model locally is easy. Download, point at a GPU, and you are chatting. But running a model is not the same as wielding it.
We wrote this around qwen3.8:27b because it is the strongest local model we have run, full stop. That is the whole point of picking it: on the agentic coding benchmarks in its own model card it sits at or above Anthropic's Opus 4.6 Max, so self-hosting it is no longer the cheaper, weaker option. It is the reason the move off a monthly subscription onto your own hardware is finally a real choice rather than a compromise.
A few rows from the model card, Qwen's own reporting:
| Benchmark | Qwen3.6-27B | Qwen3.8-27B | Opus 4.6 Max |
|---|---|---|---|
| SWE-bench Pro (agentic coding) | 53.5 | 61.7 | 53.4 |
| QwenSWEBench (software engineering) | 49.3 | 79.0 | 63.8 |
| Terminal Bench 2.1 (terminal coding) | 63.4 | 73.0 | 78.2 |
| IFBench (instruction following) | 69.1 | 79.5 | 62.5 |
| LiveCodeBench v6 (competitive coding) | 83.9 | 90.3 | 88.8 |
A 27-billion parameter model, running on hardware you own, matching or beating a frontier API model on most of the rows that matter for engineering work. The one row it loses, and the reasoning rows, is where the frontier models still lead, and nobody is pretending otherwise.
Most local stacks ship one default profile for every task: low temperature, tight filters, a short context. For qwen3.8:27b that profile is wrong, and the model card says so. It ships two explicit sampling profiles - one for thinking mode, one for instruct mode - and the thinking profile is the one an agent wants.
Here is what the card recommends, and what each knob is doing.
We run it with Ollama, which exposes the full set of sampling knobs as Modelfile parameters, so the values below are ones you can copy into a configuration and actually use.
The two profiles
| Parameter | Thinking (agent) | Instruct (chat) |
|---|---|---|
| temperature | 1.0 | 0.7 |
| top_p | 0.95 | 0.80 |
| top_k | 20 | 20 |
| min_p | 0.0 | 0.0 |
| presence_penalty | 0.0 | 1.5 |
| repetition_penalty | 1.0 | 1.0 |
Temperature
1.0 in thinking mode, 0.7 in instruct mode. The agent profile is warmer than the chat profile, which is the reverse of the usual default. In thinking mode the model is exploring a reasoning space, and a cold temperature narrows that search. If an agent is wandering, the card's lever is the output budget below, not a colder temperature.
Top_P
The primary nucleus filter: 0.95 while thinking, 0.80 in instruct mode.
Top_K
Fixed at 20 in both modes. It bounds the candidate vocabulary so a confused step does not reach into the long tail of unlikely tokens.
Min_P
The card sets min_p to 0.0 in both profiles. For Qwen3.8 it is not the load-bearing filter; top_p and the presence penalty do that work.
Presence Penalty: the anti-loop dial
0.0 in thinking mode, 1.5 in instruct mode. The card is explicit that presence_penalty can be tuned between 0 and 2 to reduce endless repetition, and warns that pushing it too high can cause language mixing and a slight drop in quality. Leave it at 0.0 while the model is reasoning, keep it at 1.5 for direct answers, and nudge it higher only if a specific loop appears on a specific task.
Repetition Penalty
1.0 in both modes, i.e. disabled. The card handles repetition with the presence penalty.
The agent levers
Three switches do more for a multi-step agent than any sampling value:
Thinking mode. On by default. Disable it per request when you want a fast, direct answer instead.
Reasoning effort. reasoning_effort tunes depth: xhigh (the default) for complex work that needs thorough analysis, medium when you want to balance accuracy against speed and cost.
Preserved thinking. On by default. The card retains the reasoning trace across messages and calls it beneficial for agent scenarios where decision consistency matters, and it improves KV cache reuse. Keep it on for a multi-step agent.
Context and output budget
qwen3.8:27b natively supports up to 262,144 tokens of context; the card points to RoPE scaling (e.g. YaRN) for longer workloads, and the hosted Qwen Cloud service is set up for 1M.
For agentic work the card recommends reserving room for the answer, not just the prompt: up to 262,144 tokens for the reasoning trace and 131,072 for the final response. A truncated reasoning trace is a truncated answer.
The profile we run
| Setting | Value |
|---|---|
| Model | qwen3.8:27b |
| Mode | thinking (on) |
| temperature | 1.0 |
| top_p | 0.95 |
| top_k | 20 |
| min_p | 0.0 |
| presence_penalty | 0.0 |
| repetition_penalty | 1.0 |
| reasoning_effort | xhigh (default) |
| preserved thinking | on |
| context | up to 262,144 |
| output budget | 262,144 reasoning / 131,072 final |
Why Self-Hosted
A closed API hands you a model and a temperature slider. Some of the card's knobs - the sampling profile per mode, the output budget split between reasoning and final answer, the preserved-thinking switch - are simply not things you get to set.
The economics follow the same logic. A subscription is priced per token or per seat, so every agentic run is a line item, and a workload that reasons for hours a day turns the model into the thing your bill grows with. On your own hardware the token meter effectively does not exist: the cost is the machine, paid once, and ten times the prompts move the electricity bill by a rounding error. "Unlimited tokens" stops being a marketing line and becomes what it always should have been, a consequence of owning the inference.
Running qwen3.8:27b on your own hardware gives you the full parameter surface, the ability to test the card's recommendations against your own workloads, and the guarantee that no prompt, no token, and no reasoning trace ever leaves the building.
The model card is the spec. Tune the engine, and the ghost in the machine stops looping, starts shipping, and stays in your building.
Need a local AI stack that actually performs?
Contact us at support@sevenseven.tech or fill out our contact form.