We are going to build a lightweight head-to-head LLM evaluation harness that sends the same coding prompt to four different models and scores the responses automatically. It is useful for any team that needs to pick a production model based on evidence rather than marketing claims. Everything runs against Oxlo.ai's request-based API, so long system prompts and multi-turn threads do not inflate the cost.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Set your key as an environment variable so it does not leak into shell history.
export OXLO_API_KEY="YOUR_OXLO_API_KEY"Step 1: Configure the client and candidate models
I start by importing the OpenAI SDK and pointing it at Oxlo.ai. I define a CANDIDATES list with four real model IDs available on the platform. I chose Llama 3.3 70B as the general-purpose workhorse, Qwen 3 32B because it handles multilingual reasoning and agent workflows well, Kimi K2.6 for its advanced chain-of-thought context, and DeepSeek V3.2 because it is optimized for coding and reasoning and sits on a free tier, which lets me add a fourth variable without burning budget. Because Oxlo.ai charges per request rather than per token, I can stuff long system prompts into every call and the cost stays flat. That matters when you are running comparison loops dozens of times a day. You can see the exact plan details at https://oxlo.ai/pricing. I also set a low temperature and a max token limit to keep the playing field level.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY"),
)
CANDIDATES = [
"llama-3.3-70b",
"qwen-3-32b",
"kimi-k2.6",
"deepseek-v3.2",
]Step 2: Run batch inference across candidates
Next I need a function that fires the same user prompt at every candidate and collects the text. I keep the signature simple: model name in, response string out. I use a try/except block so one model timeout does not kill the whole batch. Oxlo.ai does not cold-start popular models, so the loop moves quickly. I set temperature to 0.2 because I want deterministic enough outputs to compare structure and logic, not creative variance. I also truncate any answer that goes past 1024 tokens so a single verbose model does not drown the judge context later. In production you might swap this for asyncio, but a synchronous loop is clearer for a tutorial.
def get_answer(model: str, prompt: str) -> str:
try:
response = client.chat.completions.create(
model=model,
messages=[
{
"role": "system",
"content": "You are a helpful coding assistant.",
},
{"role": "user", "content": prompt},
],
temperature=0.2,
max_tokens=1024,
)
return response.choices[0].message.content.strip()
except Exception as exc:
return f"ERROR: {exc}"Step 3: Write the judge prompt
Raw text is hard to compare at scale, so I use a dedicated judge model to score every answer. The judge needs a strict rubric or it will hallucinate scores. I focus on three dimensions that matter for code generation: correctness, explanation clarity, and completeness. I also explicitly request JSON so I can parse the result programmatically. I keep the rubric in a module-level string so I can tweak it without touching the rest of the pipeline. This prompt is the most important part of the project. If the criteria drift, the leaderboard becomes meaningless.
JUDGE_PROMPT = """You are an expert software engineer grading coding answers.
You will receive one user question and four candidate answers from different AI models.
Score each answer on three criteria from 1 to 10:
1. correctness: does the code work and is the logic sound?
2. clarity: is the explanation easy to follow?
3. completeness: does it cover edge cases and trade-offs?
Return strictly JSON in this format:
{
"scores": {
"model_name": {"correctness": int, "clarity": int, "completeness": int},
...
},
"winner": "model_name",
"reasoning": "one sentence explaining the winner"
}
Be critical. Reserve 10s for truly excellent answers."""Step 4: Score responses with a judge model
The judge itself is just another chat completion, but I use Kimi K2.6 because its 131K context window lets me feed in four full answers plus the original prompt without truncation. I pack the candidate outputs into a single user message, ask for scores from 1 to 10, and request a winner field. I set response_format to JSON mode so the model is constrained to valid JSON. This is where Oxlo.ai's flat per-request pricing pays off again. The judge message is enormous, yet the cost is identical to a one-sentence ping. On token-based providers, a long judge prompt with four embedded responses would be expensive. Here, it is just one more request. I parse the returned JSON with the standard library and return the dictionary.
import json
def judge_responses(prompt: str, responses: dict) -> dict:
payload = f"User question:\n{prompt}\n\nCandidate answers:\n"
for model_name, answer in responses.items():
payload += f"\n--- {model_name} ---\n{answer}\n"
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_PROMPT},
{"role": "user", "content": payload},
],
response_format={"type": "json_object"},
temperature=0.2,
)
raw = response.choices[0].message.content
return json.loads(raw)Step 5: Assemble the evaluation harness
Finally, I wire everything into a small CLI script. I pick a nontrivial coding task: implementing a thread-safe LRU cache in Python and explaining the locking strategy. I print each candidate's answer with a header, then send the bundle to the judge. I also slice the output with [:500] for the preview so the terminal stays readable. The whole script is under 120 lines, but it gives me a reproducible baseline I can run every time a new model drops on Oxlo.ai. The final JSON dump is a permanent artifact I can check into git to track model drift over time.
if __name__ == "__main__":
user_prompt = (
"Write a Python function that implements a thread-safe LRU cache "
"and explain the locking strategy you chose."
)
print("Collecting answers...")
answers = {}
for model in CANDIDATES:
print(f" -> {model}")
answers[model] = get_answer(model, user_prompt)
print("\n--- Raw Answers ---")
for model, text in answers.items():
print(f"\n{model}:\n{text[:500]}...")
print("\nJudging...")
verdict = judge_responses(user_prompt, answers)
print("\n--- Verdict ---")
print(json.dumps(verdict, indent=2))Run it
Save the file as eval_harness.py, export your OXLO_API_KEY, and run the script. On my last execution, DeepSeek V3.2 produced the most concise working implementation, Kimi K2.6 gave the deepest explanation of lock granularity, and Llama 3.3 70B balanced both. The judge awarded the highest clarity score to Kimi K2.6 and the highest correctness score to DeepSeek V3.2. Qwen 3 32B returned a solid answer with excellent comments. Your exact scores will vary slightly because of sampling, but the ranking usually stabilizes after three runs. If you see an ERROR in an answer, double-check the model ID spelling against Oxlo.ai's documentation. The free tier includes 60 requests per day, which is enough for ten full evaluations, and upgrade details are at https://oxlo.ai/pricing.
$ python eval_harness.py
Collecting answers...
-> llama-3.3-70b
-> qwen-3-32b
-> kimi-k2.6
-> deepseek-v3.2
--- Raw Answers ---
llama-3.3-70b:
Here is a thread-safe LRU cache using threading.RLock...
qwen-3-32b:
You can implement this with collections.OrderedDict and a threading.Lock...
kimi-k2.6:
A thread-safe LRU cache requires careful consideration of lock granularity...
deepseek-v3.2:
```python
from threading import Lock
...
```
Judging...
--- Verdict ---
{
"scores": {
"llama-3.3-70b": {"correctness": 9, "clarity": 8, "completeness": 8},
"qwen-3-32b": {"correctness": 9, "clarity": 8, "completeness": 9},
"kimi-k2.6": {"correctness": 9, "clarity": 10, "completeness": 9},
"deepseek-v3.2": {"correctness": 10, "clarity": 8, "completeness": 8}
},
"winner": "kimi-k2.6",
"reasoning": "Best balance of clarity and completeness with correct code."
}Wrap-up
Two concrete next steps. First, replace the hardcoded prompt with a JSONL file of real production queries and batch-process them overnight to build a living leaderboard. Store the results in SQLite so you can query historical trends. Second, swap the judge to DeepSeek R1 671B MoE if you are evaluating reasoning-heavy tasks, or to GPT-Oss 120B for a large open-source second opinion. Both are available on Oxlo.ai with the same request-based pricing and OpenAI-compatible endpoints, so the only change is the model string. You can also add argparse so teammates can run ad-hoc comparisons without editing code. Because Oxlo.ai uses standard SDK patterns, integrating this harness into an existing MLOps pipeline takes minutes.
