# temporal-agent-optimizer

Temporal-based durable workflow for benchmarking LLM agent configurations across
latency, token usage, and task quality using local inference on an NVIDIA RTX PRO
6000 Blackwell.

Inference is served by **SGLang** rather than vLLM: `Qwen3.8-27B` (NVFP4 W4A4,
DFlash2 speculative decoding) on `127.0.0.1:8090`. A second backend, llama.cpp
serving the same model on an RTX 3090 at `127.0.0.1:8080`, is used as the
alternate-model arm.

## Layout

| File | Role |
|---|---|
| `activities.py` | Activities: prompt assembly, inference, grading. All nondeterminism lives here. |
| `workflow.py` | `AgentOptimizationWorkflow` (single task) and `BenchmarkWorkflow` (full suite). |
| `worker.py` | Worker on task queue `agent-optimization`. |
| `run_experiment.py` | The three-configuration smoke experiment. |
| `run_benchmark.py` | The five-configuration benchmark over the case suite. |
| `cases.py` | 24 self-authored server failure-triage cases. |
| `results/` | Measured output, JSON and Markdown. |

## Running it

```bash
temporal server start-dev            # terminal 1, UI on :8233
source .venv/bin/activate && source env.sh && python worker.py   # terminal 2
source .venv/bin/activate && source env.sh && python run_benchmark.py   # terminal 3
```

## The task

Single-label classification of a server failure description into one of
`MEMORY, THERMAL, POWER, FIRMWARE, INTERCONNECT, STORAGE`. Each case ships a
retrieval-style context of six reference snippets in a rotating order, so
truncating context removes real information rather than filler.

Grading is strict: the answer must name the expected label and no competing
label, so "not MEMORY, this is THERMAL" does not score as a MEMORY pass.

Every case and reference snippet was written from scratch for this project. No
proprietary logs, code, or specifications are used.

## Results

24 cases x 5 configurations = 120 durable inference Activities, one Workflow.

| Configuration | Accuracy | Median latency | P95 latency | Prompt tok | Completion tok | Tokens / correct |
|---|---:|---:|---:|---:|---:|---:|
| A full context, 256 out | 95.8% | 0.52 s | 1.03 s | 452 | 127 | 604 |
| B reduced context, 256 out | 95.8% | 0.54 s | 1.08 s | 190 | 124 | 328 |
| C reduced context, 64 out | 8.3% | 0.33 s | 0.37 s | 190 | 64 | 3050 |
| D specialized prompt, 64 out, thinking off | 100.0% | 0.08 s | 0.09 s | 159 | 3 | 162 |
| E alternate model (3090), thinking off | 100.0% | 0.67 s | 0.69 s | 159 | 3 | 162 |

### What the numbers say

**Reasoning tokens, not context, dominated the cost.** Configurations A and B
spent about 124 completion tokens per case, of which roughly 120 were reasoning
tokens the grader never sees. Cutting context from 452 to 190 prompt tokens
(A to B) left accuracy and latency unchanged, because the prompt was never the
bottleneck.

**Config C is the instructive failure.** Its 8.3% accuracy is not
misclassification. In 22 of 24 cases the model burned its entire 64-token budget
on reasoning and returned an empty answer, `finish_reason: length`. Capping
output on a reasoning model without disabling reasoning converts a cheap
configuration into a broken one, and a naive substring grader would have
reported this as a quality problem rather than a truncation problem.

**Turning reasoning off is the whole win.** Configuration D holds 100% accuracy
at 3 completion tokens and 0.08 s median latency: 6.5x faster than baseline and
3.7x cheaper per correct answer. This task is pattern recognition against a
known label set, and the reasoning trace bought nothing.

**Backend matters more than model here.** Configurations D and E run identical
prompts against the same model weights. SGLang on the RTX PRO 6000 answers in
0.08 s, llama.cpp on the 3090 in 0.67 s, at identical accuracy: an 8x latency
gap from serving stack and hardware alone.

## Durability and retry behavior

Two controlled experiments, both visible in the Temporal UI.

**Worker failure mid-Activity** (`durability-test-001`). With a 25-second
heartbeating delay injected ahead of the inference call, the Worker was killed
while `call_llm` was in flight. The Workflow stayed alive in the Temporal
service. On Worker restart the Activity was redelivered as attempt 2 and the
Workflow completed. The recorded result carries `"attempt": 2`.

**Deterministic Activity failure** (`retry-test-001`). `call_llm` raises on
attempt 1 only. The event history shows each `ActivityTaskStarted` at attempt 2
carrying `lastFailure: "Injected test failure (attempt 1)"`, and all three
configurations completed on retry.

### The idempotency caveat

Temporal retried the inference request, which is exactly the problem. An LLM
call is not idempotent: a retry after a timeout may duplicate a request the
server already accepted and is still processing, paying for it twice and
possibly returning a different answer at nonzero temperature.

This project bounds the damage rather than solving it. `maximum_attempts=3` caps
duplicate inference, the heartbeat lets Temporal detect a dead Worker in seconds
instead of waiting out a 5-minute `start_to_close_timeout`, and temperature 0
keeps retried answers stable. A production system would need a request key the
inference server can deduplicate on, or a cache checked before the call, so a
retry can return the first attempt's result instead of re-running it.