← John Curreri

Experiment · September 2026

The expensive part of my agent wasn't the context. It was the thinking.

A durable Temporal workflow that benchmarks LLM agent configurations against local inference. Five configurations, 120 runs, one classification task with a real optimization objective.

Temporal · Python Qwen3.8-27B, local 120 runs RTX PRO 6000 + RTX 3090

I wanted hands-on experience with Temporal, and I wanted the project to be something other than a chatbot demo. So I built a durable workflow that benchmarks LLM agent configurations against local inference, and pointed it at a classification task with a real optimization objective: reduce latency and token consumption without losing accuracy.

The result surprised me. The lever I expected to matter did nothing, and the lever I almost didn't test was worth a 6.5x latency improvement.

The setup

Temporal separates durable orchestration from nondeterministic work. The Workflow decides what happens in what order and survives process death. The Activities do the things that can fail, block, or return a different answer twice: HTTP calls, wall-clock timing, evaluation. Getting that boundary right is most of what learning Temporal consists of.

So the Workflow loops over configurations and cases. Three Activities do the work: build the prompt, call the model, grade the answer. Inference runs locally on an RTX PRO 6000 Blackwell through SGLang, serving Qwen3.8-27B at NVFP4 with speculative decoding. A second arm runs the same model through llama.cpp on an RTX 3090.

The task is single-label classification of a server hardware failure description into one of six root causes: memory, thermal, power, firmware, interconnect, or storage. I wrote 24 cases and six reference snippets from scratch. Each case ships a retrieval-style context of all six snippets in a rotating order, so truncating the context removes real information rather than filler.

Grading is strict. The answer must name the expected label and no competing label. A plain substring test would score "not memory, this is thermal" as a memory pass, and that leniency would have hidden the most interesting result in the whole experiment.

Five configurations, 120 runs

Seconds per case. Lower is better. Configuration D is 0.08 s against a 0.52 s baseline.

Show data table

Configuration	Median latency	P95 latency
A full context, 256 out	0.52 s	1.03 s
B reduced context, 256 out	0.54 s	1.08 s
C reduced context, 64 out	0.33 s	0.37 s
D specialized prompt, 64 out, thinking off	0.08 s	0.09 s
E alternate model on the 3090, thinking off	0.67 s	0.69 s

Context was not the bottleneck

Configuration A to B cuts the prompt from 452 tokens to 190, a 58% reduction. Accuracy does not move. Latency does not move either.

That is worth sitting with, because context trimming is the reflex optimization. It is the thing everyone reaches for first, and on this task it bought nothing measurable. Cost per correct answer did improve, from 604 tokens to 328, but entirely on the input side. If you are paying per input token that matters. If you are optimizing latency, it was wasted effort.

Meanwhile configurations A and B were each spending about 124 completion tokens per case, of which roughly 120 were reasoning tokens that the grader never sees and the user would never read. The model was writing four times more invisible text than the visible answer, on a task whose correct output is a single word.

Mean completion tokens per case, split into reasoning and answer.

Reasoning tokens Answer tokens

Show data table

Configuration	Reasoning	Answer	Total completion
A full context, 256 out	123.8	3.6	127.4
B reduced context, 256 out	120.0	3.5	123.5
C reduced context, 64 out	63.8	0.0	63.8
D specialized prompt, 64 out, thinking off	0.0	2.8	2.8
E alternate model on the 3090, thinking off	0.0	2.8	2.8

Configuration C reported a mean of 65.6 reasoning tokens against a 63.8-token mean completion count. Server-side accounting counts a few tokens the completion total does not. The chart plots the completion budget, all of which C spent reasoning.

Configuration C is the instructive failure

C scores 8.3%. It is tempting to read that as the model getting confused by a short context, which is the story I expected to write.

It is not what happened. In 22 of the 24 cases the model spent its entire 64-token budget on reasoning and returned an empty string, with a finish reason of length. It never produced an answer at all. Its 3,050 tokens per correct answer is the worst number in the table by a factor of five, for a configuration that looked like the cheap one.

If I had graded on substring match and not recorded finish_reason, I would have logged this as a quality regression and gone looking for the wrong cause.

Out of 24 runs per configuration, how many hit the output cap before producing an answer.

Show data table

Configuration	Truncated	Of
A full context, 256 out	1	24
B reduced context, 256 out	1	24
C reduced context, 64 out	22	24
D specialized prompt, 64 out, thinking off	0	24
E alternate model on the 3090, thinking off	0	24

Turning reasoning off was the entire win

Configuration D disables thinking, adds a system prompt telling the model to reply with exactly one label, and keeps the 64-token cap that broke configuration C.

It scores 100%. Completion tokens drop from 124 to 3. Median latency drops from 0.52 s to 0.08 s, and P95 from 1.03 s to 0.09 s, which is the number I care about more: the distribution collapses. Cost per correct answer falls from 604 tokens to 162.

The reasoning trace was not merely expensive on this task. It was worth negative accuracy, because it was what pushed configuration A and B into their single truncation failure each. This is pattern recognition against a known label set. There is nothing to reason about, and letting the model reason anyway cost 6.5x latency to arrive at a slightly worse answer.

The generalization is not "turn reasoning off." It is that reasoning is a cost you should have to justify per task, and classification-shaped work usually cannot justify it.

Serving stack beat model choice

Configurations D and E send byte-identical prompts to the same model weights. The only difference is where inference runs.

SGLang on the RTX PRO 6000 answers in 0.08 s. llama.cpp on the 3090 answers in 0.67 s. Same accuracy, same token counts, 8x latency gap, entirely from the serving stack and the hardware under it.

If you are evaluating models by benchmarking them on whatever runtime is convenient, this is the confound sitting in your data.

What Temporal actually bought me

Two controlled experiments, both recorded in the workflow event history.

First, worker failure mid-inference. I injected a 25-second heartbeating delay ahead of the model call and killed the worker process while the Activity was in flight. The Workflow stayed alive in the Temporal service, holding all state. On worker restart the Activity was redelivered as attempt 2 and the run completed. The saved result carries "attempt": 2, which is the whole point: the orchestration outlived the process running it.

Second, deterministic Activity failure. I made the inference Activity raise on its first attempt only. Every ActivityTaskStarted at attempt 2 in the history carries the injected failure message, and all configurations completed on retry.

The caveat that matters more than the demo

Temporal cheerfully retried a non-idempotent operation, which is exactly the problem.

An LLM call is not safe to retry. A retry after a timeout may duplicate a request the server already accepted and is still working on. You pay twice, and at nonzero temperature you may get a different answer the second time. The Activity succeeded on attempt 2 here, but nothing in the system knew whether attempt 1 had also reached the model.

This project bounds the damage rather than fixing it. Three maximum attempts caps duplicate inference. A heartbeat lets Temporal detect a dead worker in seconds rather than waiting out a five-minute timeout. Temperature zero keeps retried answers stable. A production system would need something this one does not have: a request key the inference server can deduplicate on, or a cache checked before the call, so a retry returns the first attempt's result instead of running it again.

"Temporal retries failed work" is the easy sentence. Knowing which of your work is safe to retry is the part that takes thought.

What this is and isn't

This is a small experimental project. I built a Python agent-optimization workflow with Temporal against local inference infrastructure, separated nondeterministic inference and evaluation into Activities, used a durable Workflow to compare configurations across latency, token usage, and answer quality, and explored the failure-recovery and retry model deliberately rather than incidentally.

It is not production Temporal experience, and I would not claim it is. It is enough to have opinions about Workflows, Activities, workers, task queues, and where the idempotency landmines are buried.

Source and raw data

Every number above comes from these files. Nothing is estimated or rounded beyond the table.

Running it

pip install -r requirements.txt
temporal server start-dev                  # terminal 1, UI on :8233
source env.sh && python worker.py          # terminal 2
source env.sh && python run_benchmark.py   # terminal 3

env.sh points at 127.0.0.1:8090. Edit it, and the MODELS registry at the top of activities.py, to match the endpoints on your machine.