Compiling a Workflow into a 3B Model

TL;DR: Compiling a 3,500-token Standard Operating Procedure (Workflow) into a QLoRA-tuned 3B model hit 93.0% end-to-end accuracy (vs. 92.0% on Gemini 2.5 Pro) while cutting input tokens by 91.3% (5,200 → 450) and cost by 8.4x ($1.85 → $0.22 per 1k incidents).

End-to-End Success Rate (500 Test Scenarios — Higher is Better)
─────────────────────────────────────────────────────────────────────────────
Fine-Tuned 3B SLM (This Run)  [████████████████████████████░░]  93.0% (465/500)
Gemini 2.5 Pro (With Cache)   [███████████████████████████░░░]  92.0% (460/500)
Gemini 2.5 Pro (Raw API)      [███████████████████████████░░░]  90.0% (450/500)
Untuned Qwen 2.5 3B Base      [██████░░░░░░░░░░░░░░░░░░░░░░░░]  20.0% (100/500)

Average Input Tokens per Incident (Lower is Better)
─────────────────────────────────────────────────────────────────────────────
Prompt-Driven Setups (SOP)    [██████████████████████████████]  5,200 tokens
Fine-Tuned 3B SLM (Compiled)  [███░░░░░░░░░░░░░░░░░░░░░░░░░░░]    450 tokens (-91.3%)

Cost per 1,000 Incidents (Lower is Better)
─────────────────────────────────────────────────────────────────────────────
Gemini 2.5 Pro (Raw API)      [██████████████████████████████]  $3.50+
Gemini 2.5 Pro (With Cache)   [████████████████░░░░░░░░░░░░░░]  $1.85
Fine-Tuned 3B SLM (This Run)  [██░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  $0.22  (8.4x cheaper)

I got an idea to run that takes things in the opposite direction from what most people are doing right now with language models. Here’s repository for reference arpitbbhayani/workflow-slm.

Most AI agent setups today rely on prompt engineering. You take a SOP (workflow), paste thousands of tokens of instructions into the system prompt, and send that entire document back and forth on every turn of a conversation. It works, but it got me wondering: what happens if we do the opposite?

Instead of giving the model any procedural instructions in the prompt at all, could we compile the entire multi-step decision tree directly into the weights of a small 3-billion-parameter model?

Incident triage was a simple workflow I tried to convert into an SLM. This was just a fun experiment I decided to run to see if it would work, how hard it would be to train, and what the resulting performance and token numbers would look like.

Workflow

Incident triage was a simple workflow I chose because it operates cleanly as a finite state machine where each step depends on the output of the previous step:

An alert arrives as noisy text: [FIRING] checkout api 5xx rate way above normal in us-east, started about 12 min ago.
Extract the canonical service name, symptom, region, and start time.
Query metric telemetry to check whether the anomaly is real or transient noise.
If metrics are critical, query recent releases across a forty-five-minute correlation window to locate suspect deploys.
Query service logs to match error signatures and stack traces.
Conclude the root cause, culprit deployment identifier, and remediation runbook.
Compute incident severity and on-call routing using deterministic code.
Draft a concise status update citing only verified evidence.

In a traditional prompt-driven setup, the operational specification for this procedure takes about 3,500 tokens of Markdown. Across five or six conversational turns, that means sending roughly 5,200 input tokens per incident to an API.

Zero Procedural Prompt

The idea was to strip the system prompt down to almost nothing. In my experiment, the system prompt contains zero instructions, zero schemas, and zero runbooks. It is just a single trigger string: incident_triage_v1.

Prompt Setup (Baseline — ~5,200 input tokens / incident)

> 3,500-Token SOP Manual + Alert + Tools --> Action JSON

Compiled Setup (This Experiment — ~450 input tokens / incident)

> "incident_triage_v1" + Alert + Tools --> Action JSON

The goal was to see if the neural network could learn the state machine so thoroughly that it would execute the diagnostic turns in the correct sequence purely from its weights.

Building A Simulator And Generating Training Data

To train and test this cleanly, I built a deterministic simulator with twelve microservices and nine root causes, such as bad code deploys, configuration errors, database connection pool exhaustion, and transient metric spikes.

I also wrote an oracle script that knows the mathematically optimal sequence through the state machine. The oracle generated 5,116 multi-turn training conversations across five different alert styles (PagerDuty alerts, Slack messages, email alerts, SMS messages, and messy freeform notes).

Every training trace had to pass eight programmatic checks: valid step ordering, strict JSON schemas, grounded entity extraction, true root cause identification, valid runbook selection, verified culprit commit hashes, log evidence confirmation, and hallucination-free channel summaries.

The Training Setup: Loss Masking On Assistant Turns

I used Qwen/Qwen2.5-3B-Instruct as the base model and trained it locally on my desktop with an NVIDIA RTX 3060 graphics card (12 GB of VRAM) using QLoRA.

The one crucial technical detail during fine-tuning was token loss masking.

In standard causal language modeling, loss is calculated across all tokens. In a multi-turn tool-calling dataset, you do not want the model to learn to predict what the diagnostic tools will return. If you do not mask tool outputs, the model wastes its capacity trying to memorize random telemetry numbers and arbitrary commit hashes.

To prevent this, the dataset collator assigned label $-100$ to all tokens representing the user alert, system triggers, and tool responses, leaving only the assistant actions active during loss calculation. Gradients only flow through the decisions of the assistant:

$\mathcal{L} = -\sum_{t \in \text{assistant tokens}} \log P(x_t \mid x_{<t})$

The base model was quantized in 4-bit NormalFloat (NF4) format. I attached low-rank adapters (LoRA) ( $r = 16$ , $\alpha = 32$ ) across all linear projection layers, resulting in an adapter file of just 115 MB. The entire training run took three epochs and finished in under four hours.

What An Interactive Run Looks Like

Once trained, I wired the adapter into a simple command-line runtime loop. When you pass it a raw alert string, here is what happens turn by turn:

Turn 1: The model parses the messy alert and extracts canonical keys:

{
  "step": "extract_alert",
  "service": "checkout-api",
  "symptom": "high_error_rate",
  "region": "us-east",
  "started_at": null,
  "started_minutes_ago": 12
}

Turn 2: Without any prompt telling it what to do next, the model calls the metric tool:

{
  "step": "get_metrics",
  "service": "checkout-api",
  "metric": "error_rate",
  "window_minutes": 30
}

The simulator returns an error rate of 0.18 against a baseline of 0.01, marking the status as critical.

Turn 3: Recognizing the metric anomaly, the model inspects recent releases:

{
  "step": "get_recent_deploys",
  "service": "checkout-api",
  "window_minutes": 120
}

The simulator finds deployment d-7731 rolled out fifteen minutes prior.

Turn 4: The model inspects the error logs:

{
  "step": "query_logs",
  "service": "checkout-api",
  "window_minutes": 30
}

The simulator returns NullPointerException stack traces.

Turn 5: The model concludes the root cause:

{
  "step": "conclude",
  "root_cause": "bad_deploy",
  "runbook_id": "RB-101",
  "culprit": "d-7731",
  "log_signature": "unhandled_exception"
}

A deterministic Python function runs to assign incident severity and paging actions:

{
  "severity": "sev1",
  "action": "page_oncall_immediately",
  "team": "team-checkout",
  "runbook_title": "Roll back the deploy"
}

Turn 6: The model drafts the final channel broadcast:

{
  "step": "draft_summary",
  "text": "SEV1: checkout-api root cause bad_deploy. Follow RB-101. Team team-checkout. Culprit: d-7731."
}

The Experimental Numbers

I evaluated the trained model on 500 held-out test scenarios and compared the results alongside Gemini 2.5 Pro and an untuned Qwen 2.5 3B model. The numbers surprised me:

The fine-tuned 3B model achieved a 93% end-to-end success rate, right on par with Gemini 2.5 Pro (92%).
The untuned 3B model with the full prompt failed on most traces (20% success), getting overwhelmed by the long procedural instructions.
Because the 3,500-token manual was absent from the prompt, input token volume dropped from 5,200 down to 450 tokens per incident (a 91.3% reduction).
At standard local GPU or small-model inference rates, triaging 1,000 incidents cost $0.22 compared to $1.85 on frontier prompt caching, an 8.4x difference.

What I Learned From The Experiment

This was a fun technical exploration that showed compiling a multi-turn state machine into model weights is entirely feasible on everyday hardware:

A 3B model can internalize a complex, branching procedural sequence without needing continuous prompt reminders.
Proper loss masking during training is essential to stop the model from trying to predict simulated environment outputs.
The tradeoff is simple: prompts are easy to edit when rules change, while compiled weights give you high speed, zero prompt overhead, and predictable local execution.

Footnote

It was an entertaining holiday build that proved you do not always need a massive prompt to get disciplined, multi-turn reasoning out of a language model.