Model Reliability Test

All T5/T6 retries after Pass 2 showed "models not available" — attributed to rate-limiting but actually caused by PowerShell's Set-Content -Encoding UTF8 writing a BOM (0xEF BB BF) to ~/.supratim/models.json. JSON.parse rejected the file silently. Fixed by using [System.Text.UTF8Encoding]::new($false) throughout the harness.

qwen3-coder:480b is the only model that completed all 6 tasks. devstral-2:123b finished T6 in 55 s (vs 251 s) but looped on T5 like Sarvam. Sarvam 105b never completed cross-file T6 in any pass. On Ollama free, the trade-off is speed vs loop reliability, not rupees.

Every turn in the 30b T5 run ended with tool_result_count: 1 — the model never stopped dispatching tools. All HTTP responses were 200. The API was healthy. After turn 1 (1,277 output tokens), every subsequent turn collapsed to 22–49 output tokens — just enough to emit another tool call, never a final text response.

Root cause: sarvam-30b cannot decide when to stop calling tools on multi-step edit tasks. It keeps dispatching rather than synthesising a conclusion. The earlier "silent empty response" description was correct in effect but wrong in cause — the model was not silent, it was spinning.

Setting maxTokens: 8192 in user config causes the Sarvam API to reject the first real request (after the 20-token validation call succeeds), surfacing as "Sarvam models are not available." This is not a config issue — the API hard-rejects max_tokens > 4096 on the starter tier. It cannot be worked around.

Pass 3 showed 105b collapsing to 21-token tool calls on both T5 and T6 after 2–6 productive turns. The signature is identical to 30b's loop:

105b makes more productive attempts and the tsc gate causes brief recoveries, but it cannot sustain convergence across 30 turns on multi-file tasks. T6 (cross-file) collapsed after turn 2 and produced 14–19 tokens per turn for 28 remaining turns.

The turn-limit guard is essential for both models. Pass 3 confirmed: 105b T5 cost ₹0.92 at 30 turns; without the guard it would have run indefinitely.

After 30b's 185-turn T5 run (180 s of continuous API calls), the account was rate-limited for over 75 minutes. All retry attempts failed during this window. The turn-limit guard would have cut the 30b loop at 30 turns (~25 s, ~₹0.06) instead of 185 turns.

PowerShell 5.x Set-Content -Encoding UTF8 writes a BOM (0xEF 0xBB 0xBF) prefix. The eval harness wrote it during the MaxTokens patch-and-restore step. JSON.parse failed with "Unexpected token ''" — surfaced as "models not available." All harness writes now use [System.Text.UTF8Encoding]::new($false).

Every --print run now emits a cost summary to stderr. Previously evaluation runs had no visibility into spend.