Live Model Testing

This guide verifies WebTest AI's model router, session workflows, discovery, and guarded maintenance against real model providers.

Unit tests use fake transports so they are deterministic and safe. Live model testing is a separate smoke layer for provider compatibility and model behavior.

Provider Options

Ollama

Local model runner. WebTest AI talks to it through the ollama adapter at http://127.0.0.1:11434. This is the best first path for open-source and local-first testing.

Moonshot API

Official hosted Kimi API from Moonshot AI. Use this when you want full hosted Kimi performance without local hardware. This should be tested through an OpenAI-compatible or future dedicated Moonshot profile, depending on endpoint support.

OpenRouter

Hosted gateway for many model families. Use this for quick cross-model comparison with one API key. WebTest AI should use the openrouter-compatible adapter profile.

Recommended Real-Model Matrix

Start with these Chinese/open-source reasoning and agentic models:

Useful model references:

• Qwen3: https://ollama.com/library/qwen3
• DeepSeek-R1: https://ollama.com/library/deepseek-r1
• Kimi K2: https://ollama.com/library/kimi-k2
• Kimi K2 Thinking: https://ollama.com/library/kimi-k2-thinking
• GLM4: https://ollama.com/library/glm4
• Yi: https://ollama.com/library/yi

Ollama Setup

Install and start Ollama, then pull the models you want to test:

ollama serve
ollama pull qwen3:8b
ollama pull deepseek-r1:8b
ollama pull kimi-k2:1t-cloud
ollama pull kimi-k2-thinking:cloud
ollama pull glm4
ollama pull yi:9b

Large/cloud-tagged models may require network access, an Ollama account, or provider-side availability. If a pull fails, skip that row and record the provider error.

Local Config Template

For the recommended first local smoke test, use the checked-in Qwen3 config at examples/config/ollama-qwen3.config.json.

The ollama adapter disables model thinking for structured JSON calls when a profile declares "reasoning": true. This keeps Qwen3-style reasoning traces out of machine-readable responses while still documenting that the model is reasoning-capable.

To compare other models, copy examples/config/ollama-qwen3.config.json to a temporary local file and change only the profile model value.

Generic temporary config shape:

{
  "models": {
    "activeProfile": "live-model",
    "profiles": {
      "live-model": {
        "provider": "ollama",
        "model": "qwen3:8b",
        "endpoint": "http://127.0.0.1:11434",
        "apiKeyEnv": null,
        "capabilities": {
          "structuredJson": true,
          "reasoning": true,
          "toolCalling": false,
          "streaming": false,
          "vision": false
        },
        "limits": {
          "timeoutMs": 240000,
          "retries": 0,
          "maxInputBytes": 120000,
          "maxOutputTokens": 1024,
          "maxSessionTurns": 3
        }
      }
    },
    "writePolicy": {
      "roots": ["specs", "artifacts", ".webtest-ai"],
      "extensions": [".md", ".json", ".js"]
    }
  }
}

For each model in the matrix, change only models.profiles.live-model.model.

Smoke 1: V2 Intent E2E

The maintained smoke command is gated by WEBTEST_AI_LIVE_MODEL_SMOKE=1, so default CI and local test runs never require a model server:

WEBTEST_AI_LIVE_MODEL_SMOKE=1 npm run test:models:live
WEBTEST_AI_LIVE_MODEL_SMOKE=1 WEBTEST_AI_LIVE_MODEL_CONFIG=/tmp/webtest-ai-ollama.json npm run test:models:live

With the flag enabled, the smoke runs a V2 intent test through the fake driver. It does not launch a real browser or require an HTTP app server. The configured model is still called for both bounded JSON model paths:

• intent.plan_goal: turns a human goal into allowed actions only: open, click, fill, select, wait, or submit.
• intent.assert_outcome: evaluates Assert outcome against bounded fake-driver URL, accessibility, and network evidence.

Pass criteria:

• the test result status is passed
• model telemetry includes exactly intent.plan_goal then intent.assert_outcome
• the intent plan includes at least one bounded action and stop: true
• the semantic assertion matches above the configured outcome confidence threshold

Common failures:

• disabled: activeProfile is missing or null.
• error with provider message: Ollama is not running, model is not pulled, or provider timed out.
• empty: model returned non-JSON for either intent path. Increase prompt strictness, timeout, or retry count.
• Intent planner returned no bounded actions: the model did not produce an executable plan shape.
• Semantic outcome ... was not confirmed: the model did not match the outcome against the bounded fixture evidence.

Smoke 2: V2 Intent Matrix

Use the matrix runner when you want the same V2 intent smoke to run against several temporary configs without hand-editing environment variables:

WEBTEST_AI_LIVE_MODEL_MATRIX=1 npm run test:models:matrix
WEBTEST_AI_LIVE_MODEL_MATRIX=1 WEBTEST_AI_LIVE_MODEL_CONFIGS=examples/config/ollama-qwen3.config.json,/tmp/webtest-ai-deepseek.json npm run test:models:matrix

WEBTEST_AI_LIVE_MODEL_CONFIGS is a comma-separated list. Each config is passed to the maintained V2 intent smoke with WEBTEST_AI_LIVE_MODEL_SMOKE=1. The command runs every row, prints a concise pass/fail summary, and exits non-zero when any model fails.

Smoke 3: Local Commerce Demo With A Real Model

The bundled intent demo normally uses a deterministic local sidecar so CI does not need a model server. To rehearse the same local whitelabel storefronts with a real configured model, run a narrow target/spec first:

node demo-site/intent-demo-runner.js --intent \
  --live-model \
  --config examples/config/ollama-qwen3.config.json \
  --target northwind-en \
  --spec specs/intent-whitelabel-demo.md

Keep the first run narrow because every goal and semantic outcome calls the configured model. Once that path is stable, expand to more targets or the full commerce showcase.

Smoke 4: Discovery Workflow

This verifies the session layer and webtest-ai discover command.

Start the demo site in another terminal:

npm run demo:site

Then run:

node ./src/cli/index.js discover \
  --url http://127.0.0.1:4010 \
  --config ./examples/config/ollama-qwen3.config.json \
  --dry-run

Pass criteria:

• Discovery Status: proposal with one or more proposed flows, or
• Discovery Status: no-op without provider errors.

no-op means the integration worked but the model did not propose a useful flow.

Smoke 5: Guarded Maintenance

This verifies that model-proposed writes stay policy-gated. Use a safe target under artifacts first:

node - <<'NODE'
const path = require("path");
const { runMaintenanceWorkflow } = require("./src/autonomy/maintenance");
const config = require("./examples/config/ollama-qwen3.config.json");

runMaintenanceWorkflow({
  config,
  baseDir: process.cwd(),
  targetPaths: ["artifacts/live-maintenance/proposal.md"],
  apply: false,
  context: {
    instruction: "Propose a tiny markdown test note only. Do not include secrets."
  }
}).then((result) => {
  console.log(JSON.stringify({
    status: result.status,
    success: result.success,
    proposedWrites: result.proposedWrites.map((write) => ({
      path: write.path,
      allowed: write.allowed,
      reason: write.reason
    })),
    blockedWrites: result.blockedWrites,
    warnings: result.warnings
  }, null, 2));
}).catch((error) => {
  console.error(error);
  process.exitCode = 1;
});
NODE

Pass criteria:

• status: "proposal" or status: "no-op"
• no write is applied because apply: false
• proposed writes include allowed: true only for paths covered by models.writePolicy

Do not use apply: true until proposal behavior is reviewed.

Recording Results

Track each model with:

• model name
• provider path
• router smoke status
• discovery status and number of proposed flows
• maintenance status and blocked/applied write counts
• average latency or timeout behavior
• JSON reliability notes

Example:

Safety Notes

• Do not store API keys in config files. Use apiKeyEnv.
• Do not commit temporary live-model configs unless they contain no secrets.
• Keep apply: false for first maintenance runs.
• Use narrow writePolicy.roots for auto-maintenance experiments.
• Prompts and model responses are intentionally excluded from modelCalls telemetry.