Live model testing

The navigator chooses one action from a fresh browser observation. Jev or another configured selector resolves ambiguous element targets. The runtime checks the fixed success conditions in the browser.

Local model

Start Ollama, then pull a model in another terminal:

ollama serve
ollama pull qwen3:4b

From this repository:

export WEBTEST_AI_MODEL_CONFIG=examples/config/goal-agent-cpu.config.json
node src/cli/index.js demo local headed smoke

The CPU template uses one worker and longer request deadlines. Ollama readiness inference runs before browser tests and records its usage separately. Remove smoke to run all twelve goals. Adjust the model in your config to one installed on your machine.

Hosted model with mocked Jev

Use examples/config/goal-agent-openrouter.config.json for an OpenAI-compatible gateway. Set the environment variable named in its apiKeyEnv and select a model available to your account.

export WEBTEST_AI_MODEL_CONFIG=examples/config/goal-agent-openrouter.config.json
node src/cli/index.js demo mock-jev headed smoke

The navigator is live; Jev selection is a deterministic local fixture. Reports identify each role separately. This mode exercises the live navigator without a Jev account.

Live Jev selection

Use examples/config/goal-agent-jev.config.json, configure the navigator, and set TYPESAFE_API_KEY:

export WEBTEST_AI_MODEL_CONFIG=examples/config/goal-agent-jev.config.json
node src/cli/index.js demo live headed smoke

Jev chooses among fresh element references. It does not plan the test journey. Exact unique labels can resolve locally without a selector call.

External agent session

node src/cli/index.js demo session headed

Keep the command running. Set WEBTEST_AI_SESSION_CONTROL to the printed control-file path in the agent's terminal. Read requests and submit a single allowed action from the current observation:

node demo-site/sessionControl.js next
printf '%s' '{"id":"<request-id>","action":{"action":"done","reason":"Ready for verification"}}' | node demo-site/sessionControl.js reply

The reply above illustrates the protocol only. The agent must choose actions from the actual goal and observation. The runtime independently verifies completion. Session mode uses mocked Jev and never falls back to the navigator fixture. See the demo guide for the full workflow.

When installed through npm, use npx webtest-ai demo session headed and the helper under node_modules/@asserthive/webtest-ai/demo-site/sessionControl.js.

Live release evaluation

WEBTEST_AI_MODEL_CONFIG=examples/config/goal-agent.config.json npm run test:models:live
WEBTEST_AI_LIVE_MODEL_CONFIGS=examples/config/goal-agent.config.json,examples/config/goal-agent-jev.config.json npm run test:models:matrix

The live gate runs all twelve goals in a real browser and rejects simulated calls. Missing configuration fails explicitly. These commands are separate from the credential-free release checks and may incur provider charges.

Record model identity, wrong-target rate, false passes, abstentions, latency, and cost per completed goal. Passing the scripted demo does not establish live model accuracy.

Usage and cost

Reports distinguish navigator and selector calls and retain configured and returned model identities. Token totals use endpoint metadata. If a subscription session does not return usage, the report says usage was not reported. It does not invent token counts or equate subscription usage with zero cost.

Configure per-million-token rates in the model profile to estimate API cost. Infrastructure estimates are separate. Never place API keys in committed config files.

Legacy authoring workflows

The optional legacy planner, discovery, and maintenance workflows use models.activeProfile. They do not configure the bounded navigator's agent.profile. See legacy intent mode and discovery.