By Kintu gk

1. Install Pi

Use Node.js 24 or newer for this demo project.

Install the Pi version used in the video:

npm install -g --ignore-scripts @earendil-works/[email protected]

Check your installation:

pi --version

The project's launch scripts expect 1.0.0. or newer Pi installation instructions.

2. Connect Your Main Model

Open Pi:

pi

Inside Pi, connect your provider:

/login

Select your model with:

/model

The four recorded runs used openai/gpt-6.1-sol with high reasoning. To launch with those settings:

pi --model openai/gpt-6.1-sol --thinking high

You can choose another model available through your provider. Pi model and authentication options.

Do you prefer watching instead? Check out the full breakdown and live benchmark video here:

3. Enable Code Mode

For a new project, create the configuration directory:

mkdir -p .pi

Create .pi/settings.json with:

{
  "sessionDir": ".pi/sessions",
  "defaultTools": ["+tool_search", "+codemode"]
}

If that file already exists, merge these settings into it. The + prefix adds tools to the defaults.

Start Pi from the project directory:

pi --approve

--approve trusts the project's configuration for that invocation.

Code Mode lets the agent write JavaScript that calls tools, filters results, and calculates totals. The main model receives the script's printed output. Returning compact summaries can reduce how much tool data enters its context.

Enable Code Mode.

4. Add the NHTSA MCP Server

The vehicle demo uses NHTSA Vehicle Safety MCP, pinned to 0.9.6. It uses public vehicle data and needs no API key.

In a new project, install it with:

npm install --save-exact --ignore-scripts @cyanheads/[email protected]

If you already have the video's demo repository and lockfile, use:

npm ci --ignore-scripts

Create .pi/mcp.json:

{
  "mcpServers": {
    "nhtsa": {
      "command": "node",
      "args": [
        "./node_modules/@cyanheads/nhtsa-vehicle-safety-mcp-server/dist/index.js"
      ],
      "cwd": ".",
      "env": {
        "MCP_TRANSPORT_TYPE": "stdio",
        "MCP_LOG_LEVEL": "warn"
      },
      "description": "Read NHTSA vehicle safety profiles, complaints and investigations.",
      "exposure": "hidden",
      "toolExposure": {
        "nhtsa_get_vehicle_safety": "direct",
        "nhtsa_search_complaints": "direct",
        "nhtsa_search_investigations": "direct"
      }
    }
  }
}

This exposes the three tools used in the vehicle test. They can be called directly or through Code Mode. The repo uses an absolute cwd; the example above uses . so it works from your project root.

Inside Pi, inspect the connection with:

/mcp

After editing configuration outside Pi, run:

/reload

Pi MCP configuration.

5. Give the Agent a Code Mode Task

Paste this into Pi for a small example:

Use Code Mode to compare NHTSA complaint counts for the 2023
Toyota RAV4 and Honda CR-V.

For each vehicle, call nhtsa_search_complaints with modelYear: 2023,
limit: 10, and offset: 0.

Keep full tool responses inside Code Mode. Inspect structuredContent.

Return only the vehicle name, totalCount, number of complaint records
returned, and number of returned records where crash or fire is true.

Report failed calls or missing fields. Do not replace missing data with zero.

Label the crash/fire count as belonging to the returned page only.

Do not interpret these counts as a scientific safety ranking.

The agent writes and executes the JavaScript for you.

For reference, this is a Code Mode script for one vehicle. Ask Pi to execute it through codemode; it uses Pi's tool environment:

const result = await tools.mcp__nhtsa__nhtsa_search_complaints({
  make: "Toyota",
  model: "RAV4",
  modelYear: 2023,
  limit: 10,
  offset: 0
});

const data = result.structuredContent;

if (
  result.isError ||
  !data ||
  !Number.isFinite(data.totalCount) ||
  !Array.isArray(data.complaints)
) {
  throw new Error("Complaint data failed or is incomplete");
}

text({
  vehicle: "2023 Toyota RAV4",
  totalComplaints: data.totalCount,
  recordsReturned: data.complaints.length,
  crashOrFireOnReturnedPage: data.complaints.filter(
    complaint => complaint.crash === true || complaint.fire === true
  ).length
});

Only the compact object is printed to the main model.

6. Useful Code Mode Helpers

Helper

What it does

tools.<name>(args)

Calls a tool

text(value)

Prints a result to the main model

searchTools(query)

Finds available tools

describeTool(name)

Shows a tool's arguments and return declaration

ALL_TOOLS

Lists callable tool names and descriptions

store(key, value)

Saves small JSON state between Code Mode calls

load(key)

Reads saved state

Discover the vehicle tools:

text(await searchTools("NHTSA vehicle safety complaints"));

Inspect one tool before calling it:

text(await describeTool("mcp__nhtsa__nhtsa_search_complaints"));

For independent calls, Promise.allSettled() keeps successful results available when another call fails. Check both rejected calls and MCP responses with isError.

Keep stored state small: identifiers, counters, and summaries. The video prompts ask the agent to reduce each item immediately and avoid storing combined raw responses.

Code Mode runs in a JavaScript sandbox. Access to external services goes through tools.

Code Mode API reference.

7. Connect GitHub for the Issue Demo

The second test uses GitHub's official remote MCP server.

If you use GitHub CLI, authenticate with:

gh auth login

For a standalone Pi setup, make its token available to the MCP configuration:

export GITHUB_MCP_TOKEN="$(gh auth token)"

Alternatively, set GITHUB_MCP_TOKEN to your GitHub personal access token in the launching terminal.

Merge this github entry alongside nhtsa under mcpServers in .pi/mcp.json:

{
  "mcpServers": {
    "github": {
      "url": "https://api.githubcopilot.com/mcp/",
      "headers": {
        "Authorization": "Bearer ${GITHUB_MCP_TOKEN}",
        "X-MCP-Tools": "search_issues,issue_read",
        "X-MCP-Readonly": "true"
      },
      "description": "Read GitHub issues and comments.",
      "exposure": "codemode"
    }
  }
}

The configuration references the environment variable. The GitHub tools start with Code Mode exposure; tool search can also load them for direct calls.

The demo launchers handle authentication automatically, checking GITHUB_TOKEN, then GH_TOKEN, then your existing gh auth token session.

Reload Pi after changing the config; restart it if you exported credentials in another terminal.

8. Run the Four Video Setups

These Bash commands use the video project's scripts and prompt files. Run them from the repository root on Linux, macOS, or WSL; the Winnow scripts below are configured for the Linux/NVIDIA machine used in the video.

Check the MCP setup:

./scripts/check-mcps.sh

Then choose a run:

Run

Launch command

Prompt to paste

Vehicle test: plain MCP

./scripts/pi-plain.sh

prompts/test-1-vehicle.md

Vehicle test: MCP + Code Mode

./scripts/pi-codemode.sh

prompts/test-1-vehicle-codemode.md

GitHub test: LLM baseline

./scripts/pi-llm-baseline.sh

prompts/test-2-llm-baseline.md

GitHub test: Code Mode + Winnow

./scripts/pi-winnow.sh

prompts/test-2-winnow.md

To supply the prompt file at launch instead of pasting it:

./scripts/pi-codemode.sh @prompts/test-1-vehicle-codemode.md

Each launcher creates a fresh named session using openai/gpt-6.1-sol and high reasoning.

The vehicle prompts share the same sample and scoring rules, with instructions adapted to their execution method.

Winnow is used in the GitHub test. That comparison changes both orchestration and semantic classification: the baseline model judges threads itself; the other run uses Code Mode and asks Winnow for decisions.

9. Optional: Add Local Winnow Decisions

The project includes .pi/extensions/winnow.ts, which registers winnow_decide for Code Mode and calls:

http://127.0.0.1:8091/v1/systemone

Winnow is a local classifier. Pi's main model continues to plan the task and write the final answer.

The repo's setup expects an existing model file at:

/home/kgk256/ai/models/Winnow-12B/Winnow-12B-Q8_0.gguf

Download the Q8 GGUF from the Winnow model repository if needed, then update the model path in scripts/start-winnow.sh for your machine.

Build the pinned runtime:

./scripts/install-winnow-runtime.sh

This script builds commit 77d14580c6732ca2f3745750c1dc1fd446d8bcee; it does not download model weights.

It requires the runtime's build prerequisites and sets CUDA architecture 120 for the video's RTX 5060 Ti. Adjust the build settings for your GPU; other platforms can follow the Winnow setup instructions.

Start the server in a separate terminal and leave it running:

./scripts/start-winnow.sh

In the project terminal, check it and launch the agent:

./scripts/check-winnow.sh
./scripts/pi-winnow.sh @prompts/test-2-winnow.md

The prompt asks Winnow three questions per thread:

Question

Type

Meaning

frustrated

noul

Probability that clear frustration is expressed

severity

score

Expected severity on an ordered 0–4 scale

category

choice

Selected cause of frustration

Code Mode aggregates the decisions and sends compact results to the main model.

The completed run made 100 Winnow requests containing 300 typed questions.

Winnow decision API.

Afterward, stop the server:

./scripts/stop-winnow.sh

10. Inspect and Resume Sessions

Inside Pi, inspect session statistics:

/session

The video project saves JSONL receipts under .pi/sessions/.

Open the session picker from the project root:

pi --approve --session-dir "$PWD/.pi/sessions" --resume

Or reopen a specific receipt:

pi --approve --session-dir "$PWD/.pi/sessions" --session SESSION_ID

Replace SESSION_ID with its saved ID. Use a fresh session for each new benchmark measurement.

Pi session options.

11. Read the Recorded Results

The saved runs from 2 October 2026 produced:

Run

Main-model uncached input tokens

Output coverage

Plain MCP

95,132

4 of 10 SUVs ranked

MCP + Code Mode

20,686

10 of 10 SUVs ranked

LLM baseline

91,698

50 of 100 threads processed

Code Mode + Winnow

30,020

100 of 100 threads processed

Both vehicle runs made 30 NHTSA calls. Six visibly truncated investigation responses left six vehicles unranked in the plain run.

These input figures sum uncached input across requests. Pi's /session Input display also includes cached input. The dashboard shows both, along with runtime, output tokens, and API cost estimates. Winnow's local token usage is tracked separately from the main-model totals.

The GitHub runs completed different numbers of threads, and human classification accuracy was not measured. The figures describe these recorded runs; they do not isolate the effect of Code Mode from Winnow or establish a general speed or accuracy advantage.

Open benchmark/results/index.html in your browser for the complete report.

To rebuild it from the original four receipts:

python3 scripts/build-results.py

The first build downloads fonts. The generated page embeds its data and fonts for offline viewing.

For more open-source AI tutorials, local model setups, and benchmarks, check out my YouTube channel.

Keep reading

View more
caret-right
© 2026 Kintu.
beehiivPowered by beehiiv