1
# Loooom context evals3
Test whether personal context changes an agent's answers in the ways you expect.4
Supply `AGENTS.md`, `SOUL.md`, `ME.md`, or other text files. Write prompts and5
observable checks. Run each prompt with and without the context. Save both6
answers and their results in a local JSON report.8
This package is agent-neutral, uses Node.js 22 or newer, and has no dependencies.9
It is released under the [MIT license](LICENSE). The package source is in this10
repository; an npm registry release is not required for the commands below.12
## Try the offline example14
From the Loooom repository root:16
```sh17
node plugins/context-evals/bin/loooom-context-eval.mjs \18
--suite plugins/context-evals/examples/suite.json \19
--context plugins/context-evals/examples/AGENTS.md \20
--context plugins/context-evals/examples/SOUL.md \21
--context plugins/context-evals/examples/ME.md \22
--responses plugins/context-evals/examples/responses.synthetic.json \23
--output /tmp/loooom-context-report.json24
```26
The example uses **handwritten synthetic answers**. It makes no network calls.27
It demonstrates 0/3 baseline cases and 3/3 context cases. These values do not28
measure a real model. Both the report and terminal output state that this is a29
synthetic replay. Choose a new output path for each run; existing files are never30
overwritten.32
A [saved synthetic report](examples/report.synthetic.json) is included for33
report viewers and local import tests. It was produced by the same replay CLI.34
Its string checks are demonstration data, not a model quality score.36
You can also install from this source checkout with37
`npm install -g ./plugins/context-evals`. This adds the `loooom-context-eval`38
command. To use the package with an agent, read [SKILL.md](SKILL.md) or copy the39
whole package directory into a skill directory supported by that agent.40
The CLI works without a skill loader.42
## Write checks for your context44
Start with preferences that have observable results. For example, check a45
specific command, required wording, a time zone, or a word limit. Keep prompts46
the same in both conditions.48
```json49
{50
"schemaVersion": 1,51
"name": "My work preferences",52
"cases": [53
{54
"id": "project-check",55
"prompt": "How should I check this project?",56
"assertions": [57
{ "type": "contains", "value": "npm run check" },58
{ "type": "notContains", "value": "Great question", "caseSensitive": false },59
{ "type": "maxWords", "value": 40 }60
]61
}62
]63
}64
```66
| Type | Rule |67
| --- | --- |68
| `contains` | The answer includes the supplied text as a substring. |69
| `notContains` | The answer does not include the supplied text as a substring. |70
| `exact` | The entire answer equals the supplied text, including whitespace. |71
| `maxWords` | The answer has no more than `value` whitespace-separated tokens. |73
Text checks are case-sensitive by default. Set `caseSensitive: false` to use74
Unicode lowercase conversion on both strings. The CLI does not trim text or75
remove Markdown before a text check. Text values must be non-empty. `maxWords`76
accepts an integer from 0 to 100000. An empty answer has zero words. Each case77
passes only when all of its assertions pass. Unknown fields, assertion types,78
and duplicate IDs are errors. There is no JavaScript evaluation, regular79
expression execution, or model-based judge.81
Substring checks have limits: an answer that says "do not use npm run check"82
still contains that command. Pair checks with review of the saved answers.83
`maxWords` is a count of whitespace-separated tokens, not linguistic words.85
## Run a model87
Use an endpoint that supports the OpenAI-compatible `POST /chat/completions`88
wire format. This is a transport interface; no specific model vendor or agent89
is required. The base URL must include the API prefix, such as `/v1`.91
Set these environment variables through your shell or secret manager:93
- `LOOOOM_EVAL_BASE_URL`: the API root. HTTPS is required, except on localhost.94
- `LOOOOM_EVAL_MODEL`: a model name accepted by that endpoint.95
- `LOOOOM_EVAL_API_KEY`: an optional bearer token. Omit it for an endpoint that96
does not need authentication.98
Then omit `--responses`:100
```sh101
node plugins/context-evals/bin/loooom-context-eval.mjs \102
--suite ./my-context-suite.json \103
--context ./AGENTS.md \104
--context ./SOUL.md \105
--context ./ME.md \106
--output /tmp/loooom-context-live-report.json107
```109
Live mode sends the selected files and prompts to your configured endpoint.110
The endpoint can charge for these requests. Reports contain the full context,111
prompts, and returned answers. Review them before sharing. Authorization112
headers are not saved. The CLI does not upload reports to Loooom.114
Each case uses two fresh chat requests. Both use the same prompt, model,115
`temperature: 0`, and output token limit. The baseline system message is:117
> Answer the user request. Follow any personal context supplied below.119
The context condition appends the selected files in command-line order, with120
file labels. The first case runs baseline first; the next runs context first,121
and this order alternates. No conversation history, tools, agent memory,122
automatic file discovery, or retries are added. Each selected file must exist.124
Options:126
- `--timeout-ms`: 100–120000 per request; default 30000.127
- `--max-output-tokens`: 32–8192; default 1024. Sent as `max_tokens`.128
- `--help` and `--version` need no input files or credentials.130
Limits: 100 cases, 20 assertions per case, 16 context files, 256 KiB of combined131
context, 32,768 characters per prompt, and 1 MiB per JSON input file. Provider132
responses are limited to 1 MiB and answer text to 128 KiB. Runs stop after five133
minutes, at the next request or checkpoint boundary. Calls are sequential.134
Providers that do not accept these chat completion fields need an adapter.136
## Replay saved answers138
Pass a JSON file with one pair of answers per suite case. Array order can differ139
from the suite order. All case IDs must match exactly.141
```json142
{143
"schemaVersion": 1,144
"synthetic": false,145
"description": "Answers exported from my prior run.",146
"results": [147
{148
"caseId": "project-check",149
"baseline": "Run npm test.",150
"context": "Run npm run check."151
}152
]153
}154
```156
Set `synthetic: true` for invented or edited demonstration answers. Even with157
`synthetic: false`, replay cannot verify the answers' source or that the context158
was used to generate them. The report labels this limitation. A replay report159
does not count as a new model run.161
## Read the report163
Reports use `schemaVersion: 1` and contain:165
- `evidence`: `mode` (`live` or `replay`), `synthetic`, and a plain-language label.166
- `context`: each file's name, full content, and SHA-256, plus the exact combined167
`text` and its SHA-256. File order and labels affect the combined hash.168
- `suite`: SHA-256 of the input file bytes and the parsed `definition`.169
- `replay`: replay file SHA-256 and definition, when used.170
- `config`: baseline instruction, model endpoint where used, request limits,171
and condition order. No API key or request headers.172
- `results`: one row per case with `caseId`, `prompt`, `baseline`, and `context`.173
Each completed condition holds its raw `output`, `rawResponse` for live174
calls, `finishReason`, request body, timing, individual `assertions`, and175
overall `passed` value. Each assertion has `passed`; word checks also have176
an `actual` count.177
- `summary.baseline` and `summary.context`: `passed`, `total`,178
`assertionsPassed`, and `assertionsTotal`.179
- `summary.delta`: `casesPassed` and `assertionsPassed`, calculated as context180
minus baseline. It is `null` until all pairs are complete.181
- `status`: `running`, `complete`, or `error`, with start/end time and error182
metadata.184
The CLI reserves a new report path before any provider call. It saves a valid185
JSON checkpoint after each completed answer, with file permissions `0600`.186
Each checkpoint uses an atomic rename. A failed request stops the run and keeps187
prior answers. Unfinished conditions remain `null`; missing answers are not188
counted as passed or failed. Invalid provider responses and HTTP error bodies189
are not scored or saved as answers. On a forced process kill, the latest report190
can remain `running`; that report is incomplete.192
Exit codes: **0** when all context cases pass, **1** when at least one context193
case fails, and **2** for input or runtime errors. A failed baseline is expected194
in many tests and does not cause exit 1.196
## Interpret the result198
The result measures only the checks you wrote. It does not establish general199
quality, truth, safety, or personal fit. A single paired run has no confidence200
interval, and temperature zero does not guarantee repeatable answers. Context201
can help one check and harm another. Read every failed assertion and the raw202
answers. Run independent repeats with new report paths to study variation.204
The first version does not run an installed agent's full tool loop, change your205
context files, or search for improved instructions. It measures a controlled206
chat request with and without supplied context.208
## Development210
From the repository root:212
```sh213
node --test tests/context-evals.test.mjs214
```216
The tests use offline fixtures and an in-memory HTTP transport substitute. They217
make no network calls and need no API keys. The suite checks replay evidence,218
validation, report preservation, paired request construction, provider errors,219
and timeout behavior.221
## Next Steps223
Run the offline example. Write checks for your own context. Then run your chosen224
model and review the saved answers before you change your instructions.226
### Browser report limits228
Loooom's `/evals` reader accepts reports up to 64 MiB, including interrupted229
`running` checkpoints. It labels incomplete runs and leaves missing answers230
unscored. For larger reports, split the suite into smaller runs or inspect the231
JSON locally. Reports contain request and response evidence, so their size232
can be much larger than the context files.