Try their rules. Share yours.
Share AGENTS.md
Back to personal evals

Context eval guide.

Run from source. The package is not yet published.

context-evals / README.md
1# Loooom context evals
2
3Test whether personal context changes an agent's answers in the ways you expect.
4Supply `AGENTS.md`, `SOUL.md`, `ME.md`, or other text files. Write prompts and
5observable checks. Run each prompt with and without the context. Save both
6answers and their results in a local JSON report.
7
8This package is agent-neutral, uses Node.js 22 or newer, and has no dependencies.
9It is released under the [MIT license](LICENSE). The package source is in this
10repository; an npm registry release is not required for the commands below.
12## Try the offline example
14From the Loooom repository root:
16```sh
17node plugins/context-evals/bin/loooom-context-eval.mjs \
18 --suite plugins/context-evals/examples/suite.json \
19 --context plugins/context-evals/examples/AGENTS.md \
20 --context plugins/context-evals/examples/SOUL.md \
21 --context plugins/context-evals/examples/ME.md \
22 --responses plugins/context-evals/examples/responses.synthetic.json \
23 --output /tmp/loooom-context-report.json
24```
26The example uses **handwritten synthetic answers**. It makes no network calls.
27It demonstrates 0/3 baseline cases and 3/3 context cases. These values do not
28measure a real model. Both the report and terminal output state that this is a
29synthetic replay. Choose a new output path for each run; existing files are never
30overwritten.
32A [saved synthetic report](examples/report.synthetic.json) is included for
33report viewers and local import tests. It was produced by the same replay CLI.
34Its string checks are demonstration data, not a model quality score.
36You can also install from this source checkout with
37`npm install -g ./plugins/context-evals`. This adds the `loooom-context-eval`
38command. To use the package with an agent, read [SKILL.md](SKILL.md) or copy the
39whole package directory into a skill directory supported by that agent.
40The CLI works without a skill loader.
42## Write checks for your context
44Start with preferences that have observable results. For example, check a
45specific command, required wording, a time zone, or a word limit. Keep prompts
46the same in both conditions.
48```json
50 "schemaVersion": 1,
51 "name": "My work preferences",
52 "cases": [
53 {
54 "id": "project-check",
55 "prompt": "How should I check this project?",
56 "assertions": [
57 { "type": "contains", "value": "npm run check" },
58 { "type": "notContains", "value": "Great question", "caseSensitive": false },
59 { "type": "maxWords", "value": 40 }
60 ]
61 }
62 ]
64```
66| Type | Rule |
67| --- | --- |
68| `contains` | The answer includes the supplied text as a substring. |
69| `notContains` | The answer does not include the supplied text as a substring. |
70| `exact` | The entire answer equals the supplied text, including whitespace. |
71| `maxWords` | The answer has no more than `value` whitespace-separated tokens. |
73Text checks are case-sensitive by default. Set `caseSensitive: false` to use
74Unicode lowercase conversion on both strings. The CLI does not trim text or
75remove Markdown before a text check. Text values must be non-empty. `maxWords`
76accepts an integer from 0 to 100000. An empty answer has zero words. Each case
77passes only when all of its assertions pass. Unknown fields, assertion types,
78and duplicate IDs are errors. There is no JavaScript evaluation, regular
79expression execution, or model-based judge.
81Substring checks have limits: an answer that says "do not use npm run check"
82still contains that command. Pair checks with review of the saved answers.
83`maxWords` is a count of whitespace-separated tokens, not linguistic words.
85## Run a model
87Use an endpoint that supports the OpenAI-compatible `POST /chat/completions`
88wire format. This is a transport interface; no specific model vendor or agent
89is required. The base URL must include the API prefix, such as `/v1`.
91Set these environment variables through your shell or secret manager:
93- `LOOOOM_EVAL_BASE_URL`: the API root. HTTPS is required, except on localhost.
94- `LOOOOM_EVAL_MODEL`: a model name accepted by that endpoint.
95- `LOOOOM_EVAL_API_KEY`: an optional bearer token. Omit it for an endpoint that
96 does not need authentication.
98Then omit `--responses`:
100```sh
101node plugins/context-evals/bin/loooom-context-eval.mjs \
102 --suite ./my-context-suite.json \
103 --context ./AGENTS.md \
104 --context ./SOUL.md \
105 --context ./ME.md \
106 --output /tmp/loooom-context-live-report.json
107```
109Live mode sends the selected files and prompts to your configured endpoint.
110The endpoint can charge for these requests. Reports contain the full context,
111prompts, and returned answers. Review them before sharing. Authorization
112headers are not saved. The CLI does not upload reports to Loooom.
114Each case uses two fresh chat requests. Both use the same prompt, model,
115`temperature: 0`, and output token limit. The baseline system message is:
117> Answer the user request. Follow any personal context supplied below.
119The context condition appends the selected files in command-line order, with
120file labels. The first case runs baseline first; the next runs context first,
121and this order alternates. No conversation history, tools, agent memory,
122automatic file discovery, or retries are added. Each selected file must exist.
124Options:
126- `--timeout-ms`: 100–120000 per request; default 30000.
127- `--max-output-tokens`: 32–8192; default 1024. Sent as `max_tokens`.
128- `--help` and `--version` need no input files or credentials.
130Limits: 100 cases, 20 assertions per case, 16 context files, 256 KiB of combined
131context, 32,768 characters per prompt, and 1 MiB per JSON input file. Provider
132responses are limited to 1 MiB and answer text to 128 KiB. Runs stop after five
133minutes, at the next request or checkpoint boundary. Calls are sequential.
134Providers that do not accept these chat completion fields need an adapter.
136## Replay saved answers
138Pass a JSON file with one pair of answers per suite case. Array order can differ
139from the suite order. All case IDs must match exactly.
141```json
143 "schemaVersion": 1,
144 "synthetic": false,
145 "description": "Answers exported from my prior run.",
146 "results": [
147 {
148 "caseId": "project-check",
149 "baseline": "Run npm test.",
150 "context": "Run npm run check."
151 }
152 ]
154```
156Set `synthetic: true` for invented or edited demonstration answers. Even with
157`synthetic: false`, replay cannot verify the answers' source or that the context
158was used to generate them. The report labels this limitation. A replay report
159does not count as a new model run.
161## Read the report
163Reports use `schemaVersion: 1` and contain:
165- `evidence`: `mode` (`live` or `replay`), `synthetic`, and a plain-language label.
166- `context`: each file's name, full content, and SHA-256, plus the exact combined
167 `text` and its SHA-256. File order and labels affect the combined hash.
168- `suite`: SHA-256 of the input file bytes and the parsed `definition`.
169- `replay`: replay file SHA-256 and definition, when used.
170- `config`: baseline instruction, model endpoint where used, request limits,
171 and condition order. No API key or request headers.
172- `results`: one row per case with `caseId`, `prompt`, `baseline`, and `context`.
173 Each completed condition holds its raw `output`, `rawResponse` for live
174 calls, `finishReason`, request body, timing, individual `assertions`, and
175 overall `passed` value. Each assertion has `passed`; word checks also have
176 an `actual` count.
177- `summary.baseline` and `summary.context`: `passed`, `total`,
178 `assertionsPassed`, and `assertionsTotal`.
179- `summary.delta`: `casesPassed` and `assertionsPassed`, calculated as context
180 minus baseline. It is `null` until all pairs are complete.
181- `status`: `running`, `complete`, or `error`, with start/end time and error
182 metadata.
184The CLI reserves a new report path before any provider call. It saves a valid
185JSON checkpoint after each completed answer, with file permissions `0600`.
186Each checkpoint uses an atomic rename. A failed request stops the run and keeps
187prior answers. Unfinished conditions remain `null`; missing answers are not
188counted as passed or failed. Invalid provider responses and HTTP error bodies
189are not scored or saved as answers. On a forced process kill, the latest report
190can remain `running`; that report is incomplete.
192Exit codes: **0** when all context cases pass, **1** when at least one context
193case fails, and **2** for input or runtime errors. A failed baseline is expected
194in many tests and does not cause exit 1.
196## Interpret the result
198The result measures only the checks you wrote. It does not establish general
199quality, truth, safety, or personal fit. A single paired run has no confidence
200interval, and temperature zero does not guarantee repeatable answers. Context
201can help one check and harm another. Read every failed assertion and the raw
202answers. Run independent repeats with new report paths to study variation.
204The first version does not run an installed agent's full tool loop, change your
205context files, or search for improved instructions. It measures a controlled
206chat request with and without supplied context.
208## Development
210From the repository root:
212```sh
213node --test tests/context-evals.test.mjs
214```
216The tests use offline fixtures and an in-memory HTTP transport substitute. They
217make no network calls and need no API keys. The suite checks replay evidence,
218validation, report preservation, paired request construction, provider errors,
219and timeout behavior.
221## Next Steps
223Run the offline example. Write checks for your own context. Then run your chosen
224model and review the saved answers before you change your instructions.
226### Browser report limits
228Loooom's `/evals` reader accepts reports up to 64 MiB, including interrupted
229`running` checkpoints. It labels incomplete runs and leaves missing answers
230unscored. For larger reports, split the suite into smaller runs or inspect the
231JSON locally. Reports contain request and response evidence, so their size
232can be much larger than the context files.