Superloom

Appearance

Developer-Environment Pitfalls (AI Journal) ​

Audience: AI coding agents (Cascade, Copilot, Cursor) and humans debugging a specific CI, test, or terminal failure. Not first-read material. Start with the sibling philosophy docs; come here only when a concrete failure mode needs a confirmed fix.

Shape: Every entry is Symptom → Cause → Lesson/Fix. When a new failure mode is discovered, add the entry here first (per the Golden Rule in AGENTS.md), then recompile the compact rule into AGENTS.md (a derived artifact, never edited directly).

Scope: This file covers the docs/dev/ domain: AI tool-bridge, CI/CD publishing, local module testing. Architecture-level pitfalls (module migration, refactors) live in docs/testing/pitfalls-migration.md.

On This Page ​

AI Terminal & Shell Bridge ​

Canonical journal of every shell, terminal, and tool-bridge failure mode that AI assistants (Cascade, Copilot, etc.) have hit while working on this codebase.

Why AI agents fail differently from humans ​

A human at a real terminal sees the prompt change to dquote> and fixes the quoting. An AI tool-bridge sees nothing, hits the proposed-command timeout, and the entire turn appears to "hang". The same root cause produces different symptoms depending on who is at the keyboard.

Three structural differences matter:

  1. No PTY interactivity: the bridge cannot type into a sub-prompt. Once the shell enters dquote>, heredoc>, quote>, cmdsubst>, or bracket> continuation mode, there is no way out except killing the process.
  2. Output is captured asynchronously: anything that paginates (less, more, git log without --no-pager, man) blocks forever waiting for keyboard input that will never arrive.
  3. The current working directory is per-call: every run_command resets to whatever Cwd the tool-call specifies. There is no persistent session between calls, so cd in one call has no effect on the next call.

Every pitfall below is a specific consequence of one of these three. The mitigation is always the same: avoid asking the shell to do anything interactive.

Shell-bridge pitfalls ​

S1. Heredocs hang the bridge ​

Symptom: A cat <<'EOF' ... EOF command never returns. The user sees "no progress" until the tool-call times out.

Cause: Heredoc content with backticks, $, !, parentheses, or unterminated quotes can confuse zsh's parser. The shell enters a continuation prompt waiting for more input. There is no way to send the closing token through the bridge.

Fix: Never use heredocs through the bridge.

  • For file content: use write_to_file (or edit) directly.
  • For terminal-only sinks (e.g., appending to a gitignored file): write the content to a temp file with write_to_file, then cat /tmp/file >> /path/to/target.

S2. Multi-line git commit -m "…" ​

Symptom: The command appears in the approval popup with the closing " on a different line. After the user approves, nothing happens - shell sits in dquote> mode.

Cause: zsh treats every newline inside an open "…" string as part of the string. With special characters in the body (`, $, !, (...)), the parser also opens nested continuation states. None of them can be closed through the bridge.

Fix: Always pass commit messages on a single logical line.

NeedPattern
Single-line summarygit commit -m "feat(module): one-line summary"
Summary + body paragraphsgit commit -m "feat(module): summary" -m "Paragraph one." -m "Paragraph two."
Long structured bodywrite_to_file to /tmp/commit-msg then git commit -F /tmp/commit-msg && rm /tmp/commit-msg

The same rule applies to gh pr create, aws ssm put-parameter, and any other tool that takes a quoted message.

S3. Multi-line content embedded inside any quoted shell argument ​

Symptom: Same as S2 but for non-git commands. The approval popup renders the multi-line argument unreadably; after approval, the shell hangs.

Cause: Any time a run_command payload spans multiple lines because of an embedded quoted string, the shell-bridge sees the same dquote> failure mode.

Fix: Route the multi-line content through write_to_file to a temp file first, then have the shell read from the file with a single-line command.

S4. Backticks inside a double-quoted argument ​

Symptom: git commit -m "feat: closes issue \#123`"either runs the contents of the backticks as a command substitution, or hangs incmdsubst>` mode.

Cause: Inside "…", backticks always start command substitution. Escaping them with \` only sometimes works depending on shell version.

Fix: Use single quotes around the message, or replace the backticks with single quotes.

S5. Tilde expansion inside quotes ​

Symptom: cp ~/.npmrc "…" works; cp "~/.npmrc" "…" silently fails with "no such file".

Cause: Tilde is a shell metacharacter that only expands outside quotes (and not in all positions even there).

Fix: Use $HOME (which is a regular variable and expands inside double quotes) or omit the surrounding quotes for the path.

S6. Escaped brackets inside a regex character class make a sweep silently match nothing ​

Symptom: A new sweep grep returns no output and is read as "clean", but the pattern it was written to catch is demonstrably present in the file. The sweep passes forever and enforces nothing.

Cause: Inside a POSIX bracket expression a backslash is a literal character, not an escape. A class written as [A-Za-z_.\[\]'] therefore closes at the first unescaped ] - the one intended as an escaped literal - so the class becomes {A-Za-z, _, ., \, [} and the remaining '] is parsed as pattern text. The regex still compiles, grep still exits cleanly, and nothing ever matches.

Fix: Never put escaped brackets in a character class. Prefer a negated class that needs no escaping, and always prove a new sweep fires before trusting it:

bash
grep -nE "typeof [^ ]+ (!==|===) '(number|function)'" path/to/file.js

Lesson: Every new sweep grep is validated against a file that is known to contain a violation, before it is added to a workflow's sweep battery. A sweep that has never been shown to produce a hit is unverified, and a silently-empty sweep is worse than no sweep because it reports the codebase as conformant. This is the inverse of the usual grep convention where empty output is the pass condition: empty output only means "pass" once the pattern has been proven capable of failing.

Process and pager pitfalls ​

P1. Output paginators (less, more, man, vi) ​

Symptom: A command runs forever with no visible progress.

Cause: Paginators wait for keyboard input. There is no keyboard.

Fix: Never invoke an interactive viewer. The environment runs commands with PAGER=cat so most pager-aware tools (git, systemctl, journalctl) cooperate, but commands that ignore PAGER need explicit flags:

CommandFlag
git loggit log -n 20 (or --no-pager)
git diffgit --no-pager diff
journalctljournalctl --no-pager
systemctl statussystemctl --no-pager status …

P2. Long-running foreground processes ​

Symptom: npm run dev, node server.js, tail -f, docker compose logs -f block the bridge until they exit, which they never will.

Cause: A Blocking: true run_command waits for process exit. A foreground server never exits.

Fix: Use Blocking: false with a small WaitMsBeforeAsync (e.g., 2-3 seconds) so the tool returns after the startup output is captured. Then later, use command_status with the returned CommandId to fetch more output. Always remember to stop the background process at the end of the task.

P3. npm install of a watch-script package adding a postinstall daemon ​

Symptom: npm install completes, but a hidden postinstall script forks node …watcher.js & that keeps file descriptors open.

Cause: Some packages (rare, but they exist) start a background watcher in postinstall. The bridge sees the install command return but the descriptor lingers.

Fix: This is unusual in this codebase. If it ever happens, kill the dangling process via pkill -f <package> or restart the IDE.

P4. Command output exceeding the bridge buffer ​

Symptom: npm test finishes but the captured output is truncated mid-line.

Cause: Very long stdout streams can exceed the IDE's capture buffer, especially when test runners print verbose output for hundreds of tests.

Fix: Pipe to tail -N or grep to keep only what matters: npm test 2>&1 | tail -30, npm test 2>&1 | grep -E "^(ℹ|✖|✔)". The full log is still in npm-debug.log if needed.

P5. Background command + mismatched CommandId → false "command failed" conclusion ​

Symptom: A short, quick command (git commit, git push, gh api ...) is launched. Subsequent command_status polls return command <N> not found in trajectory, and read_terminal returns empty. The agent concludes the command failed and asks the user to run it manually - but the command actually succeeded (the commit is in git log, already pushed to origin).

Cause: The command was launched in background mode (Blocking: false) and the bridge returned a Background command ID (e.g. 506). The agent then polled command_status with a guessed or adjacent ID (395, 397, …) that never existed, producing the not found in trajectory error. Short commands also complete and clear before read_terminal can capture anything, so the terminal read comes back empty. None of this is evidence of failure - only that the command's output was never observed.

Fix:

  • For short, quick commands (git, gh, npm view, file checks), always use Blocking: true. Output is returned synchronously in the tool result.
  • If a command genuinely must run in the background, poll command_status with the exact CommandId the launch returned - never a guessed or incremented number.
  • Before reporting that a state-changing command failed, verify the actual state (git log --oneline -3, git status --short, gh api "$BASE/versions"). Absence of captured output is not proof of failure.

Reserve Blocking: false for genuinely long-running foreground processes (servers, tail -f, docker compose logs -f) per P2 - not for fast one-shot commands whose output you need immediately.

Working-directory pitfalls ​

W1. Missing Cwd runs from the repo root ​

Symptom: npm install reports ETARGET No matching version found for @superloomdev/... even though the package version is correct.

Cause: The bridge resets to the repo root for every run_command unless Cwd is explicitly passed. The repo-root package.json has a different dependency tree from each module's _test/package.json.

Fix: Every module-scoped command (npm install, npm test, docker compose …) must pass Cwd set to the module's _test/ directory.

bash
# Wrong - silently runs from repo root
# run_command: { CommandLine: "npm install" }

# Right
# run_command: { CommandLine: "npm install", Cwd: ".../js-server-helper-foo/_test" }

W2. cd <path> && <command> does not persist between calls ​

Symptom: A first call does cd src/foo && npm install; a second call assumes the cwd is src/foo and does npm test, but it runs from the repo root.

Cause: Each run_command is a fresh shell. There is no session.

Fix: Always pass Cwd to every call. Never rely on a previous cd. The user's AGENTS.md explicitly says never propose a cd command for the same reason.

W3. Relative paths in tool calls ​

Symptom: read_file({ file_path: "src/foo.js" }) fails or reads from the wrong directory.

Cause: Most tools require absolute paths. Relative paths are undefined behavior.

Fix: Always pass absolute paths to file tools (read_file, edit, write_to_file, find_by_name, grep_search).

Docker lifecycle pitfalls ​

D1. Manually starting Docker before npm test ​

Symptom: Bind for 127.0.0.1:NNNN failed: port is already allocated, or tests fail immediately with ECONNRESET.

Cause: pretest runs docker compose down -v --remove-orphans for its own compose project name; it does not touch a manually started container. Then pretest tries to bind the same port and fails.

Fix: Pick one owner of the Docker lifecycle: pretest already owns it. Locally and in CI, never run a separate docker run or docker compose up for the same service before npm test.

D2. Stale containers from a crashed prior run ​

Symptom: First test in a fresh session fails because a container from yesterday is still up but has stale state.

Fix: pretest already runs docker compose down -v --remove-orphans. If pretest itself fails on the start step, run that command manually from the same _test/ directory.

D3. docker compose up -d without --wait ​

Symptom: Tests start before the database accepts connections; intermittent ECONNREFUSED on the first request.

Fix: Always use docker compose up -d --wait. The healthcheck must be a real readiness probe (see docs/dev/testing-local-modules.md "Healthcheck Philosophy").

D4. docker compose --wait returns immediately for a service with no healthcheck ​

Symptom: --wait reports the container as Healthy 0.5 s after start, even though the application inside is still initializing.

Cause: When a service has no healthcheck: block, Docker treats "container is running" as healthy. --wait honours that.

Fix: Either define a real healthcheck, or have the test code retry the first connection a few times. DynamoDB Local is the typical case here - the image has no curl/wget/nc to probe with, so the test setup absorbs a brief retry instead.

Test-environment pitfalls ​

T1. AWS SDK calls without dummy credentials ​

Symptom: A test that exercises the AWS SDK takes 1-2 seconds and fails with no clear error.

Cause: With no credentials in the env, the SDK walks the default credential provider chain. The last step of that chain is the EC2/ECS instance metadata service at http://169.254.169.254. There is no metadata service on a developer machine or a GitHub Actions runner, so the chain times out.

Fix: Every AWS test must inject dummy credentials via the _test/package.json test script:

json
"test": "AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 node --test test.js"

The dummies do not need to be valid - they just need to exist so the SDK skips the metadata lookup.

T2. node --test test.js directly, without pretest ​

Symptom: Tests fail immediately with connection errors against the database or queue.

Cause: pretest did not run, so the container is not up.

Fix: Always use npm test. The lifecycle scripts exist for a reason.

T3. Concurrent top-level describe blocks ​

Symptom: "test did not finish before its parent and was cancelled".

Cause: Node's built-in test runner runs top-level describe blocks concurrently. Suites that share lazy-init state (DB pool, AWS SDK client) race each other on the first call.

Fix: Wrap stateful suites in a single outer describe('Module', { concurrency: false }, …). See docs/dev/testing-local-modules.md "Test Concurrency".

T4. Healthcheck passes during a transient-ready window ​

Symptom: Tests pass locally, fail in CI with Connection lost: The server closed the connection.

Cause: A healthcheck that returns "ready" too early. MySQL's two-phase init is the classic case: mysqladmin ping -u root passes during phase 1, then the server restarts in phase 3 and drops every live connection.

Fix: Probe with the credentials, database, and transport the tests will use. See docs/dev/testing-local-modules.md "Healthcheck Philosophy".

File-tool vs terminal pitfalls ​

F1. Using cat to read large files ​

Symptom: Output is truncated at the bridge buffer limit, or the call times out.

Fix: Use read_file with offset + limit. Never cat files larger than ~200 lines.

F2. Using sed / awk / tr to edit files ​

Symptom: The replacement either misses the target line, mangles whitespace, or partially succeeds.

Cause: Stream editors are error-prone for surgical edits, especially when the target string contains regex metacharacters or quotes.

Fix: Use the edit or multi_edit tool with old_string set to the exact unique substring. The tool guarantees an exact match or a clear error.

F3. Editing a file with a heredoc through cat > file <<EOF ​

Same root cause as S1. Always use write_to_file or edit instead.

F4. Reading gitignored files via read_file ​

Symptom: read_file returns "file is gitignored" for __dev__/.env.dev.

Cause: The IDE's file tools intentionally refuse gitignored paths to prevent accidentally exposing secrets in chat output.

Fix: For gitignored files, use cat /path/to/file via run_command (which is allowed but visible in the approval popup). For writes, write to a temp file via write_to_file and then mv or cat >> it into place.

Auto-run / safety pitfalls ​

A1. Auto-running a destructive command ​

Symptom: The agent flips SafeToAutoRun: true on rm -rf, git push --force, docker volume rm, or npm publish and the user has no chance to review.

Cause: The agent over-trusts a previous successful execution and decides the next call is "obviously fine".

Fix: Auto-run is reserved for read-only operations and idempotent reads (git status, git log -n 20, npm test, docker ps). Anything that mutates state on disk, in a remote registry, or in a long-running service must always require user approval, even if the user has previously approved similar commands. The user's AGENTS.md explicitly forbids npm publish regardless.

A2. Running npm publish directly ​

Symptom: A package is published from the developer's laptop instead of from CI, with the wrong author or unsigned provenance.

Cause: The agent saw a successful test run and decided to publish.

Fix: Publishing in this codebase is CI-only via .github/workflows/ci-publish-helper-modules.yml. Bumping the version in package.json and pushing to main is the only trigger. The CI workflow's content guard skips only when the packed shasum matches what the registry already serves (entry 26).

A3. Force-pushing to a shared branch ​

Symptom: A git push --force rewrites main's history, losing other contributors' commits.

Fix: Never force-push without explicit user approval. The user's AGENTS.md lists this as Never in the boundaries section. Use --force-with-lease if the user explicitly asks to amend.

A4. Modifying .env files ​

Symptom: A pre-existing .env file gets overwritten and the developer loses their local credentials.

Fix: The agent's allowed write locations are spelled out in AGENTS.md "Boundaries". .env is in the Never category (except __dev__/.env, which is the user's personal workspace). For new env keys, update .env.example files only.

Verification-integrity pitfalls ​

Failures where the agent's report is wrong rather than its code. These are the most expensive class because they consume the user's trust in every other claim.

V1. Reporting a gate as passed without running it ​

Symptom: The agent states a validation workflow is complete ("/finalize-docs converged", "docs verified") after having only made edits and run one adjacent command. A later audit finds defects the gate's own passes were designed to catch.

Cause: Substituting a cheap proxy signal for the gate. A green website build gets treated as evidence for link integrity, terminology consistency, and rule mirroring, none of which it tests.

Fix: A gate is passed only when its steps were individually executed and each produced its required evidence. If the passes were not run, the report says so. Name the specific commands and file reads that produced each count. When a workflow demands per-pass evidence, an unproduced count is a failed pass, not an assumed one.

V2. Inventing a plausible mapping to close a finding ​

Symptom: Asked to fix a table row that pointed at a non-existent target, the agent wrote a different target that exists, with a parenthetical implying the content lives there. It did not. The visible error became an invisible false assertion.

Cause: Optimizing for a row that looks resolved instead of one that is true. A vague parenthetical ("folded into the X rules") reads as authoritative and survives review.

Fix: Read the destination and confirm the content is actually present before asserting a mapping. When the honest answer is "not represented anywhere", write that; an explicit gap is a working input to the next gate, while a fabricated mapping silently suppresses it. Applies to any derived-artifact index: section maps, coverage tables, traceability matrices.

V3. Green build accepted as anchor evidence ​

Symptom: Renamed a heading, updated its inbound cross-file link, left the same-file On This Page anchor stale. The site built clean three times.

Cause: VitePress does not fail on unresolved same-file fragments, so the build cannot detect them.

Fix: Verify anchors by deriving slugs from the target file's actual headings and comparing, including a file's own table of contents. Renaming a heading invalidates every inbound anchor, and the ones in the same file are the easiest to miss.

V4. Fabricated precision in a gate report ​

Symptom: The 2026-07-31 P3 run of /finalize-docs reported Rules in source inventory: 423 and Rules mirrored in AGENTS.md: 417 without counting either figure. The 6 findings the run surfaced were genuine and independently verified; the totals were not. A reader comparing the report to the actual source files would find no extraction that produced 423 and no count that produced 417.

Cause: The gate was run honestly - findings were checked, evidence was produced for each - and then the result was decorated with measured-looking numbers that were estimated rather than counted. This is distinct from V1 (the gate was not skipped) and V2 (no mapping was invented). The softer failure mode is that a real pass produced real findings, and then precision was added after the fact to make the report look more rigorous than the method supports.

Fix: Any number in a gate report is a claim requiring the same evidence as a pass/fail verdict. A total that is not the arithmetic sum of counted parts must not be emitted. The P3 output block now enforces this with the identity A + B + C + D = N; a total that does not equal the sum of its parts is a detectable error rather than a plausible-looking number.

V5. A transitional repository policy absorbed as permanent constitution doctrine ​

Symptom: The constitution stated that a shasum mismatch at the publish guard is fixed by deleting the registry version, and that the version must never be bumped to clear one. Both were true only of repositories running a temporary pre-release convention that pins every package at one version. Every other repository read the rule and got the destructive remedy: deleting a version consumers had already resolved, breaking their lockfiles. A second copy of the same leak sat in agent-configuration.md, which asserted a package version floor as a framework rule.

Cause: A journaling run captured a real failure while a temporary policy was in force, and the policy travelled into the rule as an unstated premise. Nothing in the sentence marked it as conditional, so it read as universal. The existing passes cannot see this: each statement of the rule agreed with the others, the links resolved, the tables matched their detail sections, and the site built. Rule agreement checks whether repeated statements match each other, not whether the matched statement sits at the right layer.

Fix: A rule that only holds under a policy names the policy and names where the policy is declared. Split the two halves. The permanent half states the mechanism: a guard compares a fingerprint of content. The policy-dependent half states the remedy and defers it to the repository's own standing-rule file. The default remedy is the unsurprising one, so a repository that declares nothing still gets correct behavior. When journaling a failure found under a temporary convention, write the convention into the entry's cause as a condition, never into the rule as a premise. The detection question at validation time: for every "never do X" in the constitution, name the repository where it is false. If one exists, the rule is policy-dependent and mislayered.

V6. Raw fixture comparisons conceal a broken runtime pipeline ​

Symptom: Profile tests and lint pass while the real engine rejects every profile, a bridge overwrites dotted siblings, and rendered typography overrides authored metrics. Tests for an unavailable package export skip after catching any import error, so the acceptance report stays green.

Cause: Tests compare selected raw literals rather than executing the engine, bridge, validation, utility generation, and component branches together. Negative controls compare two reference values instead of mutating the implementation input and exercising the real assertion. Static searches miss dynamic token names and lowercase semantic aliases. Filtering test output through a shell pipeline can also replace the test process's failure status with the filter's success status.

Fix: Test the actual public pipeline and assert emitted and rendered values. Preserve strict missing-token failures, including dynamic branches. Required import errors remain failures, not automatic skips; park dependent acceptance until its registry artifact exists. Run commands without output filters or preserve their exit status with pipefail. After a same-version release, verify the committed consumer lockfiles, not only the regenerated working copies. Existing smoke counts do not certify newly added behavior.

V7. A stale dev server serves output from a previous dependency state ​

Symptom: A Vite dev server started hours ago on port 5173 continues serving stale module state after a dependency is republished and node_modules is refreshed. A new vite invocation silently moves to port 5174 or 5175 because the configured port is occupied, so the developer inspects the wrong server and believes the build is current. Playwright, which builds its own production preview on port 4173, passes its tests while the dev server the developer is looking at serves a completely different layout.

Cause: Vite's dev server caches transformed modules in memory and does not detect that the installed packages changed underneath it. Its port-increment behavior makes the stale server and the fresh server coexist without warning. A production preview build produces correct output from the current tree, but that proves nothing about the dev server the developer is interacting with.

Fix: Production Playwright must always use a fresh server (reuseExistingServer: false in CI and npm run verify). A build identity endpoint (git SHA + lockfile hash) lets a freshness check compare the served identity to the identity computed before launch. A dev:fresh command checks the configured port, fails loudly with PID and remedy on a mismatch, and never silently increments the port. Killing a stale process remains an explicit human action.

V8. Presence-only E2E tests pass while the rendered UI is geometrically broken ​

Symptom: Tasks and Notes E2E suites assert that expected text appears, that adding data works, and that navigation works. All pass. The Notes banner renders at 42px/60px (a stale caption02 token), producing a 300px banner and a 762px page. A typography regression suite with an 84px global ceiling also passes because 42 is under 84. No test measures computed font size, line height, overflow, overlap, or page geometry.

Cause: Text presence and interaction success are necessary but not sufficient. A global font-size ceiling catches only extreme blowups and lets any wrong-but-smaller value through. Computed typography and layout geometry are deterministic in a fixed browser environment and can be asserted exactly, but only if a test reads them.

Fix: Application-level acceptance uses four complementary gates: contract (exact generated token values), functional readiness (response, mount, console/request errors, interaction), geometric layout (exact typography, spacing, overflow, overlap, reflow at fixed viewports), and visual regression (toHaveScreenshot baselines). Deterministic values use exact assertions, not ranges. A global ceiling does not replace a semantic expectation.

V9. Concurrent verification processes corrupt a shared generated test artifact ​

Symptom: A standalone browser suite reports a failure from a production mutation it did not plant, while the fire suite reports that unrelated mutations did not fire or recover. Rerunning both commands sequentially passes.

Cause: The fire harness (npm run fire) and the browser harness (a bundle rebuild plus Playwright) share one generated bundle. The fire process mutates production source and rebuilds that bundle for each negative control; a concurrent browser build overwrites it between the fire process's mutation, assertion, restoration, and clean recovery check. Each process then reads an artifact produced from a different source state than the one it believes it owns.

Fix: Commands that mutate source or rewrite a shared generated artifact run exclusively and sequentially. In particular, never overlap a fire suite with bundle generation, Playwright, snapshot generation, or another fire suite when they share an output path. Parallelize only gates whose complete read/write sets are disjoint; a distinct process is not isolation when the filesystem artifact is shared.

Rule Delivery ​

1. AI attribution trailers appear in commits despite a documented ban ​

Symptom: Commits in codebase-js-helper-modules carried Co-Authored-By: Devin and Generated with [Devin] trailers, even though the no-attribution rule already existed in codebase-superloom/AGENTS.md:78 and docs/ai/agent-configuration.md:115.

Cause: codebase-js-helper-modules had no AGENTS.md at all. The rule was never in the agent's context while it committed there. The CLI's built-in commit template, which appends attribution trailers by default, won because no project-level rule overrode it. A rule that exists only in the constitution repo is not in force in the repos where work actually happens.

Fix: Every repo an agent commits to must carry the no-attribution rule in its own AGENTS.md, stated in full (not by reference). The rule must explicitly say it overrides any tool's built-in commit template. Three of five workspace repos had no AGENTS.md; all four now have self-contained copies with the rule stated verbatim.

Generalization: A rule's reach equals the set of files an agent actually reads. A rule present in a repo the agent never opens during a session is invisible. Delivery is not authoring; a rule undelivered is a rule unenforced.

CI/CD Publishing ​

Each entry below maps a CI symptom to its root cause and the durable fix. The sibling philosophy doc is cicd-publishing.md; this section is the journal of real failures that shaped those rules. For the local CI parity contract, see local-ci-parity.md.

1. Bind for 127.0.0.1:NNNN failed: port is already allocated in CI ​

Cause: A workflow step started a container with docker run -d --name foo -p 127.0.0.1:NNNN:NNNN ... before the test step ran npm test. pretest then runs docker compose down -v (which does not touch the standalone container) followed by docker compose up -d --wait, which collides on the same port.

Lesson: Pick one owner of the Docker lifecycle. pretest already owns it both locally and in CI. Do not duplicate it with a workflow-level docker run step or a services: declaration that targets the same port - pick one and remove the other.

2. PROTOCOL_CONNECTION_LOST only in CI ​

Cause: The healthcheck passed too early. On the developer's laptop the service finished init fast enough to be truly ready when the healthcheck passed; on a slower CI runner, the false-positive moment was wide enough that the test connected before init was complete, and the service then dropped the connection during a real init step.

Lesson: See testing-local-modules.md -> Healthcheck Philosophy. The healthcheck must probe at the same level the test will use (credentials, database, transport). Add start_period and enough retries that the total budget covers a cold-pull, cold-start initialization.

3. Push triggers tests but not publish, even though version was bumped ​

Cause (historical): The previous detect logic compared HEAD~1:package.json to HEAD:package.json. After a force-push or a reset that left the version the same on both sides, the diff was empty and publish was skipped - even when the registry did not have the package.

Fix: The detect job now asks the registry directly instead of diffing git history. It compares the packed shasum against npm view <name>@<version> dist.shasum, so publish is scheduled whenever the version is absent or the content differs (entry 26).

4. CI runs tests for every module on every commit ​

Cause: test_modules was incorrectly populated - typically because the regex used \w+ instead of [\w-]+ and matched module paths greedily, or because someone pushed a single commit that touched every module's package.json.

Lesson: Keep the path regex hyphen-aware: src/helper-modules-[\w-]+/js-[\w-]+. If a single commit truly does touch every module (e.g., a sweep), wide test coverage is the correct outcome.

5. npm publish runs but 409 You cannot publish over the previously published versions ​

Cause: A workflow that publishes on every main push, without checking version bump or registry state.

Lesson: Use the unified detect -> publish-* pipeline. Both gates compare the packed shasum against the registry's dist.shasum, which is the canonical way to avoid this error. A gate that checks version presence alone swallows the failure instead (entry 26). Do not rebuild a separate publish workflow.

6. 403 Forbidden on npm publish ​

Cause: Missing permissions: packages: write on the publish job, or repository-level workflow permission set to "Read repository contents permission".

Lesson: Both must be set:

  1. Job-level permissions: { contents: read, packages: write }
  2. Repo-level Read and write permissions in Settings -> Actions -> General

7. CI test fails with npm error notarget No matching version found for @superloomdev/... ​

Cause: The CI step ran npm install from the wrong directory (typically the module root instead of _test/). Each _test/ directory has its own package.json with its own dependency tree. The repo-root package.json does not declare the test deps.

Lesson: In CI, set working-directory: to the module root and use cd _test && npm install && npm test for the test step. The same rule applies locally - always pass Cwd to _test/ for AI agents and scripts.

8. CI test fails with ERR_MODULE_NOT_FOUND for a helper that exists in the repo ​

Symptom: A test-* CI job fails with a Node.js ERR_MODULE_NOT_FOUND stack trace pointing to a file inside another helper module (e.g. js-server-helper-nosql-mongodb/mongodb.js), despite that module being present in src/. The error occurs even though file:../../js-server-helper-nosql-mongodb is listed in the _test/package.json dependencies.

Cause: file: path dependencies copy the directory contents at npm install time but do not run npm install inside the linked package. In CI, the runner checks out a fresh clone; the linked helper's own node_modules/ are absent, so any import inside it that needs its own npm deps (e.g. the mongodb driver) fails immediately with ERR_MODULE_NOT_FOUND.

This works locally only because the helper was previously installed in its own directory during development. CI never does that.

Lesson: Never use file: paths in _test/package.json for helper modules that have their own npm dependencies. Use registry version ranges ("^1.0.0") instead. The file:../ self-reference (pointing to the module under test) is the one legitimate exception. npm installs it as a directory link and the module itself has no transitive runtime deps outside what the test loader provides. For every other shared helper (storage, database, cloud), always pin to the published registry version.

Quick rule: file: is allowed only for "[module-under-test]": "file:../". Everything else must be a registry semver range.

9. Tests Fail in CI ​

The publish job has needs: [detect, test-*], so it never runs if tests fail. Fix the tests and push again. The detect job will pick the same set of unpublished modules and try again.

10. Invalid workflow file: ... You have an error in your yaml syntax on line N ​

Cause: A bash assignment inside a run: | block scalar contained an embedded literal newline:

yaml
run: |
  PUBLISH_MODULES="$PUBLISH_MODULES$MODULE
"

YAML block scalars (|) require every non-empty line to be indented at least to the block's indent level. The closing " on its own line had zero leading spaces - less than the block's indent - which terminates the block scalar early and fails YAML parsing.

Lesson: Never embed a literal newline inside a bash string assignment within a YAML run: | block. Use bash's $'\n' escape (or printf '%s\n', or a bash array) so every line of YAML respects the block's indentation:

yaml
run: |
  PUBLISH_MODULES="${PUBLISH_MODULES}${MODULE}"$'\n'

This applies anywhere YAML uses block scalars: GitHub Actions run:, Docker Compose command:, Helm chart values, CI configs, etc.

11. publish-* jobs silently skip when an upstream publish-* is also skipped, breaking the test→publish chain ​

Symptom: A test-* job runs and succeeds. The next publish-* job in the chain is reported as skipped (zero steps, zero seconds). Multiple downstream test-* jobs then run against a stale registry version and fail with cryptic runtime errors like TypeError: mongo.createIndex is not a function. The bumped helper version that supplied the new API was never actually published.

Cause: GitHub Actions evaluates an implicit success() check on every job that does not start its if: with a status-check function (always(), failure(), cancelled(), !cancelled()). And success() is transitive. It returns false if any job in the upstream needs graph (not just the direct needs) has the result skipped, failure, or cancelled. Quote from GitHub's docs: "If a job fails or is skipped, all jobs that need it are skipped unless they use a conditional expression that causes the job to continue."

A strictly-sequential test→publish pipeline like ci-publish-helper-modules.yml mixes:

  • test-* jobs that use if: always() && !cancelled() && ... → run regardless of upstream skips
  • publish-* jobs that previously used if: >- needs.detect.outputs.publish_modules != '[]' && contains(...) → no override, so the implicit success() applied

In fresh-state recovery runs (every module needs publishing), nothing in the chain is skipped, so this never fires. In steady-state runs where some modules are already on the registry, those modules' publish-* jobs legitimately skip; that skip silently propagates downstream, disabling every subsequent publish-* even though their direct needs (detect + the matching test-*) all succeeded.

Concrete failure that surfaced this: A run with publish_modules = [auth, logger, nosql-aws-dynamodb, nosql-mongodb, verify]:

  • publish-storage-aws-s3 correctly skipped (s3 already on registry, not in publish_modules)
  • test-nosql-aws-dynamodb ran and succeeded (uses always())
  • publish-nosql-aws-dynamodb SKIPPED (even though it was in publish_modules and its direct needs succeeded) because success() walked the transitive chain back to publish-storage-aws-s3 and saw skipped
  • All downstream publish-* skipped for the same reason
  • mongodb@1.1.0 and dynamodb@1.1.0 never reached the registry
  • Then test-verify, test-logger, test-auth ran with npm install resolving ^1.0.0 to the stale registry 1.0.0, which lacked the createIndex / createTable / deleteRecordsByFilter APIs the 1.1.0 source code calls. TypeError at first use

Lesson: Every publish-* job in a chained pipeline must override the implicit success() check and assert its own dependencies explicitly:

yaml
publish-foo:
  needs: [detect, test-foo]
  if: |
    !cancelled() &&
    needs.detect.result == 'success' &&
    needs['test-foo'].result == 'success' &&
    needs.detect.outputs.publish_modules != '[]' &&
    needs.detect.outputs.publish_modules != '' &&
    contains(needs.detect.outputs.publish_modules, 'js-server-helper-foo')

Two parts that both matter:

  1. !cancelled() && disables the implicit transitive success() check (GitHub's recommended alternative to always() for normal jobs, per the official docs).
  2. needs.detect.result == 'success' && needs['test-foo'].result == 'success' restores the safety the implicit check used to give us, but scoped to the direct needs only. Hyphenated job IDs require bracket notation (needs['test-foo'], not needs.test-foo; the latter is parsed as subtraction).

Defence in depth: pin _test/package.json to the version your code actually requires. When a helper bumps its own version (e.g. mongodb@1.1.0 adds createIndex) and downstream modules start calling the new API, every consuming _test/package.json must pin that same ^1.1.0, not the older ^1.0.0. Two reasons:

  • If the upstream publish ever fails, ^1.1.0 causes npm install to fail with a clean E404 No matching version instead of installing the stale 1.0.0 and surfacing as a TypeError deep inside a test.
  • The version pin documents the hard floor required by the source. Anyone reading the test deps sees exactly what the module needs.

Quick rules:

  • _test/package.json registry pins must match the API surface the source code uses, not the lowest published version.
  • Every publish-* job in a chained workflow must start its if: with !cancelled() (or always(), failure(), cancelled()) and re-assert its direct needs.<job>.result == 'success'. Otherwise a single skipped sibling silently disables the rest of the publish chain.

12. Main module and its adapters are both unpublished: bootstrap order ​

Situation: Any module that ships a main package plus separate adapter packages (e.g. js-server-helper-auth + js-server-helper-auth-store-sqlite) can reach a state where both the main module and one or more adapters have unpublished breaking changes. Neither side can reference the other via a registry semver range because the version does not exist yet.

The correct sequence:

  1. Test adapters locally against a file: path of the main module. In each adapter's _test/package.json, temporarily point the main module dependency at the local checkout (file:../../js-server-helper-<name>). Run the full adapter test suite. This validates the new API contract end-to-end without touching the registry.

  2. Publish the main module (bump version in package.json, push to main; CI publishes it).

  3. Switch adapter _test/package.json deps from file: to the newly published registry version (e.g. "^2.0.0"). Re-run the adapter test suite against the live registry version. This confirms the package round-trips correctly through npm pack/publish and resolves correctly for real consumers.

  4. Publish the adapters (bump versions, push to main; CI publishes them).

Why the re-test in step 3? file: deps are copied at npm install time, bypassing npm's pack/unpack pipeline entirely. A module that works locally via file: can silently fail after publishing if the files field in package.json is wrong, a required file is gitignored, or a main entry resolves differently after packing. Step 3 catches this class of error before any consumer is affected.

Lesson: The temporary file: reference is the one legitimate exception to the rule that _test/package.json only uses file: for the module under test itself. It is only valid during the bootstrap window and must be replaced with a registry pin before the adapters are published. Add a comment so the intent is clear:

json
"@superloomdev/js-server-helper-<name>": "file:../../js-server-helper-<name>"

Replace with "^<version>" after the main module is published (step 3).

14. test-verify CI fails with Cannot find module '../stores/sqlite' after stores/ was deleted ​

Symptom: test-verify in CI fails with ERR_MODULE_NOT_FOUND pointing to ../stores/sqlite inside the verify module's own _test/test-sqlite.js. The stores/ directory was intentionally deleted as dead code in the same commit that introduced standalone js-server-helper-verify-store-* packages.

Cause: Two related problems compounded:

  1. The internal test file _test/test-sqlite.js still referenced the deleted ../stores/sqlite path. It was not updated to use the npm package @superloomdev/js-server-helper-verify-store-sqlite.

  2. The five js-server-helper-verify-store-* adapter packages had no CI jobs at all. ci-publish-helper-modules.yml skipped directly from publish-sql-postgres to test-verify. This meant the adapters were never published to the registry, so _test/package.json could not resolve them even after the test file was fixed.

Additionally, the adapter package.json files had been (incorrectly) set to private: true, which would have prevented npm publish even if CI jobs existed.

Fix:

  1. In _test/test-sqlite.js: replace import('../stores/sqlite') with import('@superloomdev/js-server-helper-verify-store-sqlite').
  2. In _test/package.json: add "@superloomdev/js-server-helper-verify-store-sqlite": "^1.2.0" to dependencies.
  3. In all five adapter package.json files: set "private": false so npm publish works.
  4. In ci-publish-helper-modules.yml: insert ten new jobs (modules 17-21), one test-verify-store-* + publish-verify-store-* pair per adapter, in the sequential chain before test-verify. Update test-verify's needs to [detect, publish-verify-store-dynamodb].

Lesson: Deleting an internal directory (stores/) that is still referenced by the module's own test suite, while simultaneously introducing replacement standalone packages, requires two coordinated changes: (a) update every import that pointed at the old path, and (b) ensure the new packages exist in the CI pipeline and are published before any consumer runs npm install. A quick grep for the deleted path before committing prevents the first problem; checking whether each new adapter has CI jobs prevents the second.

13. CI fails on lint after local tests pass: pre-publish checklist not followed ​

Symptom: npm test in _test/ passes locally (all green). The version is bumped and pushed to main. CI runs npm run lint from the module root as a separate step before tests and fails with no-trailing-spaces or a stale eslint-disable-line directive. The publish job never runs even though all functional tests would have passed.

Cause: The local test command (node --test test.js from _test/) never invokes ESLint. Lint is a separate npm run lint script defined in the module root's package.json, not in _test/package.json. CI always runs lint before tests; local workflow often only runs tests. Any whitespace issue or stale disable comment that was present (or introduced) before the push goes undetected locally.

Lesson: Before bumping the version in package.json and pushing to main, always run the full pre-publish sequence from the module root, not just from _test/:

bash
# From the module root
npm run lint          # must exit 0
# From _test/
npm install && npm test  # must exit 0

Both must be green before the version bump commit. If lint fails, fix it first. Do not push a lint-broken version and rely on CI to catch it. The lint fix will require a second commit and re-trigger the entire CI pipeline, wasting pipeline time and adding noise to the git log.

Quick rule: Treat npm run lint (module root) + npm test (_test/) as a single inseparable gate. Never bump a version without both passing locally.

15. VitePress / Vue compiler crashes on bare angle-bracket placeholders in markdown ​

Symptom: The Deploy Website workflow fails during vitepress build with an error like:

[vite:vue] [plugin vite:vue] docs/architecture/<file>.md (NNN:1):
  Element is missing end tag.
SyntaxError: [plugin vite:vue] ...

The reported line number is often far below the line that actually caused the problem. The parser walks until it gives up, then reports the position at which it abandoned the parse.

Cause: VitePress feeds rendered markdown through Vue's compiler so that authors can use Vue components inside markdown. The compiler does not unconditionally trust markdown's HTML-escaping: any <lowercase-or-PascalCase-name> inside the rendered output is treated as a Vue/HTML element open tag and triggers the close-tag search. Three concrete failure modes seen on this codebase:

  1. Bare placeholders outside any code formatting. A bullet like - <Domain Operations> *(convenience layer)* parses as opening element <Domain with attribute Operations, then waits forever for </Domain>.
  2. Placeholders inside a fenced code block tagged ```markdown. Vue still scans the block's content for tags because the markdown language hint enables an additional templating pass. Placeholders such as <sibling-1> and <one-line description> inside a markdown-tagged fence will trip the same error.
  3. Placeholders inside backticks are usually safe (they render as HTML-escaped <code> content), but a few corner cases (extremely long backtick spans, nested code formatting, generic-looking names like <T>) have produced the same failure.

Lesson: Treat angle-bracket placeholders in markdown as a build-breaking smell. Three durable rules:

  1. Never use <...> placeholders outside backticks in prose. Use square brackets ([name]), curly braces ({name}), or plain capitalized phrases ("Domain operations") instead.
  2. For verbatim copy-paste templates inside fenced code blocks, prefer the text language hint. Switching ```markdown to ```text disables the secondary Vue scan and lets the placeholders survive untouched. If the syntax-highlighting loss is unacceptable for a particular block, replace the angle-bracket placeholders with square-bracket ones at template-author time and keep the markdown hint.
  3. Verify documentation changes locally before pushing when the change touches any VitePress-rendered file in docs/. Run npm run build from website/. That is the same pipeline CI runs (vitepress build . after sync-docs). Watch for the Element is missing end tag family of errors. Local build is fast (single-digit seconds) and catches the failure before it occupies a CI runner.

This pitfall is distinct from the helper-modules CI chain (entries 1–14): it lives in ci-deploy-website.yml, not ci-publish-helper-modules.yml, and a website-deploy failure does not block module publishing. The two pipelines are independent. But the same commit that triggers helper-module publishing will also trigger website deploy if it touches any docs/ file, so a documentation-side bug is a hidden cost on every push that updates rules.

16. Deleting a GitHub Packages package: org-scoped vs user-scoped endpoint ​

Symptom: gh api --method DELETE /users/OWNER/packages/npm/PACKAGE/versions/ID returns success (exit 0, empty body) but the package is still on the registry. Repeated attempts appear to "do nothing", and the agent cannot tell whether the delete worked.

Cause: superloomdev packages are owned by a GitHub organization, not a user. Organization-owned packages live under /orgs/OWNER/.... The /users/OWNER/... endpoint targets the user-account namespace; for an org-owned package it returns a misleading no-op instead of an error. Two further traps: the API path uses the bare package name without the @scope/ prefix (js-server-helper-http-gateway, not @superloomdev/js-server-helper-http-gateway), and deleting the last remaining version deletes the whole package.

Lesson: Determine the owner type first, then use the matching endpoint and always verify with a 404.

bash
gh api /users/OWNER --jq '.type'        # "Organization" or "User"
# org:        BASE=/orgs/OWNER/packages/npm/PACKAGE_NAME
# own user:   BASE=/user/packages/npm/PACKAGE_NAME
gh api "$BASE/versions"                  # list / confirm it exists

For same-version republish, delete only the specific version ID, never all versions. Bulk deletion is forbidden by the autonomous execution protocol. The correct deletion is:

bash
gh api "$BASE/versions?per_page=100" --jq '.[] | select(.name == "VERSION") | .id'
# Require exactly one ID, then:
gh api --method DELETE "$BASE/versions/VERSION_ID"
gh api "$BASE/versions?per_page=100" --jq '.[] | select(.name == "VERSION") | .id'
# Must return empty

If the verify step returns a JSON array, the delete did not work - re-check the owner type and path.

Confirmed behavior after deletion: GitHub Packages accepts republishing a previously-used version name once the version is deleted - version names are not permanently burned. Combined with the detect job's content guard, this enables same-version republish. The historical 44-module wipe-and-republish was a one-time re-baseline under a different policy; current plans delete only the exact named version. Round-trip proof after republish: npm install @scope/[name]@[version] from a scratch directory must resolve.

17. git add . bundles unrelated modules into one commit ​

Symptom: A single commit touches many modules at once. CI's per-module detect job then schedules tests/publish for every changed module path in that commit, and git history no longer maps one module refactor to one commit ("bulk commits").

Cause: git add . (or git add -A) stages every modified file in the working tree, not just the module currently being worked on. During a multi-module sweep (e.g. refactoring a parent plus its adapters), in-progress edits to sibling modules get swept into the same commit.

Lesson: Stage only the directory of the module being committed, so each module refactor is one independent commit:

bash
git add src/helper-modules-server/js-server-helper-<name>/
git commit -m "refactor(<name>): one-line summary"

This keeps the CI publish trigger scoped to a single module, makes history bisectable, and lets one module be reverted without disturbing others. Always run git status --short before committing and confirm only the intended module's files are staged.

23. Extension module's test job 404s on its base package: needs pointed at the base's test job, not its publish job ​

Symptom: During a full-registry republish, an extension module's test job (e.g. test-[base]-ext-[framework]) fails with npm error 404 Not Found ... @scope/[base]@^1.0.0 while the base module's own test and publish jobs are green in the same run.

Cause: The extension's _test/package.json installs the base module from the registry (a registry semver range, not file:). The extension's test job was chained with needs: [detect, test-[base]], so it started as soon as the base's tests passed - racing the base's publish job. On a fresh registry the base package does not exist yet when the extension's npm install runs.

Lesson: A test-* job whose _test/package.json installs another in-repo package from the registry must needs that package's publish-* job, never its test-* job. In steady-state runs the difference is invisible (the base is already published, both orderings pass), so the bug only surfaces during bootstrap or a registry re-baseline - review every extension and adapter chain against this rule before a fresh-state push.

Coverage-sweep footnote: when verifying that every module has CI jobs by extracting paths from the workflow file, the extraction pattern must cover the full name alphabet. A grep -oE character class of [a-z-]+ silently drops any path containing a digit (storage-aws-s3), producing false "missing from CI" findings. Include digits ([a-z0-9-]+) and cross-check the extracted count against the known module total before acting on the diff.

24. npm error 409 Conflict - Package file checksum mismatch after deleting and republishing a package ​

Symptom: CI test job fails with npm error code E409 and npm error 409 Conflict - GET https://npm.pkg.github.com/download/@scope/package/version/hash - Package file checksum mismatch. The package was just republished at the same version after a registry deletion.

Cause: Lock files (package-lock.json) in consuming repos cache the old tarball integrity hash. When the package is deleted from the registry and republished at the same version, the new tarball has a different hash. npm ci compares the cached hash in the lock file against the registry tarball and rejects the mismatch.

Lesson: After deleting and republishing a package at the same version, every consuming repo's lock file must be regenerated:

bash
rm -rf node_modules package-lock.json && npm install

This applies to all lock files in the repo (root, _test/, hosts/, etc.). CI will fail until the updated lock files are committed and pushed.

25. Shared devDependency package missing from registry breaks every downstream npm ci ​

Symptom: Every module's CI test job fails with ERR_MODULE_NOT_FOUND for @superloomdev/js-helper-eslint-config or a checksum mismatch, even though only the config package was deleted and republished.

Cause: When a shared devDependency (like js-helper-eslint-config) is consumed by every module, deleting it from the registry breaks all downstream installs until CI republishes it. If the config package's own test and publish jobs are not chained ahead of all other jobs in the workflow, downstream modules race the republish and fail.

Lesson: A shared devDependency that every module installs must be the head of the CI chain. Its test-* and publish-* jobs must complete before any other module's test-* job starts. In ci-publish-helper-modules.yml, chain test-eslint-config and publish-eslint-config before the first downstream test-* job. Never delete a shared devDependency version without immediately repushing to trigger its CI republish. Bulk deletion of all versions is forbidden; delete only the exact named version when the active plan authorizes it.

26. A version-existence publish guard turns an unmoved version into a green run that ships nothing ​

Symptom: A commit whose source changed is pushed, CI reports green, and the registry still serves the old tarball. A later re-trigger of the same commit fails with 409 You cannot publish over the previously published versions, the honest error the first run swallowed. The red run is then easy to misread as "the published version is corrupt", which invites a destructive registry operation to clear it.

Cause: The publish path guards npm publish with a registry-existence check: it reads npm view "$PKG@$VERSION" and sets needs_publish=false when the version is present. The guard exists to make re-runs of an unchanged commit idempotent, which is legitimate. But it cannot distinguish "nothing to do, the registry already has this exact content" from "the source changed and the version did not move with it". Those two need opposite outcomes, and the version name does not carry enough information to tell them apart. Any repository whose version can stay put across a source change is exposed, whether because a bump was forgotten or because the repository republishes at a fixed version by policy.

Fix/Lesson: The guard compares content, not version presence: pack the working tree, read the registry's dist.shasum, and skip only when the shasums match. A mismatch fails loudly, naming both shasums; the remedy follows the repository's release policy, which is normally to bump the version. Both the detect gate and the per-job guard must compare, because a detect job filtering on version presence drops the module before its publish job runs. The same comparison is the only honest post-publish verification, because a version appearing in the registry listing does not prove the new content shipped. Generalized rule: when a guard's job is to detect "already done", its input must be a fingerprint of the work, never a name that the work happens to reuse. Positive form of the rule: cicd-publishing.md - The Publish Guard Compares Content, Not Version Presence; the underlying principle: engineering-philosophy.md - Idempotency Guards Compare Fingerprints.

27. A local parity runner that extracts CI steps by name pattern reports false confidence ​

Symptom: Local verify is green, then CI goes red on a step the local runner never ran, or that it coincidentally covered without mapping.

Cause: The runner extracted workflow steps with a G[0-9]+ name pattern, so 23 of 55 named steps in one repository and 15 of 23 in another were invisible. Nothing reported them as unmapped; silence read as coverage.

Fix/Lesson: Enumerate every workflow step structurally, map each to a local gate or a signed unreplicable reason from the closed set, and fail on any step that is neither. A new CI step with no mapping turns the runner red instead of escaping to GitHub. Positive form of the rule: local-ci-parity.md - Enumerate Every Step.

28. A test passes locally because a sibling install exists that its CI job never performs ​

Symptom: CI fails with ENOENT for a path under another package's node_modules, while the same test passes locally every time.

Cause: The local runner installs every package into one shared working tree, so a test in src/_test could read hosts/web/node_modules. The CI test job installs src/_test only, from a bare checkout, so the path does not exist there.

Fix/Lesson: Replay install-scoped jobs in a snapshot that contains only that job's installs, and add a cheap grep gate for cross-package path references. The snapshot is the working tree as git sees it, not a worktree at HEAD, so it tests the edits about to be pushed. Positive form of the rule: local-ci-parity.md - Isolate the Install Scope.

29. The documented pre-push command is not the command that ran ​

Symptom: CI goes red on gates the local runner does replay, even though the developer ran a command and it passed.

Cause: AGENTS.md said "run npm run verify before every push" as prose. npm run lint was run instead, and nothing distinguished the two. Lint is a subset of verify, so it passed while the full gate would have failed.

Fix/Lesson: The full runner writes a content-hashed stamp on success, and a tracked pre-push hook refuses a push when the stamp is missing or stale. The hook is an additional local gate; CI still runs as before. Positive form of the rule: local-ci-parity.md - Enforce the Pre-Push Command.

Local Module Testing ​

Each entry below maps a symptom to its root cause and the durable fix. The sibling philosophy doc is testing-local-modules.md (healthcheck philosophy, test concurrency rules, module reference table); this section is the journal of real failures that shaped those rules.

1. npm error notarget No matching version found for @superloomdev/... ​

Cause: npm install ran from the repo root or the module root instead of _test/. Each _test/ directory has its own package.json with a different dependency tree.

Lesson: Always cd into _test/ first. Tools that automate this (AI agents, scripts) must always pass Cwd explicitly to the _test/ directory - omitting it silently runs from the repo root.

bash
# Wrong: from the module root
npm test

# Correct: from _test/
npm install && npm test

2. ERR_MODULE_NOT_FOUND for a package that should be installed ​

Cause: An import in the test uses the wrong scoped package name - typically missing the category prefix (@superloomdev/js-server-helper-postgres instead of @superloomdev/js-server-helper-sql-postgres).

Lesson: The npm package name must match the full directory name, including every category prefix. Grep for the bare name (grep -r "js-server-helper-postgres" _test/) before assuming the install is broken.

3. Manually starting Docker before npm test ​

Symptom: Bind for 127.0.0.1:NNNN failed: port is already allocated, or tests fail immediately with ECONNRESET / socket hang up, many tests cancelled.

Cause: pretest runs docker compose down -v --remove-orphans first. That command only manages containers from its own compose project name - it does not touch a manually started container. Then docker compose up tries to bind the same port and fails. (The same conflict exists in CI when a workflow step runs docker run ahead of npm test.)

Lesson: Pick one owner of the Docker lifecycle. pretest already owns it. Locally and in CI, never run a separate docker run or docker compose up for the same service before npm test.

4. node --test test.js directly without pretest ​

Cause: pretest did not run; no container is up. Tests fail immediately with connection errors.

Lesson: Use npm test, not node --test test.js. The lifecycle scripts exist for a reason.

5. Stale container from a previous failed run ​

Cause: A prior run died before its posttest could clean up.

Fix: pretest already handles this with docker compose down -v --remove-orphans. If pretest itself fails because of a deeper conflict, clean up manually from the same _test/ directory:

bash
docker compose down -v --remove-orphans

6. Tests pass locally, fail in CI with Connection lost: The server closed the connection ​

Cause: A healthcheck that returns "ready" too early. On Docker Desktop the service finishes init fast enough that the false-positive moment never overlaps with the test run; on a constrained CI runner, init takes longer, the healthcheck passes during a transient-ready window, and the server then drops connections during a real init step (e.g., MySQL's restart in normal mode).

Lesson: See testing-local-modules.md Healthcheck Philosophy. Probe with the credentials, database, and transport the tests will use. Add start_period and enough retries that the total budget covers a cold-pull, cold-start initialization.

7. test did not finish before its parent and was cancelled ​

Cause: Concurrent execution of top-level describe() blocks. A lazy-init resource was created mid-test by a parallel block, leaving the cancelled block in a half-initialized state.

Lesson: Wrap the suite in describe('Module', { concurrency: false }, ...). See testing-local-modules.md Test Concurrency.

8. && sleep 2 (or 5) in pretest ​

Cause: A previous developer added a sleep to mask an unreliable healthcheck.

Lesson: Sleeps are never the right fix. They paper over bad healthchecks, slow every developer's iteration, and still fail under load. Remove the sleep, then fix the healthcheck so docker compose up -d --wait truly waits until the service is ready.

9. AI agents or scripts hanging on a multi-line shell command ​

Cause: Heredocs (cat <<'EOF' ... EOF) and multi-line -m arguments to git commit cause zsh to enter dquote> continuation mode when the closing quote falls on a different line. Special characters (backticks, $, !, (...)) make this worse.

Lesson: For commit messages: prefer a single-line -m, or stack multiple -m flags, or use -F /tmp/file after writing the file via a non-shell tool. For file content: never use heredocs through a shell bridge; write to a file with the editor tool and cat it from there. Full journal: AI Terminal & Shell Bridge -> S1/S2.

10. AWS SDK test hangs ~1-2s, then success: false with no clear error ​

Symptom: A test that exercises an AWS SDK function (getSignedUrl, S3Client.send, DynamoDBClient.send, etc.) takes 1-2 seconds and fails with success === true assertion failing. Stack trace points to your module's catch block but the underlying error is silent or absorbed.

Cause: No AWS credentials are passed to the SDK. The SDK walks the default credential provider chain: env vars -> shared config file -> EC2/ECS instance metadata (http://169.254.169.254). On a developer machine and on a GitHub Actions runner there is no instance metadata service, so the chain times out. The 1-2 second test duration is the metadata-service connection timeout.

Lesson: Every test that uses an AWS SDK client must inject dummy credentials, even when no real network call is made (URL signing, command construction, etc.). The dummies do not need to be valid - they just need to exist so the SDK does not enter the default-chain code path.

The canonical pattern is to set them in the test script of _test/package.json:

json
"test": "AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 ... node --test test.js"

For service-specific env vars (S3_ACCESS_KEY, DYNAMODB_ENDPOINT, etc.), use the names the module's loader reads. The same env vars must also exist in __dev__/.env.dev and docs/dev/.env.dev.example per the four-file rule.

11. MongoDB replica-set healthcheck reports healthy before the node is PRIMARY ​

Symptom: Tests pass on a developer's macOS machine and fail in CI with the first few writeRecord / getRecord (after writing) calls returning success: false or result.document === null. Later writes in the same suite succeed. The mongodb helper's own _test and any downstream consumer's _test:mongodb are both vulnerable.

Cause: A naive replica-set healthcheck:

yaml
test: ["CMD", "mongosh", "--eval", "try { rs.status().ok } catch(e) { rs.initiate({...}).ok }"]

returns truthy as soon as rs.initiate() returns. But on a single-node replica set the node still spends another 1-2 s in SECONDARY → STARTUP2 → PRIMARY before it accepts writes. docker compose up --wait returns "healthy" mid-election; the test process opens its driver and fires writes immediately. On Docker Desktop / macOS the local stack happens to be fast enough that the test loop hits PRIMARY by chance; on the slower hosted GitHub Actions runner the first writes land mid-election and fail with not master-style errors that the helper catches and surfaces as success: false.

Lesson: The healthcheck must verify the same readiness the tests require (write-ready primary, not just replica-set initialized). Use db.hello().isWritablePrimary and quit(1) so docker keeps retrying until the node is actually primary:

yaml
test:
  - CMD
  - mongosh
  - --quiet
  - --eval
  - "try { rs.status() } catch(e) { rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'localhost:27017' }] }) }; if (!db.hello().isWritablePrimary) quit(1)"

This is the same probe-the-application-protocol principle as MySQL's two-phase init (entry 6). rs.status().ok is the equivalent of mysqladmin ping -u root (server alive but not yet ready for the test workload).

The general rule for any service that has an init phase distinct from "process up": the healthcheck must succeed only after the test-relevant phase has completed (PRIMARY for replica sets, test_user reachable for MySQL, test_db schema applied for Postgres, etc.).

12. verify.generateAndStore returns COOLDOWN_ACTIVE with cooldown_seconds: 0 under concurrent calls ​

Symptom: js-server-helper-verify tests fail intermittently in CI (rarely locally) with COOLDOWN_ACTIVE errors even though the test explicitly sets cooldown_seconds: 0. The concurrency test in _test/shared-store-suite.js ("concurrent createPin calls with cooldown:0 all land") is the most reliable reproducer; downstream consumers of verify see the same flake on CI.

Cause: The cooldown gate in generateAndStore computed now - existing.created_at without short-circuiting when cooldown_seconds === 0. Under concurrency, two requests can share a microsecond-close instance.time while the store's created_at (captured a few milliseconds earlier by the first winning write) is ahead of the second caller's now. The signed diff is negative, diff < cooldown_seconds is trivially true for any positive threshold; and even for 0, the strictly-less-than check fires when the diff is negative. The caller sees a cooldown error that has no real cooldown semantically.

The bug was dormant until the MongoDB PRIMARY-election fix (entry 11) stabilized CI enough that the race became reproducible; before that, the flake was attributed to the mongodb healthcheck and the true cause went unseen.

Lesson: A "cooldown disabled" configuration must short-circuit before any arithmetic on timestamps. The canonical fix is a single explicit check at the top of the gate:

js
// verify.js generateAndStore
if (options.cooldown_seconds === 0) {
  // cooldown disabled; do not consult existing.created_at
} else if (existing && existing.created_at) {
  const diff = now - existing.created_at;
  if (diff < options.cooldown_seconds) {
    return { success: false, error: CONFIG.ERRORS.COOLDOWN_ACTIVE };
  }
}

Generalized rule: any time a helper has a "feature disabled when N === 0" semantics, the disabled branch must bypass every downstream computation that uses N or any state N would have produced. Never rely on 0 < diff < N to be false when N === 0, because diff can be negative under concurrent non-monotonic time sources.

Applies anywhere a rate-limit, throttle, TTL, or cooldown is configurable with a "zero = off" value: verify.cooldown_seconds, auth.LAST_ACTIVE_UPDATE_INTERVAL_SECONDS, logger retention, and any future equivalent. Audit the gate when introducing any such option.

15. Repo-wide ERR_MODULE_NOT_FOUND after a "remove scope prefix" style commit ​

Symptom: All _test/ suites fail immediately at import 'js-helper-utils' with ERR_MODULE_NOT_FOUND. No code logic has changed; only import ... from '@superloomdev/...' statements were rewritten to import ... from '...' in a mass search-and-replace commit. node_modules/ still contains only scoped packages (@superloomdev/js-helper-utils, etc.); the unscoped names do not exist.

Cause: npm installs packages under their full published name (the "name" field in package.json, which includes the @superloomdev/ scope). An import statement must use that exact name. Removing the scope prefix from import statements without simultaneously adding npm aliases (or renaming the packages on the registry) breaks every module that was changed.

Lesson: Never strip scope prefixes from import statements unless the packages are also republished without the scope (or aliased via package.json imports map). The safe rule: import specifiers must always match "name" in the target package's package.json exactly. Fix is a single sed to restore the @superloomdev/ prefix across all .js files under src/, excluding node_modules/.

bash
find src/ -name "*.js" -not -path "*/node_modules/*" | \
  xargs sed -i '' \
    "s/from 'js-helper-/from '@superloomdev\/js-helper-/g; \
     s/from 'js-server-helper-/from '@superloomdev\/js-server-helper-/g; \
     s/from 'js-client-helper-/from '@superloomdev\/js-client-helper-/g"

16. Store contract method name drift between logger.js and test fixtures ​

Symptom: Logger unit tests fail with TypeError: store.addLog is not a function even though the memory store is correctly imported. Grep shows zero matches for addRecord in the source, yet the error points at logger.js:147.

Cause: When logger.js store contract method names were renamed (addRecord→addLog, listByEntity→getLogsByEntity, listByActor→getLogsByActor, initializeStore→setupNewStore, cleanupExpiredRecords→cleanupExpiredLogs), the rename was applied partially: logger.js and the adapter packages were updated but the inline captureStore and minimalStore stubs scattered throughout _test/test.js and the memory-store.js fixture were not. node_modules held a correct symlinked copy, but the directly-required ../logger.js used the new names, so the inline stubs were invisible mismatches.

Lesson: When renaming store-contract methods in the core module, search for every store stub in _test/test.js, not just the shared fixtures, because inline anonymous objects appear throughout the test file and are missed by a module-level rename. Grep pattern: import ... from '../logger.js' -> addRecord|listByEntity|listByActor|initializeStore|cleanupExpiredRecords. The same applies to any module that has inline store stubs in its test file (auth, verify).

17. Store contract method renames not propagated to adapter _test/test.js files: all adapter CI jobs fail ​

Symptom: All test-verify-store-* CI jobs fail with TypeError: store.initialize is not a function. All test-auth-store-{sqlite,postgres,mysql} fail with TypeError: auth.createSchema is not a function. Tests pass locally only if stale node_modules from a prior install are present.

Cause: Two separate renames were applied to the core module source and adapter store.js files but not propagated to the _test/test.js files:

  1. verify.js + all verify-store-*/store.js: store.initialize() → store.setupNewStore() (v2.1.x → v2.2.0). The _test/test.js stubs in all 5 verify-store adapters were not updated. Because verify v2.2.0 was never published (CI was failing at the time), ^2.1.0 in _test/package.json resolved to v2.1.x on the registry (which uses the old name). This masked the problem locally but was exposed as soon as CI installed fresh.
  2. auth.js: auth.createSchema() → auth.setupNewStore(). The 3 SQL auth-store adapter _test/test.js files still called auth.createSchema().

Lesson: When renaming any store-contract or module-public method, grep all _test/test.js files across every adapter before committing, not just the module source and its own tests. Pattern: grep -rn "old_method_name" src/. Adapter test files are not auto-updated by renaming the source. Also: a new major-bump on a core module will not reach adapters until it is both published and the adapter _test/package.json pin is bumped.

18. contains() substring matching causes parent modules to trigger when only adapters need publishing ​

Symptom: When only an adapter module (e.g., js-server-helper-verify-store-sqlite) needs to be published, the CI workflow also runs jobs for the parent module (js-server-helper-verify). The base module is detected as needing publish even though npm view shows it already exists on the registry. This causes unnecessary republishing of parent modules when only their adapters should be processed.

Cause: GitHub Actions contains() function performs substring matching, not exact string matching. The detect job outputs JSON arrays like:

json
["src/helper-modules-server/js-server-helper-verify-store-sqlite"]

The job condition contains(needs.detect.outputs.test_modules, 'helper-modules-server/js-server-helper-verify') returns true because 'helper-modules-server/js-server-helper-verify' is a substring of 'src/helper-modules-server/js-server-helper-verify-store-sqlite'.

Fix: Parse the JSON array first, then check for exact element matching:

yaml
# WRONG: Substring matching
contains(needs.detect.outputs.test_modules, 'helper-modules-server/js-server-helper-verify')

# CORRECT: Exact array element matching
contains(fromJSON(needs.detect.outputs.test_modules), 'src/helper-modules-server/js-server-helper-verify')

Lesson: When a workflow outputs JSON arrays from the detect job and downstream jobs need to check if a specific value is in that array:

  1. Always wrap the output with fromJSON() to convert the JSON string back to an actual array
  2. Use the full path (including src/ prefix) in the match string to ensure exact matching
  3. Never rely on contains() with a JSON string directly - it performs substring matching on the serialized JSON, not element matching on the array

This applies to any job condition checking against detect outputs:

yaml
# Pattern for all module jobs
if: contains(fromJSON(needs.detect.outputs.test_modules), 'src/helper-modules-category/js-module-name')
if: contains(fromJSON(needs.detect.outputs.publish_modules), 'src/helper-modules-category/js-module-name')

19. Test loader uses relative source path instead of the _test/package.json alias ​

Symptom: A test-* CI job fails with Error: Cannot find module 'cookie' (or any transitive dep of the module under test). The require stack points at files inside the module's source directory (e.g. src/helper-modules-server/js-server-helper-http-gateway/parts/cookies.js), not at _test/node_modules/.... Tests pass on the developer's machine because node_modules/ from a previous module-level npm install is still present, but break in a clean CI environment.

Cause: The test loader (or test file) was written as import ... from '../adapter.js' or import ... from '../../sibling-module/main.js' instead of using the alias declared in _test/package.json. The CI workflow runs npm install inside _test/, which only populates _test/node_modules/. Node's import resolution resolves the relative path to the source directory, where the module's transitive deps were never installed (CI never runs npm install at that location).

Why aliases exist: Every _test/package.json declares aliases like:

json
"dependencies": {
  "helper-foo": "file:../",
  "helper-foo-adapter-bar": "npm:@superloomdev/js-server-helper-foo-adapter-bar@^1.0.0"
}

These aliases exist precisely so the loader code stays identical between local-source (resolved via file:../) and published-package (resolved via npm:) contexts. When npm install runs in _test/, it copies the aliased package into _test/node_modules/ AND installs that package's transitive deps right next to it. Bypassing the alias defeats this entirely.

Fix: Always import helpers in _test/ files via the alias declared in _test/package.json:

javascript
// WRONG - relative path to source directory
import HttpGateway        from '../http-gateway.js';
import HttpGatewayAdapter from '../adapter.js';

// CORRECT - alias from _test/package.json
import HttpGateway        from 'helper-http-gateway';
import HttpGatewayAdapter from 'helper-http-gateway-adapter-express';

Exception: Internal files not exposed via the package's main entry (e.g. parts/cookies.js, parts/params.js testing internals) may continue to use relative paths. They are not addressable through the alias because they are not part of the package's public surface.

Lesson: Treat import style as part of test hygiene, not as a stylistic choice. The HTTP Gateway trio (js-server-helper-http-gateway + 2 adapters) is the canonical reference.

20. Singleton pattern prevents test isolation ​

Symptom: Tests interfere with each other, last test's configuration affects subsequent tests, cannot run tests in parallel, difficult to mock dependencies.

Cause: Module uses singleton pattern but takes external dependencies (libs, config, adapters). The singleton shares global state across all tests, making isolation impossible.

Lesson: Use factory pattern for any module that takes external dependencies. Only use singleton pattern for pure utility modules with zero dependencies (like js-helper-utils). Factory pattern enables:

  • Independent instances per test with different configurations
  • Parallel test execution without conflicts
  • Mock dependency injection per test scenario
  • Multiple instances for different use cases

Example of the problem:

javascript
// WRONG: Singleton with dependencies
let CONFIG;
export default function loader (config) {
  CONFIG = config;  // Last caller wins!
  return singletonInstance;
};

// PROBLEM: Test A sets {log_level: 'debug'}
// PROBLEM: Test B sets {log_level: 'error'}  
// PROBLEM: Both tests now use 'error' level

Example of the solution:

javascript
// CORRECT: Factory pattern (fixed-slots shape - see module-structure.md)
export default function loader (shared_libs, config) {
  const Lib = { Utils: shared_libs.Utils };
  const CONFIG = Object.assign({}, (await import('./[module].config.js')).default, config || {});
  const ERRORS = (await import('./[module].errors.js')).default;
  const Validators = (await import('./[module].validators.js')).default(Lib, ERRORS);
  Validators.validateConfig(CONFIG);
  return createInterface(Lib, CONFIG, ERRORS, Validators);
};

// BENEFIT: Each test gets isolated instance
const testInstance1 = createModule({log_level: 'debug'});
const testInstance2 = createModule({log_level: 'error'});

21. Stale package-lock.json causes wrong-version install after file: swap or version reset ​

Symptom: npm install && npm test in _test/ fails with a runtime error that does not match the source code. For example, CONFIG.ADAPTER must be an adapter factory function even though the source was already refactored to the new CONFIG.Adapter object pattern. Or: a freshly deleted-and-republished 1.0.0 still installs the old pre-refactor tarball. Or: npm error 409 Conflict - Package file checksum mismatch on a package that has not changed.

Cause: Two compounding sources produce the same class of failure:

  1. Stale file: path in lock file. During bootstrap testing, _test/package.json is temporarily changed to "helper-foo": "file:../../js-server-helper-foo" so tests can run against local source before the parent module is published. After restoring package.json to a registry semver range, the package-lock.json still has "resolved": "../../js-server-helper-foo". A subsequent npm install reads the lock, sees the path is satisfied, and skips the registry fetch, silently installing the local source (or nothing, if the path no longer exists).

  2. Version reset to 1.0.0 hits a cached tarball. When a module is deleted from the registry and re-published at 1.0.0, a stale node_modules/ or lock file from the previous 1.0.0 epoch resolves to the old tarball. The npm cache keeps the old hash; the registry now serves a different tarball for the same version string, producing a 409 Conflict / checksum mismatch.

Lesson: During module refactoring (where file: swaps and version resets to 1.0.0 are routine), always run rm -rf node_modules package-lock.json before npm install:

bash
# From the module's _test/ directory
rm -rf node_modules package-lock.json && npm install && npm test

This clean-install form is mandatory during module refactoring. It is not required for general module development, where a plain npm install is correct and repeated lock-file deletion would be wasteful. See testing-local-modules.md Step 1.

Secondary lesson: If npm install fails with E409 Conflict / checksum mismatch even after a clean install, the registry is temporarily inconsistent (a known GitHub Packages transient bug). Wait 30 to 60 seconds, then re-run rm -rf node_modules package-lock.json && npm install. Do not use --legacy-peer-deps. It masks the real error, writes incorrect lock entries, and creates new failures on the next run. The clean install is the correct fix.

22. ESLint no-unused-vars on intentionally reserved module-scope variables ​

Symptom: npm run lint fails with error 'CONFIG' is assigned a value but never used no-unused-vars (or similar for Lib, React, ThemeContext, etc.) even though the variable is deliberately kept as a reserved injection slot for future use.

Cause: Every module in this codebase declares its full set of dependency injection slots at the top of the file as a standard structural contract - even slots that are not yet wired. This keeps the injection surface explicit, consistent, and visually scannable across every module. ESLint does not know the variable is intentionally reserved; it sees an unread assignment and flags it.

Why not remove it or rename to _CONFIG? Removing breaks the structural contract and creates an inconsistency when the slot gets wired later. Renaming to _CONFIG changes the actual variable name and deviates from the naming convention used across all modules (Lib, CONFIG, React, etc.).

Fix: Add an inline // eslint-disable-line no-unused-vars comment at the end of the reserved declaration. This is the established convention for module-scope let vars across all modules (see docs/languages/js/module-structure.md Singleton Part Shape):

js
// Injected dependencies, set by the loader (module-scope).
let Lib;
let CONFIG; // eslint-disable-line no-unused-vars -- reserved for future knobs
let React;

Never use a file-level /* eslint-disable */ for this - it suppresses the rule for the entire file and hides real bugs. The inline directive is the only correct form.

Lesson: The module-scope injection-slot block (let Lib; let CONFIG; ...) is a deliberate structural pattern, not dead code. When adding a new slot that is not yet used, add // eslint-disable-line no-unused-vars on the same line. Remove the directive when the slot is wired.

23. CI fails after push - local tests ran against stale node_modules ​

Symptom: All tests pass locally. Code is committed and pushed. CI fails immediately with ERR_MODULE_NOT_FOUND, version mismatch, or checksum errors that were never seen on the developer's machine.

Cause: Local node_modules/ contained hoisted dependencies from a previous repo session or a file: link that masked a missing peer dependency. CI does a fresh npm ci from a clean checkout, so it sees the real dependency state. The local test "passed" only because the stale node_modules/ happened to have the right files from a previous install.

Lesson: Always run rm -rf node_modules package-lock.json && npm install before running tests locally. This is the default, not an exception. The few seconds saved by skipping the fresh install are never worth a broken CI run, a fix commit, and wasted pipeline time. See testing-local-modules.md - Pre-Commit Protocol for the full protocol.

24. _data/ directory mixes dev scripts with generated data, violating the single-concern rule ​

Symptom: A module has a _data/ directory containing both generate.js (a dev script) and basic.country-data.js (a generated data file). The generated file is a JS module (export default {...}), not pure JSON. The _data/ directory is not a recognized archetype in file-archetypes.md or module-structure.md.

Cause: The module was created without consulting the existing data/ convention. The _data/ directory was invented ad-hoc to hold both the generator script and its output, mixing two concerns (dev tooling and runtime data) in one directory.

Lesson: Generated reference data lives in data/ as pure JSON (see module-structure.md - Static Data Files). Dev scripts that produce generated data live in scripts/ (see module-structure.md - Dev Scripts). The _data/ directory is not a recognized archetype and must not be used. When a module needs generated data, the generator script outputs JSON to data/, the JSON file is committed and imported at runtime, and the script is excluded from the published tarball.

25. Repo-bound workflows miss code quality issues in other repos ​

Symptom: The js-helper-module audit workflow catches Lib.Utils gaps, step-comment gaps, and type-guard violations in codebase-js-helper-modules. But the same gaps exist in codebase-rnw-components/parts/a11y.js (19 raw !== null && !== undefined checks, no step comments) and were never caught.

Cause: The workflow files lived in codebase-js-helper-modules/.devin/workflows/, so they could only be invoked when working inside that repo. The components library has the same module structure (entry file, companion files, parts/ factories with shared_libs) but the workflow was never run against it.

Lesson: Workflows that apply to any repo with the Superloom module structure belong in the constitution repo (codebase-superloom/.devin/workflows/), not in a single dependent repo. Repo-specific workflows stay in their repo; repo-agnostic workflows move to the constitution. When a workflow is moved, its language must be generalized (Cwd references, module path examples) so it works from any repo root.

26. JSDoc content indentation drifts to 4 spaces when the docs say 4 but reference modules use 0 ​

Symptom: JSDoc blocks across ~90+ files in codebase-js-helper-modules and ~14 files in codebase-rnw-components use wrong indentation for description text, @param, @return lines, and closers. The docs (code-formatting.md) said "4 spaces from the /* delimiter" but every reference module (money.js, utils.js, debug.js) uses 0-space (flush-left) indentation at top level. Nested JSDoc blocks (inside object literals) have content at the /* column, not at a fixed offset.

Cause: Three gaps, found over three passes:

  1. The docs said "4 spaces" but reference modules use 0 spaces. The create/fix workflow's JSDoc awk check only verified the code after the JSDoc closer matched the closer's indentation, not the content inside the block. The audit workflow had no JSDoc content indentation check at all.
  2. A first fix attempt used git grep -nE "^ @param|^ @return|^ @description" which caught @param/@return lines but missed description prose lines (lines not starting with @). Replaced with an awk that tracks /*...*/ block context.
  3. The awk hardcoded ^ (exactly 4 spaces) as the detection pattern. This worked for top-level blocks (content at col 4, should be col 0) but failed for nested blocks: a JSDoc block at col 2 with content at col 0 was over-stripped, and a JSDoc block at col 4 with content at col 8 was missed entirely. The fix: detect the /* column per block and flag any content or closer not at that column. The indentation is relative to the /* column, not absolute.

Lesson: JSDoc content (description, @param, @return, notes, closer) is at the same column as the /* delimiter. The indentation is relative, not absolute. The docs must match the reference modules, not the other way around. The audit and create/fix workflows must use an awk-based sweep that detects the /* column per block and flags any content or closer not at that column. A hardcoded column assumption (whether grep ^ @param or awk ^ [^ ]) misses nested blocks and over-strips or under-strips content. Continuation lines (indented more than /* col + 4 for readability alignment) are excluded from the check.

27. Engine TTL sentinels (-1, -2) leak through wrapper purity when the driver returns them as-is ​

Symptom: A getTtl call on a key with no expiry returns -1, and on an absent key returns -2. These are Redis/Valkey engine sentinels, not application values. A caller that receives -1 cannot tell whether it means "no expiry" or "the TTL is literally minus one second" (impossible, but the type is Number). A caller that receives -2 may interpret it as a real TTL.

Cause: The ioredis client returns the raw TTL command response, which is -1 for "key exists, no expiry" and -2 for "key does not exist". The wrapper passes these through without translation, violating wrapper purity (the envelope must not leak engine-specific sentinels).

Fix/Lesson: The driver maps both sentinels to null before returning. null means "no TTL known" for both cases. A caller that needs to distinguish "no expiry" from "absent" calls getKeyExists. The same principle applies to any engine that uses sentinel values for exceptional states: the wrapper translates them to null or a boolean, never passes them through. The test suite includes a regression test that asserts ttl_seconds is never -1 or -2.

28. Hand-mirrored workflow regexes drift from CI while the count-parity check stays green ​

Symptom: A local verifier script copies a workflow's git grep regex into a shell function and asserts only that the count of copied gates matches the workflow's count. After an edit to the workflow regex, both the copied gate and the count-parity check pass locally, while CI fails on the real gate.

Cause: The local verifier is not derived from the workflow file. It hand-mirrors the regex, so an edit to the workflow is not reflected in the copy. The count-parity check only compares the number of gates, not their bodies, so changing a regex leaves the count unchanged and the check green. The defect is invisible until CI runs.

Fix/Lesson: The local verifier reads the workflow file and extracts the current gate body, so a workflow edit is reflected without a second hand-mirror edit. A count-parity check is removed; it does not catch body drift. The verifier is proven by a deliberate mutation: edit the workflow regex, plant a matching scratch file, and confirm the verifier fails for the right reason, then restore both.

Prevention: A workflow policy gate must have a local entry point that executes the workflow's source-of-truth command. See testing-local-modules.md - Local CI parity.

29. git grep is blind to untracked files, so an empty result is not evidence ​

Symptom: A new source file contains a pattern the workflow's git grep gate forbids. The local verifier runs git grep and reports the gate clean, because the file is untracked and invisible to the index. CI later adds the file on push and the gate fails.

Cause: git grep searches tracked and indexed content only. A brand-new untracked file is invisible to it, so a forbidden pattern in an untracked file produces an empty result that reads as compliance.

Fix/Lesson: The verifier fails when nonignored untracked files exist, or the operator runs git add -N path/to/new-file before trusting a clean git grep result. The verifier is proven by planting an untracked scratch file with a forbidden pattern and confirming the verifier fails naming the file, then removing the probe.

Prevention: A git grep result is not valid while matching untracked files are invisible. See testing-local-modules.md - Local CI parity.

30. An unbounded RNW list mounts the full roster, so a row-count assertion does not prove virtualization ​

Symptom: A FlatList conversion claims virtualization, but CI measures the full roster mounted on first paint while a local run measures a partial window. The local assertion passes and CI fails.

Cause: Under React Native Web, an unbounded FlatList expands to fit its content and never clips, so every row mounts regardless of viewport. The local viewport happened to be smaller and the partial count looked like windowing. The threshold was loosened until CI passed, converting a real signal into a green check.

Fix/Lesson: FlatList and VirtualizedList under RNW must have a bounded height to make windowing possible. A test that claims virtualization asserts an exact or fixed upper bound on the mounted-row count below the full roster and reaches a late row by scrolling the real scroll node. A local-versus-CI row-count difference is evidence of a layout or viewport precondition, not permission to relax the threshold.

Prevention: See rn-testing.md - List windowing under React Native Web.

31. Install claims that were never run were written into evidence ​

Symptom: An evidence block records "clean install: exit 0" for a step that was never executed, and downstream decisions are made on the false evidence.

Cause: The agent wrote the expected result as if it had observed it, skipping the actual command. The evidence read as complete, and the wave proceeded on a claim with no backing output.

Fix/Lesson: Every evidence line is the pasted output of a command the agent ran in this session, not a prediction. Rule 2.4 and 2.17 exist because of this. If a command was not run, its evidence line is absent, not "expected" or "should be".

Prevention: Paste the actual terminal output. An evidence line with no pasted output is not evidence.

32. Audit sweeps reported through &&/|| chains gave wrong verdicts ​

Symptom: A sweep reports "clean" when it has hits, because the &&/|| chain masks the exit code of the actual grep.

Cause: Shell chaining reuses the last command's exit code, not the grep's. A grep that finds hits exits 1, but grep ... && echo clean prints clean only when grep exits 0; grep ... || echo clean prints clean when grep exits 1 (found hits). Both forms invert the verdict under different conditions.

Fix/Lesson: Rule 2.6. Run the grep alone, capture its exit code, and report the hit count. A sweep is clean when it prints zero hits, not when a chained echo says so.

Prevention: Never chain a sweep through &&/|| to a verdict. Run, count, report.

33. CI skipped 150 of 154 jobs and was read as "all green" ​

Symptom: A CI run is reported as "all green" when 150 of 154 jobs were skipped. The four that ran passed; the 150 that did not run were invisible in the summary.

Cause: The CI matrix has skip conditions that drop jobs under certain paths. A summary that counts only completed jobs reads a partial run as complete. Rule 2.13.

Fix/Lesson: A CI run is green when every declared job for the commit's changed paths ran and succeeded. Skipped jobs are reported as skipped, not as passed. The job count in evidence is ran: N, skipped: M, total: N+M.

Prevention: gh run view <id> --json jobs --jq '.jobs[] | .name + ":" + .conclusion' lists every job including skipped ones. Read the full list, not the conclusion summary.

34. Literal comparisons of profile data passed while the engine rejected every profile ​

Symptom: A test compares two literal objects and passes, while the real engine rejects every profile the test claims to validate. The test is green and the feature is broken.

Cause: The test's expected value was copied from the implementation's output, not from the specification. Both sides of the deepEqual come from the same code path, so disabling the production line changes both sides equally and the test stays green. Rule 2.38.

Fix/Lesson: The expected side of every deepEqual is a literal copied from the specification, a fixture file, or a rule re-implemented in the test from the plan's words. It is never the output of the library function the test exists to check, and never a second call to the same function with the same input.

Prevention: When writing a test, ask: "if I delete the production line, does this test fail?" If not, the expected value is not independent.

35. git grep -E with \b on macOS matches nothing ​

Symptom: A git grep -E sweep whose pattern contains \b prints nothing on macOS and reads as clean, when the same pattern matches on Linux.

Cause: BSD regex has no word boundary. A -E sweep whose pattern contains \b prints nothing and reads as clean. Run word-boundary sweeps with -P, and probe that -P matches a known word before trusting an empty result.

Fix/Lesson: Use -P for word boundaries, and probe the pattern against a known word before trusting an empty result.

Prevention: Add a positive control to every word-boundary sweep: a known word the pattern must match, run first, that proves the pattern works.

36. A scoped sweep must be proven to stay in scope ​

Symptom: A script that accepts a module path and never passes it to git grep returns repository-wide hits; readers skim them and real hits hide among them.

Cause: The scoping argument was accepted but not threaded into the grep. The sweep reports every match in the repo as if it were scoped to the module.

Fix/Lesson: After scoping a sweep, assert that zero hit lines fall outside the scope. A sweep that cannot prove it stayed in scope is a repository-wide sweep.

Prevention: Every scoped sweep ends with a scope assertion: if any hit line is outside the module path, exit 1.

37. A test that asserts a field name the implementation exposes, rather than the shape the specification names, passes a defect ​

Symptom: A test asserts that a field named foo exists on the result, and passes. The specification names the field bar. The implementation exposed the wrong name and the test did not catch it.

Cause: The test's assertion was written from the implementation's surface, not from the specification's shape. The test proves the implementation is self-consistent, not that it is correct.

Fix/Lesson: Assert specification shapes with deepEqual against the specification's literal. A field name the implementation exposes that the specification does not name is a defect the test should fail on, not pass on.

Prevention: Write the expected shape from the spec, then implement to it. Never copy the implementation's field names into the test.

38. A checkpoint declared from memory skipped a wave ​

Symptom: An agent declares a checkpoint "proceed" from recollection, and a wave that was never released is treated as released. The next wave starts on a false premise.

Cause: The agent computed checkpoint eligibility from memory rather than from the plan text and the recorded evidence. The memory was wrong; the evidence was never written.

Fix/Lesson: Compute checkpoint eligibility from the plan text and the recorded evidence, never from recollection. Run checkpoint-check.sh <plan> <n> and read its output.

Prevention: A checkpoint is a gate, not a judgment. The script is the authority; memory is not.

39. Documentation written by one hand from the specification while the code was written by another from the keyboard diverged on a field name ​

Symptom: The docs say color.interactive; the code reads color.primary. Both are green in their own suites; the divergence is invisible until a consumer reads both.

Cause: The agent that changed the public surface did not write its documentation in the same pass. The doc agent worked from the spec; the code agent worked from the keyboard. Rule 23.

Fix/Lesson: The agent that changes a public surface writes its documentation in the same pass. A second agent's summary is not evidence and is not read.

Prevention: One agent, one pass, surface and docs together. A doc change without a code change in the same commit is a smell; a code change without a doc change in the same commit is a defect.

40. A shadow* collapse dropped layers and spread on native for as long as React Native lacked boxShadow ​

Symptom: A shadow approximation that collapsed multi-layer shadows to a single layer and dropped spread was kept past the React Native version that added boxShadow. Native rendered the wrong shadow for months after the approximation's reason ended.

Cause: The approximation was written when React Native could not render boxShadow. When React Native gained the capability, the approximation was not retired because no one re-checked the floor.

Fix/Lesson: When a platform gains a capability, retire the approximation and its loss report in the same release; an approximation kept past its reason is a silent defect.

Prevention: Every exception in the plan's exception register is revisited at every floor change. An approximation with no revisit date is a defect.

41. A warm npm cache served a tarball the registry had already deleted, so local npm ci passed while CI returned 403 ​

Symptom: Every CI test job fails at npm ci with npm error 403 Forbidden - GET https://npm.pkg.github.com/download/@scope/package/1.0.0/<shasum> and Permission permission_denied: read_package, while the same lockfile installs cleanly on the developer machine. The token has read access; the 403 looks like a permissions problem and is not one.

Cause: A same-version republish (delete, then publish 1.0.0 again) changes the tarball shasum. Every consumer lockfile still pins resolved: .../1.0.0/<old shasum>. GitHub Packages answers 403, not 404, for a deleted tarball. Locally, npm's content-addressable cache serves the old tarball by integrity hash without contacting the registry, so npm ci cannot see that the pin is dead. A verify script that runs npm ci against the warm cache is not CI-faithful for this failure.

Fix/Lesson: After any same-version republish, refresh every lockfile in every consumer, enumerated by path (root, src/_test, hosts/web, hosts/expo), not "the ones that changed". Two permanent gates: a lockfile freshness check that compares each pinned @scope shasum with npm view <name>@<version> dist.tarball and fails on any mismatch, run in CI before the first npm ci and mirrored in the local verify script; and npm ci --cache "$(mktemp -d)" in the verify script so the local cache can never mask a deleted tarball. Evidence for "CI green" is a gh run watch conclusion on the pushed commit, never a local pass.

42. npm unpublish returns E405 on GitHub Packages ​

Symptom: npm unpublish @scope/package@1.0.0 --registry=https://npm.pkg.github.com fails with npm error code E405 and 405 Method Not Allowed.

Cause: GitHub Packages does not implement the npm unpublish endpoint. Version removal is a GitHub Packages API operation, not an npm one.

Fix/Lesson: Query the version id with gh api /orgs/<org>/packages/npm/<package>/versions --jq '.[0].id', delete it with gh api --method DELETE /orgs/<org>/packages/npm/<package>/versions/<id>, then push so CI republishes. Never npm unpublish, never a local npm publish.

43. A React Native Web <input> keeps its intrinsic minimum width and overflows its wrapper ​

Symptom: A password, number, or search field renders wider than its wrapper by a few pixels at narrow widths (an inner <input> of 186px inside a 180px wrapper), so the wrapper's border or adjacent control is pushed out or clipped.

Cause: React Native Web renders TextInput as <input>, and a browser <input> has an intrinsic width with min-width: auto. Flex flex: 1 lets it grow but not shrink below that intrinsic width.

Fix/Lesson: The TextInput atom sets minWidth: 0, so every composite that embeds it can shrink. Patching each composite individually leaves the next one broken.

44. An SVG icon adapter that rewrites root fill and stroke turns stroke glyphs into filled shapes ​

Symptom: Checkmarks, sync arrows, and every *-outline glyph render as filled wedges or blobs; filled glyphs look correct, so the defect is not uniform and is easy to attribute to the icon set.

Cause: Stroke glyphs depend on their own classes or attributes (fill: none, a stroke width) that the adapter did not load, while the adapter set fill and stroke on the root element for every glyph. Open polylines received a fill.

Fix/Lesson: An adapter sets color through the one attribute the icon set documents (fill for filled sets, stroke for stroke sets, or the set's own color prop) and never both. Every host maps semantic names through one committed manifest, and an unmapped name is a console.error, not a silent placeholder, so the zero-console-error gate sees it. A ? placeholder that passes every gate and lands in a regenerated visual baseline is the failure this entry exists to prevent.

45. A unit test read a sibling host's node_modules, passed locally where every host was installed, and failed in the CI job that installs only its own directory ​

Symptom: src/_test passes locally and in the local verify script; the CI test job fails with ENOENT ... hosts/web/node_modules/@scope/package/data/manifest.json from a unit test.

Cause: The test resolved a data file through hosts/web/node_modules. On the developer machine every host had been installed by an earlier gate, so the path existed. The CI test job installs src/_test only.

Fix/Lesson: A test directory reads installed packages from its own node_modules and declares them in its own package.json. Anything a test needs from another directory of the repo is either a tracked source file or a declared dependency. A CI-faithful check for this class is to run the unit suite with every other node_modules moved aside.

46. A generated artifact hand-written to agree with the implementation ​

Symptom: A check comparing the implementation against a "generated" oracle passes while both are wrong.

Cause: The oracle's values were transcribed from the source by hand, so the comparison is self-confirming. A hand-written oracle cannot detect upstream drift because it has no source to regenerate from.

Evidence: button.height read 40 in both the oracle and the spec sheet while @carbon/styles declared the root at $default: 'lg' (48). The oracle agreed with the implementation because the oracle was written from the implementation.

Fix/Lesson: Parse the source, record a method per value (parsed, inherited, transcribed, none), and regenerate in CI with a copy-aside byte compare. Detection question: for every generated artifact, name the command that reproduces it from source, and if none exists it is not generated.

47. A check whose pattern cannot match the real artifact format ​

Symptom: A validator reports clean forever and its selftest passes.

Cause: The selftest plants a fixture in a format the real artifact never uses. The validator's pattern matches the fixture but not the real artifact, so the check is inert on real input.

Evidence: The plan close checker's blank-evidence rule matched zero of 14 genuinely blank lines because real plans write - Label: (a markdown list item) and the pattern required a line starting with a letter. The selftest used SelftestBlankLabel: (no dash), a format no real plan uses.

Fix/Lesson: Every checker rule ships a fixture derived from a real artifact; a rule that cannot be shown firing on real input is treated as absent. Detection question: does the selftest's fixture format match the real artifact's format? If not, the rule is inert.

48. Full-roster verification after every component edit stalled the loop ​

Symptom: Each component fix cost a full browser suite over the whole library, minutes per iteration, and the fix loop stopped converging: sessions ran out of context before a batch closed.

Cause: One verification speed. The suite that proves a release was the same suite that ran after a one-line edit, so the cost of checking one component was the cost of checking all of them, multiplied by every template.

Evidence: A component library's fidelity pass ran its complete Playwright suite for every row under every profile after each defect fix; a batch of eight components took the better part of a day of wall-clock verification and the plan tracking it grew past 150 KB.

Fix/Lesson: Three speeds, and no speed does another's work: a per-component check (lint, purity gate, the component's own tests) after every edit; a batch (browser gates for the batch's families, measurement, screenshots, docs regeneration) every 8-12 components; a full verify (clean installs, entire suite, fire runner, baselines, native gate) at milestones only. Detection question: what does the smallest verification command run, and is it proportional to the change?

49. Reference geometry transcribed from upstream stylesheets by judgment ​

Symptom: A spec sheet agreed with its oracle, both agreed with the upstream design system's documentation, and the rendered component still measured differently from the upstream component in a browser.

Cause: The oracle values were read out of the upstream's SCSS by a person (or a model) resolving mixins, defaults and density overrides in their head. Each resolution was a judgment; a wrong one produced a confident number with a parsed label. Nothing compared the number to a real render.

Evidence: A button height oracle read the root size default from one SCSS file while a component-level layout.use override in another file set a different default; both files were "parsed", and the resolved value was the reader's guess at which won.

Fix/Lesson: Measure the rendered upstream component. For a web reference, render the pinned upstream package in the same browser and read computed geometry; for a React Native reference, parse the component's style objects, not its documentation. Transcription from a stylesheet is recorded as transcribed with a reason when it cannot be avoided, and it never earns the parsed label. Detection question: for each reference value, name the command that measured it from a render.

50. A silent icon fallback drew one design system's glyphs under another's theme ​

Symptom: Every icon rendered, no console error, every zero-error gate green - and the second theme showed Carbon's glyphs on every component, for weeks.

Cause: The icon manifest had a column per host adapter, not per theme, and every host resolved every semantic name to the same icon set. Switching the theme switched colors, radii and type, and left the glyphs alone. Because a glyph did render, nothing was missing to report.

Evidence: The icon manifest carried a Carbon column and a legacy column but none for the second theme; both host adapters imported the Carbon icon package. A notification under the second theme drew the second theme's colors around IBM's 32-grid warning triangle.

Fix/Lesson: Icons are theme tokens: each template carries its own glyphs, the component library declares the icon names it requires, and a theme missing one fails to build naming every missing icon. Never draw a glyph from another set as a fallback; a fallback that renders something is invisible to every automated gate. Detection question: switch the template and diff the glyphs - if none changed, the icons were not coming from the theme.

51. A visual baseline recorded from the first render ratified its defects ​

Symptom: Pixel-exact regression tests passed on every run while the component they guarded had a visible layout defect from its first day.

Cause: The baseline screenshot was recorded from the first render, before any check had said the render was right. From then on the regression test asserted "unchanged", and unchanged included the defect. A later fix failed the test, and the failure was read as a regression.

Evidence: A field composite recorded its baseline with a nested border (wrapper and inner input both framed); the frame-ownership fix six batches later broke the baseline and was initially reverted as a visual regression.

Fix/Lesson: A baseline is the last gate, never the first. Record it only after the structural, measurement, perceptual-diff and accessibility layers pass and the batch's contact sheet has been reviewed; a baseline update is a reviewed event with the diff attached, never an automatic re-snapshot. Detection question: for each baseline, name the passing checks that preceded its recording.

52. An enforcement job read packages it never installed, and the local replay ran it against an install left by an earlier gate ​

Symptom: The local verify script replays every CI grep gate and reports them all green; on CI the gates job fails one of them with ENOENT ... _test/node_modules/<package>/metadata.json.

Cause: A data-validation gate lived in the job that holds the git grep gates, which checks out and runs scripts but installs nothing. The script it called resolves the pinned upstream packages from _test/node_modules. Locally the replay ran the gate in the working tree, where _test/node_modules existed from an earlier test run, so the missing install step was invisible. Entry 45 is the same class for a unit-test job.

Fix/Lesson: A CI job installs everything its steps read; a step that reads installed packages sits after an install step in the same job. The local replay reproduces the job's filesystem, not the developer's: before replaying a job's steps, set aside every node_modules that job does not itself install, and restore them afterwards. Detection question: for each gate that runs a script, which install step in its own job provides that script's imports?

53. A custom-scheme deep link opened with simctl openurl stopped at the iOS "Open in" prompt ​

Symptom: A simulator gate builds and installs the app, xcrun simctl openurl booted "app://..." returns ok, and nothing happens: no report, no route; the timeout screenshot shows the home screen with an "Open in [App]?" dialog.

Cause: Current iOS asks the user to confirm a custom URL scheme opened from outside the app. simctl openurl delivers the URL to SpringBoard, which shows that dialog, and no simctl command can tap it. The same link on the Android emulator (am start -a VIEW -d) needs no confirmation, so a driver written against Android passes there and stalls on iOS. Reading the CLI output alone misleads: the opener reports success, and an unrelated line (the dev-client URL expo run:ios prints) looks like the cause.

Fix/Lesson: Look at the failure screenshot before reading logs; the device shows the state, the log shows what was attempted. On iOS, drive an automated walk without a deep link: launch the app with simctl launch and have it pull its instructions from the test server (a build-time flag such as an EXPO_PUBLIC_* variable enables that mode and names the server), or use a tap-capable driver. Keep the deep-link path under test where the platform does not prompt.

54. A render-comparison gate compared generated ids, so one gate failed on every labelled component and another could never fail ​

Symptom: The first components that label a part through a generated id (a checkbox, a text field, a select) failed the accessibility-identity gate under every template, with trees that differ only in aria-labelledby=_r_1_ against _r_0_. At the same time the enum-coverage gate passed a planted edit that made both values of an enum draw the same frame.

Cause: Both gates render the same component more than once and compare the output: one compares accessibility trees across templates, the other compares innerHTML across enum values. React's useId returns a new id on every mount, so every render differs in its id and id-reference attributes. The identity gate therefore always saw a difference, and the coverage gate always saw a difference too, which is exactly the result it needed to pass. Icon, the only component built before them, has no ids, so neither gate had met one.

Fix/Lesson: A gate that compares two renders normalizes what legitimately differs between mounts before comparing: serialize an id reference as the position of the element it points at (@3, or @missing), or blank id and id-reference attribute values. Prove the gate still fires on a planted defect after adding the normalization; a comparison gate that has never seen its subject vary can be vacuous. Detection question: does anything in the compared output change on remount even when the inputs are identical?

55. A pixel comparison between two pages measured the pages, not the component ​

Symptom: A component whose measured geometry matched its rendered reference to the pixel still differed from it by 5 to 20 percent in the perceptual gate, and the contact sheet showed text in a serif fallback.

Cause: Four page-level differences, each invisible to a geometry check. The test page loaded no font files, so text fell back to the browser default. The reference page's stylesheet set the body font, and the harness's own body rule overrode it. The reference stylesheet's reset zeroed heading margins and changed line heights, so every cell after the first sat at a fractional offset on one page and a whole-pixel one on the other, and each glyph edge was anti-aliased differently. The reference stylesheet also asked for grayscale font smoothing, which the test page did not.

Fix/Lesson: Before comparing pixels across two pages, make the pages identical except for the component: serve the pinned font files and declare every family both sides name (including a native-style family name a mobile reference uses); set no body font in the harness; give every layout box around a cell an explicit whole-pixel size, or place each cell on a fixed row pitch; apply the reference's text smoothing to both pages; wait for document.fonts.ready before reading anything. Detection question: with the component removed, would the two pages render identically?

56. A cross-test gate lost its data when a sibling test failed ​

Symptom: The browser accessibility-identity gate failed with "expected at least one key, received 0" the first time any other browser test in the file failed; it had passed while everything else passed.

Cause: Three per-template blocks filled a module-level map in their beforeAll hooks, and a later test compared the map. A failing test makes the test runner discard the worker and start a new one for the remaining tests, which re-evaluates the module with an empty map.

Fix/Lesson: A test never reads state that another test or another block's hook produced; it collects what it compares inside itself. Detection question: does this test still pass when run alone with -g?

57. A focus ring drawn on one part left the browser's own ring on the focused element ​

Symptom: A brand-reach test that collected every drawn outline in a component cell after focusing it found two rings on a checkbox and a select: the theme's ring on the box or frame, and a one-pixel rgb(0, 95, 204) ring on the focused pressable. Unit tests had asserted the theme's ring and the absence of an inline outline width on the pressable, and passed.

Cause: The component drew its focus presentation on a child or parent of the focusable element, but the focusable element itself kept the user agent's :focus-visible outline. The web view resets that outline only on the elements it styles itself (a text input), not on a pressable view. A unit test reading inline styles cannot see a user-agent style.

Fix/Lesson: When a focus ring is drawn on a part other than the focusable element, the focusable element sets outlineStyle: 'none', and the unit test asserts it. A browser gate that proves focus presentation lists every element in the cell with a drawn outline after a keyboard focus and expects exactly the theme's ring. Detection question: after Tab reaches the component, how many outlines does the cell draw?

58. Measuring a web-component reference read its boxes mid-animation and through the wrong tree ​

Symptom: The web-component reference's measurement of a floating label reported fractional positions that changed between runs (y: -4.68, then 13.24), a slotted label's text style as the page body's, and a grouped selector that matched nothing.

Cause: Three properties of a shadow-DOM reference. Its label animates after the value arrives, and document.getAnimations() does not list animations running inside shadow roots, so a readiness check that waited on the document's animations saw none. Slotted text is styled by an element inside the shadow root, not by the host whose text nodes it is. A comma-separated selector list cannot be split on a shadow-crossing combinator as one string.

Fix/Lesson: Give a shadow-DOM reference a settle time before polling for quiet, read a slotted text's style from the shadow element that styles it (styleOf), and resolve selector alternatives one at a time, each crossing shadow roots segment by segment. Detection question: does the measurement give the same numbers on two consecutive runs, and is the font family the one the reference's own stylesheet names?

59. An app test host that had only ever rendered one component broke when the catalog grew ​

Symptom: The application's offline test host passed for months, then failed the first time the component library shipped more than one component: every text-input cell reported ReferenceError: document is not defined from inside the web build of React Native, and, once that was supplied, the server-driven walker test ran the process out of memory after all its assertions had passed.

Cause: Two latent defects that a one-component catalog could not reach. The web build's text input reads document.activeElement in a layout effect without a DOM guard; a test renderer runs effects, and Node has no document. Separately, the autopilot screen bumped a poll counter on every poll and re-rendered the whole walk with it; with one component a render took under the 20 ms test poll interval, with 62 cells it took longer, so poll updates queued faster than renders could drain them inside act. The same re-render ran every second on the device.

Fix/Lesson: A test host supplies exactly the global a dependency reads unguarded (here a frozen document with activeElement: null, and no window, so the library's DOM detection stays off), never a whole DOM it does not need. State that changes often (a poll count, a clock) must not re-render an expensive subtree: build that subtree once per real input (useMemo on the command) so React skips it. A test whose expected count is "the whole catalog" must compute it for the subset the scenario walks. Detection question: does the host still pass when the catalog is ten times larger, and does any interval-driven state re-render a list?

60. A native geometry gate failed on every cell because its web baseline was wrong ​

Symptom: The first native milestone run walked all 62 cells on iOS and Android with zero errors, then failed 143-150 width checks per platform against the web walker: text-bearing cells were 3-19 points wider natively, fields and selects 100-150 points narrower, under every template alike.

Cause: Four defects, three of them on the side treated as the reference. The web host's committed IBM Plex Sans files were HTML pages saved under .ttf names, so the browser drew a fallback face; the font check passed because it compared token names, not what the browser loaded. The web walker measured before its declared faces finished loading, so even the second template's family was measured in the fallback. Neither host carried the mono families, so code text was drawn in each platform's own system monospace. And the one real component defect: the field and select grew from flex: 1, a zero basis in native layout, so in a container sized to its content they collapsed to their padding, while a browser input's intrinsic width hid it on the web.

Fix/Lesson: A cross-platform comparison is only as good as its baseline: before blaming the platform under test, prove the reference drew the faces it names (document.fonts status after an explicit load, not the token), load every declared face before the first render, and carry every family a type set names on every host, from the same binaries. A component that fills its container states the frame it is shown in, and grows from its content (flexBasis: 'auto'), never from zero. Detection question: when every cell disagrees in the same direction under every template, is the baseline drawing what it names?

61. Android ignored every runtime-loaded font, and Linux Chromium hinted the baseline ​

Symptom: After the web baseline drew its real fonts, the native gate still failed Android on monospace (91pt where the web measured 122pt) and on a few labels, and a percentage allowance seemed needed for short strings. Android widths were identical under IBM Plex Sans and the second template's family; iOS widths differed between them as the web's did.

Cause: Two platform behaviors. The runtime font loader registers each face under an alias (IBM Plex Sans_normal), and Android resolves a fontFamily only by registered name, so it drew its system sans for every theme family; iOS resolves the family name inside the font file, so it worked there, and sans hid the fault because Plex Sans and the system sans are within a few points of each other. Separately, Chromium on Linux hints glyph advances to whole pixels by default, so the CI web walker measured Error text at 63px where the font's own advances give 65.19px, which macOS, iOS and Android all draw.

Fix/Lesson: On Android, embed each theme family at build time as one font family with a face per weight (the expo-font config plugin's android.fonts), so fontFamily plus fontWeight resolve to the right file; carry exactly the weights the type sets name. Run any browser that produces cross-platform text measurements unhinted (--font-render-hinting=none). Reproduce a CI-only measurement in the CI image before reasoning about it, and never size a tolerance from residuals whose causes are unexplained. Detection question: does the same cell measure differently under two templates that name different families, on every platform?

62. A visual review passed twice while a planned gate stood unbuilt ​

Symptom: Every automated gate was green through a milestone, two reviewer reads of the contact sheets passed, and the danger button under two templates was drawn in the disabled gray: indistinguishable from a disabled button at 1.71:1 contrast. An outlined button's label under one template was 1.29:1. The gate definition had named a contrast check from the start; no test implemented it.

Cause: A reviewer reads a sheet for shape and placement and sees what the design intends, not what the pixels say; two states that differ in name read as different. The plan listed the clause among the gates, but nothing tied the clause to a test, so its absence was invisible: a green suite says nothing about checks that were never written.

Fix/Lesson: Every clause in a gate definition is a row in a census fixture naming the test title that enforces it, and a test asserts each title exists and each spec declares exactly the listed number of tests. A reviewer's catch becomes a check before the fix (text contrast composited over the real backdrop, enabled-vs-disabled paint distinctness), and the check is proven to fail on the defective data before the data is corrected. Read a sheet by asking, per cell, which color makes this state different from its neighbors; if the answer is "none", the gate is missing a rule.

63. A part left out of a reference comparison hid a control that no longer read as itself ​

Symptom: A select under one template drew no trailing indicator in any state, on web and on both simulators, and looked exactly like a text field. Every gate was green, including the measurement against that template's own reference component, which does draw an arrow in every state.

Cause: Two decisions compounded. The shape enum for the indicator named presence (shown/hidden), not shape, so a template whose indicator was a different drawing than the component's glyph had only "hidden" to pick. And the comparison against that reference listed the parts to measure by hand; the indicator was left out "because the reference draws its own arrow, not a glyph", so the one check that would have seen it never looked.

Fix/Lesson: Which drawing a part shows is an icon role (dropdown_indicator), never an enum: the component asks for the role and each template supplies its own drawing, the reference component's own shape included. An enum decides structure or behavior, and a value no template picks is not offered. Every part measured against one reference is measured against every other reference, or listed as omitted with a reason and the upstream selector that proves the reference draws nothing there; that probe is asserted, so an omission goes stale the moment the reference draws the part. Read a sheet by asking whether each control still reads as itself under every template, not only whether its colors are right.

64. Fidelity gaps arrived one review at a time ​

Symptom: Each milestone review found one more class of difference against a reference (a missing indicator, then a 4px inset, then a glyph half a pixel low), and each find cost a full republish and native run. The automated gates were green every time.

Cause: The gates compared only what they were given: resting states, the light scheme, a hand-written part list, CSS properties rather than pixels. Hover, keyboard focus, press and dark mode were never compared, so every review found the next thing outside that window. And a per-system difference that no token could state (a focus ring inside or outside the edge, an outline that thickens on focus, a translucent disabled color) lived in component branches or not at all.

Fix/Lesson: Before a fix round, a census compares every built row against every reference in every sample state, and every enabled state again while hovered, keyboard focused and pressed, under light and dark, with no expected gap set aside; the whole list is resolved in one amendment, and the same comparison then stands as a gate. Differences a template cannot state become role-grid cells, filled by every template from its own system. Compare a ring or a shadow by its pixels, not by the CSS that draws it: systems draw the same ring as an outline, a border with inset shadows, or a separate element.

65. A reference page measured the wrong thing and the gate agreed with a defect ​

Symptom: The checkbox box was drawn 3px left and 1px above the reference's box, the web-component reference's error colors never matched, and a disabled label compared as white: all with a passing or misleading measurement.

Cause: The harness read a positioned pseudo-element at its offset but not its margin; themed the web-component reference only with the colors the template's mapping table named, so every other system color fell back to the component's own defaults; and read a slotted label's style from the wrong shadow element and its opacity along the light-DOM parent rather than the slot it renders in.

Fix/Lesson: A reference page is themed with the full scheme the template was generated from (the template records it), positions include margins, styles are read from the element that paints them, and opacities are walked along the rendering tree (assignedSlot, then parent, then shadow host). When a measurement and a screenshot disagree, sample the pixels; the pixel is the truth the gate must reproduce.

66. An exemption one comparison honored and another did not ​

Symptom: After a clean install brought in a newer browser, the fidelity gate failed a text field's invalid states against the web-component reference by a fraction of a percent over its pixel budget, in both schemes, while every part comparison agreed.

Cause: The reference omits the field's error icon with a reason and a probe proving it never draws one, and ours draws the template's icon by design. The part comparison skipped the omitted part; the pixel comparison of the same screenshots still counted the icon, so the state sat just under the budget until a change in rasterization tipped it over.

Fix/Lesson: An exemption is declared once and applied by every comparison that reads the same thing. The omission carries mask, and the pixel comparison leaves that part's box out of the comparison. A mask that would cover the part the component is compared around is rejected, with a fire row proving the rejection. A state that passes only by a margin smaller than a browser update is a defect waiting to surface; diagnose it from the saved screenshots before raising any budget.

67. A reach check written as an allowlist failed on correct output ​

Symptom: A brand-reach check committed one step earlier failed on its first run against the published packages: a button ring's page-color line, a checkbox outline darkening on focus and an invalid field thickening its error outline were all reported as wrong.

Cause: The check could not run until the packages were published, so it was committed untested. It listed the colors focus may add (the brand's, plus the page color found by walking up from the focused element, which starts at the button's own fill), so every legitimate color a template draws on focus counted as a defect, and a focus shown by width alone counted as drawing nothing.

Fix/Lesson: A reach check compares the same parts with and without the layer: a color the unlayered template draws there in a value the layer overrides must become the layer's value; anything else the template draws is its own business. Rings are read as zero-offset, zero-blur spread layers, never elevation shadows. A check whose inputs are not yet published is run against a local link before it is committed, and is proven by a planted defect of the class it guards (here, a role cell written as a literal the brand cannot reach).

68. Local verify and CI tested different library commits ​

Symptom: A consumer's full local verify passed (every phase, every e2e test); the push-triggered CI failed its unit tests at once with theme is missing required tokens naming a token the new contract had removed.

Cause: The consumer links the library from a sibling checkout. Locally that checkout sat on the milestone branch; the CI workflow checked out the library with no ref, so it got main, which still required the old contract. While the theme packages kept their contract, the mismatch was invisible. A same-version republish that changes the contract makes main and the registry incompatible until the library branch merges.

Fix/Lesson: A sibling dependency checked out by CI is pinned in one tracked file that every workflow reads, and the local verify refuses to start unless the sibling checkout is exactly that ref as last pushed, with a clean tree. A local pass and a CI pass then describe the same commit by construction. The pin returns to main in the same change that merges the branch it names.

69. Every native screenshot showed a system dialog, and the gate passed ​

Symptom: The native gate was green on Android (every report complete, every assertion passing), but all 21 Android family screenshots showed "Pixel Launcher isn't responding" over a dimmed app. The previous run's screenshots were clean.

Cause: An emulator under load reports the launcher as not responding, and the prompt takes focus over the app. The measurements come from inside the app, so they were unaffected; the screenshots are taken from outside, and nothing checked what they showed. Only the reviewer reads them.

Fix/Lesson: Evidence a reviewer is meant to read is checked before it is taken. The driver switches error prompts off, closes system dialogs, and requires the app's own activity to hold focus (from dumpsys window) before each screenshot; a screenshot of any other window fails the walk. A green gate whose evidence is unreadable is not a pass.

70. A primary-reference layout choice hard-coded where the web-component reference has no counterpart ​

Symptom: The contact-sheet read found the second template's extra-large and 2x-large buttons drawing their label at the top of a pill. Every gate was green.

Cause: The primary reference keeps a tall button's label where the default height puts it, at the top, and the component carried that rule in code. The web-component reference has no tall button sizes, so no fidelity comparison covers those states, and the vendor choice sat in a generic component unseen.

Fix/Lesson: A layout choice a component makes for only some sizes or kinds is a design-system decision, so it is an anatomy enum the template answers, even when only one reference states it. When writing a branch in a component, ask whether a second design system could answer it differently; if so, it belongs in the template, not in code.

71. A pixel gate passed on macOS and failed on Linux CI ​

Symptom: The library's full local verify passed, including every browser fidelity test. The first push-triggered CI run of those tests failed four of them: a sampled pixel below a checkbox read rgb(195, 234, 255) against white, and two text-field states measured 2.03-2.1% against a 2% budget. A Linux container on the same machine reproduced the exact numbers.

Cause: Linux Chromium draws text with LCD subpixel antialiasing and hinting; macOS draws grayscale and never hints. The page's -webkit-font-smoothing: antialiased has an effect on macOS only. Colored fringes and shifted glyph edges changed the pixels near text on one side of the comparison more than the other. The gates had only ever run on macOS, where verify runs, so the difference was never seen.

Fix/Lesson: A pixel gate fixes how text is rasterized, the same way it fixes fonts and viewport. Launch the browser with --disable-lcd-text and --font-render-hinting=none so text is grayscale and unhinted on every OS. Before the first CI run of a new pixel gate, run it once in the CI's Linux browser image, because a local verify on another OS does not replay its rasterizer.

72. An accessibility prop that worked everywhere else crashed the Android Release build ​

Symptom: The native gate's Android job timed out on every theme with no report and the timeout screenshot showed the launcher, while iOS and web passed. logcat showed the process dying about a second after launch.

Cause: React Native backs accessibilityRole with the narrow TalkBack enum and throws IllegalArgumentException on any ARIA name it lacks (option, listbox, tooltip, dialog, and several more). iOS tolerates the same values, so the crash appeared only in the Android Release build, and the driver suppressed the error dialog that would have shown it.

Fix/Lesson: Semantic names go through role, which maps the full ARIA vocabulary and degrades to a null mapping rather than a throw; accessibilityRole is only for values in the TalkBack set. A native walk that produces no report is a crash until logcat proves otherwise, so the driver captures logcat and the app pid on every timeout.

73. An Android text box was three points taller than the token line height ​

Symptom: Every tooltip cell measured three points taller on Android than on web and iOS (45x19 against 45x16), all templates and all states alike.

Cause: The tooltip wrapped its string anchor in a bare React Native Text with no theme type, so each platform drew it in its own system font at its own default line height. Every other text run went through the library's Text atom and matched. The first fix set includeFontPadding: false, which shrank the box and hid the real cause.

Fix/Lesson: Every text run in a component renders through the theme's Text atom (or an explicit typeStyle), never a bare platform Text. The anchor now does, with no platform style, and it measures within a point of the web on Android and iOS. When one component's text disagrees across platforms and the rest agree, look for the run that skipped the theme before switching off a platform default.

74. A fixed-position popover shrank to a column of single glyphs on Android ​

Symptom: Android screenshots showed the tooltip bubble drawing its label one character per line, though every measurement the gate checked on the cell itself was plausible. The second template's dialog also ran off the screen, centered on its 640-wide showcase stage.

Cause: React Native has no position: fixed. On native the layer fell back to position: absolute inside the component's own wrapper, and no overlay host was mounted anywhere. A column flex parent bounds an absolute child's cross axis to its own width (the 45-wide anchor), and measureInWindow window coordinates were applied inside that box. The first fix repaired it per component, branching on the platform inside the tooltip, which breaks the rule that only a dispatcher reads the platform.

Fix/Lesson: Overlays render through one OverlayHost mounted at the app root, and the positioning behaviors own every layer mode. Inside a host a layer lives in the host's space on every platform: a covering scrim fills the host, and a menu sits at its x/y in it. On native, an anchored layer converts the anchor's window rect into host space and flips or clamps against the host's size. A host inside a bounded box (a showcase stage) keeps its layers in that box. Unhosted, a native overlay warns once. Prove the host is mounted before trusting any overlay screenshot.

75. A debug short-circuit switched off the theme contract check for a whole batch ​

Symptom: createSystem built under a theme package that lacked the newest tokens without complaint, and every gate stayed green.

Cause: While a contract amendment was unpublished, a session edited if (!presence.empty) to if (false && !presence.empty) so local runs could proceed, and the edit stayed in the working tree. No gate fires on a disabled check, because its silence looks the same as a pass.

Fix/Lesson: Read every line of a working tree you inherit before committing it, and grep added lines for false &&, true ||, .only( and .skip(. Unblock an unpublished dependency by publishing it or linking it locally, never by disabling the check that would notice.

76. A stylesheet loaded for one reference reached into the other reference's shadow DOM ​

Symptom: The web-component reference's dialog actions measured 60px tall instead of 40, so the template answered an actions height of 100 instead of 80. The Modal action selector only came out right after the web-component reference page was decontaminated.

Cause: The web-component reference rendered on the same page as the primary reference's stylesheet, whose global *{box-sizing:inherit} reset beat the web components' own :host rules inside their shadow trees.

Fix/Lesson: Each reference renders on a page that loads only its own styles. The web-component reference page loads no primary stylesheet. When a reference measures an implausible number, check what else on the page can reach it before writing that number into a template.

77. A borrowed cousin token hid a fidelity gap behind a deleted comparison ​

Symptom: Menu shortcuts crowded their labels on every contact sheet. Later, the web-component reference's Menu measured 121 wide against 112 after its width comparison was restored.

Cause: Menu read the field family's icon gap and the label's own type for its item gap and shortcut. The width comparisons that would have failed were deleted with a "waits on" note, which is an expected gap kept outside expected-gaps.json.

Fix/Lesson: Each distinct visual run gets its own contract cell (control.menu_item_gap, type.menu_item_shortcut, color.menu_item_shortcut), and each template answers it from its own source. A comparison is never removed to wait for a cell; the gap goes into expected-gaps.json, where it expires.

78. Web-only styles rendered differently on iOS and Android without failing any gate ​

Symptom: Native screenshots showed the second template's switch drawing a state-layer halo at rest, a check mark on an unselected small switch, and every tooltip caret at the bubble's left edge. Geometry gates passed on all three platforms.

Cause: visibility: 'hidden' is a web style that React Native ignores. A box centered with left: 0, right: 0, margin: 'auto' centers in CSS, but native layout keeps it at left: 0. The second attempt, left: '50%' with a negative half-width margin, still landed one padding short, because native layout resolves an absolute child's percentage against the parent's content box, not its padding box. The walker measures cell boxes, not paint inside them.

Fix/Lesson: Hide with opacity or a transparent fill. Center an absolute part with a full-span absolute row (left: 0, right: 0) that centers an in-flow child, using no percentages or auto margins. The purity gate now rejects visibility in shipped source. Native screenshots get a layer-4 read every milestone, because a matching box can still hide a native-only drawing.

79. An anchored layer flipped and never clamped on native because the viewport was zero ​

Symptom: A bottom-placed tooltip drew above its anchor on iOS and Android, and a menu opened near the right edge ran off the screen.

Cause: The positioning behaviors read window.innerWidth, which is undefined on native, so every placement computed against a 0x0 viewport. The hosted menu also measured from its open effect, before the host had mounted its surface.

Fix/Lesson: On native the viewport comes from Dimensions (and the host's measured size when hosted). A layer the host mounts later measures from its own onLayout, not from the effect that opened it.

80. The test page's own cell border pushed a perceptual diff over budget ​

Symptom: The TextArea dark-scheme comparison against the web-component reference failed at 2.01% against a 2% budget on one machine and 2.02% on the Linux CI image, after passing at 2.00% on another. Every dark state sat between 1.4% and 2.0%, while the light states stayed under 0.9%.

Cause: The measurement page lays cells out on a fixed 160px row pitch, and each cell draws a dashed border. A TextArea is taller than the row, so the cell's bottom border crossed the field. The shot is cropped to the field, and the field sits at a different offset in each cell, so the border appeared at a different height in each image. On a dark backdrop the line counts as a mismatch; on a light one it falls under the pixel threshold.

Fix/Lesson: A measurement cell keeps its border's width but paints it transparent, so no measured position moves and no harness pixel enters a shot. A guard test fails if a measurement cell paints a border, outline or shadow. When one scheme's diffs all sit well above the other's for the same component, look for pixels the harness paints before tuning the component or the budget.

81. A unit test passed only while a timer had not fired yet ​

Symptom: A tooltip test failed in about one run in four on a faster machine, and had passed every run before. Nothing in the component or the test had changed.

Cause: react-native-web answers measureInWindow from a setTimeout, and React's act waits for microtasks, not timers. The test read the caret's edge right after rendering. When the timer had not fired, it saw the unmeasured placement. When it had, the zero-size test anchor flipped the placement and the caret moved to the other edge.

Fix/Lesson: A test of a measured layout waits for the measure explicitly (an act around a resolved timer) and asserts the measured result. An assertion that holds only before an asynchronous callback lands is a race, however often it passes.

82. A pseudo-element's line measured at its host's edge, not where it was drawn ​

Symptom: An inset divider measured its line 16 points away from the reference's, although both drew the line at the same place on screen.

Cause: The measurement read a pseudo-element's position as its left offset inside its host, which is right only for a positioned pseudo-element. The reference's line was an in-flow ::before inside a host with 16px of inline padding, so it started inside the padding, and the harness reported it at the host's edge.

Fix/Lesson: A positioned pseudo-element sits at its offset inside the host; an in-flow one sits inside the host's border and padding plus its own margin. When a measurement disagrees with what the screenshot shows, check how the harness locates the part before changing the component.

83. A large link icon drew a different glyph than the reference ​

Symptom: The large link's trailing icon measured 14.4 points tall at the top against the reference's 12.5, while the small and medium links matched.

Cause: The reference renders its icon component at its default 16px glyph and its stylesheet scales the box to 20px. The icon atom picked the set's own 20px glyph for a 20-point box, which is drawn on a different grid.

Fix/Lesson: The glyph a set draws and the box it is drawn in are two sizes. The icon atom takes glyphSize apart from size, and a component passes the glyph size its reference draws. Compare the painted path, not the box, when an icon disagrees.

84. Hover and focus scrolled a tag's content sideways ​

Symptom: A dismissible tag measured its label, close control and glyph 10 points left of the reference, but only while hovered, focused or pressed. At rest it matched.

Cause: The tag clipped its content with overflow: hidden, which makes the element a scroll container. Its close control's transparent press target was wider than the tag's end, and the pointer move and focus() both scroll a target into view, so the browser scrolled the tag's content by the overhang.

Fix/Lesson: A press target larger than its face must not sit inside an ancestor with overflow: hidden. Clip the text itself (numberOfLines with flexShrink), not the box. A shift that appears only in interaction states and equals a target's overhang is a scroll, not a layout bug.

85. A size every template shares held a label every template sizes differently ​

Symptom: Under the second template, every small tag drew its label crossing the pill's outline. No gate failed.

Cause: The small tag's height was a spec sum of two scale tokens, the same 18 points in every template, while the label's type set came from the template (a 20-point line there). The text overflowed a box that did not clip it, and the structural gates only checked the cell's bounds.

Fix/Lesson: A dimension that must hold template-sized content is a template cell, never a sum every template shares. The catch became the structural check that text stays inside the box painted behind it, shown failing on the old templates first.

86. The native assert guessed what the web would draw in a narrower room ​

Symptom: Each new family broke the native gate in a new way: an aspect-ratio cell on the smaller phone, a toast's height, a progress row wider than the screen.

Cause: The web reference was walked at desktop width and the native phones lay cells out in less room. The assert compensated with rules (a capped width bound, a height floor) that guessed the web's answer in a room it never measured, and every guess failed for some shape.

Fix/Lesson: Compare like with like. The web reference is walked once per native platform at that platform's grid width, the assert refuses reports of different widths, and cells compare directly within the pixel tolerances. A cell body also gives its content a definite maximum width, so a percentage maximum resolves the same way on both platforms.

87. A design system's token file and its rendered component disagreed ​

Symptom: Every interactive tag measured 1.1 points wider than the web-component reference's chip on an 11-character label, and its container read transparent upstream although the chip was visibly filled.

Cause: The chips' token file names a label tracking, but the chips' styles never apply it, so the rendered label computes letter-spacing: normal. The chips also paint their container on a ::before layer, so the element the reference measured was transparent.

Fix/Lesson: The rendered component is the reference, not its token file. When a measurement disagrees with a token-derived value, read the computed style of what actually paints, pseudo-elements included, and correct the template to the rendered value with that measurement cited.

88. Steps that shrink in CSS held their width natively ​

Symptom: A three-step progress indicator stayed 384 points wide on a phone with 344 points of room, while the reference shrinks its steps.

Cause: CSS flex items default to flex-shrink: 1, so the reference's 128-point steps shrink toward their 112-point minimum. React Native defaults flexShrink to 0, so the same declared width never gives.

Fix/Lesson: Porting a flex layout from CSS means stating the shrink the CSS default implied. A row of fixed-width items that the reference lets shrink takes flexShrink: 1 with the reference's minimum width.

89. A state just under the pixel budget hid a real difference ​

Symptom: The button set's three-button state passed the perceptual gate on macOS at 1.91% and failed on Linux CI at 2.04%, after the milestone merge. Every run on each platform gave the same number.

Cause: Ours centered each wide button's label; the reference holds it at the start padding. No measured part covered the label's position inside a seat, so only the pixel budget saw it, and the budget sat between the two platforms' anti-aliasing.

Fix/Lesson: A deterministic percentage near the budget is a difference until its diff image says otherwise; look at the images before calling it raster noise. The fix measured each seat's label as a part, so the measure gate now fails the same defect on every platform.

90. Browser cleanup erased the Node evidence from the same run ​

Symptom: A batch finished its Node tests, then its browser tests passed, but the evidence collector could no longer find the Node TAP file.

Cause: Playwright's outputDir was the common results directory. Starting the browser runner emptied that directory, including the earlier phase's TAP file.

Fix/Lesson: Give destructive runner cleanup its own child directory. Keep durable phase reports outside it and test that running the later phase preserves the earlier phase's evidence.

91. Capturing a screenshot changed the interaction being compared ​

Symptom: The measured focus outline agreed with the reference, but the screenshot showed a different outline on the last long-list sample.

Cause: Capture centered the cell again after opening its menu. The menu had increased the scrollable extent, so the second centering moved the page under a stationary pointer. A row received pointer entry and acquired a highlight after measurement but before capture.

Fix/Lesson: Position the cell before entering the requested phase. Preserve that viewport through measurement and capture; assert that capture leaves both the viewport and active descendant unchanged.

92. An additional action inherited its owner's interaction phases ​

Symptom: A count-chip clear action was clicked as though it opened a menu, while a decorative caret caused an actionability timeout.

Cause: The driver reused the main field's phase list for every extra target. It also treated a pointer-transparent painted caret as the element that must receive the event.

Fix/Lesson: Carry each target's own phase contract through every collector. A painted target can supply a measured point while its real governing control receives the event; the driver must not invent an impossible locator action.

93. Equal focus colors were mistaken for the same semantic role ​

Symptom: A sparse brand's normal focus color reached the controls, but a probe still reported a leaked template color on an inverse count chip.

Cause: The normal and inverse focus roles happened to share an unbranded RGB value. The probe inferred alias dependency from value equality, although the inverse role was independent. A second version incorrectly forced every control on an inverse surface to use inverse focus.

Fix/Lesson: Compare against the built role palette in the actual surface context. An inverse surface may carry normal or inverse focus. Calibrate that distinction with a normal-surface leak that still fails, an explicit inverse override, and missing-focus and browser-ring failures that remain failures.

94. Fractional line heights accumulated a native-only layout error ​

Symptom: Android groups and notifications were two to four points taller than the same font and theme on the web; the widths agreed.

Cause: A source type product was 18.00008. React Native's Android line-height span applied ceil to each line, producing 19 pixels at density one. Integer-valued themes did not expose the difference.

Fix/Lesson: Keep source tokens exact. Resolve final native text emission through the injected physical pixel grid, while retaining the web's own unit conversion. Test more than one density and prove that raw theme values are unchanged; do not round every token or enlarge a layout tolerance.

95. Automatically installed peers drifted outside the host SDK's versions ​

Symptom: Fresh installs succeeded, but the native host resolved animation peers newer than the versions its Expo SDK declared.

Cause: The host did not declare those peers, so npm selected newer versions through transitive peer requirements. A successful install did not prove SDK compatibility.

Fix/Lesson: Inspect the installed SDK's compatibility manifest and declare the host-owned compatible versions. Verify the resolved versions after a clean install rather than suppressing peer warnings or changing the SDK to fit an accidental resolution.

96. Native geometry passed while the scene itself was visually wrong ​

Symptom: Every component measured correctly, but dark native text appeared on a white page, status text crossed the notch, and a later Android route showed an earlier route's tooltips.

Cause: Component styles were themed but their host scene was not. Safe-area layout was absent. A global portal outside the navigation stack still displayed layers owned by a retained inactive screen.

Fix/Lesson: Review the actual simulator images before closure. A scene owns its page background, foreground, safe area and contained overlay lifetime; component geometry cannot prove those. Add a paint assertion that rejects the old image and a retained-scene ownership regression, while keeping warm navigation coverage.

97. A stopped verifier left its mutation runner alive ​

Symptom: A focused test failed against a source mutation nobody had just made, after the full verifier appeared to have stopped.

Cause: Terminating the parent and browser processes had left the separate mutation-runner child alive. It continued planting mutations while another check read the same working tree.

Fix/Lesson: Treat the verifier as an owned process tree, not one PID. Stop and verify every mutating child, preserve the incomplete run as incomplete, and check byte restoration before starting another gate.

98. A roster-sized screenshot proof shared one fixed test deadline ​

Symptom: The full browser suite passed locally, but CI timed out inside the pixel-pipeline proof after 30 seconds. The remaining comparisons passed.

Cause: One test captured every visible sample sequentially under one deadline. Roster growth increased its work without increasing test isolation. The reported screenshot stability wait was the active operation when the whole test expired, not proof of a permanently unstable component.

Fix/Lesson: Parameterize the proof by component and keep every visible sample, self-comparison, blank comparison and stale-gap assertion. Assert exact sample coverage per component. Keep the existing timeout and pixel thresholds; narrow the invisible-icon mutation to its named component proof.

99. A failing assertion over two DOM nodes killed the CI runner, not the test ​

Symptom: A mutation-check CI job died with exit 143 ("the runner has received a shutdown signal") at the same row on three consecutive runs, each time at a row whose planted edit makes a focus assertion fail. The full suite passed locally on a 32 GB machine, and the row fired locally. The job's logs were sometimes unavailable because the run was still in progress or the Actions service was degraded, so the first two occurrences were misread as infrastructure.

Cause: The assertion was assert.equal(document.activeElement, rows[2]). When it fails, node:assert formats a diff of both operands, and a DOM element in a Node DOM implementation is a very large circular graph. Formatting it exhausts the 7 GB GitHub runner, the kernel kills the runner process, and the job ends with 143 instead of a failing test. Locally the same formatting fits in memory, so the test fails normally and the mutation row "fires".

Fix/Lesson: Pass primitive operands to assertions whose values are DOM nodes: assert.equal(document.activeElement === rows[2], true, 'ArrowDown focuses gamma'). Strict identity is preserved; only the operand type changes. In the mutation checker, a child that dies without a normal test failure (a signal, an exit status other than 1, or output without a failing-test marker) is not a proof that the mutation was caught - refuse it and name the row. Put a --max-old-space-size ceiling on the children in CI and mirror it locally so a blow-up fails the child with a heap error instead of taking the runner down. A failing row that CI cannot name three times in a row is a defect in the harness's failure path, not bad luck.

Adding a New Entry ​

Whenever a new failure mode is discovered:

  1. Reproduce it once. Confirm the root cause is what you think it is.
  2. Add an entry to the right section above with Symptom, Cause, Fix/Lesson. Keep the numbering continuous within a section.
  3. If the rule is brief enough to live in the AGENTS.md compact summaries (Safe Terminal Patterns, healthcheck rules, etc.), recompile the compact mirror (AGENTS.md is a derived artifact, never edited directly).
  4. Commit the journal entry and the summary together so they never drift.

Doc drift is the slowest bug to find. No exceptions. Every new lesson goes here first, then propagates.

Cross-reference rules:

  • Pitfalls that belong to the architecture domain (docs/foundations/, docs/modules/, docs/server/, docs/testing/ - module migration, refactors) go in ../languages/js/pitfalls-migration.md, not here.
  • The philosophy docs cicd-publishing.md and testing-local-modules.md keep only the positive rules (what to do). Symptoms and root causes always live here.
  • Anchors in this file are stable. AGENTS.md and other cross-references rely on them. Never rename an H2 or H3 after it is published.

Released under the MIT License.

Copyright © 2025-present Superloom