Superloom

Appearance

Developer-Environment Pitfalls (AI Journal) ​

Audience: AI coding agents (Cascade, Copilot, Cursor) and humans debugging a specific CI, test, or terminal failure. Not first-read material. Start with the sibling philosophy docs; come here only when a concrete failure mode needs a confirmed fix.

Shape: Every entry is Symptom → Cause → Lesson/Fix. When a new failure mode is discovered, add the entry here first (per the Golden Rule in AGENTS.md), then recompile the compact rule into AGENTS.md (a derived artifact, never edited directly).

Scope: This file covers the docs/dev/ domain: AI tool-bridge, CI/CD publishing, local module testing. Architecture-level pitfalls (module migration, refactors) live in docs/testing/pitfalls-migration.md.

On This Page ​

AI Terminal & Shell Bridge ​

Canonical journal of every shell, terminal, and tool-bridge failure mode that AI assistants (Cascade, Copilot, etc.) have hit while working on this codebase.

Why AI agents fail differently from humans ​

A human at a real terminal sees the prompt change to dquote> and fixes the quoting. An AI tool-bridge sees nothing, hits the proposed-command timeout, and the entire turn appears to "hang". The same root cause produces different symptoms depending on who is at the keyboard.

Three structural differences matter:

  1. No PTY interactivity: the bridge cannot type into a sub-prompt. Once the shell enters dquote>, heredoc>, quote>, cmdsubst>, or bracket> continuation mode, there is no way out except killing the process.
  2. Output is captured asynchronously: anything that paginates (less, more, git log without --no-pager, man) blocks forever waiting for keyboard input that will never arrive.
  3. The current working directory is per-call: every run_command resets to whatever Cwd the tool-call specifies. There is no persistent session between calls, so cd in one call has no effect on the next call.

Every pitfall below is a specific consequence of one of these three. The mitigation is always the same: avoid asking the shell to do anything interactive.

Shell-bridge pitfalls ​

S1. Heredocs hang the bridge ​

Symptom: A cat <<'EOF' ... EOF command never returns. The user sees "no progress" until the tool-call times out.

Cause: Heredoc content with backticks, $, !, parentheses, or unterminated quotes can confuse zsh's parser. The shell enters a continuation prompt waiting for more input. There is no way to send the closing token through the bridge.

Fix: Never use heredocs through the bridge.

  • For file content: use write_to_file (or edit) directly.
  • For terminal-only sinks (e.g., appending to a gitignored file): write the content to a temp file with write_to_file, then cat /tmp/file >> /path/to/target.

S2. Multi-line git commit -m "…" ​

Symptom: The command appears in the approval popup with the closing " on a different line. After the user approves, nothing happens - shell sits in dquote> mode.

Cause: zsh treats every newline inside an open "…" string as part of the string. With special characters in the body (`, $, !, (...)), the parser also opens nested continuation states. None of them can be closed through the bridge.

Fix: Always pass commit messages on a single logical line.

NeedPattern
Single-line summarygit commit -m "feat(module): one-line summary"
Summary + body paragraphsgit commit -m "feat(module): summary" -m "Paragraph one." -m "Paragraph two."
Long structured bodywrite_to_file to /tmp/commit-msg then git commit -F /tmp/commit-msg && rm /tmp/commit-msg

The same rule applies to gh pr create, aws ssm put-parameter, and any other tool that takes a quoted message.

S3. Multi-line content embedded inside any quoted shell argument ​

Symptom: Same as S2 but for non-git commands. The approval popup renders the multi-line argument unreadably; after approval, the shell hangs.

Cause: Any time a run_command payload spans multiple lines because of an embedded quoted string, the shell-bridge sees the same dquote> failure mode.

Fix: Route the multi-line content through write_to_file to a temp file first, then have the shell read from the file with a single-line command.

S4. Backticks inside a double-quoted argument ​

Symptom: git commit -m "feat: closes issue \#123`"either runs the contents of the backticks as a command substitution, or hangs incmdsubst>` mode.

Cause: Inside "…", backticks always start command substitution. Escaping them with \` only sometimes works depending on shell version.

Fix: Use single quotes around the message, or replace the backticks with single quotes.

S5. Tilde expansion inside quotes ​

Symptom: cp ~/.npmrc "…" works; cp "~/.npmrc" "…" silently fails with "no such file".

Cause: Tilde is a shell metacharacter that only expands outside quotes (and not in all positions even there).

Fix: Use $HOME (which is a regular variable and expands inside double quotes) or omit the surrounding quotes for the path.

S6. Escaped brackets inside a regex character class make a sweep silently match nothing ​

Symptom: A new sweep grep returns no output and is read as "clean", but the pattern it was written to catch is demonstrably present in the file. The sweep passes forever and enforces nothing.

Cause: Inside a POSIX bracket expression a backslash is a literal character, not an escape. A class written as [A-Za-z_.\[\]'] therefore closes at the first unescaped ] - the one intended as an escaped literal - so the class becomes {A-Za-z, _, ., \, [} and the remaining '] is parsed as pattern text. The regex still compiles, grep still exits cleanly, and nothing ever matches.

Fix: Never put escaped brackets in a character class. Prefer a negated class that needs no escaping, and always prove a new sweep fires before trusting it:

bash
grep -nE "typeof [^ ]+ (!==|===) '(number|function)'" path/to/file.js

Lesson: Every new sweep grep is validated against a file that is known to contain a violation, before it is added to a workflow's sweep battery. A sweep that has never been shown to produce a hit is unverified, and a silently-empty sweep is worse than no sweep because it reports the codebase as conformant. This is the inverse of the usual grep convention where empty output is the pass condition: empty output only means "pass" once the pattern has been proven capable of failing.

Process and pager pitfalls ​

P1. Output paginators (less, more, man, vi) ​

Symptom: A command runs forever with no visible progress.

Cause: Paginators wait for keyboard input. There is no keyboard.

Fix: Never invoke an interactive viewer. The environment runs commands with PAGER=cat so most pager-aware tools (git, systemctl, journalctl) cooperate, but commands that ignore PAGER need explicit flags:

CommandFlag
git loggit log -n 20 (or --no-pager)
git diffgit --no-pager diff
journalctljournalctl --no-pager
systemctl statussystemctl --no-pager status …

P2. Long-running foreground processes ​

Symptom: npm run dev, node server.js, tail -f, docker compose logs -f block the bridge until they exit, which they never will.

Cause: A Blocking: true run_command waits for process exit. A foreground server never exits.

Fix: Use Blocking: false with a small WaitMsBeforeAsync (e.g., 2-3 seconds) so the tool returns after the startup output is captured. Then later, use command_status with the returned CommandId to fetch more output. Always remember to stop the background process at the end of the task.

P3. npm install of a watch-script package adding a postinstall daemon ​

Symptom: npm install completes, but a hidden postinstall script forks node …watcher.js & that keeps file descriptors open.

Cause: Some packages (rare, but they exist) start a background watcher in postinstall. The bridge sees the install command return but the descriptor lingers.

Fix: This is unusual in this codebase. If it ever happens, kill the dangling process via pkill -f <package> or restart the IDE.

P4. Command output exceeding the bridge buffer ​

Symptom: npm test finishes but the captured output is truncated mid-line.

Cause: Very long stdout streams can exceed the IDE's capture buffer, especially when test runners print verbose output for hundreds of tests.

Fix: Pipe to tail -N or grep to keep only what matters: npm test 2>&1 | tail -30, npm test 2>&1 | grep -E "^(ℹ|✖|✔)". The full log is still in npm-debug.log if needed.

P5. Background command + mismatched CommandId → false "command failed" conclusion ​

Symptom: A short, quick command (git commit, git push, gh api ...) is launched. Subsequent command_status polls return command <N> not found in trajectory, and read_terminal returns empty. The agent concludes the command failed and asks the user to run it manually - but the command actually succeeded (the commit is in git log, already pushed to origin).

Cause: The command was launched in background mode (Blocking: false) and the bridge returned a Background command ID (e.g. 506). The agent then polled command_status with a guessed or adjacent ID (395, 397, …) that never existed, producing the not found in trajectory error. Short commands also complete and clear before read_terminal can capture anything, so the terminal read comes back empty. None of this is evidence of failure - only that the command's output was never observed.

Fix:

  • For short, quick commands (git, gh, npm view, file checks), always use Blocking: true. Output is returned synchronously in the tool result.
  • If a command genuinely must run in the background, poll command_status with the exact CommandId the launch returned - never a guessed or incremented number.
  • Before reporting that a state-changing command failed, verify the actual state (git log --oneline -3, git status --short, gh api "$BASE/versions"). Absence of captured output is not proof of failure.

Reserve Blocking: false for genuinely long-running foreground processes (servers, tail -f, docker compose logs -f) per P2 - not for fast one-shot commands whose output you need immediately.

Working-directory pitfalls ​

W1. Missing Cwd runs from the repo root ​

Symptom: npm install reports ETARGET No matching version found for @superloomdev/... even though the package version is correct.

Cause: The bridge resets to the repo root for every run_command unless Cwd is explicitly passed. The repo-root package.json has a different dependency tree from each module's _test/package.json.

Fix: Every module-scoped command (npm install, npm test, docker compose …) must pass Cwd set to the module's _test/ directory.

bash
# Wrong - silently runs from repo root
# run_command: { CommandLine: "npm install" }

# Right
# run_command: { CommandLine: "npm install", Cwd: ".../js-server-helper-foo/_test" }

W2. cd <path> && <command> does not persist between calls ​

Symptom: A first call does cd src/foo && npm install; a second call assumes the cwd is src/foo and does npm test, but it runs from the repo root.

Cause: Each run_command is a fresh shell. There is no session.

Fix: Always pass Cwd to every call. Never rely on a previous cd. The user's AGENTS.md explicitly says never propose a cd command for the same reason.

W3. Relative paths in tool calls ​

Symptom: read_file({ file_path: "src/foo.js" }) fails or reads from the wrong directory.

Cause: Most tools require absolute paths. Relative paths are undefined behavior.

Fix: Always pass absolute paths to file tools (read_file, edit, write_to_file, find_by_name, grep_search).

Docker lifecycle pitfalls ​

D1. Manually starting Docker before npm test ​

Symptom: Bind for 127.0.0.1:NNNN failed: port is already allocated, or tests fail immediately with ECONNRESET.

Cause: pretest runs docker compose down -v --remove-orphans for its own compose project name; it does not touch a manually started container. Then pretest tries to bind the same port and fails.

Fix: Pick one owner of the Docker lifecycle: pretest already owns it. Locally and in CI, never run a separate docker run or docker compose up for the same service before npm test.

D2. Stale containers from a crashed prior run ​

Symptom: First test in a fresh session fails because a container from yesterday is still up but has stale state.

Fix: pretest already runs docker compose down -v --remove-orphans. If pretest itself fails on the start step, run that command manually from the same _test/ directory.

D3. docker compose up -d without --wait ​

Symptom: Tests start before the database accepts connections; intermittent ECONNREFUSED on the first request.

Fix: Always use docker compose up -d --wait. The healthcheck must be a real readiness probe (see docs/dev/testing-local-modules.md "Healthcheck Philosophy").

D4. docker compose --wait returns immediately for a service with no healthcheck ​

Symptom: --wait reports the container as Healthy 0.5 s after start, even though the application inside is still initializing.

Cause: When a service has no healthcheck: block, Docker treats "container is running" as healthy. --wait honours that.

Fix: Either define a real healthcheck, or have the test code retry the first connection a few times. DynamoDB Local is the typical case here - the image has no curl/wget/nc to probe with, so the test setup absorbs a brief retry instead.

Test-environment pitfalls ​

T1. AWS SDK calls without dummy credentials ​

Symptom: A test that exercises the AWS SDK takes 1-2 seconds and fails with no clear error.

Cause: With no credentials in the env, the SDK walks the default credential provider chain. The last step of that chain is the EC2/ECS instance metadata service at http://169.254.169.254. There is no metadata service on a developer machine or a GitHub Actions runner, so the chain times out.

Fix: Every AWS test must inject dummy credentials via the _test/package.json test script:

json
"test": "AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 node --test test.js"

The dummies do not need to be valid - they just need to exist so the SDK skips the metadata lookup.

T2. node --test test.js directly, without pretest ​

Symptom: Tests fail immediately with connection errors against the database or queue.

Cause: pretest did not run, so the container is not up.

Fix: Always use npm test. The lifecycle scripts exist for a reason.

T3. Concurrent top-level describe blocks ​

Symptom: "test did not finish before its parent and was cancelled".

Cause: Node's built-in test runner runs top-level describe blocks concurrently. Suites that share lazy-init state (DB pool, AWS SDK client) race each other on the first call.

Fix: Wrap stateful suites in a single outer describe('Module', { concurrency: false }, …). See docs/dev/testing-local-modules.md "Test Concurrency".

T4. Healthcheck passes during a transient-ready window ​

Symptom: Tests pass locally, fail in CI with Connection lost: The server closed the connection.

Cause: A healthcheck that returns "ready" too early. MySQL's two-phase init is the classic case: mysqladmin ping -u root passes during phase 1, then the server restarts in phase 3 and drops every live connection.

Fix: Probe with the credentials, database, and transport the tests will use. See docs/dev/testing-local-modules.md "Healthcheck Philosophy".

File-tool vs terminal pitfalls ​

F1. Using cat to read large files ​

Symptom: Output is truncated at the bridge buffer limit, or the call times out.

Fix: Use read_file with offset + limit. Never cat files larger than ~200 lines.

F2. Using sed / awk / tr to edit files ​

Symptom: The replacement either misses the target line, mangles whitespace, or partially succeeds.

Cause: Stream editors are error-prone for surgical edits, especially when the target string contains regex metacharacters or quotes.

Fix: Use the edit or multi_edit tool with old_string set to the exact unique substring. The tool guarantees an exact match or a clear error.

F3. Editing a file with a heredoc through cat > file <<EOF ​

Same root cause as S1. Always use write_to_file or edit instead.

F4. Reading gitignored files via read_file ​

Symptom: read_file returns "file is gitignored" for __dev__/.env.dev.

Cause: The IDE's file tools intentionally refuse gitignored paths to prevent accidentally exposing secrets in chat output.

Fix: For gitignored files, use cat /path/to/file via run_command (which is allowed but visible in the approval popup). For writes, write to a temp file via write_to_file and then mv or cat >> it into place.

Auto-run / safety pitfalls ​

A1. Auto-running a destructive command ​

Symptom: The agent flips SafeToAutoRun: true on rm -rf, git push --force, docker volume rm, or npm publish and the user has no chance to review.

Cause: The agent over-trusts a previous successful execution and decides the next call is "obviously fine".

Fix: Auto-run is reserved for read-only operations and idempotent reads (git status, git log -n 20, npm test, docker ps). Anything that mutates state on disk, in a remote registry, or in a long-running service must always require user approval, even if the user has previously approved similar commands. The user's AGENTS.md explicitly forbids npm publish regardless.

A2. Running npm publish directly ​

Symptom: A package is published from the developer's laptop instead of from CI, with the wrong author or unsigned provenance.

Cause: The agent saw a successful test run and decided to publish.

Fix: Publishing in this codebase is CI-only via .github/workflows/ci-publish-helper-modules.yml. Bumping the version in package.json and pushing to main is the only trigger. The CI workflow's content guard skips only when the packed shasum matches what the registry already serves (entry 26).

A3. Force-pushing to a shared branch ​

Symptom: A git push --force rewrites main's history, losing other contributors' commits.

Fix: Never force-push without explicit user approval. The user's AGENTS.md lists this as Never in the boundaries section. Use --force-with-lease if the user explicitly asks to amend.

A4. Modifying .env files ​

Symptom: A pre-existing .env file gets overwritten and the developer loses their local credentials.

Fix: The agent's allowed write locations are spelled out in AGENTS.md "Boundaries". .env is in the Never category (except __dev__/.env, which is the user's personal workspace). For new env keys, update .env.example files only.

Verification-integrity pitfalls ​

Failures where the agent's report is wrong rather than its code. These are the most expensive class because they consume the user's trust in every other claim.

V1. Reporting a gate as passed without running it ​

Symptom: The agent states a validation workflow is complete ("/finalize-docs converged", "docs verified") after having only made edits and run one adjacent command. A later audit finds defects the gate's own passes were designed to catch.

Cause: Substituting a cheap proxy signal for the gate. A green website build gets treated as evidence for link integrity, terminology consistency, and rule mirroring, none of which it tests.

Fix: A gate is passed only when its steps were individually executed and each produced its required evidence. If the passes were not run, the report says so. Name the specific commands and file reads that produced each count. When a workflow demands per-pass evidence, an unproduced count is a failed pass, not an assumed one.

V2. Inventing a plausible mapping to close a finding ​

Symptom: Asked to fix a table row that pointed at a non-existent target, the agent wrote a different target that exists, with a parenthetical implying the content lives there. It did not. The visible error became an invisible false assertion.

Cause: Optimizing for a row that looks resolved instead of one that is true. A vague parenthetical ("folded into the X rules") reads as authoritative and survives review.

Fix: Read the destination and confirm the content is actually present before asserting a mapping. When the honest answer is "not represented anywhere", write that; an explicit gap is a working input to the next gate, while a fabricated mapping silently suppresses it. Applies to any derived-artifact index: section maps, coverage tables, traceability matrices.

V3. Green build accepted as anchor evidence ​

Symptom: Renamed a heading, updated its inbound cross-file link, left the same-file On This Page anchor stale. The site built clean three times.

Cause: VitePress does not fail on unresolved same-file fragments, so the build cannot detect them.

Fix: Verify anchors by deriving slugs from the target file's actual headings and comparing, including a file's own table of contents. Renaming a heading invalidates every inbound anchor, and the ones in the same file are the easiest to miss.

V4. Fabricated precision in a gate report ​

Symptom: The 2026-07-31 P3 run of /finalize-docs reported Rules in source inventory: 423 and Rules mirrored in AGENTS.md: 417 without counting either figure. The 6 findings the run surfaced were genuine and independently verified; the totals were not. A reader comparing the report to the actual source files would find no extraction that produced 423 and no count that produced 417.

Cause: The gate was run honestly - findings were checked, evidence was produced for each - and then the result was decorated with measured-looking numbers that were estimated rather than counted. This is distinct from V1 (the gate was not skipped) and V2 (no mapping was invented). The softer failure mode is that a real pass produced real findings, and then precision was added after the fact to make the report look more rigorous than the method supports.

Fix: Any number in a gate report is a claim requiring the same evidence as a pass/fail verdict. A total that is not the arithmetic sum of counted parts must not be emitted. The P3 output block now enforces this with the identity A + B + C + D = N; a total that does not equal the sum of its parts is a detectable error rather than a plausible-looking number.

V5. A transitional repository policy absorbed as permanent constitution doctrine ​

Symptom: The constitution stated that a shasum mismatch at the publish guard is fixed by deleting the registry version, and that the version must never be bumped to clear one. Both were true only of repositories running a temporary pre-release convention that pins every package at one version. Every other repository read the rule and got the destructive remedy: deleting a version consumers had already resolved, breaking their lockfiles. A second copy of the same leak sat in agent-configuration.md, which asserted a package version floor as a framework rule.

Cause: A journaling run captured a real failure while a temporary policy was in force, and the policy travelled into the rule as an unstated premise. Nothing in the sentence marked it as conditional, so it read as universal. The existing passes cannot see this: each statement of the rule agreed with the others, the links resolved, the tables matched their detail sections, and the site built. Rule agreement checks whether repeated statements match each other, not whether the matched statement sits at the right layer.

Fix: A rule that only holds under a policy names the policy and names where the policy is declared. Split the two halves. The permanent half states the mechanism: a guard compares a fingerprint of content. The policy-dependent half states the remedy and defers it to the repository's own standing-rule file. The default remedy is the unsurprising one, so a repository that declares nothing still gets correct behavior. When journaling a failure found under a temporary convention, write the convention into the entry's cause as a condition, never into the rule as a premise. The detection question at validation time: for every "never do X" in the constitution, name the repository where it is false. If one exists, the rule is policy-dependent and mislayered.

V6. Raw fixture comparisons conceal a broken runtime pipeline ​

Symptom: Profile tests and lint pass while the real engine rejects every profile, a bridge overwrites dotted siblings, and rendered typography overrides authored metrics. Tests for an unavailable package export skip after catching any import error, so the acceptance report stays green.

Cause: Tests compare selected raw literals rather than executing the engine, bridge, validation, utility generation, and component branches together. Negative controls compare two reference values instead of mutating the implementation input and exercising the real assertion. Static searches miss dynamic token names and lowercase semantic aliases. Filtering test output through a shell pipeline can also replace the test process's failure status with the filter's success status.

Fix: Test the actual public pipeline and assert emitted and rendered values. Preserve strict missing-token failures, including dynamic branches. Required import errors remain failures, not automatic skips; park dependent acceptance until its registry artifact exists. Run commands without output filters or preserve their exit status with pipefail. After a same-version release, verify the committed consumer lockfiles, not only the regenerated working copies. Existing smoke counts do not certify newly added behavior.

V7. A stale dev server serves output from a previous dependency state ​

Symptom: A Vite dev server started hours ago on port 5173 continues serving stale module state after a dependency is republished and node_modules is refreshed. A new vite invocation silently moves to port 5174 or 5175 because the configured port is occupied, so the developer inspects the wrong server and believes the build is current. Playwright, which builds its own production preview on port 4173, passes its tests while the dev server the developer is looking at serves a completely different layout.

Cause: Vite's dev server caches transformed modules in memory and does not detect that the installed packages changed underneath it. Its port-increment behavior makes the stale server and the fresh server coexist without warning. A production preview build produces correct output from the current tree, but that proves nothing about the dev server the developer is interacting with.

Fix: Production Playwright must always use a fresh server (reuseExistingServer: false in CI and npm run verify). A build identity endpoint (git SHA + lockfile hash) lets a freshness check compare the served identity to the identity computed before launch. A dev:fresh command checks the configured port, fails loudly with PID and remedy on a mismatch, and never silently increments the port. Killing a stale process remains an explicit human action.

V8. Presence-only E2E tests pass while the rendered UI is geometrically broken ​

Symptom: Tasks and Notes E2E suites assert that expected text appears, that adding data works, and that navigation works. All pass. The Notes banner renders at 42px/60px (a stale caption02 token), producing a 300px banner and a 762px page. A typography regression suite with an 84px global ceiling also passes because 42 is under 84. No test measures computed font size, line height, overflow, overlap, or page geometry.

Cause: Text presence and interaction success are necessary but not sufficient. A global font-size ceiling catches only extreme blowups and lets any wrong-but-smaller value through. Computed typography and layout geometry are deterministic in a fixed browser environment and can be asserted exactly, but only if a test reads them.

Fix: Application-level acceptance uses four complementary gates: contract (exact generated token values), functional readiness (response, mount, console/request errors, interaction), geometric layout (exact typography, spacing, overflow, overlap, reflow at fixed viewports), and visual regression (toHaveScreenshot baselines). Deterministic values use exact assertions, not ranges. A global ceiling does not replace a semantic expectation.

Rule Delivery ​

1. AI attribution trailers appear in commits despite a documented ban ​

Symptom: Commits in codebase-js-helper-modules carried Co-Authored-By: Devin and Generated with [Devin] trailers, even though the no-attribution rule already existed in codebase-superloom/AGENTS.md:78 and docs/ai/agent-configuration.md:115.

Cause: codebase-js-helper-modules had no AGENTS.md at all. The rule was never in the agent's context while it committed there. The CLI's built-in commit template, which appends attribution trailers by default, won because no project-level rule overrode it. A rule that exists only in the constitution repo is not in force in the repos where work actually happens.

Fix: Every repo an agent commits to must carry the no-attribution rule in its own AGENTS.md, stated in full (not by reference). The rule must explicitly say it overrides any tool's built-in commit template. Three of five workspace repos had no AGENTS.md; all four now have self-contained copies with the rule stated verbatim.

Generalization: A rule's reach equals the set of files an agent actually reads. A rule present in a repo the agent never opens during a session is invisible. Delivery is not authoring; a rule undelivered is a rule unenforced.

CI/CD Publishing ​

Each entry below maps a CI symptom to its root cause and the durable fix. The sibling philosophy doc is cicd-publishing.md; this section is the journal of real failures that shaped those rules. For the local CI parity contract, see local-ci-parity.md.

1. Bind for 127.0.0.1:NNNN failed: port is already allocated in CI ​

Cause: A workflow step started a container with docker run -d --name foo -p 127.0.0.1:NNNN:NNNN ... before the test step ran npm test. pretest then runs docker compose down -v (which does not touch the standalone container) followed by docker compose up -d --wait, which collides on the same port.

Lesson: Pick one owner of the Docker lifecycle. pretest already owns it both locally and in CI. Do not duplicate it with a workflow-level docker run step or a services: declaration that targets the same port - pick one and remove the other.

2. PROTOCOL_CONNECTION_LOST only in CI ​

Cause: The healthcheck passed too early. On the developer's laptop the service finished init fast enough to be truly ready when the healthcheck passed; on a slower CI runner, the false-positive moment was wide enough that the test connected before init was complete, and the service then dropped the connection during a real init step.

Lesson: See testing-local-modules.md -> Healthcheck Philosophy. The healthcheck must probe at the same level the test will use (credentials, database, transport). Add start_period and enough retries that the total budget covers a cold-pull, cold-start initialization.

3. Push triggers tests but not publish, even though version was bumped ​

Cause (historical): The previous detect logic compared HEAD~1:package.json to HEAD:package.json. After a force-push or a reset that left the version the same on both sides, the diff was empty and publish was skipped - even when the registry did not have the package.

Fix: The detect job now asks the registry directly instead of diffing git history. It compares the packed shasum against npm view <name>@<version> dist.shasum, so publish is scheduled whenever the version is absent or the content differs (entry 26).

4. CI runs tests for every module on every commit ​

Cause: test_modules was incorrectly populated - typically because the regex used \w+ instead of [\w-]+ and matched module paths greedily, or because someone pushed a single commit that touched every module's package.json.

Lesson: Keep the path regex hyphen-aware: src/helper-modules-[\w-]+/js-[\w-]+. If a single commit truly does touch every module (e.g., a sweep), wide test coverage is the correct outcome.

5. npm publish runs but 409 You cannot publish over the previously published versions ​

Cause: A workflow that publishes on every main push, without checking version bump or registry state.

Lesson: Use the unified detect -> publish-* pipeline. Both gates compare the packed shasum against the registry's dist.shasum, which is the canonical way to avoid this error. A gate that checks version presence alone swallows the failure instead (entry 26). Do not rebuild a separate publish workflow.

6. 403 Forbidden on npm publish ​

Cause: Missing permissions: packages: write on the publish job, or repository-level workflow permission set to "Read repository contents permission".

Lesson: Both must be set:

  1. Job-level permissions: { contents: read, packages: write }
  2. Repo-level Read and write permissions in Settings -> Actions -> General

7. CI test fails with npm error notarget No matching version found for @superloomdev/... ​

Cause: The CI step ran npm install from the wrong directory (typically the module root instead of _test/). Each _test/ directory has its own package.json with its own dependency tree. The repo-root package.json does not declare the test deps.

Lesson: In CI, set working-directory: to the module root and use cd _test && npm install && npm test for the test step. The same rule applies locally - always pass Cwd to _test/ for AI agents and scripts.

8. CI test fails with ERR_MODULE_NOT_FOUND for a helper that exists in the repo ​

Symptom: A test-* CI job fails with a Node.js ERR_MODULE_NOT_FOUND stack trace pointing to a file inside another helper module (e.g. js-server-helper-nosql-mongodb/mongodb.js), despite that module being present in src/. The error occurs even though file:../../js-server-helper-nosql-mongodb is listed in the _test/package.json dependencies.

Cause: file: path dependencies copy the directory contents at npm install time but do not run npm install inside the linked package. In CI, the runner checks out a fresh clone; the linked helper's own node_modules/ are absent, so any import inside it that needs its own npm deps (e.g. the mongodb driver) fails immediately with ERR_MODULE_NOT_FOUND.

This works locally only because the helper was previously installed in its own directory during development. CI never does that.

Lesson: Never use file: paths in _test/package.json for helper modules that have their own npm dependencies. Use registry version ranges ("^1.0.0") instead. The file:../ self-reference (pointing to the module under test) is the one legitimate exception. npm installs it as a directory link and the module itself has no transitive runtime deps outside what the test loader provides. For every other shared helper (storage, database, cloud), always pin to the published registry version.

Quick rule: file: is allowed only for "[module-under-test]": "file:../". Everything else must be a registry semver range.

9. Tests Fail in CI ​

The publish job has needs: [detect, test-*], so it never runs if tests fail. Fix the tests and push again. The detect job will pick the same set of unpublished modules and try again.

10. Invalid workflow file: ... You have an error in your yaml syntax on line N ​

Cause: A bash assignment inside a run: | block scalar contained an embedded literal newline:

yaml
run: |
  PUBLISH_MODULES="$PUBLISH_MODULES$MODULE
"

YAML block scalars (|) require every non-empty line to be indented at least to the block's indent level. The closing " on its own line had zero leading spaces - less than the block's indent - which terminates the block scalar early and fails YAML parsing.

Lesson: Never embed a literal newline inside a bash string assignment within a YAML run: | block. Use bash's $'\n' escape (or printf '%s\n', or a bash array) so every line of YAML respects the block's indentation:

yaml
run: |
  PUBLISH_MODULES="${PUBLISH_MODULES}${MODULE}"$'\n'

This applies anywhere YAML uses block scalars: GitHub Actions run:, Docker Compose command:, Helm chart values, CI configs, etc.

11. publish-* jobs silently skip when an upstream publish-* is also skipped, breaking the test→publish chain ​

Symptom: A test-* job runs and succeeds. The next publish-* job in the chain is reported as skipped (zero steps, zero seconds). Multiple downstream test-* jobs then run against a stale registry version and fail with cryptic runtime errors like TypeError: mongo.createIndex is not a function. The bumped helper version that supplied the new API was never actually published.

Cause: GitHub Actions evaluates an implicit success() check on every job that does not start its if: with a status-check function (always(), failure(), cancelled(), !cancelled()). And success() is transitive. It returns false if any job in the upstream needs graph (not just the direct needs) has the result skipped, failure, or cancelled. Quote from GitHub's docs: "If a job fails or is skipped, all jobs that need it are skipped unless they use a conditional expression that causes the job to continue."

A strictly-sequential test→publish pipeline like ci-publish-helper-modules.yml mixes:

  • test-* jobs that use if: always() && !cancelled() && ... → run regardless of upstream skips
  • publish-* jobs that previously used if: >- needs.detect.outputs.publish_modules != '[]' && contains(...) → no override, so the implicit success() applied

In fresh-state recovery runs (every module needs publishing), nothing in the chain is skipped, so this never fires. In steady-state runs where some modules are already on the registry, those modules' publish-* jobs legitimately skip; that skip silently propagates downstream, disabling every subsequent publish-* even though their direct needs (detect + the matching test-*) all succeeded.

Concrete failure that surfaced this: A run with publish_modules = [auth, logger, nosql-aws-dynamodb, nosql-mongodb, verify]:

  • publish-storage-aws-s3 correctly skipped (s3 already on registry, not in publish_modules)
  • test-nosql-aws-dynamodb ran and succeeded (uses always())
  • publish-nosql-aws-dynamodb SKIPPED (even though it was in publish_modules and its direct needs succeeded) because success() walked the transitive chain back to publish-storage-aws-s3 and saw skipped
  • All downstream publish-* skipped for the same reason
  • mongodb@1.1.0 and dynamodb@1.1.0 never reached the registry
  • Then test-verify, test-logger, test-auth ran with npm install resolving ^1.0.0 to the stale registry 1.0.0, which lacked the createIndex / createTable / deleteRecordsByFilter APIs the 1.1.0 source code calls. TypeError at first use

Lesson: Every publish-* job in a chained pipeline must override the implicit success() check and assert its own dependencies explicitly:

yaml
publish-foo:
  needs: [detect, test-foo]
  if: |
    !cancelled() &&
    needs.detect.result == 'success' &&
    needs['test-foo'].result == 'success' &&
    needs.detect.outputs.publish_modules != '[]' &&
    needs.detect.outputs.publish_modules != '' &&
    contains(needs.detect.outputs.publish_modules, 'js-server-helper-foo')

Two parts that both matter:

  1. !cancelled() && disables the implicit transitive success() check (GitHub's recommended alternative to always() for normal jobs, per the official docs).
  2. needs.detect.result == 'success' && needs['test-foo'].result == 'success' restores the safety the implicit check used to give us, but scoped to the direct needs only. Hyphenated job IDs require bracket notation (needs['test-foo'], not needs.test-foo; the latter is parsed as subtraction).

Defence in depth: pin _test/package.json to the version your code actually requires. When a helper bumps its own version (e.g. mongodb@1.1.0 adds createIndex) and downstream modules start calling the new API, every consuming _test/package.json must pin that same ^1.1.0, not the older ^1.0.0. Two reasons:

  • If the upstream publish ever fails, ^1.1.0 causes npm install to fail with a clean E404 No matching version instead of installing the stale 1.0.0 and surfacing as a TypeError deep inside a test.
  • The version pin documents the hard floor required by the source. Anyone reading the test deps sees exactly what the module needs.

Quick rules:

  • _test/package.json registry pins must match the API surface the source code uses, not the lowest published version.
  • Every publish-* job in a chained workflow must start its if: with !cancelled() (or always(), failure(), cancelled()) and re-assert its direct needs.<job>.result == 'success'. Otherwise a single skipped sibling silently disables the rest of the publish chain.

12. Main module and its adapters are both unpublished: bootstrap order ​

Situation: Any module that ships a main package plus separate adapter packages (e.g. js-server-helper-auth + js-server-helper-auth-store-sqlite) can reach a state where both the main module and one or more adapters have unpublished breaking changes. Neither side can reference the other via a registry semver range because the version does not exist yet.

The correct sequence:

  1. Test adapters locally against a file: path of the main module. In each adapter's _test/package.json, temporarily point the main module dependency at the local checkout (file:../../js-server-helper-<name>). Run the full adapter test suite. This validates the new API contract end-to-end without touching the registry.

  2. Publish the main module (bump version in package.json, push to main; CI publishes it).

  3. Switch adapter _test/package.json deps from file: to the newly published registry version (e.g. "^2.0.0"). Re-run the adapter test suite against the live registry version. This confirms the package round-trips correctly through npm pack/publish and resolves correctly for real consumers.

  4. Publish the adapters (bump versions, push to main; CI publishes them).

Why the re-test in step 3? file: deps are copied at npm install time, bypassing npm's pack/unpack pipeline entirely. A module that works locally via file: can silently fail after publishing if the files field in package.json is wrong, a required file is gitignored, or a main entry resolves differently after packing. Step 3 catches this class of error before any consumer is affected.

Lesson: The temporary file: reference is the one legitimate exception to the rule that _test/package.json only uses file: for the module under test itself. It is only valid during the bootstrap window and must be replaced with a registry pin before the adapters are published. Add a comment so the intent is clear:

json
"@superloomdev/js-server-helper-<name>": "file:../../js-server-helper-<name>"

Replace with "^<version>" after the main module is published (step 3).

14. test-verify CI fails with Cannot find module '../stores/sqlite' after stores/ was deleted ​

Symptom: test-verify in CI fails with ERR_MODULE_NOT_FOUND pointing to ../stores/sqlite inside the verify module's own _test/test-sqlite.js. The stores/ directory was intentionally deleted as dead code in the same commit that introduced standalone js-server-helper-verify-store-* packages.

Cause: Two related problems compounded:

  1. The internal test file _test/test-sqlite.js still referenced the deleted ../stores/sqlite path. It was not updated to use the npm package @superloomdev/js-server-helper-verify-store-sqlite.

  2. The five js-server-helper-verify-store-* adapter packages had no CI jobs at all. ci-publish-helper-modules.yml skipped directly from publish-sql-postgres to test-verify. This meant the adapters were never published to the registry, so _test/package.json could not resolve them even after the test file was fixed.

Additionally, the adapter package.json files had been (incorrectly) set to private: true, which would have prevented npm publish even if CI jobs existed.

Fix:

  1. In _test/test-sqlite.js: replace import('../stores/sqlite') with import('@superloomdev/js-server-helper-verify-store-sqlite').
  2. In _test/package.json: add "@superloomdev/js-server-helper-verify-store-sqlite": "^1.2.0" to dependencies.
  3. In all five adapter package.json files: set "private": false so npm publish works.
  4. In ci-publish-helper-modules.yml: insert ten new jobs (modules 17-21), one test-verify-store-* + publish-verify-store-* pair per adapter, in the sequential chain before test-verify. Update test-verify's needs to [detect, publish-verify-store-dynamodb].

Lesson: Deleting an internal directory (stores/) that is still referenced by the module's own test suite, while simultaneously introducing replacement standalone packages, requires two coordinated changes: (a) update every import that pointed at the old path, and (b) ensure the new packages exist in the CI pipeline and are published before any consumer runs npm install. A quick grep for the deleted path before committing prevents the first problem; checking whether each new adapter has CI jobs prevents the second.

13. CI fails on lint after local tests pass: pre-publish checklist not followed ​

Symptom: npm test in _test/ passes locally (all green). The version is bumped and pushed to main. CI runs npm run lint from the module root as a separate step before tests and fails with no-trailing-spaces or a stale eslint-disable-line directive. The publish job never runs even though all functional tests would have passed.

Cause: The local test command (node --test test.js from _test/) never invokes ESLint. Lint is a separate npm run lint script defined in the module root's package.json, not in _test/package.json. CI always runs lint before tests; local workflow often only runs tests. Any whitespace issue or stale disable comment that was present (or introduced) before the push goes undetected locally.

Lesson: Before bumping the version in package.json and pushing to main, always run the full pre-publish sequence from the module root, not just from _test/:

bash
# From the module root
npm run lint          # must exit 0
# From _test/
npm install && npm test  # must exit 0

Both must be green before the version bump commit. If lint fails, fix it first. Do not push a lint-broken version and rely on CI to catch it. The lint fix will require a second commit and re-trigger the entire CI pipeline, wasting pipeline time and adding noise to the git log.

Quick rule: Treat npm run lint (module root) + npm test (_test/) as a single inseparable gate. Never bump a version without both passing locally.

15. VitePress / Vue compiler crashes on bare angle-bracket placeholders in markdown ​

Symptom: The Deploy Website workflow fails during vitepress build with an error like:

[vite:vue] [plugin vite:vue] docs/architecture/<file>.md (NNN:1):
  Element is missing end tag.
SyntaxError: [plugin vite:vue] ...

The reported line number is often far below the line that actually caused the problem. The parser walks until it gives up, then reports the position at which it abandoned the parse.

Cause: VitePress feeds rendered markdown through Vue's compiler so that authors can use Vue components inside markdown. The compiler does not unconditionally trust markdown's HTML-escaping: any <lowercase-or-PascalCase-name> inside the rendered output is treated as a Vue/HTML element open tag and triggers the close-tag search. Three concrete failure modes seen on this codebase:

  1. Bare placeholders outside any code formatting. A bullet like - <Domain Operations> *(convenience layer)* parses as opening element <Domain with attribute Operations, then waits forever for </Domain>.
  2. Placeholders inside a fenced code block tagged ```markdown. Vue still scans the block's content for tags because the markdown language hint enables an additional templating pass. Placeholders such as <sibling-1> and <one-line description> inside a markdown-tagged fence will trip the same error.
  3. Placeholders inside backticks are usually safe (they render as HTML-escaped <code> content), but a few corner cases (extremely long backtick spans, nested code formatting, generic-looking names like <T>) have produced the same failure.

Lesson: Treat angle-bracket placeholders in markdown as a build-breaking smell. Three durable rules:

  1. Never use <...> placeholders outside backticks in prose. Use square brackets ([name]), curly braces ({name}), or plain capitalized phrases ("Domain operations") instead.
  2. For verbatim copy-paste templates inside fenced code blocks, prefer the text language hint. Switching ```markdown to ```text disables the secondary Vue scan and lets the placeholders survive untouched. If the syntax-highlighting loss is unacceptable for a particular block, replace the angle-bracket placeholders with square-bracket ones at template-author time and keep the markdown hint.
  3. Verify documentation changes locally before pushing when the change touches any VitePress-rendered file in docs/. Run npm run build from website/. That is the same pipeline CI runs (vitepress build . after sync-docs). Watch for the Element is missing end tag family of errors. Local build is fast (single-digit seconds) and catches the failure before it occupies a CI runner.

This pitfall is distinct from the helper-modules CI chain (entries 1–14): it lives in ci-deploy-website.yml, not ci-publish-helper-modules.yml, and a website-deploy failure does not block module publishing. The two pipelines are independent. But the same commit that triggers helper-module publishing will also trigger website deploy if it touches any docs/ file, so a documentation-side bug is a hidden cost on every push that updates rules.

16. Deleting a GitHub Packages package: org-scoped vs user-scoped endpoint ​

Symptom: gh api --method DELETE /users/OWNER/packages/npm/PACKAGE/versions/ID returns success (exit 0, empty body) but the package is still on the registry. Repeated attempts appear to "do nothing", and the agent cannot tell whether the delete worked.

Cause: superloomdev packages are owned by a GitHub organization, not a user. Organization-owned packages live under /orgs/OWNER/.... The /users/OWNER/... endpoint targets the user-account namespace; for an org-owned package it returns a misleading no-op instead of an error. Two further traps: the API path uses the bare package name without the @scope/ prefix (js-server-helper-http-gateway, not @superloomdev/js-server-helper-http-gateway), and deleting the last remaining version deletes the whole package.

Lesson: Determine the owner type first, then use the matching endpoint and always verify with a 404.

bash
gh api /users/OWNER --jq '.type'        # "Organization" or "User"
# org:        BASE=/orgs/OWNER/packages/npm/PACKAGE_NAME
# own user:   BASE=/user/packages/npm/PACKAGE_NAME
gh api "$BASE/versions"                  # list / confirm it exists

For same-version republish, delete only the specific version ID, never all versions. Bulk deletion is forbidden by the autonomous execution protocol. The correct deletion is:

bash
gh api "$BASE/versions?per_page=100" --jq '.[] | select(.name == "VERSION") | .id'
# Require exactly one ID, then:
gh api --method DELETE "$BASE/versions/VERSION_ID"
gh api "$BASE/versions?per_page=100" --jq '.[] | select(.name == "VERSION") | .id'
# Must return empty

If the verify step returns a JSON array, the delete did not work - re-check the owner type and path.

Confirmed behavior after deletion: GitHub Packages accepts republishing a previously-used version name once the version is deleted - version names are not permanently burned. Combined with the detect job's content guard, this enables same-version republish. The historical 44-module wipe-and-republish was a one-time re-baseline under a different policy; current plans delete only the exact named version. Round-trip proof after republish: npm install @scope/[name]@[version] from a scratch directory must resolve.

17. git add . bundles unrelated modules into one commit ​

Symptom: A single commit touches many modules at once. CI's per-module detect job then schedules tests/publish for every changed module path in that commit, and git history no longer maps one module refactor to one commit ("bulk commits").

Cause: git add . (or git add -A) stages every modified file in the working tree, not just the module currently being worked on. During a multi-module sweep (e.g. refactoring a parent plus its adapters), in-progress edits to sibling modules get swept into the same commit.

Lesson: Stage only the directory of the module being committed, so each module refactor is one independent commit:

bash
git add src/helper-modules-server/js-server-helper-<name>/
git commit -m "refactor(<name>): one-line summary"

This keeps the CI publish trigger scoped to a single module, makes history bisectable, and lets one module be reverted without disturbing others. Always run git status --short before committing and confirm only the intended module's files are staged.

23. Extension module's test job 404s on its base package: needs pointed at the base's test job, not its publish job ​

Symptom: During a full-registry republish, an extension module's test job (e.g. test-[base]-ext-[framework]) fails with npm error 404 Not Found ... @scope/[base]@^1.0.0 while the base module's own test and publish jobs are green in the same run.

Cause: The extension's _test/package.json installs the base module from the registry (a registry semver range, not file:). The extension's test job was chained with needs: [detect, test-[base]], so it started as soon as the base's tests passed - racing the base's publish job. On a fresh registry the base package does not exist yet when the extension's npm install runs.

Lesson: A test-* job whose _test/package.json installs another in-repo package from the registry must needs that package's publish-* job, never its test-* job. In steady-state runs the difference is invisible (the base is already published, both orderings pass), so the bug only surfaces during bootstrap or a registry re-baseline - review every extension and adapter chain against this rule before a fresh-state push.

Coverage-sweep footnote: when verifying that every module has CI jobs by extracting paths from the workflow file, the extraction pattern must cover the full name alphabet. A grep -oE character class of [a-z-]+ silently drops any path containing a digit (storage-aws-s3), producing false "missing from CI" findings. Include digits ([a-z0-9-]+) and cross-check the extracted count against the known module total before acting on the diff.

24. npm error 409 Conflict - Package file checksum mismatch after deleting and republishing a package ​

Symptom: CI test job fails with npm error code E409 and npm error 409 Conflict - GET https://npm.pkg.github.com/download/@scope/package/version/hash - Package file checksum mismatch. The package was just republished at the same version after a registry deletion.

Cause: Lock files (package-lock.json) in consuming repos cache the old tarball integrity hash. When the package is deleted from the registry and republished at the same version, the new tarball has a different hash. npm ci compares the cached hash in the lock file against the registry tarball and rejects the mismatch.

Lesson: After deleting and republishing a package at the same version, every consuming repo's lock file must be regenerated:

bash
rm -rf node_modules package-lock.json && npm install

This applies to all lock files in the repo (root, _test/, hosts/, etc.). CI will fail until the updated lock files are committed and pushed.

25. Shared devDependency package missing from registry breaks every downstream npm ci ​

Symptom: Every module's CI test job fails with ERR_MODULE_NOT_FOUND for @superloomdev/js-helper-eslint-config or a checksum mismatch, even though only the config package was deleted and republished.

Cause: When a shared devDependency (like js-helper-eslint-config) is consumed by every module, deleting it from the registry breaks all downstream installs until CI republishes it. If the config package's own test and publish jobs are not chained ahead of all other jobs in the workflow, downstream modules race the republish and fail.

Lesson: A shared devDependency that every module installs must be the head of the CI chain. Its test-* and publish-* jobs must complete before any other module's test-* job starts. In ci-publish-helper-modules.yml, chain test-eslint-config and publish-eslint-config before the first downstream test-* job. Never delete a shared devDependency version without immediately repushing to trigger its CI republish. Bulk deletion of all versions is forbidden; delete only the exact named version when the active plan authorizes it.

26. A version-existence publish guard turns an unmoved version into a green run that ships nothing ​

Symptom: A commit whose source changed is pushed, CI reports green, and the registry still serves the old tarball. A later re-trigger of the same commit fails with 409 You cannot publish over the previously published versions, the honest error the first run swallowed. The red run is then easy to misread as "the published version is corrupt", which invites a destructive registry operation to clear it.

Cause: The publish path guards npm publish with a registry-existence check: it reads npm view "$PKG@$VERSION" and sets needs_publish=false when the version is present. The guard exists to make re-runs of an unchanged commit idempotent, which is legitimate. But it cannot distinguish "nothing to do, the registry already has this exact content" from "the source changed and the version did not move with it". Those two need opposite outcomes, and the version name does not carry enough information to tell them apart. Any repository whose version can stay put across a source change is exposed, whether because a bump was forgotten or because the repository republishes at a fixed version by policy.

Fix/Lesson: The guard compares content, not version presence: pack the working tree, read the registry's dist.shasum, and skip only when the shasums match. A mismatch fails loudly, naming both shasums; the remedy follows the repository's release policy, which is normally to bump the version. Both the detect gate and the per-job guard must compare, because a detect job filtering on version presence drops the module before its publish job runs. The same comparison is the only honest post-publish verification, because a version appearing in the registry listing does not prove the new content shipped. Generalized rule: when a guard's job is to detect "already done", its input must be a fingerprint of the work, never a name that the work happens to reuse. Positive form of the rule: cicd-publishing.md - The Publish Guard Compares Content, Not Version Presence; the underlying principle: engineering-philosophy.md - Idempotency Guards Compare Fingerprints.

27. A local parity runner that extracts CI steps by name pattern reports false confidence ​

Symptom: Local verify is green, then CI goes red on a step the local runner never ran, or that it coincidentally covered without mapping.

Cause: The runner extracted workflow steps with a G[0-9]+ name pattern, so 23 of 55 named steps in one repository and 15 of 23 in another were invisible. Nothing reported them as unmapped; silence read as coverage.

Fix/Lesson: Enumerate every workflow step structurally, map each to a local gate or a signed unreplicable reason from the closed set, and fail on any step that is neither. A new CI step with no mapping turns the runner red instead of escaping to GitHub. Positive form of the rule: local-ci-parity.md - Enumerate Every Step.

28. A test passes locally because a sibling install exists that its CI job never performs ​

Symptom: CI fails with ENOENT for a path under another package's node_modules, while the same test passes locally every time.

Cause: The local runner installs every package into one shared working tree, so a test in src/_test could read hosts/web/node_modules. The CI test job installs src/_test only, from a bare checkout, so the path does not exist there.

Fix/Lesson: Replay install-scoped jobs in a snapshot that contains only that job's installs, and add a cheap grep gate for cross-package path references. The snapshot is the working tree as git sees it, not a worktree at HEAD, so it tests the edits about to be pushed. Positive form of the rule: local-ci-parity.md - Isolate the Install Scope.

29. The documented pre-push command is not the command that ran ​

Symptom: CI goes red on gates the local runner does replay, even though the developer ran a command and it passed.

Cause: AGENTS.md said "run npm run verify before every push" as prose. npm run lint was run instead, and nothing distinguished the two. Lint is a subset of verify, so it passed while the full gate would have failed.

Fix/Lesson: The full runner writes a content-hashed stamp on success, and a tracked pre-push hook refuses a push when the stamp is missing or stale. The hook is an additional local gate; CI still runs as before. Positive form of the rule: local-ci-parity.md - Enforce the Pre-Push Command.

Local Module Testing ​

Each entry below maps a symptom to its root cause and the durable fix. The sibling philosophy doc is testing-local-modules.md (healthcheck philosophy, test concurrency rules, module reference table); this section is the journal of real failures that shaped those rules.

1. npm error notarget No matching version found for @superloomdev/... ​

Cause: npm install ran from the repo root or the module root instead of _test/. Each _test/ directory has its own package.json with a different dependency tree.

Lesson: Always cd into _test/ first. Tools that automate this (AI agents, scripts) must always pass Cwd explicitly to the _test/ directory - omitting it silently runs from the repo root.

bash
# Wrong: from the module root
npm test

# Correct: from _test/
npm install && npm test

2. ERR_MODULE_NOT_FOUND for a package that should be installed ​

Cause: An import in the test uses the wrong scoped package name - typically missing the category prefix (@superloomdev/js-server-helper-postgres instead of @superloomdev/js-server-helper-sql-postgres).

Lesson: The npm package name must match the full directory name, including every category prefix. Grep for the bare name (grep -r "js-server-helper-postgres" _test/) before assuming the install is broken.

3. Manually starting Docker before npm test ​

Symptom: Bind for 127.0.0.1:NNNN failed: port is already allocated, or tests fail immediately with ECONNRESET / socket hang up, many tests cancelled.

Cause: pretest runs docker compose down -v --remove-orphans first. That command only manages containers from its own compose project name - it does not touch a manually started container. Then docker compose up tries to bind the same port and fails. (The same conflict exists in CI when a workflow step runs docker run ahead of npm test.)

Lesson: Pick one owner of the Docker lifecycle. pretest already owns it. Locally and in CI, never run a separate docker run or docker compose up for the same service before npm test.

4. node --test test.js directly without pretest ​

Cause: pretest did not run; no container is up. Tests fail immediately with connection errors.

Lesson: Use npm test, not node --test test.js. The lifecycle scripts exist for a reason.

5. Stale container from a previous failed run ​

Cause: A prior run died before its posttest could clean up.

Fix: pretest already handles this with docker compose down -v --remove-orphans. If pretest itself fails because of a deeper conflict, clean up manually from the same _test/ directory:

bash
docker compose down -v --remove-orphans

6. Tests pass locally, fail in CI with Connection lost: The server closed the connection ​

Cause: A healthcheck that returns "ready" too early. On Docker Desktop the service finishes init fast enough that the false-positive moment never overlaps with the test run; on a constrained CI runner, init takes longer, the healthcheck passes during a transient-ready window, and the server then drops connections during a real init step (e.g., MySQL's restart in normal mode).

Lesson: See testing-local-modules.md Healthcheck Philosophy. Probe with the credentials, database, and transport the tests will use. Add start_period and enough retries that the total budget covers a cold-pull, cold-start initialization.

7. test did not finish before its parent and was cancelled ​

Cause: Concurrent execution of top-level describe() blocks. A lazy-init resource was created mid-test by a parallel block, leaving the cancelled block in a half-initialized state.

Lesson: Wrap the suite in describe('Module', { concurrency: false }, ...). See testing-local-modules.md Test Concurrency.

8. && sleep 2 (or 5) in pretest ​

Cause: A previous developer added a sleep to mask an unreliable healthcheck.

Lesson: Sleeps are never the right fix. They paper over bad healthchecks, slow every developer's iteration, and still fail under load. Remove the sleep, then fix the healthcheck so docker compose up -d --wait truly waits until the service is ready.

9. AI agents or scripts hanging on a multi-line shell command ​

Cause: Heredocs (cat <<'EOF' ... EOF) and multi-line -m arguments to git commit cause zsh to enter dquote> continuation mode when the closing quote falls on a different line. Special characters (backticks, $, !, (...)) make this worse.

Lesson: For commit messages: prefer a single-line -m, or stack multiple -m flags, or use -F /tmp/file after writing the file via a non-shell tool. For file content: never use heredocs through a shell bridge; write to a file with the editor tool and cat it from there. Full journal: AI Terminal & Shell Bridge -> S1/S2.

10. AWS SDK test hangs ~1-2s, then success: false with no clear error ​

Symptom: A test that exercises an AWS SDK function (getSignedUrl, S3Client.send, DynamoDBClient.send, etc.) takes 1-2 seconds and fails with success === true assertion failing. Stack trace points to your module's catch block but the underlying error is silent or absorbed.

Cause: No AWS credentials are passed to the SDK. The SDK walks the default credential provider chain: env vars -> shared config file -> EC2/ECS instance metadata (http://169.254.169.254). On a developer machine and on a GitHub Actions runner there is no instance metadata service, so the chain times out. The 1-2 second test duration is the metadata-service connection timeout.

Lesson: Every test that uses an AWS SDK client must inject dummy credentials, even when no real network call is made (URL signing, command construction, etc.). The dummies do not need to be valid - they just need to exist so the SDK does not enter the default-chain code path.

The canonical pattern is to set them in the test script of _test/package.json:

json
"test": "AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 ... node --test test.js"

For service-specific env vars (S3_ACCESS_KEY, DYNAMODB_ENDPOINT, etc.), use the names the module's loader reads. The same env vars must also exist in __dev__/.env.dev and docs/dev/.env.dev.example per the four-file rule.

11. MongoDB replica-set healthcheck reports healthy before the node is PRIMARY ​

Symptom: Tests pass on a developer's macOS machine and fail in CI with the first few writeRecord / getRecord (after writing) calls returning success: false or result.document === null. Later writes in the same suite succeed. The mongodb helper's own _test and any downstream consumer's _test:mongodb are both vulnerable.

Cause: A naive replica-set healthcheck:

yaml
test: ["CMD", "mongosh", "--eval", "try { rs.status().ok } catch(e) { rs.initiate({...}).ok }"]

returns truthy as soon as rs.initiate() returns. But on a single-node replica set the node still spends another 1-2 s in SECONDARY → STARTUP2 → PRIMARY before it accepts writes. docker compose up --wait returns "healthy" mid-election; the test process opens its driver and fires writes immediately. On Docker Desktop / macOS the local stack happens to be fast enough that the test loop hits PRIMARY by chance; on the slower hosted GitHub Actions runner the first writes land mid-election and fail with not master-style errors that the helper catches and surfaces as success: false.

Lesson: The healthcheck must verify the same readiness the tests require (write-ready primary, not just replica-set initialized). Use db.hello().isWritablePrimary and quit(1) so docker keeps retrying until the node is actually primary:

yaml
test:
  - CMD
  - mongosh
  - --quiet
  - --eval
  - "try { rs.status() } catch(e) { rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'localhost:27017' }] }) }; if (!db.hello().isWritablePrimary) quit(1)"

This is the same probe-the-application-protocol principle as MySQL's two-phase init (entry 6). rs.status().ok is the equivalent of mysqladmin ping -u root (server alive but not yet ready for the test workload).

The general rule for any service that has an init phase distinct from "process up": the healthcheck must succeed only after the test-relevant phase has completed (PRIMARY for replica sets, test_user reachable for MySQL, test_db schema applied for Postgres, etc.).

12. verify.generateAndStore returns COOLDOWN_ACTIVE with cooldown_seconds: 0 under concurrent calls ​

Symptom: js-server-helper-verify tests fail intermittently in CI (rarely locally) with COOLDOWN_ACTIVE errors even though the test explicitly sets cooldown_seconds: 0. The concurrency test in _test/shared-store-suite.js ("concurrent createPin calls with cooldown:0 all land") is the most reliable reproducer; downstream consumers of verify see the same flake on CI.

Cause: The cooldown gate in generateAndStore computed now - existing.created_at without short-circuiting when cooldown_seconds === 0. Under concurrency, two requests can share a microsecond-close instance.time while the store's created_at (captured a few milliseconds earlier by the first winning write) is ahead of the second caller's now. The signed diff is negative, diff < cooldown_seconds is trivially true for any positive threshold; and even for 0, the strictly-less-than check fires when the diff is negative. The caller sees a cooldown error that has no real cooldown semantically.

The bug was dormant until the MongoDB PRIMARY-election fix (entry 11) stabilized CI enough that the race became reproducible; before that, the flake was attributed to the mongodb healthcheck and the true cause went unseen.

Lesson: A "cooldown disabled" configuration must short-circuit before any arithmetic on timestamps. The canonical fix is a single explicit check at the top of the gate:

js
// verify.js generateAndStore
if (options.cooldown_seconds === 0) {
  // cooldown disabled; do not consult existing.created_at
} else if (existing && existing.created_at) {
  const diff = now - existing.created_at;
  if (diff < options.cooldown_seconds) {
    return { success: false, error: CONFIG.ERRORS.COOLDOWN_ACTIVE };
  }
}

Generalized rule: any time a helper has a "feature disabled when N === 0" semantics, the disabled branch must bypass every downstream computation that uses N or any state N would have produced. Never rely on 0 < diff < N to be false when N === 0, because diff can be negative under concurrent non-monotonic time sources.

Applies anywhere a rate-limit, throttle, TTL, or cooldown is configurable with a "zero = off" value: verify.cooldown_seconds, auth.LAST_ACTIVE_UPDATE_INTERVAL_SECONDS, logger retention, and any future equivalent. Audit the gate when introducing any such option.

15. Repo-wide ERR_MODULE_NOT_FOUND after a "remove scope prefix" style commit ​

Symptom: All _test/ suites fail immediately at import 'js-helper-utils' with ERR_MODULE_NOT_FOUND. No code logic has changed; only import ... from '@superloomdev/...' statements were rewritten to import ... from '...' in a mass search-and-replace commit. node_modules/ still contains only scoped packages (@superloomdev/js-helper-utils, etc.); the unscoped names do not exist.

Cause: npm installs packages under their full published name (the "name" field in package.json, which includes the @superloomdev/ scope). An import statement must use that exact name. Removing the scope prefix from import statements without simultaneously adding npm aliases (or renaming the packages on the registry) breaks every module that was changed.

Lesson: Never strip scope prefixes from import statements unless the packages are also republished without the scope (or aliased via package.json imports map). The safe rule: import specifiers must always match "name" in the target package's package.json exactly. Fix is a single sed to restore the @superloomdev/ prefix across all .js files under src/, excluding node_modules/.

bash
find src/ -name "*.js" -not -path "*/node_modules/*" | \
  xargs sed -i '' \
    "s/from 'js-helper-/from '@superloomdev\/js-helper-/g; \
     s/from 'js-server-helper-/from '@superloomdev\/js-server-helper-/g; \
     s/from 'js-client-helper-/from '@superloomdev\/js-client-helper-/g"

16. Store contract method name drift between logger.js and test fixtures ​

Symptom: Logger unit tests fail with TypeError: store.addLog is not a function even though the memory store is correctly imported. Grep shows zero matches for addRecord in the source, yet the error points at logger.js:147.

Cause: When logger.js store contract method names were renamed (addRecord→addLog, listByEntity→getLogsByEntity, listByActor→getLogsByActor, initializeStore→setupNewStore, cleanupExpiredRecords→cleanupExpiredLogs), the rename was applied partially: logger.js and the adapter packages were updated but the inline captureStore and minimalStore stubs scattered throughout _test/test.js and the memory-store.js fixture were not. node_modules held a correct symlinked copy, but the directly-required ../logger.js used the new names, so the inline stubs were invisible mismatches.

Lesson: When renaming store-contract methods in the core module, search for every store stub in _test/test.js, not just the shared fixtures, because inline anonymous objects appear throughout the test file and are missed by a module-level rename. Grep pattern: import ... from '../logger.js' -> addRecord|listByEntity|listByActor|initializeStore|cleanupExpiredRecords. The same applies to any module that has inline store stubs in its test file (auth, verify).

17. Store contract method renames not propagated to adapter _test/test.js files: all adapter CI jobs fail ​

Symptom: All test-verify-store-* CI jobs fail with TypeError: store.initialize is not a function. All test-auth-store-{sqlite,postgres,mysql} fail with TypeError: auth.createSchema is not a function. Tests pass locally only if stale node_modules from a prior install are present.

Cause: Two separate renames were applied to the core module source and adapter store.js files but not propagated to the _test/test.js files:

  1. verify.js + all verify-store-*/store.js: store.initialize() → store.setupNewStore() (v2.1.x → v2.2.0). The _test/test.js stubs in all 5 verify-store adapters were not updated. Because verify v2.2.0 was never published (CI was failing at the time), ^2.1.0 in _test/package.json resolved to v2.1.x on the registry (which uses the old name). This masked the problem locally but was exposed as soon as CI installed fresh.
  2. auth.js: auth.createSchema() → auth.setupNewStore(). The 3 SQL auth-store adapter _test/test.js files still called auth.createSchema().

Lesson: When renaming any store-contract or module-public method, grep all _test/test.js files across every adapter before committing, not just the module source and its own tests. Pattern: grep -rn "old_method_name" src/. Adapter test files are not auto-updated by renaming the source. Also: a new major-bump on a core module will not reach adapters until it is both published and the adapter _test/package.json pin is bumped.

18. contains() substring matching causes parent modules to trigger when only adapters need publishing ​

Symptom: When only an adapter module (e.g., js-server-helper-verify-store-sqlite) needs to be published, the CI workflow also runs jobs for the parent module (js-server-helper-verify). The base module is detected as needing publish even though npm view shows it already exists on the registry. This causes unnecessary republishing of parent modules when only their adapters should be processed.

Cause: GitHub Actions contains() function performs substring matching, not exact string matching. The detect job outputs JSON arrays like:

json
["src/helper-modules-server/js-server-helper-verify-store-sqlite"]

The job condition contains(needs.detect.outputs.test_modules, 'helper-modules-server/js-server-helper-verify') returns true because 'helper-modules-server/js-server-helper-verify' is a substring of 'src/helper-modules-server/js-server-helper-verify-store-sqlite'.

Fix: Parse the JSON array first, then check for exact element matching:

yaml
# WRONG: Substring matching
contains(needs.detect.outputs.test_modules, 'helper-modules-server/js-server-helper-verify')

# CORRECT: Exact array element matching
contains(fromJSON(needs.detect.outputs.test_modules), 'src/helper-modules-server/js-server-helper-verify')

Lesson: When a workflow outputs JSON arrays from the detect job and downstream jobs need to check if a specific value is in that array:

  1. Always wrap the output with fromJSON() to convert the JSON string back to an actual array
  2. Use the full path (including src/ prefix) in the match string to ensure exact matching
  3. Never rely on contains() with a JSON string directly - it performs substring matching on the serialized JSON, not element matching on the array

This applies to any job condition checking against detect outputs:

yaml
# Pattern for all module jobs
if: contains(fromJSON(needs.detect.outputs.test_modules), 'src/helper-modules-category/js-module-name')
if: contains(fromJSON(needs.detect.outputs.publish_modules), 'src/helper-modules-category/js-module-name')

19. Test loader uses relative source path instead of the _test/package.json alias ​

Symptom: A test-* CI job fails with Error: Cannot find module 'cookie' (or any transitive dep of the module under test). The require stack points at files inside the module's source directory (e.g. src/helper-modules-server/js-server-helper-http-gateway/parts/cookies.js), not at _test/node_modules/.... Tests pass on the developer's machine because node_modules/ from a previous module-level npm install is still present, but break in a clean CI environment.

Cause: The test loader (or test file) was written as import ... from '../adapter.js' or import ... from '../../sibling-module/main.js' instead of using the alias declared in _test/package.json. The CI workflow runs npm install inside _test/, which only populates _test/node_modules/. Node's import resolution resolves the relative path to the source directory, where the module's transitive deps were never installed (CI never runs npm install at that location).

Why aliases exist: Every _test/package.json declares aliases like:

json
"dependencies": {
  "helper-foo": "file:../",
  "helper-foo-adapter-bar": "npm:@superloomdev/js-server-helper-foo-adapter-bar@^1.0.0"
}

These aliases exist precisely so the loader code stays identical between local-source (resolved via file:../) and published-package (resolved via npm:) contexts. When npm install runs in _test/, it copies the aliased package into _test/node_modules/ AND installs that package's transitive deps right next to it. Bypassing the alias defeats this entirely.

Fix: Always import helpers in _test/ files via the alias declared in _test/package.json:

javascript
// WRONG - relative path to source directory
import HttpGateway        from '../http-gateway.js';
import HttpGatewayAdapter from '../adapter.js';

// CORRECT - alias from _test/package.json
import HttpGateway        from 'helper-http-gateway';
import HttpGatewayAdapter from 'helper-http-gateway-adapter-express';

Exception: Internal files not exposed via the package's main entry (e.g. parts/cookies.js, parts/params.js testing internals) may continue to use relative paths. They are not addressable through the alias because they are not part of the package's public surface.

Lesson: Treat import style as part of test hygiene, not as a stylistic choice. The HTTP Gateway trio (js-server-helper-http-gateway + 2 adapters) is the canonical reference.

20. Singleton pattern prevents test isolation ​

Symptom: Tests interfere with each other, last test's configuration affects subsequent tests, cannot run tests in parallel, difficult to mock dependencies.

Cause: Module uses singleton pattern but takes external dependencies (libs, config, adapters). The singleton shares global state across all tests, making isolation impossible.

Lesson: Use factory pattern for any module that takes external dependencies. Only use singleton pattern for pure utility modules with zero dependencies (like js-helper-utils). Factory pattern enables:

  • Independent instances per test with different configurations
  • Parallel test execution without conflicts
  • Mock dependency injection per test scenario
  • Multiple instances for different use cases

Example of the problem:

javascript
// WRONG: Singleton with dependencies
let CONFIG;
export default function loader (config) {
  CONFIG = config;  // Last caller wins!
  return singletonInstance;
};

// PROBLEM: Test A sets {log_level: 'debug'}
// PROBLEM: Test B sets {log_level: 'error'}  
// PROBLEM: Both tests now use 'error' level

Example of the solution:

javascript
// CORRECT: Factory pattern (fixed-slots shape - see module-structure.md)
export default function loader (shared_libs, config) {
  const Lib = { Utils: shared_libs.Utils };
  const CONFIG = Object.assign({}, (await import('./[module].config.js')).default, config || {});
  const ERRORS = (await import('./[module].errors.js')).default;
  const Validators = (await import('./[module].validators.js')).default(Lib, ERRORS);
  Validators.validateConfig(CONFIG);
  return createInterface(Lib, CONFIG, ERRORS, Validators);
};

// BENEFIT: Each test gets isolated instance
const testInstance1 = createModule({log_level: 'debug'});
const testInstance2 = createModule({log_level: 'error'});

21. Stale package-lock.json causes wrong-version install after file: swap or version reset ​

Symptom: npm install && npm test in _test/ fails with a runtime error that does not match the source code. For example, CONFIG.ADAPTER must be an adapter factory function even though the source was already refactored to the new CONFIG.Adapter object pattern. Or: a freshly deleted-and-republished 1.0.0 still installs the old pre-refactor tarball. Or: npm error 409 Conflict - Package file checksum mismatch on a package that has not changed.

Cause: Two compounding sources produce the same class of failure:

  1. Stale file: path in lock file. During bootstrap testing, _test/package.json is temporarily changed to "helper-foo": "file:../../js-server-helper-foo" so tests can run against local source before the parent module is published. After restoring package.json to a registry semver range, the package-lock.json still has "resolved": "../../js-server-helper-foo". A subsequent npm install reads the lock, sees the path is satisfied, and skips the registry fetch, silently installing the local source (or nothing, if the path no longer exists).

  2. Version reset to 1.0.0 hits a cached tarball. When a module is deleted from the registry and re-published at 1.0.0, a stale node_modules/ or lock file from the previous 1.0.0 epoch resolves to the old tarball. The npm cache keeps the old hash; the registry now serves a different tarball for the same version string, producing a 409 Conflict / checksum mismatch.

Lesson: During module refactoring (where file: swaps and version resets to 1.0.0 are routine), always run rm -rf node_modules package-lock.json before npm install:

bash
# From the module's _test/ directory
rm -rf node_modules package-lock.json && npm install && npm test

This clean-install form is mandatory during module refactoring. It is not required for general module development, where a plain npm install is correct and repeated lock-file deletion would be wasteful. See testing-local-modules.md Step 1.

Secondary lesson: If npm install fails with E409 Conflict / checksum mismatch even after a clean install, the registry is temporarily inconsistent (a known GitHub Packages transient bug). Wait 30 to 60 seconds, then re-run rm -rf node_modules package-lock.json && npm install. Do not use --legacy-peer-deps. It masks the real error, writes incorrect lock entries, and creates new failures on the next run. The clean install is the correct fix.

22. ESLint no-unused-vars on intentionally reserved module-scope variables ​

Symptom: npm run lint fails with error 'CONFIG' is assigned a value but never used no-unused-vars (or similar for Lib, React, ThemeContext, etc.) even though the variable is deliberately kept as a reserved injection slot for future use.

Cause: Every module in this codebase declares its full set of dependency injection slots at the top of the file as a standard structural contract - even slots that are not yet wired. This keeps the injection surface explicit, consistent, and visually scannable across every module. ESLint does not know the variable is intentionally reserved; it sees an unread assignment and flags it.

Why not remove it or rename to _CONFIG? Removing breaks the structural contract and creates an inconsistency when the slot gets wired later. Renaming to _CONFIG changes the actual variable name and deviates from the naming convention used across all modules (Lib, CONFIG, React, etc.).

Fix: Add an inline // eslint-disable-line no-unused-vars comment at the end of the reserved declaration. This is the established convention for module-scope let vars across all modules (see docs/languages/js/module-structure.md Singleton Part Shape):

js
// Injected dependencies, set by the loader (module-scope).
let Lib;
let CONFIG; // eslint-disable-line no-unused-vars -- reserved for future knobs
let React;

Never use a file-level /* eslint-disable */ for this - it suppresses the rule for the entire file and hides real bugs. The inline directive is the only correct form.

Lesson: The module-scope injection-slot block (let Lib; let CONFIG; ...) is a deliberate structural pattern, not dead code. When adding a new slot that is not yet used, add // eslint-disable-line no-unused-vars on the same line. Remove the directive when the slot is wired.

23. CI fails after push - local tests ran against stale node_modules ​

Symptom: All tests pass locally. Code is committed and pushed. CI fails immediately with ERR_MODULE_NOT_FOUND, version mismatch, or checksum errors that were never seen on the developer's machine.

Cause: Local node_modules/ contained hoisted dependencies from a previous repo session or a file: link that masked a missing peer dependency. CI does a fresh npm ci from a clean checkout, so it sees the real dependency state. The local test "passed" only because the stale node_modules/ happened to have the right files from a previous install.

Lesson: Always run rm -rf node_modules package-lock.json && npm install before running tests locally. This is the default, not an exception. The few seconds saved by skipping the fresh install are never worth a broken CI run, a fix commit, and wasted pipeline time. See testing-local-modules.md - Pre-Commit Protocol for the full protocol.

24. _data/ directory mixes dev scripts with generated data, violating the single-concern rule ​

Symptom: A module has a _data/ directory containing both generate.js (a dev script) and basic.country-data.js (a generated data file). The generated file is a JS module (export default {...}), not pure JSON. The _data/ directory is not a recognized archetype in file-archetypes.md or module-structure.md.

Cause: The module was created without consulting the existing data/ convention. The _data/ directory was invented ad-hoc to hold both the generator script and its output, mixing two concerns (dev tooling and runtime data) in one directory.

Lesson: Generated reference data lives in data/ as pure JSON (see module-structure.md - Static Data Files). Dev scripts that produce generated data live in scripts/ (see module-structure.md - Dev Scripts). The _data/ directory is not a recognized archetype and must not be used. When a module needs generated data, the generator script outputs JSON to data/, the JSON file is committed and imported at runtime, and the script is excluded from the published tarball.

25. Repo-bound workflows miss code quality issues in other repos ​

Symptom: The js-helper-module audit workflow catches Lib.Utils gaps, step-comment gaps, and type-guard violations in codebase-js-helper-modules. But the same gaps exist in codebase-rnw-components/parts/a11y.js (19 raw !== null && !== undefined checks, no step comments) and were never caught.

Cause: The workflow files lived in codebase-js-helper-modules/.devin/workflows/, so they could only be invoked when working inside that repo. The components library has the same module structure (entry file, companion files, parts/ factories with shared_libs) but the workflow was never run against it.

Lesson: Workflows that apply to any repo with the Superloom module structure belong in the constitution repo (codebase-superloom/.devin/workflows/), not in a single dependent repo. Repo-specific workflows stay in their repo; repo-agnostic workflows move to the constitution. When a workflow is moved, its language must be generalized (Cwd references, module path examples) so it works from any repo root.

26. JSDoc content indentation drifts to 4 spaces when the docs say 4 but reference modules use 0 ​

Symptom: JSDoc blocks across ~90+ files in codebase-js-helper-modules and ~14 files in codebase-rnw-components use wrong indentation for description text, @param, @return lines, and closers. The docs (code-formatting.md) said "4 spaces from the /* delimiter" but every reference module (money.js, utils.js, debug.js) uses 0-space (flush-left) indentation at top level. Nested JSDoc blocks (inside object literals) have content at the /* column, not at a fixed offset.

Cause: Three gaps, found over three passes:

  1. The docs said "4 spaces" but reference modules use 0 spaces. The create/fix workflow's JSDoc awk check only verified the code after the JSDoc closer matched the closer's indentation, not the content inside the block. The audit workflow had no JSDoc content indentation check at all.
  2. A first fix attempt used git grep -nE "^ @param|^ @return|^ @description" which caught @param/@return lines but missed description prose lines (lines not starting with @). Replaced with an awk that tracks /*...*/ block context.
  3. The awk hardcoded ^ (exactly 4 spaces) as the detection pattern. This worked for top-level blocks (content at col 4, should be col 0) but failed for nested blocks: a JSDoc block at col 2 with content at col 0 was over-stripped, and a JSDoc block at col 4 with content at col 8 was missed entirely. The fix: detect the /* column per block and flag any content or closer not at that column. The indentation is relative to the /* column, not absolute.

Lesson: JSDoc content (description, @param, @return, notes, closer) is at the same column as the /* delimiter. The indentation is relative, not absolute. The docs must match the reference modules, not the other way around. The audit and create/fix workflows must use an awk-based sweep that detects the /* column per block and flags any content or closer not at that column. A hardcoded column assumption (whether grep ^ @param or awk ^ [^ ]) misses nested blocks and over-strips or under-strips content. Continuation lines (indented more than /* col + 4 for readability alignment) are excluded from the check.

27. Engine TTL sentinels (-1, -2) leak through wrapper purity when the driver returns them as-is ​

Symptom: A getTtl call on a key with no expiry returns -1, and on an absent key returns -2. These are Redis/Valkey engine sentinels, not application values. A caller that receives -1 cannot tell whether it means "no expiry" or "the TTL is literally minus one second" (impossible, but the type is Number). A caller that receives -2 may interpret it as a real TTL.

Cause: The ioredis client returns the raw TTL command response, which is -1 for "key exists, no expiry" and -2 for "key does not exist". The wrapper passes these through without translation, violating wrapper purity (the envelope must not leak engine-specific sentinels).

Fix/Lesson: The driver maps both sentinels to null before returning. null means "no TTL known" for both cases. A caller that needs to distinguish "no expiry" from "absent" calls getKeyExists. The same principle applies to any engine that uses sentinel values for exceptional states: the wrapper translates them to null or a boolean, never passes them through. The test suite includes a regression test that asserts ttl_seconds is never -1 or -2.

28. Hand-mirrored workflow regexes drift from CI while the count-parity check stays green ​

Symptom: A local verifier script copies a workflow's git grep regex into a shell function and asserts only that the count of copied gates matches the workflow's count. After an edit to the workflow regex, both the copied gate and the count-parity check pass locally, while CI fails on the real gate.

Cause: The local verifier is not derived from the workflow file. It hand-mirrors the regex, so an edit to the workflow is not reflected in the copy. The count-parity check only compares the number of gates, not their bodies, so changing a regex leaves the count unchanged and the check green. The defect is invisible until CI runs.

Fix/Lesson: The local verifier reads the workflow file and extracts the current gate body, so a workflow edit is reflected without a second hand-mirror edit. A count-parity check is removed; it does not catch body drift. The verifier is proven by a deliberate mutation: edit the workflow regex, plant a matching scratch file, and confirm the verifier fails for the right reason, then restore both.

Prevention: A workflow policy gate must have a local entry point that executes the workflow's source-of-truth command. See testing-local-modules.md - Local CI parity.

29. git grep is blind to untracked files, so an empty result is not evidence ​

Symptom: A new source file contains a pattern the workflow's git grep gate forbids. The local verifier runs git grep and reports the gate clean, because the file is untracked and invisible to the index. CI later adds the file on push and the gate fails.

Cause: git grep searches tracked and indexed content only. A brand-new untracked file is invisible to it, so a forbidden pattern in an untracked file produces an empty result that reads as compliance.

Fix/Lesson: The verifier fails when nonignored untracked files exist, or the operator runs git add -N path/to/new-file before trusting a clean git grep result. The verifier is proven by planting an untracked scratch file with a forbidden pattern and confirming the verifier fails naming the file, then removing the probe.

Prevention: A git grep result is not valid while matching untracked files are invisible. See testing-local-modules.md - Local CI parity.

30. An unbounded RNW list mounts the full roster, so a row-count assertion does not prove virtualization ​

Symptom: A FlatList conversion claims virtualization, but CI measures the full roster mounted on first paint while a local run measures a partial window. The local assertion passes and CI fails.

Cause: Under React Native Web, an unbounded FlatList expands to fit its content and never clips, so every row mounts regardless of viewport. The local viewport happened to be smaller and the partial count looked like windowing. The threshold was loosened until CI passed, converting a real signal into a green check.

Fix/Lesson: FlatList and VirtualizedList under RNW must have a bounded height to make windowing possible. A test that claims virtualization asserts an exact or fixed upper bound on the mounted-row count below the full roster and reaches a late row by scrolling the real scroll node. A local-versus-CI row-count difference is evidence of a layout or viewport precondition, not permission to relax the threshold.

Prevention: See rn-testing.md - List windowing under React Native Web.

31. Install claims that were never run were written into evidence ​

Symptom: An evidence block records "clean install: exit 0" for a step that was never executed, and downstream decisions are made on the false evidence.

Cause: The agent wrote the expected result as if it had observed it, skipping the actual command. The evidence read as complete, and the wave proceeded on a claim with no backing output.

Fix/Lesson: Every evidence line is the pasted output of a command the agent ran in this session, not a prediction. Rule 2.4 and 2.17 exist because of this. If a command was not run, its evidence line is absent, not "expected" or "should be".

Prevention: Paste the actual terminal output. An evidence line with no pasted output is not evidence.

32. Audit sweeps reported through &&/|| chains gave wrong verdicts ​

Symptom: A sweep reports "clean" when it has hits, because the &&/|| chain masks the exit code of the actual grep.

Cause: Shell chaining reuses the last command's exit code, not the grep's. A grep that finds hits exits 1, but grep ... && echo clean prints clean only when grep exits 0; grep ... || echo clean prints clean when grep exits 1 (found hits). Both forms invert the verdict under different conditions.

Fix/Lesson: Rule 2.6. Run the grep alone, capture its exit code, and report the hit count. A sweep is clean when it prints zero hits, not when a chained echo says so.

Prevention: Never chain a sweep through &&/|| to a verdict. Run, count, report.

33. CI skipped 150 of 154 jobs and was read as "all green" ​

Symptom: A CI run is reported as "all green" when 150 of 154 jobs were skipped. The four that ran passed; the 150 that did not run were invisible in the summary.

Cause: The CI matrix has skip conditions that drop jobs under certain paths. A summary that counts only completed jobs reads a partial run as complete. Rule 2.13.

Fix/Lesson: A CI run is green when every declared job for the commit's changed paths ran and succeeded. Skipped jobs are reported as skipped, not as passed. The job count in evidence is ran: N, skipped: M, total: N+M.

Prevention: gh run view <id> --json jobs --jq '.jobs[] | .name + ":" + .conclusion' lists every job including skipped ones. Read the full list, not the conclusion summary.

34. Literal comparisons of profile data passed while the engine rejected every profile ​

Symptom: A test compares two literal objects and passes, while the real engine rejects every profile the test claims to validate. The test is green and the feature is broken.

Cause: The test's expected value was copied from the implementation's output, not from the specification. Both sides of the deepEqual come from the same code path, so disabling the production line changes both sides equally and the test stays green. Rule 2.38.

Fix/Lesson: The expected side of every deepEqual is a literal copied from the specification, a fixture file, or a rule re-implemented in the test from the plan's words. It is never the output of the library function the test exists to check, and never a second call to the same function with the same input.

Prevention: When writing a test, ask: "if I delete the production line, does this test fail?" If not, the expected value is not independent.

35. git grep -E with \b on macOS matches nothing ​

Symptom: A git grep -E sweep whose pattern contains \b prints nothing on macOS and reads as clean, when the same pattern matches on Linux.

Cause: BSD regex has no word boundary. A -E sweep whose pattern contains \b prints nothing and reads as clean. Run word-boundary sweeps with -P, and probe that -P matches a known word before trusting an empty result.

Fix/Lesson: Use -P for word boundaries, and probe the pattern against a known word before trusting an empty result.

Prevention: Add a positive control to every word-boundary sweep: a known word the pattern must match, run first, that proves the pattern works.

36. A scoped sweep must be proven to stay in scope ​

Symptom: A script that accepts a module path and never passes it to git grep returns repository-wide hits; readers skim them and real hits hide among them.

Cause: The scoping argument was accepted but not threaded into the grep. The sweep reports every match in the repo as if it were scoped to the module.

Fix/Lesson: After scoping a sweep, assert that zero hit lines fall outside the scope. A sweep that cannot prove it stayed in scope is a repository-wide sweep.

Prevention: Every scoped sweep ends with a scope assertion: if any hit line is outside the module path, exit 1.

37. A test that asserts a field name the implementation exposes, rather than the shape the specification names, passes a defect ​

Symptom: A test asserts that a field named foo exists on the result, and passes. The specification names the field bar. The implementation exposed the wrong name and the test did not catch it.

Cause: The test's assertion was written from the implementation's surface, not from the specification's shape. The test proves the implementation is self-consistent, not that it is correct.

Fix/Lesson: Assert specification shapes with deepEqual against the specification's literal. A field name the implementation exposes that the specification does not name is a defect the test should fail on, not pass on.

Prevention: Write the expected shape from the spec, then implement to it. Never copy the implementation's field names into the test.

38. A checkpoint declared from memory skipped a wave ​

Symptom: An agent declares a checkpoint "proceed" from recollection, and a wave that was never released is treated as released. The next wave starts on a false premise.

Cause: The agent computed checkpoint eligibility from memory rather than from the plan text and the recorded evidence. The memory was wrong; the evidence was never written.

Fix/Lesson: Compute checkpoint eligibility from the plan text and the recorded evidence, never from recollection. Run checkpoint-check.sh <plan> <n> and read its output.

Prevention: A checkpoint is a gate, not a judgment. The script is the authority; memory is not.

39. Documentation written by one hand from the specification while the code was written by another from the keyboard diverged on a field name ​

Symptom: The docs say color.interactive; the code reads color.primary. Both are green in their own suites; the divergence is invisible until a consumer reads both.

Cause: The agent that changed the public surface did not write its documentation in the same pass. The doc agent worked from the spec; the code agent worked from the keyboard. Rule 23.

Fix/Lesson: The agent that changes a public surface writes its documentation in the same pass. A second agent's summary is not evidence and is not read.

Prevention: One agent, one pass, surface and docs together. A doc change without a code change in the same commit is a smell; a code change without a doc change in the same commit is a defect.

40. A shadow* collapse dropped layers and spread on native for as long as React Native lacked boxShadow ​

Symptom: A shadow approximation that collapsed multi-layer shadows to a single layer and dropped spread was kept past the React Native version that added boxShadow. Native rendered the wrong shadow for months after the approximation's reason ended.

Cause: The approximation was written when React Native could not render boxShadow. When React Native gained the capability, the approximation was not retired because no one re-checked the floor.

Fix/Lesson: When a platform gains a capability, retire the approximation and its loss report in the same release; an approximation kept past its reason is a silent defect.

Prevention: Every exception in the plan's exception register is revisited at every floor change. An approximation with no revisit date is a defect.

41. A warm npm cache served a tarball the registry had already deleted, so local npm ci passed while CI returned 403 ​

Symptom: Every CI test job fails at npm ci with npm error 403 Forbidden - GET https://npm.pkg.github.com/download/@scope/package/1.0.0/<shasum> and Permission permission_denied: read_package, while the same lockfile installs cleanly on the developer machine. The token has read access; the 403 looks like a permissions problem and is not one.

Cause: A same-version republish (delete, then publish 1.0.0 again) changes the tarball shasum. Every consumer lockfile still pins resolved: .../1.0.0/<old shasum>. GitHub Packages answers 403, not 404, for a deleted tarball. Locally, npm's content-addressable cache serves the old tarball by integrity hash without contacting the registry, so npm ci cannot see that the pin is dead. A verify script that runs npm ci against the warm cache is not CI-faithful for this failure.

Fix/Lesson: After any same-version republish, refresh every lockfile in every consumer, enumerated by path (root, src/_test, hosts/web, hosts/expo), not "the ones that changed". Two permanent gates: a lockfile freshness check that compares each pinned @scope shasum with npm view <name>@<version> dist.tarball and fails on any mismatch, run in CI before the first npm ci and mirrored in the local verify script; and npm ci --cache "$(mktemp -d)" in the verify script so the local cache can never mask a deleted tarball. Evidence for "CI green" is a gh run watch conclusion on the pushed commit, never a local pass.

42. npm unpublish returns E405 on GitHub Packages ​

Symptom: npm unpublish @scope/package@1.0.0 --registry=https://npm.pkg.github.com fails with npm error code E405 and 405 Method Not Allowed.

Cause: GitHub Packages does not implement the npm unpublish endpoint. Version removal is a GitHub Packages API operation, not an npm one.

Fix/Lesson: Query the version id with gh api /orgs/<org>/packages/npm/<package>/versions --jq '.[0].id', delete it with gh api --method DELETE /orgs/<org>/packages/npm/<package>/versions/<id>, then push so CI republishes. Never npm unpublish, never a local npm publish.

43. A React Native Web <input> keeps its intrinsic minimum width and overflows its wrapper ​

Symptom: A password, number, or search field renders wider than its wrapper by a few pixels at narrow widths (an inner <input> of 186px inside a 180px wrapper), so the wrapper's border or adjacent control is pushed out or clipped.

Cause: React Native Web renders TextInput as <input>, and a browser <input> has an intrinsic width with min-width: auto. Flex flex: 1 lets it grow but not shrink below that intrinsic width.

Fix/Lesson: The TextInput atom sets minWidth: 0, so every composite that embeds it can shrink. Patching each composite individually leaves the next one broken.

44. An SVG icon adapter that rewrites root fill and stroke turns stroke glyphs into filled shapes ​

Symptom: Checkmarks, sync arrows, and every *-outline glyph render as filled wedges or blobs; filled glyphs look correct, so the defect is not uniform and is easy to attribute to the icon set.

Cause: Stroke glyphs depend on their own classes or attributes (fill: none, a stroke width) that the adapter did not load, while the adapter set fill and stroke on the root element for every glyph. Open polylines received a fill.

Fix/Lesson: An adapter sets color through the one attribute the icon set documents (fill for filled sets, stroke for stroke sets, or the set's own color prop) and never both. Every host maps semantic names through one committed manifest, and an unmapped name is a console.error, not a silent placeholder, so the zero-console-error gate sees it. A ? placeholder that passes every gate and lands in a regenerated visual baseline is the failure this entry exists to prevent.

45. A unit test read a sibling host's node_modules, passed locally where every host was installed, and failed in the CI job that installs only its own directory ​

Symptom: src/_test passes locally and in the local verify script; the CI test job fails with ENOENT ... hosts/web/node_modules/@scope/package/data/manifest.json from a unit test.

Cause: The test resolved a data file through hosts/web/node_modules. On the developer machine every host had been installed by an earlier gate, so the path existed. The CI test job installs src/_test only.

Fix/Lesson: A test directory reads installed packages from its own node_modules and declares them in its own package.json. Anything a test needs from another directory of the repo is either a tracked source file or a declared dependency. A CI-faithful check for this class is to run the unit suite with every other node_modules moved aside.

46. A generated artifact hand-written to agree with the implementation ​

Symptom: A check comparing the implementation against a "generated" oracle passes while both are wrong.

Cause: The oracle's values were transcribed from the source by hand, so the comparison is self-confirming. A hand-written oracle cannot detect upstream drift because it has no source to regenerate from.

Evidence: button.height read 40 in both the oracle and the spec sheet while @carbon/styles declared the root at $default: 'lg' (48). The oracle agreed with the implementation because the oracle was written from the implementation.

Fix/Lesson: Parse the source, record a method per value (parsed, inherited, transcribed, none), and regenerate in CI with a copy-aside byte compare. Detection question: for every generated artifact, name the command that reproduces it from source, and if none exists it is not generated.

47. A check whose pattern cannot match the real artifact format ​

Symptom: A validator reports clean forever and its selftest passes.

Cause: The selftest plants a fixture in a format the real artifact never uses. The validator's pattern matches the fixture but not the real artifact, so the check is inert on real input.

Evidence: The plan close checker's blank-evidence rule matched zero of 14 genuinely blank lines because real plans write - Label: (a markdown list item) and the pattern required a line starting with a letter. The selftest used SelftestBlankLabel: (no dash), a format no real plan uses.

Fix/Lesson: Every checker rule ships a fixture derived from a real artifact; a rule that cannot be shown firing on real input is treated as absent. Detection question: does the selftest's fixture format match the real artifact's format? If not, the rule is inert.

Adding a New Entry ​

Whenever a new failure mode is discovered:

  1. Reproduce it once. Confirm the root cause is what you think it is.
  2. Add an entry to the right section above with Symptom, Cause, Fix/Lesson. Keep the numbering continuous within a section.
  3. If the rule is brief enough to live in the AGENTS.md compact summaries (Safe Terminal Patterns, healthcheck rules, etc.), recompile the compact mirror (AGENTS.md is a derived artifact, never edited directly).
  4. Commit the journal entry and the summary together so they never drift.

Doc drift is the slowest bug to find. No exceptions. Every new lesson goes here first, then propagates.

Cross-reference rules:

  • Pitfalls that belong to the architecture domain (docs/foundations/, docs/modules/, docs/server/, docs/testing/ - module migration, refactors) go in ../languages/js/pitfalls-migration.md, not here.
  • The philosophy docs cicd-publishing.md and testing-local-modules.md keep only the positive rules (what to do). Symptoms and root causes always live here.
  • Anchors in this file are stable. AGENTS.md and other cross-references rely on them. Never rename an H2 or H3 after it is published.

Released under the MIT License.

Copyright © 2025-present Superloom