Superloom

Appearance

Developer-Environment Pitfalls (AI Journal) ​

Audience: AI coding agents (Cascade, Copilot, Cursor) and humans debugging a specific CI, test, or terminal failure. Not first-read material. Start with the sibling philosophy docs; come here only when a concrete failure mode needs a confirmed fix.

Shape: Every entry is Symptom → Cause → Lesson/Fix. When a new failure mode is discovered, add the entry here first (per the Golden Rule in AGENTS.md), then recompile the compact rule into AGENTS.md (a derived artifact, never edited directly).

Scope: This file covers the docs/dev/ domain: AI tool-bridge, CI/CD publishing, local module testing. Architecture-level pitfalls (module migration, refactors) live in docs/testing/pitfalls-migration.md.

On This Page ​

AI Terminal & Shell Bridge ​

Canonical journal of every shell, terminal, and tool-bridge failure mode that AI assistants (Cascade, Copilot, etc.) have hit while working on this codebase.

Why AI agents fail differently from humans ​

A human at a real terminal sees the prompt change to dquote> and fixes the quoting. An AI tool-bridge sees nothing, hits the proposed-command timeout, and the entire turn appears to "hang". The same root cause produces different symptoms depending on who is at the keyboard.

Three structural differences matter:

  1. No PTY interactivity: the bridge cannot type into a sub-prompt. Once the shell enters dquote>, heredoc>, quote>, cmdsubst>, or bracket> continuation mode, there is no way out except killing the process.
  2. Output is captured asynchronously: anything that paginates (less, more, git log without --no-pager, man) blocks forever waiting for keyboard input that will never arrive.
  3. The current working directory is per-call: every run_command resets to whatever Cwd the tool-call specifies. There is no persistent session between calls, so cd in one call has no effect on the next call.

Every pitfall below is a specific consequence of one of these three. The mitigation is always the same: avoid asking the shell to do anything interactive.

Shell-bridge pitfalls ​

S1. Heredocs hang the bridge ​

Symptom: A cat <<'EOF' ... EOF command never returns. The user sees "no progress" until the tool-call times out.

Cause: Heredoc content with backticks, $, !, parentheses, or unterminated quotes can confuse zsh's parser. The shell enters a continuation prompt waiting for more input. There is no way to send the closing token through the bridge.

Fix: Never use heredocs through the bridge.

  • For file content: use write_to_file (or edit) directly.
  • For terminal-only sinks (e.g., appending to a gitignored file): write the content to a temp file with write_to_file, then cat /tmp/file >> /path/to/target.

S2. Multi-line git commit -m "…" ​

Symptom: The command appears in the approval popup with the closing " on a different line. After the user approves, nothing happens - shell sits in dquote> mode.

Cause: zsh treats every newline inside an open "…" string as part of the string. With special characters in the body (`, $, !, (...)), the parser also opens nested continuation states. None of them can be closed through the bridge.

Fix: Always pass commit messages on a single logical line.

NeedPattern
Single-line summarygit commit -m "feat(module): one-line summary"
Summary + body paragraphsgit commit -m "feat(module): summary" -m "Paragraph one." -m "Paragraph two."
Long structured bodywrite_to_file to /tmp/commit-msg then git commit -F /tmp/commit-msg && rm /tmp/commit-msg

The same rule applies to gh pr create, aws ssm put-parameter, and any other tool that takes a quoted message.

S3. Multi-line content embedded inside any quoted shell argument ​

Symptom: Same as S2 but for non-git commands. The approval popup renders the multi-line argument unreadably; after approval, the shell hangs.

Cause: Any time a run_command payload spans multiple lines because of an embedded quoted string, the shell-bridge sees the same dquote> failure mode.

Fix: Route the multi-line content through write_to_file to a temp file first, then have the shell read from the file with a single-line command.

S4. Backticks inside a double-quoted argument ​

Symptom: git commit -m "feat: closes issue \#123`"either runs the contents of the backticks as a command substitution, or hangs incmdsubst>` mode.

Cause: Inside "…", backticks always start command substitution. Escaping them with \` only sometimes works depending on shell version.

Fix: Use single quotes around the message, or replace the backticks with single quotes.

S5. Tilde expansion inside quotes ​

Symptom: cp ~/.npmrc "…" works; cp "~/.npmrc" "…" silently fails with "no such file".

Cause: Tilde is a shell metacharacter that only expands outside quotes (and not in all positions even there).

Fix: Use $HOME (which is a regular variable and expands inside double quotes) or omit the surrounding quotes for the path.

S6. Escaped brackets inside a regex character class make a sweep silently match nothing ​

Symptom: A new sweep grep returns no output and is read as "clean", but the pattern it was written to catch is demonstrably present in the file. The sweep passes forever and enforces nothing.

Cause: Inside a POSIX bracket expression a backslash is a literal character, not an escape. A class written as [A-Za-z_.\[\]'] therefore closes at the first unescaped ] - the one intended as an escaped literal - so the class becomes {A-Za-z, _, ., \, [} and the remaining '] is parsed as pattern text. The regex still compiles, grep still exits cleanly, and nothing ever matches.

Fix: Never put escaped brackets in a character class. Prefer a negated class that needs no escaping, and always prove a new sweep fires before trusting it:

bash
grep -nE "typeof [^ ]+ (!==|===) '(number|function)'" path/to/file.js

Lesson: Every new sweep grep is validated against a file that is known to contain a violation, before it is added to a workflow's sweep battery. A sweep that has never been shown to produce a hit is unverified, and a silently-empty sweep is worse than no sweep because it reports the codebase as conformant. This is the inverse of the usual grep convention where empty output is the pass condition: empty output only means "pass" once the pattern has been proven capable of failing.

Process and pager pitfalls ​

P1. Output paginators (less, more, man, vi) ​

Symptom: A command runs forever with no visible progress.

Cause: Paginators wait for keyboard input. There is no keyboard.

Fix: Never invoke an interactive viewer. The environment runs commands with PAGER=cat so most pager-aware tools (git, systemctl, journalctl) cooperate, but commands that ignore PAGER need explicit flags:

CommandFlag
git loggit log -n 20 (or --no-pager)
git diffgit --no-pager diff
journalctljournalctl --no-pager
systemctl statussystemctl --no-pager status …

P2. Long-running foreground processes ​

Symptom: npm run dev, node server.js, tail -f, docker compose logs -f block the bridge until they exit, which they never will.

Cause: A Blocking: true run_command waits for process exit. A foreground server never exits.

Fix: Use Blocking: false with a small WaitMsBeforeAsync (e.g., 2-3 seconds) so the tool returns after the startup output is captured. Then later, use command_status with the returned CommandId to fetch more output. Always remember to stop the background process at the end of the task.

P3. npm install of a watch-script package adding a postinstall daemon ​

Symptom: npm install completes, but a hidden postinstall script forks node …watcher.js & that keeps file descriptors open.

Cause: Some packages (rare, but they exist) start a background watcher in postinstall. The bridge sees the install command return but the descriptor lingers.

Fix: This is unusual in this codebase. If it ever happens, kill the dangling process via pkill -f <package> or restart the IDE.

P4. Command output exceeding the bridge buffer ​

Symptom: npm test finishes but the captured output is truncated mid-line.

Cause: Very long stdout streams can exceed the IDE's capture buffer, especially when test runners print verbose output for hundreds of tests.

Fix: Pipe to tail -N or grep to keep only what matters: npm test 2>&1 | tail -30, npm test 2>&1 | grep -E "^(ℹ|✖|✔)". The full log is still in npm-debug.log if needed.

P5. Background command + mismatched CommandId → false "command failed" conclusion ​

Symptom: A short, quick command (git commit, git push, gh api ...) is launched. Subsequent command_status polls return command <N> not found in trajectory, and read_terminal returns empty. The agent concludes the command failed and asks the user to run it manually - but the command actually succeeded (the commit is in git log, already pushed to origin).

Cause: The command was launched in background mode (Blocking: false) and the bridge returned a Background command ID (e.g. 506). The agent then polled command_status with a guessed or adjacent ID (395, 397, …) that never existed, producing the not found in trajectory error. Short commands also complete and clear before read_terminal can capture anything, so the terminal read comes back empty. None of this is evidence of failure - only that the command's output was never observed.

Fix:

  • For short, quick commands (git, gh, npm view, file checks), always use Blocking: true. Output is returned synchronously in the tool result.
  • If a command genuinely must run in the background, poll command_status with the exact CommandId the launch returned - never a guessed or incremented number.
  • Before reporting that a state-changing command failed, verify the actual state (git log --oneline -3, git status --short, gh api "$BASE/versions"). Absence of captured output is not proof of failure.

Reserve Blocking: false for genuinely long-running foreground processes (servers, tail -f, docker compose logs -f) per P2 - not for fast one-shot commands whose output you need immediately.

Working-directory pitfalls ​

W1. Missing Cwd runs from the repo root ​

Symptom: npm install reports ETARGET No matching version found for @superloomdev/... even though the package version is correct.

Cause: The bridge resets to the repo root for every run_command unless Cwd is explicitly passed. The repo-root package.json has a different dependency tree from each module's _test/package.json.

Fix: Every module-scoped command (npm install, npm test, docker compose …) must pass Cwd set to the module's _test/ directory.

bash
# Wrong - silently runs from repo root
# run_command: { CommandLine: "npm install" }

# Right
# run_command: { CommandLine: "npm install", Cwd: ".../js-server-helper-foo/_test" }

W2. cd <path> && <command> does not persist between calls ​

Symptom: A first call does cd src/foo && npm install; a second call assumes the cwd is src/foo and does npm test, but it runs from the repo root.

Cause: Each run_command is a fresh shell. There is no session.

Fix: Always pass Cwd to every call. Never rely on a previous cd. The user's AGENTS.md explicitly says never propose a cd command for the same reason.

W3. Relative paths in tool calls ​

Symptom: read_file({ file_path: "src/foo.js" }) fails or reads from the wrong directory.

Cause: Most tools require absolute paths. Relative paths are undefined behavior.

Fix: Always pass absolute paths to file tools (read_file, edit, write_to_file, find_by_name, grep_search).

Docker lifecycle pitfalls ​

D1. Manually starting Docker before npm test ​

Symptom: Bind for 127.0.0.1:NNNN failed: port is already allocated, or tests fail immediately with ECONNRESET.

Cause: pretest runs docker compose down -v --remove-orphans for its own compose project name; it does not touch a manually started container. Then pretest tries to bind the same port and fails.

Fix: Pick one owner of the Docker lifecycle: pretest already owns it. Locally and in CI, never run a separate docker run or docker compose up for the same service before npm test.

D2. Stale containers from a crashed prior run ​

Symptom: First test in a fresh session fails because a container from yesterday is still up but has stale state.

Fix: pretest already runs docker compose down -v --remove-orphans. If pretest itself fails on the start step, run that command manually from the same _test/ directory.

D3. docker compose up -d without --wait ​

Symptom: Tests start before the database accepts connections; intermittent ECONNREFUSED on the first request.

Fix: Always use docker compose up -d --wait. The healthcheck must be a real readiness probe (see docs/dev/testing-local-modules.md "Healthcheck Philosophy").

D4. docker compose --wait returns immediately for a service with no healthcheck ​

Symptom: --wait reports the container as Healthy 0.5 s after start, even though the application inside is still initializing.

Cause: When a service has no healthcheck: block, Docker treats "container is running" as healthy. --wait honours that.

Fix: Either define a real healthcheck, or have the test code retry the first connection a few times. DynamoDB Local is the typical case here - the image has no curl/wget/nc to probe with, so the test setup absorbs a brief retry instead.

Test-environment pitfalls ​

T1. AWS SDK calls without dummy credentials ​

Symptom: A test that exercises the AWS SDK takes 1-2 seconds and fails with no clear error.

Cause: With no credentials in the env, the SDK walks the default credential provider chain. The last step of that chain is the EC2/ECS instance metadata service at http://169.254.169.254. There is no metadata service on a developer machine or a GitHub Actions runner, so the chain times out.

Fix: Every AWS test must inject dummy credentials via the _test/package.json test script:

json
"test": "AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 node --test test.js"

The dummies do not need to be valid - they just need to exist so the SDK skips the metadata lookup.

T2. node --test test.js directly, without pretest ​

Symptom: Tests fail immediately with connection errors against the database or queue.

Cause: pretest did not run, so the container is not up.

Fix: Always use npm test. The lifecycle scripts exist for a reason.

T3. Concurrent top-level describe blocks ​

Symptom: "test did not finish before its parent and was cancelled".

Cause: Node's built-in test runner runs top-level describe blocks concurrently. Suites that share lazy-init state (DB pool, AWS SDK client) race each other on the first call.

Fix: Wrap stateful suites in a single outer describe('Module', { concurrency: false }, …). See docs/dev/testing-local-modules.md "Test Concurrency".

T4. Healthcheck passes during a transient-ready window ​

Symptom: Tests pass locally, fail in CI with Connection lost: The server closed the connection.

Cause: A healthcheck that returns "ready" too early. MySQL's two-phase init is the classic case: mysqladmin ping -u root passes during phase 1, then the server restarts in phase 3 and drops every live connection.

Fix: Probe with the credentials, database, and transport the tests will use. See docs/dev/testing-local-modules.md "Healthcheck Philosophy".

File-tool vs terminal pitfalls ​

F1. Using cat to read large files ​

Symptom: Output is truncated at the bridge buffer limit, or the call times out.

Fix: Use read_file with offset + limit. Never cat files larger than ~200 lines.

F2. Using sed / awk / tr to edit files ​

Symptom: The replacement either misses the target line, mangles whitespace, or partially succeeds.

Cause: Stream editors are powerful but error-prone for surgical edits, especially when the target string contains regex metacharacters or quotes.

Fix: Use the edit or multi_edit tool with old_string set to the exact unique substring. The tool guarantees an exact match or a clear error.

F3. Editing a file with a heredoc through cat > file <<EOF ​

Same root cause as S1. Always use write_to_file or edit instead.

F4. Reading gitignored files via read_file ​

Symptom: read_file returns "file is gitignored" for __dev__/.env.dev.

Cause: The IDE's file tools intentionally refuse gitignored paths to prevent accidentally exposing secrets in chat output.

Fix: For gitignored files, use cat /path/to/file via run_command (which is allowed but visible in the approval popup). For writes, write to a temp file via write_to_file and then mv or cat >> it into place.

Auto-run / safety pitfalls ​

A1. Auto-running a destructive command ​

Symptom: The agent flips SafeToAutoRun: true on rm -rf, git push --force, docker volume rm, or npm publish and the user has no chance to review.

Cause: The agent over-trusts a previous successful execution and decides the next call is "obviously fine".

Fix: Auto-run is reserved for read-only operations and idempotent reads (git status, git log -n 20, npm test, docker ps). Anything that mutates state on disk, in a remote registry, or in a long-running service must always require user approval, even if the user has previously approved similar commands. The user's AGENTS.md explicitly forbids npm publish regardless.

A2. Running npm publish directly ​

Symptom: A package is published from the developer's laptop instead of from CI, with the wrong author or unsigned provenance.

Cause: The agent saw a successful test run and decided to publish.

Fix: Publishing in this codebase is CI-only via .github/workflows/ci-helper-modules.yml. Bumping the version in package.json and pushing to main is the only trigger. The CI workflow has a safety net that skips already-published versions.

A3. Force-pushing to a shared branch ​

Symptom: A git push --force rewrites main's history, losing other contributors' commits.

Fix: Never force-push without explicit user approval. The user's AGENTS.md lists this as Never in the boundaries section. Use --force-with-lease if the user explicitly asks to amend.

A4. Modifying .env files ​

Symptom: A pre-existing .env file gets overwritten and the developer loses their local credentials.

Fix: The agent's allowed write locations are spelled out in AGENTS.md "Boundaries". .env is in the Never category (except __dev__/.env, which is the user's personal workspace). For new env keys, update .env.example files only.

CI/CD Publishing ​

Each entry below maps a CI symptom to its root cause and the durable fix. The sibling philosophy doc is cicd-publishing.md; this section is the journal of real failures that shaped those rules.

1. Bind for 127.0.0.1:NNNN failed: port is already allocated in CI ​

Cause: A workflow step started a container with docker run -d --name foo -p 127.0.0.1:NNNN:NNNN ... before the test step ran npm test. pretest then runs docker compose down -v (which does not touch the standalone container) followed by docker compose up -d --wait, which collides on the same port.

Lesson: Pick one owner of the Docker lifecycle. pretest already owns it both locally and in CI. Do not duplicate it with a workflow-level docker run step or a services: declaration that targets the same port -- pick one and remove the other.

2. PROTOCOL_CONNECTION_LOST only in CI ​

Cause: The healthcheck passed too early. On the developer's laptop the service finished init fast enough to be truly ready when the healthcheck passed; on a slower CI runner, the false-positive moment was wide enough that the test connected before init was complete, and the service then dropped the connection during a real init step.

Lesson: See testing-local-modules.md -> Healthcheck Philosophy. The healthcheck must probe at the same level the test will use (credentials, database, transport). Add start_period and enough retries that the total budget covers a cold-pull, cold-start initialization.

3. Push triggers tests but not publish, even though version was bumped ​

Cause (historical): The previous detect logic compared HEAD~1:package.json to HEAD:package.json. After a force-push or a reset that left the version the same on both sides, the diff was empty and publish was skipped -- even when the registry did not have the package.

Fix: The detect job now uses npm view <name>@<version> to ask the registry directly. If the version is missing, publish is scheduled regardless of git history.

4. CI runs tests for every module on every commit ​

Cause: test_modules was incorrectly populated -- typically because the regex used \w+ instead of [\w-]+ and matched module paths greedily, or because someone pushed a single commit that touched every module's package.json.

Lesson: Keep the path regex hyphen-aware: src/helper-modules-[\w-]+/js-[\w-]+. If a single commit truly does touch every module (e.g., a sweep), wide test coverage is the correct outcome.

5. npm publish runs but 409 You cannot publish over the previously published versions ​

Cause: A workflow that publishes on every main push, without checking version bump or registry state.

Lesson: Use the unified detect -> publish-* pipeline. The detect job's registry-existence check, plus the per-publish-job safety-net, is the canonical way to avoid this error. Do not rebuild a separate publish workflow.

6. 403 Forbidden on npm publish ​

Cause: Missing permissions: packages: write on the publish job, or repository-level workflow permission set to "Read repository contents permission".

Lesson: Both must be set:

  1. Job-level permissions: { contents: read, packages: write }
  2. Repo-level Read and write permissions in Settings -> Actions -> General

7. CI test fails with npm error notarget No matching version found for @superloomdev/... ​

Cause: The CI step ran npm install from the wrong directory (typically the module root instead of _test/). Each _test/ directory has its own package.json with its own dependency tree. The repo-root package.json does not declare the test deps.

Lesson: In CI, set working-directory: to the module root and use cd _test && npm install && npm test for the test step. The same rule applies locally -- always pass Cwd to _test/ for AI agents and scripts.

8. CI test fails with MODULE_NOT_FOUND for a helper that exists in the repo ​

Symptom: A test-* CI job fails with a Node.js MODULE_NOT_FOUND stack trace pointing to a file inside another helper module (e.g. js-server-helper-nosql-mongodb/mongodb.js), despite that module being present in src/. The error occurs even though file:../../js-server-helper-nosql-mongodb is listed in the _test/package.json dependencies.

Cause: file: path dependencies copy the directory contents at npm install time but do not run npm install inside the linked package. In CI, the runner checks out a fresh clone; the linked helper's own node_modules/ are absent, so any require() inside it that needs its own npm deps (e.g. the mongodb driver) fails immediately with MODULE_NOT_FOUND.

This works locally only because the helper was previously installed in its own directory during development. CI never does that.

Lesson: Never use file: paths in _test/package.json for helper modules that have their own npm dependencies. Use registry version ranges ("^1.0.0") instead. The file:../ self-reference (pointing to the module under test) is the one legitimate exception. npm installs it as a directory link and the module itself has no transitive runtime deps outside what the test loader provides. For every other shared helper (storage, database, cloud), always pin to the published registry version.

Quick rule: file: is allowed only for "[module-under-test]": "file:../". Everything else must be a registry semver range.

9. Tests Fail in CI ​

The publish job has needs: [detect, test-*], so it never runs if tests fail. Fix the tests and push again. The detect job will pick the same set of unpublished modules and try again.

10. Invalid workflow file: ... You have an error in your yaml syntax on line N ​

Cause: A bash assignment inside a run: | block scalar contained an embedded literal newline:

yaml
run: |
  PUBLISH_MODULES="$PUBLISH_MODULES$MODULE
"

YAML block scalars (|) require every non-empty line to be indented at least to the block's indent level. The closing " on its own line had zero leading spaces -- less than the block's indent -- which terminates the block scalar early and fails YAML parsing.

Lesson: Never embed a literal newline inside a bash string assignment within a YAML run: | block. Use bash's $'\n' escape (or printf '%s\n', or a bash array) so every line of YAML respects the block's indentation:

yaml
run: |
  PUBLISH_MODULES="${PUBLISH_MODULES}${MODULE}"$'\n'

This applies anywhere YAML uses block scalars: GitHub Actions run:, Docker Compose command:, Helm chart values, CI configs, etc.

11. publish-* jobs silently skip when an upstream publish-* is also skipped, breaking the test→publish chain ​

Symptom: A test-* job runs and succeeds. The next publish-* job in the chain is reported as skipped (zero steps, zero seconds). Multiple downstream test-* jobs then run against a stale registry version and fail with cryptic runtime errors like TypeError: mongo.createIndex is not a function. The bumped helper version that supplied the new API was never actually published.

Cause: GitHub Actions evaluates an implicit success() check on every job that does not start its if: with a status-check function (always(), failure(), cancelled(), !cancelled()). And success() is transitive. It returns false if any job in the upstream needs graph (not just the direct needs) has the result skipped, failure, or cancelled. Quote from GitHub's docs: "If a job fails or is skipped, all jobs that need it are skipped unless they use a conditional expression that causes the job to continue."

A strictly-sequential test→publish pipeline like ci-helper-modules.yml mixes:

  • test-* jobs that use if: always() && !cancelled() && ... → run regardless of upstream skips
  • publish-* jobs that previously used if: >- needs.detect.outputs.publish_modules != '[]' && contains(...) → no override, so the implicit success() applied

In fresh-state recovery runs (every module needs publishing), nothing in the chain is skipped, so this never fires. In steady-state runs where some modules are already on the registry, those modules' publish-* jobs legitimately skip; that skip silently propagates downstream, disabling every subsequent publish-* even though their direct needs (detect + the matching test-*) all succeeded.

Concrete failure that surfaced this: A run with publish_modules = [auth, logger, nosql-aws-dynamodb, nosql-mongodb, verify]:

  • publish-storage-aws-s3 correctly skipped (s3 already on registry, not in publish_modules)
  • test-nosql-aws-dynamodb ran and succeeded (uses always())
  • publish-nosql-aws-dynamodb SKIPPED (even though it was in publish_modules and its direct needs succeeded) because success() walked the transitive chain back to publish-storage-aws-s3 and saw skipped
  • All downstream publish-* skipped for the same reason
  • mongodb@1.1.0 and dynamodb@1.1.0 never reached the registry
  • Then test-verify, test-logger, test-auth ran with npm install resolving ^1.0.0 to the stale registry 1.0.0, which lacked the createIndex / createTable / deleteRecordsByFilter APIs the 1.1.0 source code calls. TypeError at first use

Lesson: Every publish-* job in a chained pipeline must override the implicit success() check and assert its own dependencies explicitly:

yaml
publish-foo:
  needs: [detect, test-foo]
  if: |
    !cancelled() &&
    needs.detect.result == 'success' &&
    needs['test-foo'].result == 'success' &&
    needs.detect.outputs.publish_modules != '[]' &&
    needs.detect.outputs.publish_modules != '' &&
    contains(needs.detect.outputs.publish_modules, 'js-server-helper-foo')

Two parts that both matter:

  1. !cancelled() && disables the implicit transitive success() check (GitHub's recommended alternative to always() for normal jobs, per the official docs).
  2. needs.detect.result == 'success' && needs['test-foo'].result == 'success' restores the safety the implicit check used to give us, but scoped to the direct needs only. Hyphenated job IDs require bracket notation (needs['test-foo'], not needs.test-foo; the latter is parsed as subtraction).

Defence in depth: pin _test/package.json to the version your code actually requires. When a helper bumps its own version (e.g. mongodb@1.1.0 adds createIndex) and downstream modules start calling the new API, every consuming _test/package.json must pin that same ^1.1.0, not the older ^1.0.0. Two reasons:

  • If the upstream publish ever fails, ^1.1.0 causes npm install to fail with a clean E404 No matching version instead of installing the stale 1.0.0 and surfacing as a TypeError deep inside a test.
  • The version pin documents the hard floor required by the source. Anyone reading the test deps sees exactly what the module needs.

Quick rules:

  • _test/package.json registry pins must match the API surface the source code uses, not the lowest published version.
  • Every publish-* job in a chained workflow must start its if: with !cancelled() (or always(), failure(), cancelled()) and re-assert its direct needs.<job>.result == 'success'. Otherwise a single skipped sibling silently disables the rest of the publish chain.

12. Main module and its adapters are both unpublished: bootstrap order ​

Situation: Any module that ships a main package plus separate adapter packages (e.g. js-server-helper-auth + js-server-helper-auth-store-sqlite) can reach a state where both the main module and one or more adapters have unpublished breaking changes. Neither side can reference the other via a registry semver range because the version does not exist yet.

The correct sequence:

  1. Test adapters locally against a file: path of the main module. In each adapter's _test/package.json, temporarily point the main module dependency at the local checkout (file:../../js-server-helper-<name>). Run the full adapter test suite. This validates the new API contract end-to-end without touching the registry.

  2. Publish the main module (bump version in package.json, push to main; CI publishes it).

  3. Switch adapter _test/package.json deps from file: to the newly published registry version (e.g. "^2.0.0"). Re-run the adapter test suite against the live registry version. This confirms the package round-trips correctly through npm pack/publish and resolves correctly for real consumers.

  4. Publish the adapters (bump versions, push to main; CI publishes them).

Why the re-test in step 3? file: deps are copied at npm install time, bypassing npm's pack/unpack pipeline entirely. A module that works locally via file: can silently fail after publishing if the files field in package.json is wrong, a required file is gitignored, or a main entry resolves differently after packing. Step 3 catches this class of error before any consumer is affected.

Lesson: The temporary file: reference is the one legitimate exception to the rule that _test/package.json only uses file: for the module under test itself. It is only valid during the bootstrap window and must be replaced with a registry pin before the adapters are published. Add a comment so the intent is clear:

json
"@superloomdev/js-server-helper-<name>": "file:../../js-server-helper-<name>"

Replace with "^<version>" after the main module is published (step 3).

14. test-verify CI fails with Cannot find module '../stores/sqlite' after stores/ was deleted ​

Symptom: test-verify in CI fails with MODULE_NOT_FOUND pointing to ../stores/sqlite inside the verify module's own _test/test-sqlite.js. The stores/ directory was intentionally deleted as dead code in the same commit that introduced standalone js-server-helper-verify-store-* packages.

Cause: Two related problems compounded:

  1. The internal test file _test/test-sqlite.js still referenced the deleted ../stores/sqlite path. It was not updated to use the npm package @superloomdev/js-server-helper-verify-store-sqlite.

  2. The five js-server-helper-verify-store-* adapter packages had no CI jobs at all. ci-helper-modules.yml skipped directly from publish-sql-postgres to test-verify. This meant the adapters were never published to the registry, so _test/package.json could not resolve them even after the test file was fixed.

Additionally, the adapter package.json files had been (incorrectly) set to private: true, which would have prevented npm publish even if CI jobs existed.

Fix:

  1. In _test/test-sqlite.js: replace require('../stores/sqlite') with require('@superloomdev/js-server-helper-verify-store-sqlite').
  2. In _test/package.json: add "@superloomdev/js-server-helper-verify-store-sqlite": "^1.2.0" to dependencies.
  3. In all five adapter package.json files: set "private": false so npm publish works.
  4. In ci-helper-modules.yml: insert ten new jobs (modules 17-21), one test-verify-store-* + publish-verify-store-* pair per adapter, in the sequential chain before test-verify. Update test-verify's needs to [detect, publish-verify-store-dynamodb].

Lesson: Deleting an internal directory (stores/) that is still referenced by the module's own test suite, while simultaneously introducing replacement standalone packages, requires two coordinated changes: (a) update every require() that pointed at the old path, and (b) ensure the new packages exist in the CI pipeline and are published before any consumer runs npm install. A quick grep for the deleted path before committing prevents the first problem; checking whether each new adapter has CI jobs prevents the second.

13. CI fails on lint after local tests pass: pre-publish checklist not followed ​

Symptom: npm test in _test/ passes locally (all green). The version is bumped and pushed to main. CI runs npm run lint from the module root as a separate step before tests and fails with no-trailing-spaces or a stale eslint-disable-line directive. The publish job never runs even though all functional tests would have passed.

Cause: The local test command (node --test test.js from _test/) never invokes ESLint. Lint is a separate npm run lint script defined in the module root's package.json, not in _test/package.json. CI always runs lint before tests; local workflow often only runs tests. Any whitespace issue or stale disable comment that was present (or introduced) before the push goes undetected locally.

Lesson: Before bumping the version in package.json and pushing to main, always run the full pre-publish sequence from the module root, not just from _test/:

bash
# From the module root
npm run lint          # must exit 0
# From _test/
npm install && npm test  # must exit 0

Both must be green before the version bump commit. If lint fails, fix it first. Do not push a lint-broken version and rely on CI to catch it. The lint fix will require a second commit and re-trigger the entire CI pipeline, wasting pipeline time and adding noise to the git log.

Quick rule: Treat npm run lint (module root) + npm test (_test/) as a single inseparable gate. Never bump a version without both passing locally.

15. VitePress / Vue compiler crashes on bare angle-bracket placeholders in markdown ​

Symptom: The Deploy Website workflow fails during vitepress build with an error like:

[vite:vue] [plugin vite:vue] docs/architecture/<file>.md (NNN:1):
  Element is missing end tag.
SyntaxError: [plugin vite:vue] ...

The reported line number is often far below the line that actually caused the problem. The parser walks until it gives up, then reports the position at which it abandoned the parse.

Cause: VitePress feeds rendered markdown through Vue's compiler so that authors can use Vue components inside markdown. The compiler does not unconditionally trust markdown's HTML-escaping: any <lowercase-or-PascalCase-name> inside the rendered output is treated as a Vue/HTML element open tag and triggers the close-tag search. Three concrete failure modes seen on this codebase:

  1. Bare placeholders outside any code formatting. A bullet like - <Domain Operations> *(convenience layer)* parses as opening element <Domain with attribute Operations, then waits forever for </Domain>.
  2. Placeholders inside a fenced code block tagged ```markdown. Vue still scans the block's content for tags because the markdown language hint enables an additional templating pass. Placeholders such as <sibling-1> and <one-line description> inside a markdown-tagged fence will trip the same error.
  3. Placeholders inside backticks are usually safe (they render as HTML-escaped <code> content), but a few corner cases (extremely long backtick spans, nested code formatting, generic-looking names like <T>) have produced the same failure.

Lesson: Treat angle-bracket placeholders in markdown as a build-breaking smell. Three durable rules:

  1. Never use <...> placeholders outside backticks in prose. Use square brackets ([name]), curly braces ({name}), or plain capitalized phrases ("Domain operations") instead.
  2. For verbatim copy-paste templates inside fenced code blocks, prefer the text language hint. Switching ```markdown to ```text disables the secondary Vue scan and lets the placeholders survive untouched. If the syntax-highlighting loss is unacceptable for a particular block, replace the angle-bracket placeholders with square-bracket ones at template-author time and keep the markdown hint.
  3. Verify documentation changes locally before pushing when the change touches any VitePress-rendered file in docs/. Run npm run build from website/. That is the same pipeline CI runs (vitepress build . after sync-docs). Watch for the Element is missing end tag family of errors. Local build is fast (single-digit seconds) and catches the failure before it occupies a CI runner.

This pitfall is distinct from the helper-modules CI chain (entries 1–14): it lives in ci-deploy-website.yml, not ci-helper-modules.yml, and a website-deploy failure does not block module publishing. The two pipelines are independent. But the same commit that triggers helper-module publishing will also trigger website deploy if it touches any docs/ file, so a documentation-side bug is a hidden cost on every push that updates rules.

16. Deleting a GitHub Packages package: org-scoped vs user-scoped endpoint ​

Symptom: gh api --method DELETE /users/OWNER/packages/npm/PACKAGE/versions/ID returns success (exit 0, empty body) but the package is still on the registry. Repeated attempts appear to "do nothing", and the agent cannot tell whether the delete worked.

Cause: superloomdev packages are owned by a GitHub organization, not a user. Organization-owned packages live under /orgs/OWNER/.... The /users/OWNER/... endpoint targets the user-account namespace; for an org-owned package it returns a misleading no-op instead of an error. Two further traps: the API path uses the bare package name without the @scope/ prefix (js-server-helper-http-gateway, not @superloomdev/js-server-helper-http-gateway), and deleting the last remaining version deletes the whole package.

Lesson: Determine the owner type first, then use the matching endpoint and always verify with a 404.

bash
gh api /users/OWNER --jq '.type'        # "Organization" or "User"
# org:        BASE=/orgs/OWNER/packages/npm/PACKAGE_NAME
# own user:   BASE=/user/packages/npm/PACKAGE_NAME
gh api "$BASE/versions"                  # list / confirm it exists
gh api "$BASE/versions" --jq '.[].id' | xargs -I {} gh api --method DELETE "$BASE/versions/{}"
gh api "$BASE/versions"                  # MUST return 404 Package not found

If the verify step returns a JSON array, the delete did not work - re-check the owner type and path.

Confirmed behavior after deletion: GitHub Packages accepts republishing a previously-used version name once the version (or the whole package) is deleted - version names are not permanently burned. Combined with the detect job's registry-existence check, this enables a full re-baseline: wipe the packages, push to main (no file changes needed), and CI republishes every module at its unchanged package.json version. Verified across a 44-module wipe-and-republish. Round-trip proof after republish: npm install @scope/[name]@[version] from a scratch directory must resolve.

17. git add . bundles unrelated modules into one commit ​

Symptom: A single commit touches many modules at once. CI's per-module detect job then schedules tests/publish for every changed module path in that commit, and git history no longer maps one module refactor to one commit ("bulk commits").

Cause: git add . (or git add -A) stages every modified file in the working tree, not just the module currently being worked on. During a multi-module sweep (e.g. refactoring a parent plus its adapters), in-progress edits to sibling modules get swept into the same commit.

Lesson: Stage only the directory of the module being committed, so each module refactor is one independent commit:

bash
git add src/helper-modules-server/js-server-helper-<name>/
git commit -m "refactor(<name>): one-line summary"

This keeps the CI publish trigger scoped to a single module, makes history bisectable, and lets one module be reverted without disturbing others. Always run git status --short before committing and confirm only the intended module's files are staged.

23. Extension module's test job 404s on its base package: needs pointed at the base's test job, not its publish job ​

Symptom: During a full-registry republish, an extension module's test job (e.g. test-[base]-ext-[framework]) fails with npm error 404 Not Found ... @scope/[base]@^1.0.0 while the base module's own test and publish jobs are green in the same run.

Cause: The extension's _test/package.json installs the base module from the registry (a registry semver range, not file:). The extension's test job was chained with needs: [detect, test-[base]], so it started as soon as the base's tests passed - racing the base's publish job. On a fresh registry the base package does not exist yet when the extension's npm install runs.

Lesson: A test-* job whose _test/package.json installs another in-repo package from the registry must needs that package's publish-* job, never its test-* job. In steady-state runs the difference is invisible (the base is already published, both orderings pass), so the bug only surfaces during bootstrap or a registry re-baseline - review every extension and adapter chain against this rule before a fresh-state push.

Coverage-sweep footnote: when verifying that every module has CI jobs by extracting paths from the workflow file, the extraction pattern must cover the full name alphabet. A grep -oE character class of [a-z-]+ silently drops any path containing a digit (storage-aws-s3), producing false "missing from CI" findings. Include digits ([a-z0-9-]+) and cross-check the extracted count against the known module total before acting on the diff.

Local Module Testing ​

Each entry below maps a symptom to its root cause and the durable fix. The sibling philosophy doc is testing-local-modules.md (healthcheck philosophy, test concurrency rules, module reference table); this section is the journal of real failures that shaped those rules.

1. npm error notarget No matching version found for @superloomdev/... ​

Cause: npm install ran from the repo root or the module root instead of _test/. Each _test/ directory has its own package.json with a different dependency tree.

Lesson: Always cd into _test/ first. Tools that automate this (AI agents, scripts) must always pass Cwd explicitly to the _test/ directory -- omitting it silently runs from the repo root.

bash
# Wrong: from the module root
npm test

# Correct: from _test/
npm install && npm test

2. MODULE_NOT_FOUND for a package that should be installed ​

Cause: A require() in the test uses the wrong scoped package name -- typically missing the category prefix (@superloomdev/js-server-helper-postgres instead of @superloomdev/js-server-helper-sql-postgres).

Lesson: The npm package name must match the full directory name, including every category prefix. Grep for the bare name (grep -r "js-server-helper-postgres" _test/) before assuming the install is broken.

3. Manually starting Docker before npm test ​

Symptom: Bind for 127.0.0.1:NNNN failed: port is already allocated, or tests fail immediately with ECONNRESET / socket hang up, many tests cancelled.

Cause: pretest runs docker compose down -v --remove-orphans first. That command only manages containers from its own compose project name -- it does not touch a manually started container. Then docker compose up tries to bind the same port and fails. (The same conflict exists in CI when a workflow step runs docker run ahead of npm test.)

Lesson: Pick one owner of the Docker lifecycle. pretest already owns it. Locally and in CI, never run a separate docker run or docker compose up for the same service before npm test.

4. node --test test.js directly without pretest ​

Cause: pretest did not run; no container is up. Tests fail immediately with connection errors.

Lesson: Use npm test, not node --test test.js. The lifecycle scripts exist for a reason.

5. Stale container from a previous failed run ​

Cause: A prior run died before its posttest could clean up.

Fix: pretest already handles this with docker compose down -v --remove-orphans. If pretest itself fails because of a deeper conflict, clean up manually from the same _test/ directory:

bash
docker compose down -v --remove-orphans

6. Tests pass locally, fail in CI with Connection lost: The server closed the connection ​

Cause: A healthcheck that returns "ready" too early. On Docker Desktop the service finishes init fast enough that the false-positive moment never overlaps with the test run; on a constrained CI runner, init takes longer, the healthcheck passes during a transient-ready window, and the server then drops connections during a real init step (e.g., MySQL's restart in normal mode).

Lesson: See testing-local-modules.md Healthcheck Philosophy. Probe with the credentials, database, and transport the tests will use. Add start_period and enough retries that the total budget covers a cold-pull, cold-start initialization.

7. test did not finish before its parent and was cancelled ​

Cause: Concurrent execution of top-level describe() blocks. A lazy-init resource was created mid-test by a parallel block, leaving the cancelled block in a half-initialized state.

Lesson: Wrap the suite in describe('Module', { concurrency: false }, ...). See testing-local-modules.md Test Concurrency.

8. && sleep 2 (or 5) in pretest ​

Cause: A previous developer added a sleep to mask an unreliable healthcheck.

Lesson: Sleeps are never the right fix. They paper over bad healthchecks, slow every developer's iteration, and still fail under load. Remove the sleep, then fix the healthcheck so docker compose up -d --wait truly waits until the service is ready.

9. AI agents or scripts hanging on a multi-line shell command ​

Cause: Heredocs (cat <<'EOF' ... EOF) and multi-line -m arguments to git commit cause zsh to enter dquote> continuation mode when the closing quote falls on a different line. Special characters (backticks, $, !, (...)) make this worse.

Lesson: For commit messages: prefer a single-line -m, or stack multiple -m flags, or use -F /tmp/file after writing the file via a non-shell tool. For file content: never use heredocs through a shell bridge; write to a file with the editor tool and cat it from there. Full journal: AI Terminal & Shell Bridge -> S1/S2.

10. AWS SDK test hangs ~1-2s, then success: false with no clear error ​

Symptom: A test that exercises an AWS SDK function (getSignedUrl, S3Client.send, DynamoDBClient.send, etc.) takes 1-2 seconds and fails with success === true assertion failing. Stack trace points to your module's catch block but the underlying error is silent or absorbed.

Cause: No AWS credentials are passed to the SDK. The SDK walks the default credential provider chain: env vars -> shared config file -> EC2/ECS instance metadata (http://169.254.169.254). On a developer machine and on a GitHub Actions runner there is no instance metadata service, so the chain times out. The 1-2 second test duration is the metadata-service connection timeout.

Lesson: Every test that uses an AWS SDK client must inject dummy credentials, even when no real network call is made (URL signing, command construction, etc.). The dummies do not need to be valid -- they just need to exist so the SDK does not enter the default-chain code path.

The canonical pattern is to set them in the test script of _test/package.json:

json
"test": "AWS_ACCESS_KEY_ID=local AWS_SECRET_ACCESS_KEY=local AWS_REGION=us-east-1 ... node --test test.js"

For service-specific env vars (S3_ACCESS_KEY, DYNAMODB_ENDPOINT, etc.), use the names the module's loader reads. The same env vars must also exist in __dev__/.env.dev and docs/dev/.env.dev.example per the four-file rule.

11. MongoDB replica-set healthcheck reports healthy before the node is PRIMARY ​

Symptom: Tests pass on a developer's macOS machine and fail in CI with the first few writeRecord / getRecord (after writing) calls returning success: false or result.document === null. Later writes in the same suite succeed. The mongodb helper's own _test and any downstream consumer's _test:mongodb are both vulnerable.

Cause: A naive replica-set healthcheck:

yaml
test: ["CMD", "mongosh", "--eval", "try { rs.status().ok } catch(e) { rs.initiate({...}).ok }"]

returns truthy as soon as rs.initiate() returns. But on a single-node replica set the node still spends another 1-2 s in SECONDARY → STARTUP2 → PRIMARY before it accepts writes. docker compose up --wait returns "healthy" mid-election; the test process opens its driver and fires writes immediately. On Docker Desktop / macOS the local stack happens to be fast enough that the test loop hits PRIMARY by chance; on the slower hosted GitHub Actions runner the first writes land mid-election and fail with not master-style errors that the helper catches and surfaces as success: false.

Lesson: The healthcheck must verify the same readiness the tests require (write-ready primary, not just replica-set initialized). Use db.hello().isWritablePrimary and quit(1) so docker keeps retrying until the node is actually primary:

yaml
test:
  - CMD
  - mongosh
  - --quiet
  - --eval
  - "try { rs.status() } catch(e) { rs.initiate({ _id: 'rs0', members: [{ _id: 0, host: 'localhost:27017' }] }) }; if (!db.hello().isWritablePrimary) quit(1)"

This is the same probe-the-application-protocol principle as MySQL's two-phase init (entry 6). rs.status().ok is the equivalent of mysqladmin ping -u root (server alive but not yet ready for the test workload).

The general rule for any service that has an init phase distinct from "process up": the healthcheck must succeed only after the test-relevant phase has completed (PRIMARY for replica sets, test_user reachable for MySQL, test_db schema applied for Postgres, etc.).

12. verify.generateAndStore returns COOLDOWN_ACTIVE with cooldown_seconds: 0 under concurrent calls ​

Symptom: js-server-helper-verify tests fail intermittently in CI (rarely locally) with COOLDOWN_ACTIVE errors even though the test explicitly sets cooldown_seconds: 0. The concurrency test in _test/shared-store-suite.js ("concurrent createPin calls with cooldown:0 all land") is the most reliable reproducer; downstream consumers of verify see the same flake on CI.

Cause: The cooldown gate in generateAndStore computed now - existing.created_at without short-circuiting when cooldown_seconds === 0. Under concurrency, two requests can share a microsecond-close instance.time while the store's created_at (captured a few milliseconds earlier by the first winning write) is ahead of the second caller's now. The signed diff is negative, diff < cooldown_seconds is trivially true for any positive threshold; and even for 0, the strictly-less-than check fires when the diff is negative. The caller sees a cooldown error that has no real cooldown semantically.

The bug was dormant until the MongoDB PRIMARY-election fix (entry 11) stabilised CI enough that the race became reproducible; before that, the flake was attributed to the mongodb healthcheck and the true cause went unseen.

Lesson: A "cooldown disabled" configuration must short-circuit before any arithmetic on timestamps. The canonical fix is a single explicit check at the top of the gate:

js
// verify.js generateAndStore
if (options.cooldown_seconds === 0) {
  // cooldown disabled; do not consult existing.created_at
} else if (existing && existing.created_at) {
  const diff = now - existing.created_at;
  if (diff < options.cooldown_seconds) {
    return { success: false, error: CONFIG.ERRORS.COOLDOWN_ACTIVE };
  }
}

Generalised rule: any time a helper has a "feature disabled when N === 0" semantics, the disabled branch must bypass every downstream computation that uses N or any state N would have produced. Never rely on 0 < diff < N to be false when N === 0, because diff can be negative under concurrent non-monotonic time sources.

Applies anywhere a rate-limit, throttle, TTL, or cooldown is configurable with a "zero = off" value: verify.cooldown_seconds, auth.LAST_ACTIVE_UPDATE_INTERVAL_SECONDS, logger retention, and any future equivalent. Audit the gate when introducing any such option.

15. Repo-wide MODULE_NOT_FOUND after a "remove scope prefix" style commit ​

Symptom: All _test/ suites fail immediately at require('js-helper-utils') with MODULE_NOT_FOUND. No code logic has changed; only require('@superloomdev/...') calls were rewritten to require('...') in a mass search-and-replace commit. node_modules/ still contains only scoped packages (@superloomdev/js-helper-utils, etc.); the unscoped names simply don't exist.

Cause: npm installs packages under their full published name (the "name" field in package.json, which includes the @superloomdev/ scope). A require() call must use that exact name. Removing the scope prefix from require() calls without simultaneously adding npm aliases (or renaming the packages on the registry) breaks every module that was changed.

Lesson: Never strip scope prefixes from require() calls unless the packages are also republished without the scope (or aliased via package.json imports map). The safe rule: require() strings must always match "name" in the target package's package.json exactly. Fix is a single sed to restore the @superloomdev/ prefix across all .js files under src/, excluding node_modules/.

bash
find src/ -name "*.js" -not -path "*/node_modules/*" | \
  xargs sed -i '' \
    "s/require('js-helper-/require('@superloomdev\/js-helper-/g; \
     s/require('js-server-helper-/require('@superloomdev\/js-server-helper-/g; \
     s/require('js-client-helper-/require('@superloomdev\/js-client-helper-/g"

16. Store contract method name drift between logger.js and test fixtures ​

Symptom: Logger unit tests fail with TypeError: store.addLog is not a function even though the memory store is correctly imported. Grep shows zero matches for addRecord in the source, yet the error points at logger.js:147.

Cause: When logger.js store contract method names were renamed (addRecord→addLog, listByEntity→getLogsByEntity, listByActor→getLogsByActor, initializeStore→setupNewStore, cleanupExpiredRecords→cleanupExpiredLogs), the rename was applied partially: logger.js and the adapter packages were updated but the inline captureStore and minimalStore stubs scattered throughout _test/test.js and the memory-store.js fixture were not. node_modules held a correct symlinked copy, but the directly-required ../logger.js used the new names, so the inline stubs were invisible mismatches.

Lesson: When renaming store-contract methods in the core module, search for every store stub in _test/test.js, not just the shared fixtures, because inline anonymous objects appear throughout the test file and are missed by a module-level rename. Grep pattern: require('../logger.js') → addRecord|listByEntity|listByActor|initializeStore|cleanupExpiredRecords. The same applies to any module that has inline store stubs in its test file (auth, verify).

17. Store contract method renames not propagated to adapter _test/test.js files: all adapter CI jobs fail ​

Symptom: All test-verify-store-* CI jobs fail with TypeError: store.initialize is not a function. All test-auth-store-{sqlite,postgres,mysql} fail with TypeError: auth.createSchema is not a function. Tests pass locally only if stale node_modules from a prior install are present.

Cause: Two separate renames were applied to the core module source and adapter store.js files but not propagated to the _test/test.js files:

  1. verify.js + all verify-store-*/store.js: store.initialize() → store.setupNewStore() (v2.1.x → v2.2.0). The _test/test.js stubs in all 5 verify-store adapters were not updated. Because verify v2.2.0 was never published (CI was failing at the time), ^2.1.0 in _test/package.json resolved to v2.1.x on the registry (which uses the old name). This masked the problem locally but was exposed as soon as CI installed fresh.
  2. auth.js: auth.createSchema() → auth.setupNewStore(). The 3 SQL auth-store adapter _test/test.js files still called auth.createSchema().

Lesson: When renaming any store-contract or module-public method, grep all _test/test.js files across every adapter before committing, not just the module source and its own tests. Pattern: grep -rn "old_method_name" src/. Adapter test files are not auto-updated by renaming the source. Also: a new major-bump on a core module will not reach adapters until it is both published and the adapter _test/package.json pin is bumped.

18. contains() substring matching causes parent modules to trigger when only adapters need publishing ​

Symptom: When only an adapter module (e.g., js-server-helper-verify-store-sqlite) needs to be published, the CI workflow also runs jobs for the parent module (js-server-helper-verify). The base module is detected as needing publish even though npm view shows it already exists on the registry. This causes unnecessary republishing of parent modules when only their adapters should be processed.

Cause: GitHub Actions contains() function performs substring matching, not exact string matching. The detect job outputs JSON arrays like:

json
["src/helper-modules-server/js-server-helper-verify-store-sqlite"]

The job condition contains(needs.detect.outputs.test_modules, 'helper-modules-server/js-server-helper-verify') returns true because 'helper-modules-server/js-server-helper-verify' is a substring of 'src/helper-modules-server/js-server-helper-verify-store-sqlite'.

Fix: Parse the JSON array first, then check for exact element matching:

yaml
# WRONG: Substring matching
contains(needs.detect.outputs.test_modules, 'helper-modules-server/js-server-helper-verify')

# CORRECT: Exact array element matching
contains(fromJSON(needs.detect.outputs.test_modules), 'src/helper-modules-server/js-server-helper-verify')

Lesson: When a workflow outputs JSON arrays from the detect job and downstream jobs need to check if a specific value is in that array:

  1. Always wrap the output with fromJSON() to convert the JSON string back to an actual array
  2. Use the full path (including src/ prefix) in the match string to ensure exact matching
  3. Never rely on contains() with a JSON string directly -- it performs substring matching on the serialized JSON, not element matching on the array

This applies to any job condition checking against detect outputs:

yaml
# Pattern for all module jobs
if: contains(fromJSON(needs.detect.outputs.test_modules), 'src/helper-modules-category/js-module-name')
if: contains(fromJSON(needs.detect.outputs.publish_modules), 'src/helper-modules-category/js-module-name')

19. Test loader uses relative source path instead of the _test/package.json alias ​

Symptom: A test-* CI job fails with Error: Cannot find module 'cookie' (or any transitive dep of the module under test). The require stack points at files inside the module's source directory (e.g. src/helper-modules-server/js-server-helper-http-gateway/parts/cookies.js), not at _test/node_modules/.... Tests pass on the developer's machine because node_modules/ from a previous module-level npm install is still present, but break in a clean CI environment.

Cause: The test loader (or test file) was written as require('../adapter.js') or require('../../sibling-module/main.js') instead of using the alias declared in _test/package.json. The CI workflow runs npm install inside _test/, which only populates _test/node_modules/. Node's require resolves the relative path to the source directory, where the module's transitive deps were never installed (CI never runs npm install at that location).

Why aliases exist: Every _test/package.json declares aliases like:

json
"dependencies": {
  "helper-foo": "file:../",
  "helper-foo-adapter-bar": "npm:@superloomdev/js-server-helper-foo-adapter-bar@^1.0.0"
}

These aliases exist precisely so the loader code stays identical between local-source (resolved via file:../) and published-package (resolved via npm:) contexts. When npm install runs in _test/, it copies the aliased package into _test/node_modules/ AND installs that package's transitive deps right next to it. Bypassing the alias defeats this entirely.

Fix: Always require helpers in _test/ files via the alias declared in _test/package.json:

javascript
// WRONG - relative path to source directory
const HttpGateway        = require('../http-gateway.js');
const HttpGatewayAdapter = require('../adapter.js');

// CORRECT - alias from _test/package.json
const HttpGateway        = require('helper-http-gateway');
const HttpGatewayAdapter = require('helper-http-gateway-adapter-express');

Exception: Internal files not exposed via the package's main entry (e.g. parts/cookies.js, parts/params.js testing internals) may continue to use relative paths. They are not addressable through the alias because they are not part of the package's public surface.

Lesson: Treat require() style as part of test hygiene, not as a stylistic choice. The HTTP Gateway trio (js-server-helper-http-gateway + 2 adapters) is the canonical reference.

20. Singleton pattern prevents test isolation ​

Symptom: Tests interfere with each other, last test's configuration affects subsequent tests, cannot run tests in parallel, difficult to mock dependencies.

Cause: Module uses singleton pattern but takes external dependencies (libs, config, adapters). The singleton shares global state across all tests, making isolation impossible.

Lesson: Use factory pattern for any module that takes external dependencies. Only use singleton pattern for pure utility modules with zero dependencies (like js-helper-utils). Factory pattern enables:

  • Independent instances per test with different configurations
  • Parallel test execution without conflicts
  • Mock dependency injection per test scenario
  • Multiple instances for different use cases

Example of the problem:

javascript
// WRONG: Singleton with dependencies
let CONFIG;
module.exports = function loader (config) {
  CONFIG = config;  // Last caller wins!
  return singletonInstance;
};

// PROBLEM: Test A sets {log_level: 'debug'}
// PROBLEM: Test B sets {log_level: 'error'}  
// PROBLEM: Both tests now use 'error' level

Example of the solution:

javascript
// CORRECT: Factory pattern (fixed-slots shape - see module-structure.md)
module.exports = function loader (shared_libs, config) {
  const Lib = { Utils: shared_libs.Utils };
  const CONFIG = Object.assign({}, require('./[module].config'), config || {});
  const ERRORS = require('./[module].errors');
  const Validators = require('./[module].validators')(Lib, ERRORS);
  Validators.validateConfig(CONFIG);
  return createInterface(Lib, CONFIG, ERRORS, Validators);
};

// BENEFIT: Each test gets isolated instance
const testInstance1 = createModule({log_level: 'debug'});
const testInstance2 = createModule({log_level: 'error'});

21. Stale package-lock.json causes wrong-version install after file: swap or version reset ​

Symptom: npm install && npm test in _test/ fails with a runtime error that does not match the source code. For example, CONFIG.ADAPTER must be an adapter factory function even though the source was already refactored to the new CONFIG.Adapter object pattern. Or: a freshly deleted-and-republished 1.0.0 still installs the old pre-refactor tarball. Or: npm error 409 Conflict - Package file checksum mismatch on a package that has not changed.

Cause: Two compounding sources produce the same class of failure:

  1. Stale file: path in lock file. During bootstrap testing, _test/package.json is temporarily changed to "helper-foo": "file:../../js-server-helper-foo" so tests can run against local source before the parent module is published. After restoring package.json to a registry semver range, the package-lock.json still has "resolved": "../../js-server-helper-foo". A subsequent npm install reads the lock, sees the path is satisfied, and skips the registry fetch, silently installing the local source (or nothing, if the path no longer exists).

  2. Version reset to 1.0.0 hits a cached tarball. When a module is deleted from the registry and re-published at 1.0.0, a stale node_modules/ or lock file from the previous 1.0.0 epoch resolves to the old tarball. The npm cache keeps the old hash; the registry now serves a different tarball for the same version string, producing a 409 Conflict / checksum mismatch.

Lesson: During module refactoring (where file: swaps and version resets to 1.0.0 are routine), always run rm -rf node_modules package-lock.json before npm install:

bash
# From the module's _test/ directory
rm -rf node_modules package-lock.json && npm install && npm test

This clean-install form is mandatory during module refactoring. It is not required for general module development, where a plain npm install is correct and repeated lock-file deletion would be wasteful. See testing-local-modules.md Step 1.

Secondary lesson: If npm install fails with E409 Conflict / checksum mismatch even after a clean install, the registry is temporarily inconsistent (a known GitHub Packages transient bug). Wait 30 to 60 seconds, then re-run rm -rf node_modules package-lock.json && npm install. Do not use --legacy-peer-deps. It masks the real error, writes incorrect lock entries, and creates new failures on the next run. The clean install is the correct fix.

22. ESLint no-unused-vars on intentionally reserved module-scope variables ​

Symptom: npm run lint fails with error 'CONFIG' is assigned a value but never used no-unused-vars (or similar for Lib, React, ThemeContext, etc.) even though the variable is deliberately kept as a reserved injection slot for future use.

Cause: Every module in this codebase declares its full set of dependency injection slots at the top of the file as a standard structural contract - even slots that are not yet wired. This keeps the injection surface explicit, consistent, and visually scannable across every module. ESLint does not know the variable is intentionally reserved; it sees an unread assignment and flags it.

Why not remove it or rename to _CONFIG? Removing breaks the structural contract and creates an inconsistency when the slot gets wired later. Renaming to _CONFIG changes the actual variable name and deviates from the naming convention used across all modules (Lib, CONFIG, React, etc.).

Fix: Add an inline // eslint-disable-line no-unused-vars comment at the end of the reserved declaration. This is the established convention for module-scope let vars across all modules (see docs/languages/js/module-structure.md Singleton Part Shape):

js
// Injected dependencies, set by the loader (module-scope).
let Lib;
let CONFIG; // eslint-disable-line no-unused-vars -- reserved for future knobs
let React;

Never use a file-level /* eslint-disable */ for this - it suppresses the rule for the entire file and hides real bugs. The inline directive is the only correct form.

Lesson: The module-scope injection-slot block (let Lib; let CONFIG; ...) is a deliberate structural pattern, not dead code. When adding a new slot that is not yet used, add // eslint-disable-line no-unused-vars on the same line. Remove the directive when the slot is wired.

Adding a New Entry ​

Whenever a new failure mode is discovered:

  1. Reproduce it once. Confirm the root cause is what you think it is.
  2. Add an entry to the right section above with Symptom, Cause, Fix/Lesson. Keep the numbering continuous within a section.
  3. If the rule is brief enough to live in the AGENTS.md compact summaries (Safe Terminal Patterns, healthcheck rules, etc.), recompile the compact mirror (AGENTS.md is a derived artifact, never edited directly).
  4. Commit the journal entry and the summary together so they never drift.

Doc drift is the slowest bug to find. No exceptions. Every new lesson goes here first, then propagates.

Cross-reference rules:

  • Pitfalls that belong to the architecture domain (docs/foundations/, docs/modules/, docs/server/, docs/testing/ - module migration, refactors) go in ../languages/js/pitfalls-migration.md, not here.
  • The philosophy docs cicd-publishing.md and testing-local-modules.md keep only the positive rules (what to do). Symptoms and root causes always live here.
  • Anchors in this file are stable. AGENTS.md and other cross-references rely on them. Never rename an H2 or H3 after it is published.

Released under the MIT License.

Copyright © 2025-present Superloom