Every lesson

The problem

A review finds a defect, the patch passes, and the next review finds a neighbouring input that reaches the same cause. The work keeps returning because each brief names only the latest example.

Why it happens

An example establishes that a rule is broken. It does not define the full boundary of that rule. A worker asked to make one example pass can reasonably stop before inspecting the other callers, formats or input states that share the mechanism.

How to work through it

1. Reproduce the finding, then write the broken rule in one sentence. Describe the mechanism rather than repeating the failing value.
2. List the relevant input dimensions: for a parser, empty, malformed and valid input are different cases. Search the shared code and its callers, and bound the repair by the behaviour this change promises.
3. Brief the repair around that class, with the original example as one row in a test table. Ask the reviewer to derive a sibling case independently.

The solution

Give the worker a cause, a bounded class and a check for that class. One shared repair may cover several findings; unrelated old defects still need a recorded disposition.

How to check it

For a future fix, confirm that the original case fails before the patch and passes after it, then test another combination that was not named in the report. A sibling failure means the claimed boundary needs another look.

Where this stops

The source reports repeated narrow rounds and supplies the wider briefing method. It does not prove that every bug needs a broad rewrite: a genuinely isolated wrong value can justify a one-off correction.

The problem

The same unchanged source can be selected for two jobs: as a reference in one task and as a rule in another. If both selections receive the same snapshot id while their stored metadata differs, an append-only store has to reject the second one.

Why it happens

The id and the immutability promise describe different objects. Hashing the source bytes alone identifies those bytes. It does not identify a record that also freezes the source role, area or other metadata.

How to work through it

1. List every field that is immutable in the stored snapshot. Separate source facts from the reason a particular task selected it.
2. Either include all immutable record fields in its identity, or keep the snapshot independent of the task and move task-specific selection metadata into the receipt.
3. Test selecting the same source in two roles, then repeat both writes. Both legitimate selections should work, and a replay should keep the same ids and bytes.

The solution

Make the identity boundary match the record boundary. A refusal on changed bytes is useful; a refusal on two supported ways of selecting the same source reveals a mismatch in the design.

How to check it

The source review reports that its original two-role reproduction succeeded after the identity was expanded. For another store, replay that sequence and also confirm that a changed immutable field cannot silently replace the old record.

Where this stops

The review establishes its recorded reproduction and recheck, not every downstream use. Content hashes do not authenticate the writer or prove the source is true.

The problem

A test puts a fake aws command on PATH and expects all calls to reach it. That expectation fails if the application also accepts an explicit binary path from its environment. The fake can be present and executable while never being called.

Why it happens

The tests inherited AWS_CLI_BIN from the session launcher. That explicit binary selection took precedence over the fake command on PATH. Plain shells and CI did not set the variable, so their successful runs missed this route.

How to work through it

1. Check how the command chooses its executable, including explicit environment overrides.
2. Remove the inherited override in the default test fixture. Let tests that need it set their own value.
3. Start a nested test run with the variable deliberately set and confirm the fixture still removes it.

The solution

Make the test environment explicit at the suite boundary. A fake on PATH only covers callers that use PATH, so test the launch environment that can override it too. A harmless worked example can use two toy executables that print different sentinel words: set the override to one and PATH to the other, then check which word appears.

How to check it

The source reports a nested-test regression that failed without the fixture and passing CI after the fix. The toy sentinel example is a suggested reproduction, not a run performed for this post.

Where this stops

This closes the recorded AWS_CLI_BIN leak. Clearing one variable does not make live cloud access impossible; other credential or executable-selection routes need their own isolation.

The problem

An operator runs a command, gets a result and still has to search the documentation to work out the next action. Knowing the exit code does not explain how to recover or continue.

Why it happens

The commands handled their own work, but the route between them lived elsewhere. Advice printed in individual scripts could also drift away from the targets and arguments that actually existed.

How to work through it

1. Write down the command routes: what can follow success, what can recover a failure and where the flow finishes.
2. Print a short Next block at command exit. Say what each suggested command does.
3. Check the routes in CI: targets exist, nodes are reachable, paths can finish and suggested arguments are read by the next target.

The solution

Keep command navigation in a checked graph and let the command print the relevant part. Retry advice should carry values the operator typed, while asking for confirmation again when the action needs it.

How to check it

The work began on October 22 with a spec, plan and graph checker. Final review fixes merged on October 23, including cancellation handling and argument-carry checks.

Where this stops

The final changes were checked on the host, but the container walk was not rerun for that final PR. A checked command graph does not prove every live operating environment.

The problem

The full verification run failed in a clean worktree: a separate checkout containing the committed files, without my local extras. The familiar checkout had looked healthy. The new changes seemed the obvious cause, but the same failure appeared on the base revision.

Why it happens

An untracked symlink, a file pointing to another location, made the test take an easier path in the main checkout. Without that local shortcut, the audit and update checks followed the failing path. A second failure came from inherited make settings changing the command output.

How to work through it

1. Run the failing test in clean checkouts of the base revision and each changed revision.
2. Compare the tracked and untracked state. Add back one suspected file and check whether it changes the result.
3. Clear inherited runner settings when testing output, then rerun through the original failing route.

The solution

Put the state a test needs in its fixture. Keep an independent clean-checkout run in the verification flow so my working directory cannot quietly supply missing setup.

How to check it

The investigation reproduced the failure on clean origin. Adding one untracked file made that test pass. Clearing inherited make settings also made the output test pass.

Where this stops

The record opened a fix slice for the fixture, the underlying bug and the output test. It does not establish that the complete fix had landed that day.

The problem

An update is supposed to stop when the application is busy. A passing test tells me the check works in the environment the test created. I still need to know what the real host does, including whether a refused update has already changed anything.

Why it happens

Fakes leave out parts of the installed system. The real pilot found an old loader and a rotated-log naming case that the existing checks had missed.

How to work through it

1. Keep the application open and record the installed state before starting the update.
2. Run the update in a supervised terminal. Capture its message and exit code, then compare the state afterwards.
3. Separate a product defect from a host configuration fact. Reproduce a defect before changing code.

The solution

Make refusal a behaviour I can demonstrate: the command explains why it stopped, returns the expected code and leaves the installation alone. Keep a short evidence record so the next person can repeat the same check.

How to check it

The pilot reported BUSY, returned exit 4 and changed nothing while the application was open. Host report values were also compared with operating-system tools.

Where this stops

This was a supervised pilot. Dashboard and fleet-path steps were not run, so it did not prove the whole deployment flow.

The problem

A repository can pass CI while its documented setup fails for a new developer. This happens when the pipeline installs dependencies through a separate path that nobody maintains as the public bootstrap.

Why it happens

Two setup scripts create two descriptions of the required environment. The familiar CI path stays exercised while the developer path quietly misses a tool, configuration step or version constraint.

How to work through it

1. Make the documented setup command the entry point for both developers and fresh CI workers. Let environment-specific choices be explicit parameters rather than a second handwritten installation sequence.
2. Run the normal checks after that setup on a clean workspace. Cache downloads where useful, but keep a check that does not depend on a pre-existing virtual environment or undeclared host package.

The solution

Use CI to rehearse the path a new contributor follows. Fixing one bootstrap then repairs both environments and makes the documentation easier to trust.

How to check it

Try the documented command in a clean supported environment and inspect failures for missing prerequisites. A successful check should require only the prerequisites the documentation actually names.

Where this stops

Hosted CI covers its own operating system and network conditions. It cannot prove every developer platform works, and caches or preinstalled tools can still mask a missing setup step.

The problem

A design document can describe every component and still fail to explain how the system handles its required features. The missing part appears when a second user, larger input or unnoticed failure reaches the design.

Why it happens

A component list shows what exists. A defence question follows one feature through load, ownership and failure paths, forcing the document to explain how those pieces work together.

How to work through it

1. List each feature the design must carry and ask a few hard questions on its behalf. Include scale, concurrency, a broken dependency and who notices when the result is wrong.
2. Answer from the design as written, linking the section that provides the mechanism. Mark an unanswered question as a gap, then change the design or name the scope cut before planning the build.

The solution

Use the defence record to turn broad confidence into specific claims that can be challenged. Repeat the exercise when a new feature changes what the design must support.

How to check it

Give the record to a reader who did not write the document. They should be able to follow the mechanism and find a concrete answer without relying on the author's spoken explanation.

Where this stops

The quality of the questions still needs judgement. A complete table is useful coverage evidence, but it cannot prove that all important failure modes have been imagined.

The problem

Finishing all requirements before exploring any architecture can hide an impossible or needlessly expensive demand. Designing first can hide the opposite problem: a neat system with no answer to an actual need.

Why it happens

Requirements constrain the design, while design choices expose missing detail and derived obligations. Working in only one direction delays that feedback until changes have more dependencies.

How to work through it

1. Write a small set of goals and a rough architecture together. Allocate each requirement across the parts that contribute to it, identifying the budget or interface each part must meet.
2. Use what the allocation reveals to refine the requirement or change the design. Record the trade-off, then repeat the loop at the next useful level of detail.

The solution

Keep intent and mechanism in conversation. For an illustrative latency target, divide the budget across processing and transport; if one part cannot meet its share, revisit the allocation or target explicitly.

How to check it

Check that each important requirement has a design path and that each major design block has a reason. An infeasible budget should appear as an open decision, not a quietly omitted number.

Where this stops

A repeated, well-understood design may need less exploration. The source's research model was not independently checked against its cited paper, so this lesson presents the working method rather than a research finding.

The problem

A command that stops without a clear status forces its caller to guess whether work succeeded. A message alone is also weak evidence if automation can only see an exit code of zero.

Why it happens

Each branch in control flow ends in a result, whether success, refusal or failure. If those endings are implicit, an unhandled path can look like an ordinary completion.

How to work through it

1. List the terminal outcomes of the operation and assign documented statuses. Keep successful empty results distinct from missing input, invalid configuration and failed work.
2. Route each terminal branch through the appropriate return or exit path, with a short message that explains the reason. Preserve the status through wrappers and pipelines.

The solution

Make the program's outcome usable by both a person and another program. The status selects the caller's next action; the message explains what the caller can do about it.

How to check it

Exercise one isolated input for each outcome. Check the structured result or exit status, and check wording only where the message itself is a public contract.

Where this stops

A library usually returns a value or raises a typed exception rather than exiting the process. The same principle applies, but the boundary decides which mechanism is appropriate.

The problem

A slow regression is not automatically a caching problem. If most time sits inside one long tool invocation, a new build scheduler cannot make that invocation finish faster.

Why it happens

Incremental build tools avoid actions whose declared inputs have not changed. They save repeated work, while hung jobs, slow first runs and hidden dependencies need different corrections.

How to work through it

1. Break elapsed time into useful execution, waiting, failed work and repeated unchanged work. Compare input revisions between repeated runs rather than treating every repeated job name as a cache opportunity.
2. Trial a small representative flow with declared source, tool and configuration inputs. Measure cold and warm runs separately and inspect outputs for deterministic behaviour.

The solution

Choose the intervention from the measured loss. Time limits and wrapper fixes may matter more than caching when jobs fail to terminate or hold allocations after useful work ends.

How to check it

Compare total elapsed and compute time before and after the trial. A credible cache benefit comes from avoided unchanged actions, not a claim that the longest action itself became faster.

Where this stops

Undeclared absolute-path reads, tool versions and nondeterministic outputs can make caching unsafe. The historical investigation remained in progress; it did not establish that a new build system was adopted.

The problem

An incremental flow may rebuild almost everything after a tiny edit to a common header. A sophisticated scheduler still sees a legitimate dependency change across the whole project.

Why it happens

When many source files include one definitions hub, they all depend on its bytes. A change to an unrelated definition invalidates every consumer because the dependency is tracked at file granularity.

How to work through it

1. Inspect the include graph and measure the actions invalidated by a representative header edit. Separate genuinely shared definitions from values used by only one subsystem.
2. Split definitions along real ownership boundaries and update includes to depend on the smallest relevant header. Keep a shared layer only for definitions that truly belong to every consumer.

The solution

Reduce unnecessary dependency edges before expecting a cache to skip much work. The scheduler can reuse only the independence that the source structure actually expresses.

How to check it

Make a local-only definition change in an illustrative trial and inspect the rebuilt action set. Unrelated subsystems should remain reusable, while real consumers still rebuild.

Where this stops

Splitting a header can create awkward include cycles or duplicate definitions if ownership is unclear. Shared interface changes should still invalidate broadly, and hidden inputs remain a separate cache risk.

The problem

A page test can prove that selecting a filter sends the intended request and still miss a server that returns the wrong rows. The mocked answer only repeats what the test author expected.

Why it happens

The browser and server are two halves of one interaction. An offline fixture can check the question the browser asks; it cannot establish how the real query behaves against real data.

How to work through it

1. Keep fast offline tests for control state and outgoing request parameters. Cover empty selections, combined filters and categories that must not disappear.
2. Replay representative requests against a controlled real API and database, using a copy or safe fixture environment. Check returned rows and counts against independently known cases.

The solution

Use both layers to locate errors. A bad outgoing parameter points to the page; a correct request with a bad result points to the server or data interpretation.

How to check it

Choose a case with included and excluded rows. The live-path check should return the intended set and matching total, not merely a successful status code.

Where this stops

Live data changes and can contain sensitive information. Record the dataset boundary, avoid modifying live stores, and do not call an offline replay proof of current server behaviour.

The problem

A mutation check can appear to tolerate a deliberate bug because the test imported old bytecode. That turns a cache problem into a false conclusion about test quality.

Why it happens

Python may validate cached bytecode using source timestamps and size. A rapid same-size edit can therefore be hard to distinguish from the source that produced the cache.

How to work through it

1. Run mutation probes in an isolated copy and remove existing bytecode caches before the probe, or direct caches to a fresh location. Confirm that the test imports the mutated source file.
2. Use python -B to avoid writing more bytecode during the probe, then restore the source and check the working tree. Keep mutation debris out of later tests.

The solution

Treat a surviving mutant as something to investigate: first establish which code executed, then judge whether the assertion detected the changed behaviour.

How to check it

Make a mutation with a clear expected failure and inspect the imported module path. After the clean-cache run, the relevant test should fail for the planted defect.

Where this stops

The short lesson preserves the historical -B advice, but -B suppresses writes; it does not guarantee existing bytecode will not be read. Import paths and already-loaded modules are separate causes to rule out.

The problem

A local environment link can slip into a commit even when .gitignore contains .venv/. The pattern looks right because the link points to a directory, but Git sees a different kind of entry.

Why it happens

A trailing slash restricts an ignore pattern to directories. A symbolic link named .venv is a link in the working tree, so a directory-only rule does not cover it.

How to work through it

1. Use .venv without the trailing slash when both a directory and a symlink should be ignored. Keep the pattern scoped to the intended repository location.
2. Check the index before adding everything. If the link is already tracked, remove it from the index deliberately; changing .gitignore alone does not untrack an existing entry.

The solution

Write ignore rules for the entry Git records, rather than the type of its target. This keeps local environment layout out of the repository across different developer setups.

How to check it

Use git check-ignore -v .venv to identify the matching rule, then inspect the staged names. A local symlink should remain untracked after the normal add command.

Where this stops

Ignore patterns are not protection for secrets already committed. A tracked environment entry needs separate cleanup, and other ignore files can affect which pattern wins.

The problem

A test that sleeps before checking the next stage can pass on a quiet machine and fail under load. Increasing the sleep makes the suite slower without proving that the dependency is ready.

Why it happens

A wall-clock delay measures elapsed time, while the next step depends on an event. Process scheduling, I/O and startup work can all change the gap between those two things.

How to work through it

1. Name the condition that makes the next step valid: a readiness response, completed task, emitted event or fully written artifact. Wait for that condition rather than a guessed delay.
2. Put a finite deadline around the wait and report the last observed state on timeout. If polling is necessary, bound it and preserve the underlying failure message.

The solution

Chain the test on completion evidence so fast machines move on promptly and slow machines get the time they actually need.

How to check it

Exercise the test with an intentionally delayed dependency and with one that never becomes ready. It should proceed after real readiness and fail clearly at the deadline when readiness never arrives.

Where this stops

Some tests deliberately study timing, and simulator time has different semantics. Even then, distinguish a required duration from a guessed pause used to hide a missing readiness signal.

The problem

A network probe can hang long enough to block the rest of a monitoring cycle.

Why it happens

urllib can wait without a useful application timeout unless one is supplied. HTTP error responses also take an exception path, so code that expects only a returned status misses part of normal probe behavior.

How to work through it

1. Supply an explicit timeout for the request.
2. Handle HTTPError as a received HTTP outcome and separate it from transport failure.
3. Bound the overall probe workflow and report which target failed.

The solution

Illustrative: a 503 response is different from a DNS failure or a socket that never responds. A useful probe records those states instead of crashing or waiting indefinitely.

How to check it

Use disposable endpoints for success, HTTP failure and delayed response. Expect a classified result within the intended allowance, including a non-success status when the server answers with an error.

Where this stops

Some redirects are followed automatically, so the original non-2xx shorthand is too broad. Catch HTTPError for responses that reach the error handler. The timeout is not necessarily a strict total wall-clock deadline across DNS, redirects and reads; a hard deadline needs a surrounding execution and cancellation strategy.

The problem

A design becomes expensive to change when its purpose can only be explained by naming today's libraries.

Why it happens

Tight coupling lets one component's interface and storage assumptions spread through the pipeline. Replacing that component then changes several unrelated parts of the system.

How to work through it

1. Describe the system in terms of roles and information flow.
2. Define contracts at those boundaries.
3. Keep component-specific translation inside adapters and check the same contract when swapping an implementation.

The solution

Illustrative: 'retrieve evidence, rank it, compose an answer' can survive a search-backend change if each stage has an explicit input and output. Backend-specific response objects crossing every boundary make that swap harder.

How to check it

Replace one component with a small alternative in a disposable example. The surrounding stages should continue to satisfy the agreed contract with limited local changes.

Where this stops

A clean diagram does not prove that a migration is cheap. Data format changes, performance assumptions and operational dependencies can still require careful transition work.

The problem

An asynchronous API can stop answering health requests while one handler waits for a synchronous subprocess.

Why it happens

async def does not make the work inside it nonblocking. A blocking function occupies the event-loop thread, preventing other coroutines on that loop from advancing.

How to work through it

1. Identify synchronous subprocess, database and file calls inside async handlers.
2. Use an async-native operation or move suitable blocking work to a bounded worker thread.
3. Keep timeouts and cancellation behavior explicit.

The solution

Illustrative: await asyncio.to_thread(blocking_probe) lets the event loop serve another request while the probe waits. An async subprocess API can provide more direct process lifecycle control.

How to check it

Run a deliberately slow operation beside a simple health request in a disposable service. The health request should remain responsive while the slow task is still in progress.

Where this stops

Thread offloading does not stop the underlying function automatically when its awaiting request is cancelled. CPU-heavy work, thread safety and pool exhaustion need their own design.

The problem

A clean exit can appear as missing data when a fallback expression treats zero like absence.

Why it happens

Python's or operator chooses its fallback for any falsy value. Exit code 0 and elapsed time 0 are valid measurements, so truthiness is the wrong missing-value test.

How to work through it

1. Define which sentinel actually means missing.
2. Use an explicit None check before choosing the fallback.
3. Carry zero unchanged through the formatter, filter and export.

The solution

Illustrative: value if value is not None else 'not recorded' displays an exit code of 0 as zero while still naming an absent record. The distinction changes how a reader interprets the run.

How to check it

Exercise None, 0 and a nonzero value through the real rendering path. Expect all three states to remain distinguishable, including a valid zero threshold in filters.

Where this stops

Empty strings and false booleans may also be valid in other fields. Choose the missing-value rule from the field's contract rather than replacing every or expression mechanically.

The problem

A diagnosis can overstate its evidence when ruling out one suspect becomes a claim to know the real cause.

Why it happens

Measurements support particular exclusions within a time window. They do not automatically establish why the whole host failed, especially when the decisive display or process state was inaccessible.

How to work through it

1. Write down the observed symptom and measurement window.
2. List what each observation can exclude.
3. Keep the remaining explanation labelled as a hypothesis and name the evidence needed to test it.

The solution

Illustrative: low service resource use during the observed window can weaken a resource-exhaustion theory, while an unrelated GUI-lock theory remains unproven.

How to check it

Ask whether another reader can trace every exclusion to an observation and every proposed cause to its test. The report should distinguish measured facts from plausible explanations.

Where this stops

Negative evidence is bounded by sampling and access. Do not turn 'not observed here' into a claim that a service could never contribute under different conditions.

The problem

A filter control can appear to work because its request returns 200 while the server ignores the requested restriction.

Why it happens

An undeclared query parameter can be discarded rather than rejected. The route exists and answers successfully, but its result describes a broader population than the interface promises.

How to work through it

1. Compare each client filter with the handler's declared parameters.
2. Use a dataset where applying the filter must change the result.
3. Check both returned rows and any summary totals against that same population.

The solution

Illustrative: a days=30 request should exclude older fixture records. A successful response containing those records reveals that route existence and filter semantics are separate checks.

How to check it

Send the same request with two deliberately different filter values. Expect the population to change as specified, and verify that companion tables use the same scope.

Where this stops

Some APIs intentionally tolerate unknown parameters. That convention still requires explicit tests for supported filters; a status-only smoke check cannot establish their behavior.

The problem

A file move can make useful history harder to follow when the same patch also rewrites its contents.

Why it happens

Git detects renames from content similarity rather than storing a permanent move record. A large simultaneous edit can make the old and new files look unrelated.

How to work through it

1. Write a move map before reorganizing paths.
2. Commit the path-only moves with unchanged contents.
3. Update imports, scripts and documentation in the next change and check their references.

The solution

Illustrative: move a script first, then change how it locates its configuration. The intermediate commit gives history tools an unchanged file to match while keeping the functional edit easy to review.

How to check it

Run git log --follow on a moved file and check that it reaches the old path. Then exercise path resolution so a clean historical move does not conceal broken runtime wiring.

Where this stops

A pure move improves rename detection but does not guarantee every history tool follows every migration. Cross-repository moves require their own history and provenance plan.

The problem

A formatter can obscure the history of a feature even when its changes were carefully separated during development.

Why it happens

Squashing combines the mechanical sweep and functional changes into one final commit. Later blame points to that combined change instead of leaving the reformat independently identifiable.

How to work through it

1. Commit formatting separately from behavior changes.
2. Choose a merge method that preserves the useful commit boundary.
3. Inspect the resulting history before relying on it for future debugging.

The solution

Illustrative: if a patch changes one condition and reformats a hundred files, preserve the formatting commit so a reader can identify and skip that mechanical change while tracing the condition.

How to check it

Check the final branch history and run blame on representative reformatted lines. The mechanical change should still have its own commit identity after integration.

Where this stops

A separate commit does not automatically make blame show the older semantic change; readers may still need to skip it. Repository merge rules can constrain the available strategy.

The problem

A configuration can match current documentation while the installed binary silently ignores the setting that was meant to enforce it.

Why it happens

Policy syntax and supported keys change between versions. Valid JSON or a successful startup does not establish that an older client understands a newer setting form.

How to work through it

1. Record the exact client version and consult the corresponding setting contract, including the accepted value type.
2. Test the intended behaviour on that pinned binary before rolling out the policy, then repeat the compatibility check when either side changes.

The solution

For a setting that accepts an array in the deployed version, do not substitute an object because a newer reference shows one. Bind the policy release to its supported client versions.

How to check it

Check effective configuration where available and exercise a harmless action the setting should prevent. Expect the pinned client to enforce the decision, not merely accept the file without an error.

Where this stops

A version pin is reproducibility, not proof of security or continued support. Keep an upgrade path and test older clients explicitly if they remain permitted in the fleet.

The problem

A language choice can become an argument about preference before anyone separates the AI ecosystem needs from the infrastructure’s performance constraints.

Why it happens

Python has strong library and integration support for AI work. A systems component may have different needs around memory, speed and deployment, so one language need not serve every layer.

How to work through it

1. Start with the libraries and integration paths the feature actually requires, using Python where that ecosystem saves substantial work.
2. Measure the systems bottleneck before considering a bounded Rust component with an explicit interface back to the rest of the application.

The solution

Keep the choice reversible. A narrow service or library boundary is easier to compare and replace than a rewrite justified only by a language’s reputation.

How to check it

Compare representative workload latency, memory use, build effort and operational complexity. Include the cost of crossing the language boundary rather than timing only the inner loop.

Where this stops

This is a stated direction of interest, not a reported benchmark or a decision to rewrite a project. A mature Python implementation may remain the best choice when it already meets the requirements.

The problem

A team can ask every job author to include the same metadata and still receive incomplete records, because each script has to remember the rule independently.

Why it happens

The requirement is shared, but enforcement sits below the common boundary. Every new caller becomes another opportunity to omit or invent the fields.

How to work through it

1. Identify the wrapper or submission boundary that every supported job already crosses.
2. Have that layer derive common metadata, validate required caller inputs and reject incomplete submissions before the lower-level work begins.

The solution

For a job record, the wrapper can attach submission identity and timing consistently while the job supplies only task-specific fields. Define which fields callers may override instead of silently accepting conflicting copies.

How to check it

Submit a minimal job through the wrapper and expect a complete standard record. Test a missing required input and an attempted protected-field override; both should produce a clear decision.

Where this stops

This works only for paths that cross the boundary. Direct execution needs to be disallowed or explicitly outside the contract, and centrally added metadata still needs a trustworthy source.

The problem

A metadata value that looks like an identifier can arrive in code as a boolean or integer, breaking comparisons and joins that expect text.

Why it happens

YAML loaders infer scalar types. Under YAML 1.1-style rules, unquoted yes, no, on and off can become booleans; an unquoted numeric id can become an integer.

How to work through it

1. Quote values whose contract is text, such as `id: "42"` and `status: "no"`.
2. After loading, validate the resulting types against the schema rather than assuming the source’s appearance establishes its meaning.

The solution

Treat quoting as part of the serialized contract. A textual id must remain the same type through parsing, validation and export, even if its characters happen to be digits.

How to check it

Load a small fixture with the actual application loader and inspect both value and type. Expect the quoted examples to remain strings, and reject an inferred integer where the schema requires a string.

Where this stops

Boolean inference differs across YAML versions and loaders. Test the one you ship; quoting text is useful even when a newer loader no longer treats “on” as a boolean.

The problem

A module can be present in the repository yet absent from the corpus an agent uses to understand it. Human reviewers and agents then see different projects.

Why it happens

A written convention helps only if readers can find it and producers follow the machine-readable contract. Missing or malformed module headers can silently remove useful context.

How to work through it

1. Define the exact header and exported-symbol contract that the corpus reader accepts, alongside a short human style guide.
2. Gate the checkable contract and compare corpus coverage with the source scope, while leaving clarity and design judgment to review.

The solution

For a public module, require its supported summary field rather than hoping the next agent infers its role from a filename. Make invalid headers visible instead of silently skipping them.

How to check it

Use a missing-header example and a malformed summary example. Expect the gate to reject them and corpus checks to expose lost modules; separately ask a reader whether the accepted summary makes sense.

Where this stops

The original phrase “reading guaranteed” overstates what standards can do. A gate can establish structure and coverage, not guarantee that a person or model reads or understands it.

The problem

A Python source scanner can reject a valid-looking file on its first character, then lose the file entirely if that parse error is swallowed.

Why it happens

A UTF-8 byte-order mark decodes as an extra leading character with the plain utf-8 codec. Passing that text directly to ast.parse can produce a SyntaxError.

How to work through it

1. When the accepted input is UTF-8 text with an optional leading BOM, read it with encoding="utf-8-sig".
2. Pass the resulting text to the parser and report any remaining parse error with the filename rather than treating it as an empty result.

The solution

For example, `text = path.read_text(encoding="utf-8-sig")` removes an optional leading BOM before `ast.parse(text)`. Apply the same input rule to every reader of that file kind.

How to check it

Prepare equivalent files with and without the BOM and expect the same parsed structure. Include a genuinely invalid file and expect an explicit error, not a missing corpus entry.

Where this stops

This codec handles an optional UTF-8 BOM; it does not detect every encoding or fix invalid source. Use it where the input contract is UTF-8, rather than guessing an arbitrary document’s encoding.

The problem

A repository can contain a format target while accepting every unformatted change. Its presence looks like enforcement without changing what passes.

Why it happens

The command is disconnected from the path that decides whether work is accepted. A tool nobody has to run is an available action, not a gate.

How to work through it

1. Add the formatter’s non-writing check mode to the normal verify target and the CI path used for changes.
2. Pin the tool version, then introduce a small formatting violation on a disposable branch to check that the acceptance path rejects it.

The solution

Keep automatic rewriting as a separate convenience command. For example, use a format-check dependency in verify so CI reports the violation rather than silently changing the submitted files.

How to check it

The deliberate violation should make the normal gate exit non-zero. After formatting it, the same gate should pass that check and continue to the other required checks.

Where this stops

A formatting gate establishes layout consistency, not readable design or correct behaviour. Large mechanical reformats also deserve a separate change so functional edits remain reviewable.

The problem

A build cache can return an old result after a header changes, even though the top-level source file and command are unchanged.

Why it happens

An include-directory option names where a tool looks, but the tool also reads files found there. If those bytes are absent from the cache key, different computations can share one identity.

How to work through it

1. Enumerate the complete input set, including resolved headers, configuration and other files the tool reads indirectly.
2. Build the key from their content and the relevant invocation, then version the key format so incomplete old entries become misses.

The solution

For a source that includes a header, changing only that header must change the computation’s key. Treat the include path as a search rule, not a substitute for the files selected through it.

How to check it

Populate a disposable cache, change an included header and request the result again. Expect a miss and recomputation. An unchanged input set should still be eligible for a hit.

Where this stops

A sound key depends on discovering every meaningful input. Hidden environment or tool-version dependencies can create the same stale-result problem even after headers are included.

The problem

A design can name its components and still leave nobody able to tell whether the result meets the goal.

Why it happens

Without a route from output back to a decision, an observed error changes nothing. The design describes production of an output but not how it learns that the output is wrong.

How to work through it

1. State the end goal without choosing tools, then name the observable result that would demonstrate it.
2. Draw the path from that observation back to the input or decision it can change, including who acts and when the loop stops.

The solution

For a generated report, identify the source totals, the comparison with those totals and the correction when they disagree. That loop is meaningful before choosing a report engine.

How to check it

Walk one plausible bad output through the drawing. Someone should be able to say how it is detected, what changes next and what result ends the correction.

Where this stops

A feedback loop does not make a goal useful or a measurement accurate. Those choices remain design questions, and some checks need a human rather than automatic retries.

The problem

A caller can handle an explicit command failure correctly while still accepting a command that reports success without doing its work.

Why it happens

An exit status describes what the process reported. It is not an independent observation that a file, service or database reached the requested state.

How to work through it

1. Replace the dependency with a stub that exits non-zero and check that the caller surfaces the error.
2. Use a second stub that exits zero but changes nothing. Ask the caller to perform the same operation and inspect its next decision.

The solution

Check the postcondition that matters. For a file writer, require the expected new content before reporting completion, rather than trusting a successful process exit.

How to check it

The first stub should trigger failure handling. The lying stub should fail the state check and must not produce a success report.

Where this stops

Choose an observable postcondition within the caller’s authority. Eventual consistency may require bounded retries, so an immediate read is not always the right oracle.

The problem

A passing test can agree with the implementation and still disagree with the requirement. That makes a green suite a weak answer to a disputed interpretation.

Why it happens

The expected result may have been copied from the same assumption that shaped the code. Reusing its helper in the test can make the same fault cancel on both sides.

How to work through it

1. Start with the requirement and write one small input whose correct output can be worked out without the implementation.
2. Compare that result with the test oracle, then deliberately break the disputed rule and see whether the test notices.

The solution

Give the test an independent basis for its answer. For a boundary rule, include values just below, at and above the boundary.

How to check it

The deliberately wrong implementation should fail for the intended reason. A passing mutation means the test has not established that rule.

Where this stops

Independence reduces shared assumptions; it does not prove the requirement itself is right. Keep ambiguous wording open for review.

The problem

A green suite can mean that a check passed, or that it never ran. A missing optional dependency can quietly turn a meaningful test into a permanent skip.

Why it happens

The harness may interpret an unavailable prerequisite as an expected condition. Unless something records the intended coverage, the same missing dependency can hide a guard on every run.

How to work through it

1. List which tests are required for the job and which are genuinely optional. Declare the required dependencies in the environment that runs those tests.
2. Make the result show the number of executed and skipped checks, with a concrete reason for each skip. Treat a missing required check as a failure.
3. Remove a required dependency in an isolated run and confirm that the job refuses to claim success. Audit similar skip patterns rather than fixing one line and assuming the class is covered.

The solution

Make absence visible at the level that decides whether the work is complete. A skip can be a useful result when it is explicit and does not silently remove promised coverage.

How to check it

The normal environment should execute the required checks. The deliberately incomplete environment should produce a clear missing-prerequisite failure, not an indistinguishable green verdict.

Where this stops

Some integrations cannot run on every host. Keep those intentional limits documented and visible instead of making every optional environment a hard failure.

The problem

A Make target can decide to skip work on its first line and still run the next line. That is surprising if the recipe is read as one ordinary shell script.

Why it happens

By default, Make starts a separate shell for each recipe line. An exit ends the shell running that line; a successful exit tells Make it may continue with the next line.

How to work through it

1. Read the recipe as separate shell invocations unless it explicitly uses a single-shell mode. Mark the lines that are supposed to share a condition or variable.
2. Put the condition and the work it controls in one shell block, or move the procedure into a script with an explicit exit contract.
3. Exercise both branches. The skip path should show that the protected command never ran, not merely that the target returned success.

The solution

Keep the decision inside one shell lifetime. A single recipe line such as `@if test "$(RUN)" = yes; then touch marker; fi` makes the marker conditional. A later independent recipe line would not share that guard.

How to check it

Use a harmless marker command in the protected position. On the skip path, expect no marker; on the normal path, expect exactly the intended work and exit status.

Where this stops

Make can be configured to run an entire recipe in one shell. Check the actual configuration before applying the default-line model.

The problem

A Python file can compile even though importing it fails. A removed function name or a missing dependency may only become visible when the module-level imports actually execute.

Why it happens

Bytecode compilation checks syntax. It does not establish that a dependency exists, that an imported name is still exported or that module initialization succeeds in the environment the product uses.

How to work through it

1. List the delivered non-test modules and the interpreter and import roots they require. Make that list explicit rather than relying on whichever modules tests happen to touch.
2. Import the declared modules in a fresh interpreter with the project environment. Capture failures per module so one early exception does not hide the rest.
3. Keep compilation and runtime tests where they add value, but make a successful import part of the build claim for this kind of Python project.

The solution

Define build in terms of the runtime contract. The useful question is whether the delivered module can be loaded where it is meant to run.

How to check it

An illustrative regression removes an exported name that another module imports. Syntax checks can stay green, while the import gate must fail and name the affected module.

Where this stops

Importing modules can execute code. Run the gate in an isolated test environment, and keep external effects out of module initialization.

The problem

A script can run locally after chmod +x and still arrive non-executable in a fresh clone.

Why it happens

Git records an executable bit in its index. When core.fileMode is false, ordinary working-tree mode changes are ignored, so chmod alone may never update the committed mode.

How to work through it

1. Inspect both the local permissions and the mode recorded by Git.
2. Use a supported index mode update when the filesystem cannot reliably track it.
3. Review and commit the mode change along with the script.

The solution

For example, an executable script should normally appear with mode 100755 in the index rather than 100644. The file’s local ls output cannot establish what another clone will receive.

How to check it

Inspect the staged mode change and launch the script from a fresh disposable clone on a compatible filesystem. That checks the committed artifact rather than the author’s workspace.

Where this stops

Disabling file-mode detection can be legitimate on some filesystems. The fix is to record the intended index mode deliberately, not to assume every environment must use the same local setting.

The problem

A template update can exit successfully while leaving important project changes in reject files instead of the resulting source.

Why it happens

With some conflict modes, tool success means the update operation completed, not that every conflict was resolved in favor of the intended project behavior. The template side may remain active.

How to work through it

1. Rehearse the update on a disposable copy with a clean recorded baseline.
2. Inspect the diff, reject files and conflict markers after the command.
3. Resolve the intended behavior and run the project’s checks before accepting the update.

The solution

For example, a local authentication customization moved into a .rej file is no longer active just because the update returned zero. Compare the resulting code and contract, not only the command status.

How to check it

Use a fixture with an intentional project/template conflict and confirm the review procedure discovers it. Exercise the preserved project behavior after resolution.

Where this stops

Conflict handling varies by tool version and options. Follow the actual migration procedure; restarting an interrupted update incorrectly can skip required migrations even when the files look plausible.

The problem

A structural RTL mistake found during subsystem integration costs more to isolate because many blocks and assumptions are already interacting.

Why it happens

Lint and static analysis can inspect local rules before a long simulation or full integration run. Earlier feedback keeps the investigation close to the change that introduced the problem.

How to work through it

1. Choose relevant lint and static-formal goals for the block.
2. Run them through a repeatable script during local development.
3. Review violations and govern waivers before accepting the block into the next integration stage.

The solution

For example, a width mismatch can be caught at a module interface before it turns into unexpected data several blocks downstream. The earlier report gives a smaller search area than a late behavioral symptom.

How to check it

Use a known-bad fixture to confirm the normal flow runs the intended rule. Check tool logs and loaded configuration, not just the wrapper’s exit status.

Where this stops

Early analysis complements simulation and later implementation checks. It does not prove that the requirements are right or that every functional path works, and no new estimate of days saved is claimed.

The problem

Two designs can solve the same immediate task while leaving very different amounts of explanation for the next maintainer.

Why it happens

A clean block diagram often exposes clear responsibilities and a small number of meaningful interfaces. A design that needs many crossing arrows may be spreading state or policy across too many owners.

How to work through it

1. Draw both candidate designs at the same level of detail.
2. Label the data, ownership and failure behavior at each boundary.
3. Prefer the simpler structure when it meets the real requirements and keeps changes local.

The solution

For example, a validation stage before storage is easier to reason about than several storage callers each applying a different part of the same rule. The diagram makes the duplicated ownership visible.

How to check it

Walk a normal request, a failure and a likely future change through the chosen diagram. Check whether a reader can locate the responsible module in the code.

Where this stops

A tidy picture can hide complexity by omitting details. Treat ease of drawing as a design test, not proof that the system is correct or that the fewest boxes always win.

The problem

A terminal footer can display a price correctly to a person while breaking code that expects the amount to be one intact string.

Why it happens

Rich’s automatic highlighter can insert ANSI formatting around numeric portions of text. Those invisible escape sequences divide values such as $0.04 in the raw output.

How to work through it

1. Decide whether the output is for display, machine parsing or both.
2. Disable automatic highlighting for text whose byte sequence must stay intact.
3. Provide a separate plain or structured output mode for machine consumers.

The solution

For example, a cost footer can use highlight=False while retaining deliberate styling around the whole line. That prevents a number-specific highlighter from modifying the middle of the amount.

How to check it

Capture raw output with the actual terminal settings and inspect the complete amount. Test colored and noncolored paths, because a nonterminal capture may omit ANSI sequences and miss the bug.

Where this stops

Other styling can still add escapes. Turning off highlighting is a targeted fix, while a supported JSON or plain-text interface gives downstream parsers a clearer contract.

The problem

A tracker can say that every requirement is tested while the repository contains no check for one of them.

Why it happens

A manually edited status is testimony. A generated trace can compare the actual requirement inventory with test markers and report the exact missing or undefined links.

How to work through it

1. Give requirements stable identifiers and identify their acceptance criteria.
2. Mark the test functions that check those criteria.
3. Generate the matrix from the current repository and fail on missing or unknown links.

The solution

For example, a requirement with three numbered criteria needs evidence for those criteria, not merely one marker naming its parent. Function-level markers also stop a removed test from remaining apparently covered by a module-wide label.

How to check it

Delete or rename a marked fixture test and confirm the trace becomes incomplete. Check that an empty requirement inventory cannot silently produce a perfect result.

Where this stops

A marker proves a claimed link exists; it does not prove the test is relevant or correct. Review the assertion behind the marker and keep the generated matrix tied to the exact revision.

The problem

A Make recipe can pass an unintended identity argument even when no developer explicitly configured that variable.

Why it happens

GNU Make supplies built-in variables. AS commonly names the assembler, so a recipe using $(AS) as a convenient “act as” parameter can receive as instead of an empty value.

How to work through it

1. Choose a task-specific variable name such as WHO for the requested identity.
2. Inspect the variable’s origin and the expanded recipe while debugging.
3. Validate the supplied identity before invoking the downstream command.

The solution

For example, --as $(AS) can become --as as. Renaming the configuration avoids a collision with Make’s toolchain namespace and makes the argument’s meaning clearer to callers.

How to check it

Use a dry-run recipe inspection with no override, then with an explicit WHO value. Confirm the default behavior and the expanded argument list before exercising a real privileged operation.

Where this stops

Built-in variables vary with Make settings and platforms, and environment overrides can add more collisions. Namespacing avoids the immediate trap; validation is still needed at the command boundary.

The problem

A review result can become misleading when a checker that never finished is scored as though it disproved the finding.

Why it happens

There are two questions: did the check complete, and what did it conclude? A crash, timeout or missing dependency answers neither true nor false about the proposed bug.

How to work through it

1. Record execution status separately from the finding verdict.
2. Use an explicit unverified state for interrupted or incomplete checks.
3. Retry or investigate that state before treating the finding as resolved.

The solution

For example, a reproducer that times out should retain its evidence and timeout reason. Calling it a false positive would erase a potentially real issue merely because the checking path failed.

How to check it

Feed the review pipeline one confirmed issue, one refuted issue and one deliberately interrupted check. Confirm that all three remain distinct in the report and any aggregate score.

Where this stops

Unverified does not mean the finding is probably correct. It means the evidence is unfinished. A later completed check may support or refute it, and the report should preserve that transition.

The problem

A value copied into code and documentation can require a release and several synchronized edits whenever the operating environment changes.

Why it happens

The same policy now has multiple owners. One copy can change while another stays plausible, leaving behavior and instructions in disagreement.

How to work through it

1. Identify values expected to vary across deployments or over time.
2. Give each value one configuration owner with validation and a documented default.
3. Have code and generated documentation read that owner rather than copying the value.

The solution

For example, a retry limit that differs by environment belongs in validated settings. A mathematical constant inside an algorithm usually does not. The distinction is whether the value represents a changeable decision.

How to check it

Change the setting in a fixture and confirm the behavior and displayed documentation agree. Reject invalid values at startup instead of accepting a configuration that cannot work.

Where this stops

Configuration adds another interface to maintain. Keep true invariants in code, avoid exposing internal details without a use case, and record which changes require a restart.

The problem

A test suite can be fast and green while depending on a fake version of storage, authentication or timing that the real system does not provide.

Why it happens

Mocks verify behavior under stated assumptions. If those assumptions drift from the real dependency, the test can keep confirming an interface that no longer exists.

How to work through it

1. Use mocks for most local logic and contract cases.
2. Choose a few critical paths where a wrong assumption would be costly.
3. Exercise those paths against the real dependency through the same interface production uses.

The solution

For example, a mocked database can confirm that save is called. An unmocked check should save, reconnect and read the data back, because persistence across a connection boundary is the property at risk.

How to check it

Compare the observed real behavior with the mock’s contract. Include authentication refusal or a relevant timing boundary rather than testing only a happy path.

Where this stops

The right balance depends on risk and test cost; a fixed mock percentage is not a universal rule. Real-dependency tests also need isolation and repeatable cleanup to remain useful.

The problem

A poor component choice becomes expensive when its assumptions spread through the rest of the system.

Why it happens

Coupling means other modules depend on details that should belong to one component. Replacing that component then requires changing callers, stored data or control flow across the system.

How to work through it

1. Define the behavior callers need before choosing the implementation.
2. Keep vendor-specific types and settings behind a small adapter.
3. Make dependencies flow through the contract rather than importing implementation details everywhere.

The solution

For example, a storage interface can expose read and write operations without requiring every caller to know a database driver’s connection object. Another backend then has one boundary to implement.

How to check it

Try a second minimal implementation in a fixture. If replacing it requires edits across the core pipeline, inspect which assumptions escaped the boundary.

Where this stops

Indirection has a cost, and an interface cannot predict every future need. Keep it tied to real behavior instead of creating speculative layers. Swappability reduces the cost of a wrong choice; it does not make every choice equally good.

The problem

Rerunning an unchanged language-model stage spends money again even when its previous accepted result is still applicable.

Why it happens

A dependency graph makes it possible to identify which results depend on changed inputs. Without that relationship, a small edit can trigger a full rebuild or leave old results looking current.

How to work through it

1. Record each stage’s inputs, recipe identity and accepted output.
2. Invalidate affected descendants when a relevant input changes.
3. Reuse unaffected accepted results and rerun stale stages in dependency order.

The solution

For example, editing one source document should invalidate its summary and consumers, while leaving unrelated summaries available. The reuse decision belongs to deterministic input tracking, not to a model’s opinion that nothing important changed.

How to check it

Change one fixture input and inspect the exact set of stages marked stale. Confirm that unrelated stages remain reusable and that changed configuration also invalidates the right results.

Where this stops

The declared dependency set must be complete. A reused model answer remains a saved accepted answer, not a promise that generating it again would produce identical text.

The problem

A filename extractor can return a plausible shorter extension instead of the full extension present in the text.

Why it happens

In a typical backtracking regular expression, alternatives are tried in order. If oa matches the start of oasis and no following boundary is required, the first successful branch can end the match early.

How to work through it

1. Put longer overlapping alternatives first and require the intended end boundary.
2. Add fixtures for every overlapping extension and for suffix text that should not match.
3. Inspect downstream stored results as well as the parser after fixing the expression.

The solution

For example, matching .oasis must not emit .oa. A boundary or full-token check protects the intended token more clearly than relying on branch order alone.

How to check it

Compare extracted full paths against expected paths, then check any derived records that the old extractor populated. A corrected parser does not automatically repair rows already saved.

Where this stops

Regex is often too weak for a complete command language. Prefer a parser, tool-reported inputs or observed file access when the surrounding syntax matters.

The problem

A randomized live-data check can report a mismatch that cannot be reproduced, leaving it unclear whether the system or the check was wrong.

Why it happens

Three independent sources of uncertainty get mixed together: generated input, changing data and a checker that may share the system’s own bug.

How to work through it

1. Record a seed and the exact generated query.
2. Run the system and the reference evaluator against the same frozen data snapshot.
3. Use a deliberately simple evaluator that shares neither the parser nor query builder.

The solution

For an illustrative filter check, compare exact row-id sets, not only counts. Equal counts can hide one missing row and one extra row. Save the snapshot identity and mismatch so a later investigation starts with the same problem.

How to check it

Replay a saved seed and snapshot and confirm the same query and result sets. Plant one known filtering defect to prove that the independent check detects it.

Where this stops

The cited design is a proposal, not evidence that a periodic production checker was installed. A simple oracle can still be wrong, so give it its own examples and boundaries.

The problem

A signal test can disturb the wrong thread when it assumes that targeting a process also selects a particular worker.

Why it happens

A process-directed signal may be delivered to any eligible thread that has not blocked it. Thread-directed delivery selects a thread, but Python’s signal-handler rules remain a separate constraint.

How to work through it

1. Write down whether the test needs process delivery or delivery to a specific thread.
2. Control signal masks explicitly and restore them during cleanup.
3. Use thread-directed delivery only with a clear understanding of the runtime’s handling rules.

The solution

For example, a test of an interrupted worker should identify that worker rather than relying on whichever thread the kernel chooses. Keep the signal action small and arrange deterministic synchronization around the expected interruption.

How to check it

Repeat the isolated test and check which thread was targeted and which handler behavior occurred. A test that merely observed some signal is too weak.

Where this stops

Python handlers normally run in the main thread, and pthread_kill is not a general way to execute Python callbacks in a worker. The source trail for this exact historical assertion is weak; treat the procedure as general guidance.

The problem

An input hash can be used to make a stronger reproducibility claim than it actually supports.

Why it happens

A digest identifies bytes supplied to a run. It says nothing by itself about random sampling, parallel execution, remote services or hidden inputs that can change the output.

How to work through it

1. Record input identities and the execution environment separately.
2. State whether a step promises repeatable bytes, equivalent meaning or only acceptable quality.
3. Keep the produced output and its own digest alongside the run evidence.

The solution

For an illustrative language-model step, store the prompt, model identifier and generation settings, then preserve the actual answer. Repeating the same prompt may produce another valid answer; that does not mean either input digest was wrong.

How to check it

Repeat a representative run and compare outputs against the promised level of agreement. If they differ, identify the source of variation instead of calling the hash a failed determinism guarantee.

Where this stops

Even deterministic code needs all relevant inputs declared. A matching hash is evidence of matching captured inputs, not proof that the capture was complete or the result correct.

The problem

A function’s calling convention is harder to review when readers must inspect its body to discover expected inputs and outputs.

Why it happens

Type hints expose part of the interface before execution. Like declared port types, they let tools check whether connected pieces agree on the values they exchange.

How to work through it

1. Annotate the public inputs and return value with useful domain types.
2. Run a type checker as a separate verification step.
3. Validate data arriving from untrusted or untyped boundaries at runtime.

The solution

For example, an annotated parser returning a result object is easier to use correctly than one returning an undocumented mixture of strings and dictionaries. The annotation helps callers, while runtime validation still handles malformed input.

How to check it

Change a fixture caller to pass an incompatible type and confirm that the configured checker reports it. Then exercise a malformed external value to check the runtime boundary.

Where this stops

Python annotations do not automatically enforce types when code runs. Broad Any annotations and unchecked dependencies weaken the guarantee, and type correctness does not establish the function’s intended behavior.

The problem

A reader should not need a repository-wide search to discover which part of a system owns a task.

Why it happens

When code is arranged by incidental helpers rather than system responsibilities, the architecture exists only in the author’s head. Changes then require rediscovering the same boundaries.

How to work through it

1. Draw the main responsibilities and the data crossing their boundaries.
2. Give each block a clear owner in the code layout.
3. Walk one real request from entry point to result and check that the diagram follows it.

The solution

For example, a pipeline with ingest, validate and export blocks should make those three responsibilities easy to locate. Shared utilities can support the blocks, but should not hide the direction of data flow or become an unexplained extra core.

How to check it

Ask a fresh reader where a validation change belongs, then compare the answer with the diagram and actual imports. Disagreement reveals a boundary or naming problem.

Where this stops

A diagram is a model, so small helpers need not each become a box. Keep the view at a useful level and revise it when responsibilities move.

The problem

A code graph or patch target can point at the wrong object even when the import statement itself is valid.

Why it happens

Python separates loading a module from binding a local name. With import a.b, the local name is a; treating a.b as the bound name loses the first step of attribute lookup.

How to work through it

1. Parse the import form and any alias before resolving a later call.
2. Track the local binding separately from the fully qualified module path.
3. Cover plain imports, aliases and from-imports with small examples.

The solution

In an illustrative module, import package.worker makes package.worker.run() reachable through package. By contrast, import package.worker as worker binds worker. A resolver needs both the spelling used locally and the target it denotes.

How to check it

Inspect the module namespace in a small fixture and compare the graph’s resolved target with Python’s actual lookup. Include a case where a package and a function share a name.

Where this stops

Dynamic imports, re-exports and attribute chains need additional rules. Mark unresolved targets rather than turning a partial static analysis into a confident but false edge.

The problem

A cache keyed only by source commit can reuse a result produced under a different external environment.

Why it happens

A flow can consume released IP, scripts, libraries and tool builds that live outside the source repository. The commit stays fixed while these inputs change, so apparently identical reruns are not identical experiments.

How to work through it

1. Inventory external inputs at each stage, including tool builds and library releases.
2. Record immutable versions or content digests in a run manifest.
3. Build the cache key from the complete declared input set.

The solution

Compare report content separately from run metadata. An illustrative report with a new build number is not a functional change; a changed timing result under an unchanged source commit is a reason to inspect the environment.

How to check it

Rerun a small stage with pinned inputs, then deliberately change one external input and confirm that the cache misses. Compare normalized reports only where removing metadata is justified.

Where this stops

Declaring inputs is necessary for safe caching, but does not make a nondeterministic tool deterministic. The historical source reports hidden-input drift; this draft does not reproduce that trial.

The problem

A design simulation can show a convincing capacity curve while saying nothing about whether the implementation returns the correct result.

Why it happens

A model executes assumptions about the system. It does not execute the application code, its database integration or its actual error handling.

How to work through it

1. Use the design model to explore queues, contention and expected bottlenecks.
2. Build separate tests that run the real application with real local dependencies.
3. Add controlled delay or failure injection where the implementation’s behavior needs checking.

The solution

A simulated database queue can reveal which service would saturate first under the chosen assumptions. A real database test asks whether the query and retry logic work when a connection is delayed or dropped.

How to check it

Label the result by what ran. Inspect assumptions for the simulation and actual response correctness for the implementation test; a load curve alone does not establish either one.

Where this stops

The source is a research comparison, not a recorded production validation. Real-code tests also cover only the workloads and faults exercised, so neither approach replaces the other.

The problem

A template change can appear to have no effect because the test harness never included the edited files.

Why it happens

A fixture that clones committed content can omit uncommitted changes. The test then exercises an older tree while its author thinks it is testing the current one.

How to work through it

1. Record the source tree or revision used by the harness.
2. Inspect the relevant file inside the actual fixture.
3. Repair the fixture input before investigating the tool’s behavior.

The solution

If a migration change is absent from the cloned fixture, rerunning the migration test only repeats the wrong experiment. Confirm that the expected bytes reached the fixture before concluding that the template tool ignores them.

How to check it

Add a temporary observable marker in a disposable test input and check its presence at the tool boundary. Then remove the marker and rerun the real behavior check on the corrected fixture.

Where this stops

Instrumenting the harness establishes what was tested, not that the tool is correct. Once the right input is confirmed, continue investigating any remaining tool or configuration issue.

The problem

A test can compare the right fields and still fail to detect that the implementation swapped them when their fixture values are equal.

Why it happens

Equal inputs make the correct and faulty mappings observationally identical. The test has no signal that distinguishes the two behaviors.

How to work through it

1. Identify output fields that could be confused or swapped.
2. Give their fixture inputs distinct values that survive rounding and transformation.
3. Check each output against its own independently derived expectation.

The solution

For two percentages, use values such as 33.3 and 50.0 rather than two paths that both produce 50.0. A swapped mapping then gives visibly wrong output instead of passing by coincidence.

How to check it

Deliberately swap the two fields in a disposable mutation and expect the test to fail. If it still passes, inspect whether formatting or shared helper logic erased the distinction.

Where this stops

Distinct values do not guarantee independence. Keep the expected calculation separate from the implementation, and include boundary cases as well as the asymmetric example.

The problem

A gate can stay green over real damage if its test path never exercises the condition it claims to reject.

Why it happens

A passing normal case shows that valid work is accepted. It does not show that the detector or final failure signal works in the opposite direction.

How to work through it

1. Choose a representative fault the gate is required to catch.
2. Inject it into an isolated fixture and run the gate.
3. Record the fault, expected rejection and actual result, then remove the fault and check recovery.

The solution

If a gate promises to reject a missing required field, delete that field in a disposable input. The gate should fail because of that omission, not because the fixture is broken in some unrelated way.

How to check it

Inspect both the local diagnostic and the final caller-visible status. A message saying “failed” is insufficient if the overall command still returns success.

Where this stops

One planted fault proves only that path. Select faults for distinct failure classes, and keep destructive injections away from live data and systems.

The problem

A query layer can return plausible rows without a known exact answer for every possible query. That makes broad parser checks difficult.

Why it happens

Some results must obey relationships even without a separate expected-answer system. These checks are often called metamorphic checks.

How to work through it

1. Choose a fixed data snapshot and a relation that should hold.
2. Run the related queries against that same snapshot.
3. Compare row sets as well as counts, and save the inputs needed to replay a failure.

The solution

For example, rows matching A AND B must be contained in the rows matching A. On a complete two-valued partition, count(A) plus count(NOT A) must equal the total. A broken parser can violate either relation.

How to check it

Deliberately alter a query interpretation in a safe test and expect a relation check to fail. A repeatable seed and snapshot make the failure useful to investigate.

Where this stops

NULL semantics, pagination and changing data can invalidate a relation’s assumptions. These checks complement independent expected answers; an implementation can preserve the chosen relations and still be wrong elsewhere.

The problem

A repository that works on Linux can fail to check out cleanly on a case-insensitive filesystem when two filenames differ only by case.

Why it happens

Git records path names distinctly, while the target filesystem may treat those paths as the same file. One working tree cannot represent both names reliably.

How to work through it

1. Normalize generated names according to one naming policy.
2. Check case-folded uniqueness before writing or accepting paths.
3. Reject a collision with both conflicting names so it can be resolved deliberately.

The solution

Foo.md and foo.md are different Git entries but can collide on a default Windows checkout. Choosing one canonical spelling before generation avoids leaving the conflict for the next machine to discover.

How to check it

Run the collision check over the full generated path set, including existing tracked files. Expect the example pair to fail even when the check runs on a case-sensitive host.

Where this stops

Case folding is one portability risk. Unicode normalization, reserved names and path-length limits can also matter; a case-only check does not establish complete cross-platform compatibility.

The problem

A fresh clone can contain the right files while losing the ownership and permission layout needed for a shared installation.

Why it happens

Git tracks a file’s executable distinction, not the full Unix mode, owner or group. Group-write and setgid intent therefore cannot live only in the working tree’s current permissions.

How to work through it

1. Keep content and executable intent in Git.
2. Express ownership and additional modes in the package or installation procedure.
3. Reapply and validate those settings when the deployment changes.

The solution

A shared directory may need a group owner and setgid so new files inherit the group. Record that rule in setup rather than assuming a clone will preserve the source directory’s mode.

How to check it

Inspect a clean installation as the intended service and team identities. Check effective access and group inheritance, not just whether the tracked scripts retained their executable bit.

Where this stops

Hooks can help restore local state but are not automatically installed by every clone. The deployment process must own their installation or enforce the permissions through another reliable mechanism.

The problem

A suite that passes in one parallel run may still depend on test order or shared state. Speed alone does not establish that parallel execution is equivalent.

Why it happens

Workers change execution order and overlap. Tests that share files, environment or mutable fixtures can interfere in ways a serial run hides.

How to work through it

1. Run the same selected suite serially.
2. Compare runs with four and sixteen workers.
3. Add a shuffled-order run and investigate differences in both outcomes and collected cases.

The solution

Treat a parallel-only failure as evidence about isolation. If one test changes a shared setting and another assumes its original value, a scheduling change can expose a real weakness in the suite.

How to check it

Compare collected, passed, failed and skipped cases, not just the final exit status. Expect the same intended coverage and outcomes across the selected execution modes.

Where this stops

These runs sample possible schedules; they do not prove every interleaving safe. Use isolated fixtures and separate checks for truly concurrent behavior, and run heavy suites on appropriate compute resources.

The problem

A green mock-based suite can confirm the intended logic while missing a wrong assumption about the external system.

Why it happens

A mock behaves as its author says it should. It cannot reveal a disagreement between that assumption and a real API, filesystem, identity or timing boundary.

How to work through it

1. Use mocks for fast checks of logic and contracts.
2. Add a fake with meaningful state, then real components in a local setup.
3. Use a small live smoke check for the critical external assumptions that lower rungs cannot establish.

The solution

For persistence, a mock can confirm that a write was requested. A real store check asks whether the data survives and is readable through the actual integration. Those are different claims and need different evidence.

How to check it

Record which rung ran and what it established. Confirm that the live check really reached the intended system rather than silently falling back to a mock.

Where this stops

Live checks need controlled inputs and appropriate authorization. More rungs do not justify unsafe production tests, and identical green results are useful only if each rung checks something distinct.

The problem

A harmless wording change can break many tests when they assert on an error sentence instead of the behavior that produced it.

Why it happens

The tests bind correctness to presentation. The message may change while the error category, state and exit behavior stay correct.

How to work through it

1. Give the result a stable code or structured field.
2. Create a fixture that isolates the condition being checked.
3. Assert on that structure and reserve text assertions for wording that is itself part of the contract.

The solution

For example, an invalid range can return an INVALID_RANGE code plus a helpful sentence. The behavior test should check the code and rejected operation, so improving the sentence does not rewrite the suite.

How to check it

Change only the human wording and expect behavior tests to stay green. Then change the error code or allow the invalid operation and expect a relevant test to fail.

Where this stops

User-facing copy can deserve dedicated checks, and some command-line interfaces promise exact text. Keep those expectations explicit rather than using message matching as a substitute for a missing structured result.

The problem

A reduced failure case can be called “the smallest” even though the reduction algorithm has proved something narrower.

Why it happens

ddmin finds a set that is 1-minimal: removing any one remaining difference loses the failure. That does not rule out a different, smaller failing set.

How to work through it

1. Reproduce the passing baseline and the failing full-change case first.
2. Reduce differences on an isolated copy while preserving the same failure test.
3. After reduction, test removal of each remaining difference and report the guarantee accurately.

The solution

If change a fails alone while b and c fail only together, a search can return {b, c}. Neither member can be removed from that result, even though {a} is a smaller failure case.

How to check it

Expect the reduced set to reproduce and every single-member removal to stop reproducing. If either original endpoint no longer behaves as recorded, stop and repair the experiment first.

Where this stops

Flaky tests and unresolved intermediate outcomes weaken the result. A 1-minimal set is useful evidence about a reproducer, not proof of the unique root cause or globally smallest explanation.

The problem

A reader can run after a writer in wall-clock time and still have no valid synchronization relationship with that write.

Why it happens

Happens-before is an ordering defined by the memory model, not by which operation appeared to finish first. Threads need an appropriate synchronization path to publish and acquire shared state.

How to work through it

1. Identify the shared data and the flag or event meant to publish it.
2. Use the required release operation after the write and matching acquire before the read.
3. Check the whole path, including any transitive synchronization through other threads.

The solution

For example, write a payload, publish readiness with release semantics, and have the reader acquire that readiness before using the payload. A delay or an observed completion time does not replace that relationship.

How to check it

Trace the actual synchronization edge between the conflicting accesses. A run that happens to return the expected value only shows one execution; it does not establish the memory-order guarantee.

Where this stops

The exact operations and scopes depend on the language and hardware memory model. Use its specified primitives rather than assuming every flag or barrier has the needed semantics.

The problem

A template substitution can alter the test fixtures as well as the production examples. The generated project then fails tests for an accidental naming collision.

Why it happens

A broad replacement does not understand which occurrence is sample data and which is a test input. Shared names make both look eligible.

How to work through it

1. Give fixtures clearly synthetic names that do not overlap the real substitution values.
2. Limit replacement to the intended files and fields.
3. Verify the generated project, including its unchanged fixture assumptions.

The solution

For example, keep a fixture’s fake account name separate from the account token that the template fills. Changing the deployment account should then leave the test’s deliberately invalid account untouched.

How to check it

Render with replacement values that resemble realistic names and inspect the fixture diff. Expect only declared template fields to change, then run the generated tree’s own tests.

Where this stops

Names reduce accidental overlap but do not replace a proper template boundary. If replacement can still edit arbitrary text, another collision can appear later.

The problem

Tests written after implementation can confirm the shape of the code while missing the behavior the plan actually promised.

Why it happens

The implementation becomes the test author’s reference. Mistakes in the code can then be copied into the expected result.

How to work through it

1. Write the intended behavior and boundary cases in the plan.
2. Derive the test cases before implementation begins.
3. Implement against those cases, then add targeted checks for new risks discovered during the work.

The solution

For a filter, the plan should say how empty input, invalid ranges and boundary values behave. Tests can encode those decisions before the parser’s current behavior has a chance to become the accidental specification.

How to check it

Compare the tests back to the plan and introduce a fault that violates one promised behavior. Expect the corresponding test to fail for the intended reason.

Where this stops

A plan can also be wrong or incomplete. Independent review and integration checks still matter; writing tests first separates two artifacts, but does not make their shared assumptions true.

The problem

A caller cannot act correctly if a negative answer and a failed attempt to obtain an answer look identical.

Why it happens

One exit status collapses two different states: the check completed and found no match, or the check could not complete at all.

How to work through it

1. Document separate results for yes, no and unable to check.
2. Return those results from every execution path.
3. Make callers branch explicitly on all three outcomes.

The solution

For example, “the file is absent” can support a normal decision, while “permission denied while reading the directory” should stop that decision. Both may look like no result unless the check preserves its error state.

How to check it

Exercise a positive fixture, a valid negative fixture and an unavailable prerequisite. Expect three distinguishable outcomes and confirm that the caller does not turn the last one into an ordinary no.

Where this stops

The numeric exit codes are an interface choice and need documentation. A nonzero code is not automatically an error if the tool contract uses it for a valid negative result.

The problem

Adding a feature becomes expensive when it requires changes throughout the central pipeline. Each new branch increases the amount of core behavior that must be understood again.

Why it happens

The extension point does not own a clear contract, so feature-specific details leak into orchestration.

How to work through it

1. Locate the part of the core that knows the new feature’s details.
2. Define the inputs, outputs and failure behavior at that boundary.
3. Move feature-specific behavior behind the boundary and let the core call the shared contract.

The solution

For example, a new model provider should implement the provider interface and register itself. The pipeline should not need a new provider-name conditional at every stage.

How to check it

Compare the next feature addition against the boundary. It should be possible to explain its integration through one contract and test that contract without rebuilding the whole pipeline.

Where this stops

Some features change the system’s actual responsibilities. In that case the core contract may need a deliberate redesign. The warning is repeated feature-specific edits, not a rule that the core can never change.

The problem

A TypeScript application can bundle successfully while containing type errors. A green build can therefore be mistaken for a stronger check than it actually provides.

Why it happens

esbuild removes type annotations and produces JavaScript. It does not check whether assignments and function calls satisfy the TypeScript types.

How to work through it

1. Keep esbuild as the bundling step.
2. Add tsc --noEmit as a separate required check.
3. Make the verification command and CI run both steps, with missing tools treated as incomplete verification.

The solution

If a function expects a number and a caller passes text, bundling may still work. The type-checking step should catch the mismatch before the bundle is accepted.

How to check it

Introduce a small deliberate type mismatch in a disposable copy. Expect the type gate to fail even if bundling succeeds, then remove the fault and expect both checks to pass.

Where this stops

Type checks do not establish runtime API correctness. Handwritten interfaces can drift from real responses, so important boundaries also need contract or integration checks.

The problem

GSan, a race sanitizer for Triton and Gluon GPU code, can miss a real race or report one that did not occur. Treating its output as a complete account of memory ordering is too strong.

Why it happens

GSan uses logical clocks: records of which accesses are ordered by synchronization. Tracking every reader’s clock is expensive. Sampling leaves some older reads untracked, while reuse of a wrapped clock buffer can confuse the recorded history.

How to work through it

1. Understand what the sanitizer records and what it samples.
2. For a report, inspect the conflicting accesses and the synchronization between them.
3. For a clean run, keep the sampling limit in mind and check critical shared-memory protocols separately.

The solution

Inspect synchronization as well as the report. For example, a producer writes a shared value, then signals completion; a consumer waits for that signal before reading it. If that synchronization orders the write before the read, those two accesses are ordered. Without the wait, their apparent order in a log does not establish safety. This is an illustrative check, not a tested kernel.

How to check it

For a suspected access pair, identify the synchronization operation that orders one before the other. If supported by the GSan configuration, more read samples or a larger clock buffer can reduce these failure modes at the cost of memory. Repeated silence still does not prove every access is ordered.

Where this stops

These limits describe GSan’s sampled clock design. Its synchronization must be instrumented and recognized; code or synchronization outside its coverage can cause missed races or misleading reports. Other sanitizers may track ordering differently.

The problem

A suite can report success even though none of its live cases ran. The result looks green while the behavior it was meant to check remains unknown.

Why it happens

The scoring path counted no failures without first requiring any completed work. An empty set therefore satisfied the success condition.

How to work through it

1. Report selected, completed, skipped and failed cases separately.
2. Require the intended cases to complete before returning a passing verdict.
3. Make missing credentials or unavailable prerequisites visible as a non-passing result.

The solution

A live evaluation with ten selected cases and ten marked “not run” should say it was not evaluated. It should never share the same success signal as ten completed, passing cases.

How to check it

Deliberately remove a required prerequisite and expect the gate to stop or report incomplete execution. Also run a known failing case to check that failure reaches the final verdict.

Where this stops

An intentionally optional suite can be skipped, but the caller must see that distinction. No completed cases means no evidence about the behavior under test.

The problem

A release procedure can fail even when each individual command is valid. The full gate may depend on release metadata that was not present during an earlier check.

Why it happens

The order of operations changes the tree and the metadata being verified. A gate run before the local release state exists has checked a different state.

How to work through it

1. Prepare the changelog and commit the release tree.
2. Create the release tag locally.
3. Run the complete required gate against that local release state.
4. Push the commit and tag only after the gate succeeds.

The solution

Treat the tag as local preparation until verification finishes. If a release check reads the version tag, it can now inspect the same version that would be pushed.

How to check it

Inspect the gate’s real result and confirm which commit the local tag resolves to. A successful earlier test run is not a substitute for this release check.

Where this stops

The original title is shorthand; the recorded procedure creates the local tag before the full gate and pushes last. Repositories with different release contracts may need another order.

The problem

A tool refuses an input, but its output contains several warnings. Matching one sentence does not show which condition caused the refusal.

Why it happens

Messages are written for people and can change. If two controlled runs differ in one input, their structured results give a cleaner causal signal.

How to work through it

1. Build a small input that is accepted.
2. Copy it and change only the condition you want to test.
3. Run both under the same environment and compare exit codes or structured result fields.

The solution

For an installer check, keep the destination and settings fixed, then add one invalid setting. If the accepted copy succeeds and the changed copy refuses, the test now targets that setting instead of guessing from wording.

How to check it

Restore the changed input and expect acceptance again. Keep unrelated warnings identical across both runs so they cannot explain the difference.

Where this stops

This isolates one condition only when other inputs and state stay fixed. Use text assertions when exact wording is itself a promised interface.

The problem

An answer key can look thorough while covering only familiar documents. If its author reads each source and turns a sentence into a question, the search system gets vocabulary clues that an ordinary reader may never supply.

Why it happens

The test is then partly measuring phrase matching. It can hide weak retrieval across several documents, and a fluent answer may pass even when it omits the facts the question actually needs.

How to work through it

1. Draft questions as someone with the problem would ask them. Keep the expected answer separate, with the source documents and a small list of facts that must be present.
2. Include questions answered by one document, questions that need several documents, and questions the indexed material cannot answer. Score those groups separately.
3. Check repeated wording and document reuse, then inspect the questions yourself. Treat phrase-overlap checks as prompts for review, because necessary technical terms may be shared.

The solution

Make the question natural and the answer key precise. Judge whether the key facts are supported, rather than whether the answer copies one preferred sentence.

How to check it

Before trusting the score, count the indexed documents and check which ones the answer set covers. For retrieval, report whether all required documents were found; a result depth the system did not return is not measured, rather than a measured zero.

Where this stops

The source records a prepared evaluation set; it leaves the owner’s first run and edits open. A no-answer question needs readable source text: an unprocessed scan cannot establish that the documents say nothing.

The problem

A cost study needs to separate input, cached input and output. A transcript may expose fields for all three, yet still record some of them before the response has finished. Summing those fields does not make them complete.

Why it happens

Streaming systems can record usage at different stages. An early event may know how much input was sent while knowing very little about the answer still being generated. The field name describes the quantity; it does not establish when the quantity was measured.

How to work through it

1. Choose a completed session and compare each usage category with the harness or provider total for that same session. Keep the model, time range and units the same.
2. Check whether the transcript records a final event. Compare visible answer size as a rough sanity check, while keeping hidden reasoning and tokenisation differences separate.
3. Use the authoritative final counters where available. If only partial telemetry survives, label its coverage. Call the cost a lower bound only after checking that no usage was counted twice.

The solution

Validate the counter before building the spreadsheet around it. Input telemetry can be useful even when output telemetry is incomplete, so judge the categories separately.

How to check it

For a new audit, reconcile one session end to end before aggregating many. A large unexplained gap means the total is not ready to support a savings claim.

Where this stops

The recorded study reported incomplete output counters; this is not a claim that every provider or transcript format behaves the same way. Character counts are estimates, and list-price calculations are not billed spend.

The problem

A skill may be selected correctly and still have its procedure skipped. If its description contains a miniature workflow, the agent can treat that summary as enough to act on.

Why it happens

The description is the routing surface: it helps choose when to load the skill. Mixing trigger conditions with abbreviated steps gives the reader a shortcut that omits the body's checks and decision points.

How to work through it

1. Write the description around the task symptoms and situations that should trigger the skill. Include a confusing near-match if it helps distinguish the skill from another available one.
2. Put the ordered procedure, prerequisites and checks in the body. Keep one maintained process so the description cannot drift into an older, shorter version of it.

The solution

Separate finding the right instructions from following them. A good trigger gets the agent to the full procedure rather than trying to replace that procedure with a sentence.

How to check it

Test should-trigger and should-not-trigger prompts, then inspect a selected run to see whether the body was actually read and followed. Routing success and process compliance are separate results.

Where this stops

A clearer description cannot guarantee obedience, and overlapping skills may still compete. Report the scope and sample of any routing evaluation instead of treating a few successes as a universal firing rate.

The problem

An agent rule file can grow with plausible warnings until the important rules become hard to find. Advice without a real case is difficult to test and difficult to remove.

Why it happens

A rule spends attention on every task where it is loaded. Without a specific failure behind it, nobody can tell whether it prevents a recurring problem or merely sounds cautious.

How to work through it

1. Capture the observed mistake and the condition that made it possible. Write the smallest correction that would have changed that case, including the scope where it applies.
2. Ask whether a script, test or hook can enforce the correction. Put deterministic checks there and leave the agent the judgement that cannot be reduced to a reliable check.

The solution

Keep the rule, its reason and the case connected. This lets a future reviewer narrow or retire the rule when the workflow changes.

How to check it

Replay the original failure as a fixture where practical. The new check should reject it while accepting a nearby valid case, so a broad prohibition does not quietly replace the real correction.

Where this stops

Some preventative controls are justified by external requirements or known hazards before a local failure occurs. Name that basis explicitly instead of inventing an observed incident.

The problem

A provider-neutral model API can accept the same request shape while different models reject different parameter values. Testing one model therefore leaves compatibility failures hidden until another is selected.

Why it happens

The common interface translates the request, but the model provider still validates its own limits. In the reported example, a one-token output limit worked for some model families and failed for another.

How to work through it

1. Build a small probe for every model exposed by the application. Keep the request within documented limits and record validation failures separately from access or availability failures.
2. Choose shared defaults from the supported range, then store any model-specific constraints in configuration. Record the reason for a changed default so a later cleanup does not undo it.

The solution

Preserve the common adapter while testing the differences behind it. The reported probe needed an output allowance of 16 tokens for the model family that rejected 1.

How to check it

Repeat the compatibility sweep when adding models or changing parameters. Every configured model should accept the supported minimal request, or be clearly excluded with its reason.

Where this stops

A minimal successful response proves invocability, not quality or complete feature parity. The reported floor is historical; new models and API changes can introduce different constraints.

The problem

A one-line request to an agent can cost more than its visible prompt suggests. Estimating from the user's sentence misses the context the agent framework sends before it can answer.

Why it happens

The request may include a long system prompt, tool definitions and a cache write. Those are input costs even when the requested answer is only a few words.

How to work through it

1. Measure the whole invocation with token and cost telemetry. Separate ordinary input, cache creation, cache reads and output so the fixed overhead is visible.
2. Compare a fresh invocation with a warm repeat under the same configuration. Use the actual billing categories rather than assuming all cached tokens have the same price.

The solution

Choose the smallest execution path that can do the task. A direct structured call may suit a tiny operation; an agent is worthwhile when its tools and judgement earn their setup cost.

How to check it

Check whether reducing the user prompt changes total billed work meaningfully. The expected result is a cost breakdown that explains the bill, including context outside the visible request.

Where this stops

Cache behaviour depends on provider, lifetime and exact prompt reuse. One reported call does not establish a universal token overhead, and saving cost must not remove context needed for correctness.

The problem

Generating many varied inputs does not tell you whether a language model answered correctly. A large test run can produce plenty of output while offering no basis for accepting any of it.

Why it happens

Constrained random testing works in hardware because a reference model supplies the expected answer. Open-ended model output needs an equivalent judgement rule before variation becomes useful evidence.

How to work through it

1. Start with a small evaluation set whose expected answers or acceptance criteria are known. Include failures that matter to the task, such as a missed confidential term or an incorrect refusal.
2. Where exact answers are unsuitable, define a judge rubric and check it against examples a person has labelled. Keep disagreements visible rather than treating the judge as unquestionable truth.

The solution

Use varied inputs to explore a defined correctness boundary. Preserve failing examples so the next change can be checked against the same cases.

How to check it

Plant a clearly wrong output and confirm the evaluation rejects it. Then inspect borderline cases to see whether the scoring rule matches the intended task.

Where this stops

A judge can share the model's blind spots, and a fixed set can be overfit. Keep some examples aside and report what the evaluation covers rather than calling the model generally correct.

The problem

A model can produce a fluent result that has no dependable relationship to the task's evidence.

Why it happens

Statistical generation responds to its context. Missing grounding and vague handoffs leave the model to supply likely text, while repeated prose handoffs can change meaning between steps.

How to work through it

1. Give each step the evidence and contract it needs.
2. Use deterministic code for decisions that can be specified exactly.
3. Reserve model judgment for the remaining ambiguous work and check its output before the next step.

The solution

Illustrative: a source-selection step should return structured document identities and evidence, rather than a persuasive paragraph that the next agent has to reinterpret as facts.

How to check it

Trace a representative answer back through its selected sources and validation steps. Missing evidence should produce a visible gap rather than an invented completion.

Where this stops

Better context does not guarantee truth. Schema validity proves structure, not factual support, and judgment steps still need appropriate review or comparison against known cases.

The problem

Retrieval can get worse after adding a ranking stage if the corpus contains navigation text and competing versions of the same evidence.

Why it happens

A reranker can only reorder what it receives. It cannot turn irrelevant or misleading source chunks into a clean evidence set, and repeated navigation can dominate the available context.

How to work through it

1. Inspect retrieved chunks before changing the model.
2. Remove or separate navigation, obsolete versions and low-value repeated material.
3. Evaluate corpus changes and ranking changes independently against the same questions.

The solution

Illustrative: a top result containing only a menu and page links offers little evidence for a technical answer. Removing that chunk can be more useful than making its ranking score more precise.

How to check it

Run retrieval-only cases and inspect relevance, source identity and useful answer content. Keep a ranking change only if the observed evidence improves for the intended tasks.

Where this stops

Curation can also remove needed context. Preserve document provenance and test difficult queries, rather than assuming that a smaller corpus or another ranking stage is always better.

The problem

An answering agent can make its own unsupported statements look authoritative by writing them into the corpus it later searches.

Why it happens

Retrieval provides provenance only if the underlying content has earned its place. Direct write-back can turn generated guesses into future evidence and create a feedback loop.

How to work through it

1. Route proposed additions to an inbox outside the trusted corpus.
2. Attach source provenance and distinguish generated interpretation.
3. Review the proposal before promotion and ingestion.

The solution

Illustrative: a meeting summary can be staged as a proposed note with timestamped source references. Until reviewed, it should not become a fact the answering agent cites as company knowledge.

How to check it

Trace a proposed write through staging, review and ingestion. An unreviewed statement should remain unavailable to the trusted-answer path, while approved material retains its provenance.

Where this stops

This is a design safeguard in the source, not evidence that the complete agent was deployed. Review also needs access controls and a way to correct or withdraw already ingested material.

The problem

A component can be easy to call while remaining too poorly understood to maintain or trust.

Why it happens

An interface hides implementation detail, but it does not remove assumptions about inputs, state and failure. Without that rough model, a caller cannot tell when the abstraction's promises stop applying.

How to work through it

1. Draw the component's inputs, outputs and state boundary.
2. Explain its normal path and its important failure modes.
3. Check its contract independently before depending on hidden internals.

The solution

Illustrative: a storage adapter may hide its database implementation, while the caller still needs to know whether a write is atomic and how failure is reported. That is enough model to reason about the boundary.

How to check it

Ask another reader to predict the outcome of a normal call, a missing input and a failure using the contract alone. Unanswerable cases identify gaps in understanding.

Where this stops

Understanding every internal line is unnecessary. The goal is a usable model and an earned contract, not a requirement to reimplement each dependency before using it.

The problem

Retrieval can confidently cite an obsolete draft when several versions of the same document share one index.

Why it happens

Similarity ranking does not inherently prefer the current revision. Repeated old text can occupy several top results and crowd out the document that now owns the answer.

How to work through it

1. Choose an authoritative version before ingestion.
2. Remove superseded copies from the searchable current corpus or place them behind an explicit history filter.
3. Inspect returned document identities and versions before judging the generated answer.

The solution

Illustrative: three similar drafts can fill the top matches while a current version ranks below them. Cleaning the corpus addresses the evidence selection before changing the model prompt.

How to check it

Run a retrieval-only question set and inspect the versions of the returned documents. Current-answer queries should resolve to the intended current source.

Where this stops

Historical research may need old versions. Retain them with explicit version metadata and scope rather than deleting the only record of past decisions.

The problem

A large API summary can spend its time recovering from rate limits instead of answering the question.

Why it happens

If raw 429 responses reach the model, it may improvise retries. The request budget is shared across sessions, so one conversation cannot reliably manage that platform constraint.

How to work through it

1. Put bounded retries and backoff inside the API tool.
2. Use the endpoint's supported page size to reduce requests.
3. Stop at a defined budget floor and return an honest partial sample when necessary.

The solution

Illustrative: increasing a supported page size can turn many small page requests into fewer large ones. The tool can then return the actual sample size instead of letting the model imply complete coverage.

How to check it

Inject a rate-limit response and a dropped connection. Expect bounded recovery, no duplicate page accumulation and an explicit partial-result signal when the budget is exhausted.

Where this stops

Retries do not create more capacity. Respect endpoint semantics and avoid replaying non-idempotent writes without a separate duplicate-protection design.

The problem

Parallel agents can produce conflicting work when they share mutable notes and assumptions without clear task boundaries.

Why it happens

More workers do not automatically create more independent progress. If every agent must infer which state is current, coordination can consume the benefit of parallel execution.

How to work through it

1. Give each agent a bounded task, the evidence it needs and an explicit output contract.
2. Define ownership of shared artifacts and a handoff point where results are reconciled before dependent work continues.

The solution

For two modules, let each worker own its implementation while agreeing on the interface first. A handoff should name the exact files, checks and unresolved questions instead of assuming the next agent remembers the conversation.

How to check it

Ask whether a fresh agent can complete one assigned task from its handoff alone. At integration, check conflicting edits and interface assumptions before treating the outputs as one working feature.

Where this stops

Isolation still needs coordination at shared boundaries. Some tasks are too coupled to split economically; one well-contextualised agent can be the simpler choice.

The problem

A rule written in a guide can disappear from daily practice while the guide still claims the project follows it.

Why it happens

Prose depends on each reader remembering and interpreting the rule at the right moment. A required check can make a decidable violation visible on the acceptance path.

How to work through it

1. For each rule, ask what concrete artifact proves compliance and whether code can decide it reliably.
2. Put decidable rules into the normal gate, keep judgment questions in review, and add a deliberately broken example proving the gate rejects the violation.

The solution

For a required metadata field, validate its presence and type instead of repeating “remember the field” in more documents. Keep a link to the reason so future maintainers know what the check protects.

How to check it

Run the negative example through the same gate used for real work. Confirm a non-zero result and a specific diagnostic, then confirm a valid case is accepted.

Where this stops

Checks can encode the wrong rule or be disconnected from real acceptance. They do not replace judgment, and the historical audit claim in the original card is a source report rather than a new comparison here.

The problem

An agent that produces output and immediately declares completion can repeat the same mistake without ever seeing the error.

Why it happens

A corrective loop needs a target, an observation of the result and a comparison that changes the next action. Asking for “better work” supplies none of those on its own.

How to work through it

1. Choose the observable acceptance condition and run the check against the produced artifact.
2. Feed the specific failing evidence back into a bounded correction step, then repeat the check on the revised artifact.

The solution

For a generated configuration, parse and validate the actual file before asking the agent to fix any reported field. Keep the validator independent of the prose claim that the file is valid.

How to check it

Deliberately include one invalid value in a disposable example. Expect the check to reject it and identify the field; completion should require a fresh check of the corrected bytes.

Where this stops

A faulty sensor drives a faulty loop. Human judgment still matters for goals and uncheckable quality, and retry limits must surface unresolved failure rather than looping indefinitely.

The problem

A document converter can preserve every word while damaging list nesting, emphasis and the relationships that make the document readable.

Why it happens

Word recall measures retained text. It cannot tell whether a phrase moved into the wrong list item or whether a heading was flattened into ordinary prose.

How to work through it

1. Keep text-retention checks, then add a separate reference for headings, lists, tables, links and emphasis.
2. Derive that reference independently from the source format, rather than copying the converter’s own assumptions into its grader.

The solution

For a nested list, compare which item owns each child, not merely whether all words appear. Give unsupported formatting an explicit receipt instead of counting silent flattening as faithful conversion.

How to check it

Use a fixture with identical words in two different nesting structures. The word score may match, but the structure check should distinguish them and identify the changed relationship.

Where this stops

A structure score is only as good as the facts and reference it checks. Independent extraction reduces shared mistakes but still needs review for omitted source semantics.

The problem

A growing conversation can make a short follow-up expensive because the next model call also carries material from earlier turns.

Why it happens

Stateless agents need context supplied on each call. A larger allowed window gives the caller room to send more; it does not make every old detail useful to the current question.

How to work through it

1. Set a routine context budget and assemble the current task, relevant evidence and recent turns deliberately.
2. Use retrieval or a labelled summary for older material, and expand the budget explicitly when the task needs the complete source.

The solution

Keep a short preamble and record what was included. If required evidence does not fit, split the task or refuse the incomplete review instead of silently clipping the source.

How to check it

Inspect request token counts and actual retained evidence over several turns. For a known question, check whether the bounded context still contains its supporting facts before comparing answer quality.

Where this stops

Resent context may receive provider caching discounts, so cost is not always full-price each turn. Large windows do not invariably degrade answers, and summaries can lose essential detail.

The problem

An agent asked for a monthly summary can spend its context fetching pages of records instead of answering the question.

Why it happens

A record-listing API exposes storage access, not the aggregation the user needs. Asking the model to collect and count everything makes pagination, completeness and arithmetic part of its reasoning burden.

How to work through it

1. Translate the question into an explicit time range, grouping and measure, then implement that calculation beside the data access code.
2. Return the grouped answer with the total population, inspected coverage and any unavailable fields, rather than returning an unbounded record dump.

The solution

For “how many tickets by category over 30 days,” expose a bounded aggregation tool. Leave interpretation and possible causes to the model after deterministic counting is complete.

How to check it

Compare the tool with a small known dataset and reconcile group totals with the declared population. Simulate a missing API page and expect partial coverage to remain visible.

Where this stops

Aggregation cannot answer every qualitative question. Keep a bounded way to inspect representative records, and do not describe incomplete collection as a complete report.

The problem

A report can carry a warning about incomplete data and still be summarised as a confident headline number.

Why it happens

The caveat is only information until the consuming step acts on it. A model can overlook a flag or quote the statistic without carrying its conditions into the answer.

How to work through it

1. Make the tool calculate coverage and sample validity itself, and return explicit reasons when a statistic is not supported.
2. Require the reporting step to inspect that status before it can use the number; for stronger guarantees, withhold invalid statistics in deterministic code.

The solution

For a small contaminated sample, return “not enough data” with the count and reason rather than a median plus a warning the next step may drop.

How to check it

Feed a report with a tempting headline and an invalid-sample flag through the consumer. Expect it to refuse the statistic and preserve the reason in the user-facing explanation.

Where this stops

A second model step can improve attention but cannot guarantee compliance. A code-enforced validity boundary is stronger when the rule can be decided mechanically.

The problem

Two starts with the same vLLM memory fraction can leave different amounts of memory available for requests, making a shared card difficult to budget.

Why it happens

The key/value cache holds attention data for request tokens. When its size is the remainder after startup measurements, a fixed memory fraction need not produce a fixed cache allocation.

How to work through it

1. Choose a cache budget with the other servers accounted for, and set kv-cache-memory-bytes on a version that supports it.
2. Keep the total budget for weights, working memory and all caches within the card’s capacity, with room for startup and other users.

The solution

Pin the cache in bytes when repeatable allocation matters. Treat the fraction as planning information once the runtime uses an explicit cache size.

How to check it

After each restart, inspect the reported cache bytes and available token capacity. After an upgrade, check both again; the same byte budget may hold a different number of tokens.

Where this stops

A cache pin fixes one allocation, not the whole process footprint or an operating-system-enforced GPU cap. Context length and concurrency still have to fit.

The problem

An answer can remain wrong after a better reranker is installed because the passage containing the answer never reached that model.

Why it happens

The first stage chooses candidates; the reranker scores only those candidates against the query. Its improved ordering cannot add a missing passage.

How to work through it

1. For a question with a known supporting passage, save the first-stage candidate list before reranking.
2. Check whether the passage is present. If absent, improve candidate coverage through chunking, search method or shortlist size before tuning the second stage.

The solution

Debug retrieval coverage separately from ranking quality. For example, moving the right passage from rank 30 to rank 2 helps only if the first stage returns at least 30 candidates.

How to check it

Measure how often known supporting passages enter the candidate set, then how often reranking places them in the final context. Those checks locate different failures.

Where this stops

A larger shortlist costs time, and better retrieval does not guarantee a faithful answer. Keep answer grounding as a separate check after ranking.

The problem

A small SQL aggregation can fail even when the model knows how to write SQL. Repeating the request with a larger model may leave the real gap untouched.

Why it happens

The agent may lack table definitions, time-window rules or a way to inspect an error. Without those inputs it is solving a different problem from the one the operator has in mind.

How to work through it

1. Give the agent the relevant schema, units and one example row, with the exact aggregation and time range requested.
2. Ask it to show the query and inspect the returned error or a small sample before producing the final summary.

The solution

For a 30-day total, make the date field, timezone and treatment of missing values explicit. Feed query errors back into the same bounded task instead of restarting without them.

How to check it

Compare a small, hand-checkable subset with the query result. Then inspect the complete query’s filters before trusting the larger total.

Where this stops

Context is a useful first diagnosis, not proof that model capability never matters. Reassess after the task, permissions and feedback path are complete.

The problem

A client expecting only an answer can receive the model’s reasoning text, or receive no answer when the token budget runs out before the answer starts.

Why it happens

The model’s output format and the server’s parser have to agree. A closing thinking marker without a matching opening marker can defeat a parser chosen for a different template.

How to work through it

1. Inspect the model’s documented chat template and choose the matching reasoning parser in the serving configuration.
2. Send a short diagnostic request and inspect the structured reasoning field, answer field and finish reason separately.

The solution

Keep reasoning and the final reply in separate response fields. Budget for both phases rather than setting the token limit to the desired answer length alone.

How to check it

The expected response has answer text in message.content and reasoning in its separate field. A length finish reason with empty content calls for a budget check before blaming the client renderer.

Where this stops

Parser names and response shapes depend on the model and server version. Do not apply one model’s parser setting to every model.

The problem

A document-answering system can look impressive when every question has an easy answer in its documents. That test set does not show whether the system knows when its evidence is missing.

Why it happens

Retrieval and answer writing are separate stages. A language model can produce a plausible answer from weak or unrelated passages, so fluent output is not a reliable signal that the documents support it.

How to work through it

1. Build a small evaluation set with questions whose answers are in the corpus and questions whose needed facts are absent. Keep the expected support separate from the generated answer.
2. Inspect the retrieved passages for each question before grading the response. Decide whether retrieval found the evidence and whether the response stayed within it.
3. Require an explicit insufficient-evidence result for the unsupported questions. Repeat the same set when changing the retriever, chunking or model.

The solution

Test the edge of the system's knowledge as deliberately as its happy path. A grounded refusal can be the correct output even when the model could write a longer answer.

How to check it

For an illustrative absent-fact question, expect the system to say it cannot establish the answer from the corpus. It should not turn a loosely related passage into a citation for the missing fact.

Where this stops

A refusal alone is not enough: retrieval may have missed a real answer. Grade retrieval and answer faithfulness separately before calling the result correct.

The problem

An agent handoff can appear detailed while omitting the decision or artifact needed to make the next step executable.

Why it happens

The receiving agent does not share the author’s unstated context. Missing input versions, acceptance criteria or stop conditions force it to search, guess or ask for work already understood elsewhere.

How to work through it

1. State the goal, current state and exact next deliverable.
2. Include the source locations, relevant decisions and checks needed to act.
3. Name unresolved questions and failure conditions rather than leaving them implicit.

The solution

For example, “finish the parser” is weaker than a handoff naming the supported grammar, changed files, remaining failing fixture and command that checks it. The latter gives the next worker a bounded problem.

How to check it

Give the handoff to a fresh reader and ask them to identify their first action and definition of done. Any required hidden fact is a concrete repair to the document.

Where this stops

Self-contained does not mean copying the entire project. Stable links and concise relevant context can be enough, and the source’s cost warning is not a measured universal growth formula.

The problem

Asking a model to own an entire job can hide which decisions need judgment and which should already have a precise answer.

Why it happens

An open-ended request combines planning, execution and checking in one uncertain loop. Routine steps then become fresh language-model decisions instead of repeatable operations.

How to work through it

1. Choose a bounded task where interpretation adds value.
2. Give it explicit input, output and acceptance rules.
3. Let deterministic code own the surrounding data handling, execution and result recording.

The solution

For example, a model can propose an explanation for a failed check while a script gathers the logs and a real test evaluates the proposed repair. The model’s answer is useful input to the process, not its final evidence of success.

How to check it

Compare complete-task outcomes and correction effort across representative cases. Check that a wrong or incomplete model answer is caught before the workflow reports completion.

Where this stops

This is a design preference, not a measured claim that AI is overvalued in every domain. Some tasks need more exploration, but they still benefit from an explicit way to judge the result.

The problem

A useful model answer can fail JSON parsing because of a formatting defect, while a permissively parsed answer can still contain invalid data.

Why it happens

Parsing and validation solve different problems. A tolerant parser may recover a representation, but only strict validation can decide whether its fields, types and values meet the consumer’s contract.

How to work through it

1. Preserve the raw model response for diagnosis.
2. Apply only a defined, bounded set of syntax repairs.
3. Validate the recovered object strictly before any downstream action, and reject ambiguous recovery.

The solution

For example, repairing a raw line break inside a string should not authorize an unknown action field. Keep the original response and record the repair so a later failure can be traced to what the model actually returned.

How to check it

Replay captured malformed answers and planted invalid fields through the real parser and validator. Confirm recoverable syntax succeeds while wrong shape and meaning still fail.

Where this stops

Syntax repair can change meaning, so keep it narrow. Some applications should retry or reject malformed output entirely; tolerant parsing is not permission to guess missing facts.

The problem

An instruction loaded at session start may have little practical effect when the relevant decision happens much later in a long context.

Why it happens

The model must select useful context among many later details. A rule far from the action can be overlooked even when it technically remains in the prompt.

How to work through it

1. Identify the decisions where a particular rule must influence behavior.
2. Place a short, relevant reminder or retrieved rule beside that decision.
3. Keep authoritative rule text in one owner so reminders do not become conflicting copies.

The solution

For example, a reviewer choosing whether a claim is supported should receive the evidence-status definitions with the claims it is judging. A general opening reminder to be careful is less specific to that choice.

How to check it

Test representative long-context cases with the rule only at the start and with the relevant reminder near the decision. Compare actual rule compliance rather than prompt length alone.

Where this stops

Context placement is a reliability aid, not a security boundary or a guaranteed attention rule. Excessive reminders can consume budget and create their own noise.

The problem

An agent can treat a protocol step as optional when the instruction leaves its trigger or failure behavior open to interpretation.

Why it happens

Words such as “consider” describe a preference. During a competing main task, a preference may be skipped because the model has no clear condition that makes the step mandatory.

How to work through it

1. Name the event that triggers the rule.
2. State the required action and the evidence that counts as completion.
3. State what to do if the action cannot complete, including any required stop.

The solution

For example, “Before reporting completion, run the named check; if it cannot run, report the blocker and do not claim a pass” gives the agent a decision it can apply. “Try to test” leaves the important branch unspecified.

How to check it

Exercise the instruction with a successful check and a missing prerequisite. Inspect whether the agent reports the actual result and follows the defined stop behavior.

Where this stops

Stronger wording helps clarify intent, but a prompt remains guidance. Put high-consequence requirements in enforceable gates rather than relying on capitalization or repetition alone.

The problem

A converter and its checker can agree perfectly because both make the same extraction mistake.

Why it happens

If the expected result comes through the same parser or intermediate conversion as the output under test, a shared loss appears on both sides and cancels out of the comparison.

How to work through it

1. Trace how the output and expected result are produced.
2. Read the original source independently for the properties being checked.
3. Use damaged fixtures that alter text, order and structure separately.

The solution

For example, if both paths reorder slides by filename, comparing their text will not expose the wrong presentation order. The expected sequence must come from the presentation’s own ordering relationships.

How to check it

Plant a one-sided defect and a shared-helper defect in disposable fixtures. The check should reveal whether its independence is real or only assumed.

Where this stops

An independent checker is still an implementation with blind spots. Token recall also ignores extra text and order, so a perfect recall score must not stand in for structural fidelity or full correctness.

The problem

A library can fail at runtime even after its Python dependencies installed successfully, because it generates native kernels on the host.

Why it happens

TorchInductor uses a C++ toolchain during compilation. An older compiler may not support the generated code, so the failure comes from the environment beneath the application.

How to work through it

1. Inspect the compiler error and identify the compilation path actually being used.
2. Choose a supported compiler or a documented eager-execution workaround.
3. Apply that choice in the lane’s environment so local runs and CI use a deliberate policy.

The solution

For example, disabling Dynamo can avoid an unsupported compilation path. That restores eager execution; it should be evaluated as a different execution mode, with its output and performance checked rather than assumed.

How to check it

Run the same representative inputs through the chosen mode and compare against the established baseline. Keep pre-existing failing cases visible instead of renaming a matching mixed result as a complete pass.

Where this stops

Compiler requirements vary with PyTorch, backend and generated kernels. The old-compiler failure in the source is a particular environment, not a universal minimum-version rule.

The problem

A carefully written rule has no effect when the agent doing the work never loads it.

Why it happens

Authoring and delivery are separate parts of the system. A file in a vault can be correct while a fresh working session sees only its installed instructions and discoverable skills.

How to work through it

1. Package the rule where the intended agent already looks for guidance.
2. Give it a stable name, trigger and version that a session can inspect.
3. Verify loading in a fresh session instead of relying on the author’s current context.

The solution

For example, a writing deck can be distributed as a skill that points to the current rules and examples. The useful test is whether a new session can find the relevant instruction when the task requires it.

How to check it

Inspect the loaded package and try a small task that exercises a specific rule. Check the baseline first: a failing baseline can make every mutation appear successfully caught.

Where this stops

Distribution does not guarantee understanding or compliance. It closes the missing-context gap; enforcement and review still need evidence of their own.

The problem

An agent chain can lose meaning when every stage interprets and rewrites the previous stage’s prose.

Why it happens

Natural language leaves room for different readings. A schema makes required fields and permitted values explicit, so a handoff can be validated before another program acts on it.

How to work through it

1. Let the human describe the goal in ordinary language.
2. Translate that goal into a checked contract with explicit unresolved questions.
3. Pass structured results between stages and reject missing or invalid fields.

The solution

For example, a task handoff can name an input artifact, required operation and acceptance check as separate fields. The next stage should not need to infer those from a paragraph that mixes instructions with background.

How to check it

Replay an accepted handoff through the consumer and test one with a missing requirement or unknown status. Validation should stop it before execution, with a useful error.

Where this stops

A schema checks shape, not truth or completeness. Human judgment still has a role in resolving ambiguous intent, and prose remains useful as explanation alongside the fields.

The problem

A page-image retrieval job can exhaust memory because its input resolution is far larger than the model’s useful working scale.

Why it happens

Vision models divide images into patches or similar units. More pixels can mean many more units per page, and batching multiplies their memory cost without guaranteeing a better retrieval result.

How to work through it

1. Read the model’s supported preprocessing and resolution guidance.
2. Resize pages while preserving aspect ratio and enough detail for the task.
3. Use an explicit batch bound and measure memory on representative pages.

The solution

For example, a high-resolution scan can remain the archival original while a smaller derived image feeds retrieval. Keep the two roles separate so reducing inference cost does not destroy the source.

How to check it

Compare retrieval on small text, tables and diagrams at the selected size, then measure peak memory for the intended batch. Check the hardest pages rather than only an average page.

Where this stops

There is no universal DPI or pixel limit. The model, preprocessing and page content determine the trade-off, so a smaller image must earn its quality as well as its memory saving.

The problem

A long agent session can spend much of its input budget rereading instructions and context that have not changed.

Why it happens

Prompt caching lets a provider reuse an eligible repeated prefix at a lower input price. The saving depends on how much is reused, cache lifetime and the provider’s current pricing.

How to work through it

1. Put stable instructions and reusable context before frequently changing material.
2. Use the provider’s supported cache boundaries and record cache-read and cache-write tokens.
3. Compare total cost over a representative session, including the initial cache write.

The solution

For example, a stable rule deck followed by the current task can reuse more input than a prompt that inserts changing status text before the deck. Check actual token categories rather than inferring a hit from identical-looking prompts.

How to check it

Compare uncached input, cache writes and cache reads for similar workloads using the correct model prices. Keep output cost separate so the source of the saving remains visible.

Where this stops

Caching is not always the largest lever. Short sessions, changing prefixes and expired entries may gain little, and model choice or smaller context may matter more.

The problem

An open-ended agent workflow makes routine work depend on repeated model decisions, so the same task can take a different path each time.

Why it happens

Many steps already have a precise answer: validate a schema, find changed inputs, run a command, or compare results. Giving those decisions to a model adds variation without adding useful judgment.

How to work through it

1. Separate exact operations from decisions that need interpretation.
2. Script the exact operations and give the model bounded inputs and outputs.
3. Validate the model’s result before the next scripted step acts on it.

The solution

For example, code can collect failing tests and their logs; a model can suggest a cause; code can then run the selected check and record what happened. The workflow owns progress and failure handling.

How to check it

Replay a known case and check that routing, validation and stop behavior remain predictable. Test malformed or incomplete model output as well as a plausible answer.

Where this stops

The 80/20 split is a design heuristic, not a measured ratio or a requirement. Tasks differ in how much judgment they need, and deterministic scaffolding does not make an incorrect model answer true.

The problem

A quality score can remain high while an important property of the output gets worse.

Why it happens

A checker measures only the questions it asks. Text recall, for example, can stay perfect when a converter changes table structure or slide order, because all the words remain present.

How to work through it

1. List the kinds of damage the checker is supposed to detect.
2. Build small examples with one known defect at a time.
3. Confirm that each defect changes the relevant verdict, and add an independent comparison where practical.

The solution

Treat the grader as another system under test. An illustrative table with its columns swapped should fail a structural check even when its text score stays unchanged. The two checks answer different questions.

How to check it

Keep a clean control beside each damaged fixture. The checker should accept the clean case and reject the damaged one for the intended reason.

Where this stops

A mutation proves sensitivity to that mutation, not complete coverage. Independent implementations can still share assumptions, so agreement alone should not be presented as a correctness proof.

The problem

A reasoning-model request can return a successful transport response with no user-facing answer when the output budget is too small.

Why it happens

The model can spend the allowed tokens on reasoning before reaching answer content. In the described API path, reasoning and content are separate response fields.

How to work through it

1. Inspect content, reasoning and the completion status separately.
2. Set a budget that leaves room for a final answer on the intended workload.
3. Treat missing answer content as an incomplete result rather than presenting transport success as an answer.

The solution

If content is null while the reasoning field contains text and generation hit its limit, increasing the answer budget may be the relevant correction. Do not discard the distinction by converting every 200 response into success.

How to check it

Use a representative reasoning request and check that it produces meaningful final content within the chosen limits. Also exercise the capped case so the caller reports it honestly.

Where this stops

Response fields and token accounting vary by provider and model. A larger budget can cost more and does not fix every empty response; inspect the actual completion reason.

The problem

A reporting agent can stall while collecting hundreds of records that a deterministic query could have reduced to a few useful totals.

Why it happens

Raw pagination spends context and model effort on counting and grouping. Those operations have exact rules that code can apply before the judgment step.

How to work through it

1. Define the summary questions and their data scope.
2. Compute counts and groups in code or an appropriate server-side aggregation.
3. Pass compact results and coverage limits to the model for explanation.

The solution

For a monthly ticket summary, return counts and a bounded set of themes rather than asking the model to read every ticket and calculate totals. Keep drill-down available for the records behind a claim.

How to check it

Compare the aggregate with a checked small dataset and confirm the report’s window, filters and completeness. The model should not describe a partial page as the whole population.

Where this stops

The source does not establish a particular time-bucketing implementation. Aggregation still depends on correct queries and access rules, and a compact summary can hide outliers unless the reporting design preserves them.

The problem

A short document’s embedding can describe its metadata more strongly than its useful prose when the header contains long hashes and bookkeeping fields.

Why it happens

The embedding sees the tokens it is given. Front matter can dominate a small file even though those fields say little about the subject a reader wants to find.

How to work through it

1. Parse the front matter as structured metadata.
2. Embed the meaningful document body.
3. Keep identifiers, provenance and filter fields alongside the vector rather than mixing every field into its text input.

The solution

A short note with two long digests can spend much of its token budget on identifiers. Removing those fields from the embedding input leaves more of the representation devoted to the note’s actual explanation.

How to check it

Inspect the exact text sent to the embedding model and its token composition. Then compare retrieval on representative content questions while confirming that metadata filters still work.

Where this stops

Some titles and concise contextual labels improve retrieval. The boundary is deliberate selection of useful context, not a requirement to discard every metadata field or the provenance needed to audit a result.

The problem

Each model configuration can look valid on its own while all the configured instances together reserve more GPU memory than the card can provide.

Why it happens

Splitting configuration across files removes the place where a shared resource limit was previously obvious. Per-file validation cannot establish a cross-file invariant.

How to work through it

1. Load every instance assigned to the same GPU.
2. Sum the relevant memory shares and validate the combined budget.
3. Reject overcommit before starting any instance and show which allocations caused it.

The solution

If three files request fractions that add to 1.67, each fraction can be individually below one while the fleet still asks for more than one card. The validator needs the whole set.

How to check it

Use a fixture whose individual files are valid but whose total exceeds the allowed budget. Expect the fleet check to reject it, then check a valid combined allocation.

Where this stops

A fraction sum is only one invariant. Model weights, caches, runtime overhead and other processes still need headroom; a valid configuration total does not prove the device is currently free.

The problem

A playbook becomes unreliable when it tries to help with every nearby task. Readers cannot tell when to use it or when it should stop.

Why it happens

Without explicit boundaries, the same instructions are applied to cases with different prerequisites and risks.

How to work through it

1. State the task the playbook owns.
2. Name the adjacent work it excludes and the conditions that require rejection.
3. Describe the inputs, expected result and route for a case outside its scope.

The solution

For example, a read-only reporting playbook can explain how to collect and summarise data while directing configuration changes to another procedure. That keeps a request for a report from quietly becoming permission to alter the system.

How to check it

Try a clear in-scope case, a nearby out-of-scope case and a case missing a prerequisite. Expect useful execution for the first and a clear boundary or handoff for the others.

Where this stops

Boundaries need maintenance as tools change. A rejection should explain the missing condition or next route; “out of scope” by itself does not help someone finish the work.

The problem

A wrong answer about a PDF table can send debugging toward the language model even when the wrong numbers were already stored in retrieval.

Why it happens

An index faithfully returns the chunks it receives. If parsing corrupted the table, retrieval and generation can both behave as designed while propagating that error.

How to work through it

1. Read the retrieved chunk exactly as stored.
2. Compare its table values and relationships with the source page.
3. Fix extraction and rebuild the affected content before changing the answer model.

The solution

If a source table says 24 nodes and 9.6 kW but the stored chunk says 12 and 4.8, a more capable model still receives the wrong evidence. The repair belongs upstream at parsing.

How to check it

After reprocessing, inspect the stored chunk again and repeat the same question. Check both that the values survived extraction and that retrieval supplied the corrected evidence.

Where this stops

Clean chunks do not prove the model is innocent. Once the evidence is correct, continue through ranking, context assembly and answer generation to find any remaining fault.

The problem

A document pipeline can pass tidy synthetic fixtures and still break on an ordinary real document. The missing complexity never entered the test.

Why it happens

Real PDFs carry layout and outline structures that small hand-made examples often omit. In the source account, a real datasheet’s outline was flattened to one level.

How to work through it

1. Keep fast synthetic cases for isolated behavior.
2. Add an authorized representative real document to a longer-running check.
3. Verify the structure that matters, such as nested sections, rather than merely checking that parsing finishes.

The solution

For a document pipeline, include a real PDF whose outline is flattened to one level. Check whether the downstream section and table handling still produces useful chunks instead of assuming that the input outline supplies a clean hierarchy.

How to check it

Compare the extracted sections and table chunks with a checked reference for that document and rerun after parser or chunker changes. A green extraction command alone does not establish that useful document structure survived.

Where this stops

One real document does not represent every format. Choose samples for known risks, keep their use and privacy appropriate, and retain focused fixtures so failures remain easy to diagnose.

The problem

Increasing a server’s advertised context length can prevent startup or reduce the number of requests it can serve at once.

Why it happens

The runtime needs cache space for the keys and values associated with input tokens. A larger maximum request consumes more of that fixed memory budget.

How to work through it

1. Read the configured maximum context length and the reported cache capacity.
2. Check that one full-length request fits.
3. Choose the supported context and concurrency together, then verify the deployed configuration.

The solution

For illustration, a cache that holds 32,000 tokens can fit more short requests than requests near a 16,000-token limit. Advertising a limit above the cache’s total capacity cannot be rescued by testing only short prompts.

How to check it

Expect startup to report enough capacity for the maximum length. Then exercise representative lengths and inspect admission or memory behavior before treating a concurrency estimate as operational capacity.

Where this stops

The arithmetic is a planning model. Model architecture, runtime version and scheduling affect the real capacity, and token capacity should come from the running configuration’s report.

The problem

A model server’s memory footprint can change when startup profiling sees a different amount of free GPU memory.

Why it happens

A fixed key/value cache budget reserves a stated number of bytes and skips that profiling choice. The token capacity those bytes provide still depends on the runtime and model.

How to work through it

1. Derive a safe cache budget with other instances paused.
2. Pin the cache in bytes and keep the model and context configuration recorded.
3. After an upgrade, read the runtime’s new token-capacity report before setting concurrency.

The solution

Treat bytes as the reservation and tokens as the runtime’s interpretation of it. If an upgrade reports fewer tokens for the same byte budget, update the concurrency expectation rather than assuming the old capacity still holds.

How to check it

Inspect startup output and the combined GPU footprint. Confirm that at least one maximum-length request fits and size simultaneous requests from the pinned configuration’s report.

Where this stops

Re-derive the budget when the model, context limit, memory share or card changes. A byte pin makes one allocation repeatable; it does not freeze every other source of GPU memory use.

The problem

A campaign total can look precise while leaving out runs that did not record usage. Automated runs are easy to miss because nobody is watching their individual invoices.

Why it happens

The total can only include the usage records that exist. Missing reporting creates unknown spend, not zero spend.

How to work through it

1. Give each run a stable identifier and a usage record.
2. Include automated and delegated work in the same accounting boundary.
3. Label each amount as billed, estimated or notional, and surface missing records.

The solution

Have the run report its own totals when it finishes. A campaign report can then add known amounts and name the runs whose cost is still unknown, instead of silently treating those runs as free.

How to check it

Compare the run list with the usage-record list and investigate unmatched entries. Cross-check one representative record against the harness or billing source before trusting a large aggregate.

Where this stops

Token counts and billed cost are different quantities. Pricing, caching and incomplete streaming records can affect conversion, so an estimate should keep its label.

The problem

A required gate becomes difficult to trust when the same unchanged answer can pass once and fail on the next run.

Why it happens

A language-model judge adds variable interpretation and generation to the scoring step. That variability can obscure whether the candidate itself changed.

How to work through it

1. State the properties the gate must establish.
2. Score machine-checkable properties with deterministic code.
3. Keep judgment-based review visible as a separate verdict with its evidence and limits.

The solution

For a structured output, validate required fields and exact invariants in code. The same input should receive the same structural verdict, leaving a reviewer to judge qualities such as clarity that the schema cannot establish.

How to check it

Run the scorer twice on identical fixtures and expect identical results. Add a known invalid fixture and confirm that it is rejected for the intended rule.

Where this stops

Deterministic scoring is only as good as its rules. Free-form quality may still need human or bounded model review; a stable score does not prove that every important property was checked.

The problem

An embedding server can refuse to start even though it will never generate an output token. Budgeting only for model weights leaves out working memory required by the serving runtime.

Why it happens

In vLLM, the embedding deployment still reserves key and value cache for input tokens. Its configured input length must fit the available cache.

How to work through it

1. Read the runtime’s startup memory and cache report.
2. Check the maximum embedding input length against the reported capacity.
3. Adjust the cache budget or supported input length, then restart and inspect the new report.

The solution

Treat the embedder as a separate memory consumer. A small model sharing a card with a chat server can still need enough reserved cache for its longest accepted input.

How to check it

Expect startup to succeed with capacity for at least one maximum-length request. Then check an input at the supported length rather than relying only on a short test sentence.

Where this stops

This is a serving-runtime requirement, not a claim that every embedding architecture uses persistent generation-style cache. Check the actual backend and version before copying a memory budget.

The problem

A search can find the words on a page and still miss the diagram that explains them. Text extraction drops the positions, shapes and connections that made the page useful.

Why it happens

A text vector represents extracted words. A page-image model represents visual patches, so it can preserve signals that never became text.

How to work through it

1. Parse the document once, keeping both text chunks and page images.
2. Embed the text and page images into separate collections, joined by a stable document identifier.
3. Query both tracks and merge their results with the visual model’s scoring method.

The solution

Keep the text path and add a visual path beside it. For example, a block diagram with very little text can then be retrieved as a page rather than disappearing from the searchable content.

How to check it

Use a question whose answer depends on a diagram. Check that retrieval returns the right page and that the answer points to the relevant visual evidence.

Where this stops

Visual retrieval adds storage and model cost. A retrieved page still needs to be read correctly; finding it does not prove the answer.

The problem

A command-line evaluation can work in a developer checkout and fail on a managed host. The harness supplies its own tool configuration, while the host already has an exclusive managed configuration that controls which servers may start.

Why it happens

The harness assumes it owns tool setup. The managed policy assigns that ownership elsewhere. Adding more per-run flags does not resolve the conflict, and a test against a separate checkout may miss faults in the installed revision.

How to work through it

1. Inspect the supported managed configuration and determine who owns server selection. Check the installed revision and entry point before starting a paid evaluation.
2. When managed configuration is authoritative, exercise that installed route. Refuse a missing or mismatched installation before making model calls.
3. Keep each user’s identity on the supported server path and verify an idempotent installation check. Record the tested revision so later results name the code that actually ran.

The solution

Let deployment policy determine the evaluation entry point. The test should verify the same installation boundary users rely on, while keeping its own data and expected results separate.

How to check it

The source reports that the managed-install check was repeated without changing the installation. For another host, deliberately supply a missing or wrong revision and confirm the preflight stops before evaluation spend.

Where this stops

This concerns the recorded exclusive managed setup, not every tool server or managed policy. A successful installation check is not proof of answer quality, complete isolation or a successful evaluation run.

The problem

A personal statusline was supposed to replace the default. Instead, both printed. The two lines differed in one field, which pointed to two producers rather than one producer repeating its output.

Why it happens

The script used a pipeline to feed input into exec. In this Bash execution path, exec replaced the pipeline child. The parent script continued and printed its own statusline afterwards.

How to work through it

1. Reproduce the duplicate output with a small personal script whose output is easy to recognise.
2. Write a failing test that checks the entire output: the personal line appears and the default line does not.
3. Replace the pipe with exec bash "$user_sl" <<<"$input" so the handover happens in the parent script.

The solution

Use a redirect or here-string when exec needs to replace the current script. A replacement contract needs an exclusive-output assertion; checking that the new output appears leaves half the contract untested.

How to check it

Two failing tests were added before the change. The fix merged to main. The source also records four full-suite failures reproduced on unchanged main.

Where this stops

This correction uses Bash; a here-string adds a newline, so use a suitable input redirect if exact bytes matter. The fix merged, but the privileged deployment remained with the operator and is not established by this record.

The problem

A job that ends at exactly the scheduler's maximum runtime may have been stopped by the limit rather than completed. Raising the ceiling can make the same defect consume more resources for longer.

Why it happens

The scheduler enforces elapsed time, not useful progress. A runaway simulation can keep a core busy, while a broken wrapper can hold an allocation after the real tool has finished.

How to work through it

1. Compare elapsed time with the configured cap, exit reason, CPU use and tool log progress. Separate a long advancing run from a stuck tool and from an idle allocation.
2. Set realistic per-stage deadlines from run history and make missing time limits a visible submission problem. Ensure the wrapper releases resources even if its client disappears.

The solution

Fix the termination and progress mechanism before extending the limit. A longer allowance is justified by evidence of useful work, not by the fact that the previous run reached its ceiling.

How to check it

Use controlled cases for a slow advancing task, a stalled task and a disconnected client. The wrapper should preserve useful work within policy and reclaim resources when progress or ownership is lost.

Where this stops

CPU activity alone cannot prove progress. Some legitimate runs are long, and the historical exact-cap pattern is a diagnostic clue rather than proof that every capped job was hung.

The problem

A client can time out and retry while the first attempt is still consuming downstream capacity. During overload, the extra attempt adds work precisely when the service has the least room for it.

Why it happens

A timeout ends the caller's wait; it does not necessarily cancel the remote operation. Several layers retrying independently can multiply one original request into many overlapping attempts.

How to work through it

1. Map where retries occur and set one overall deadline and attempt budget. Retry only transient failures and ensure repeated operations are safe, using an idempotency key where the service supports it.
2. Add bounded backoff and jitter, and propagate cancellation where it actually works. Limit concurrent attempts so a failing dependency cannot create an unbounded queue of replacement work.

The solution

Treat retries as extra load that needs a budget. A visible final failure can be healthier than silently keeping every timed-out operation alive.

How to check it

In a controlled delay test, inspect downstream attempts as well as client errors. The number of active operations should remain bounded, and a timeout should not cause repeated side effects.

Where this stops

Backoff cannot create capacity, and some servers ignore cancellation. Simulations can expose a risky mechanism but do not establish that production code has survived it; a real integration test is a separate check.

The problem

Summing process memory listings can make a database look much larger than the memory it uniquely occupies. Stopping the largest-looking process on that basis may target the wrong source of pressure.

Why it happens

Resident memory can include pages shared by several processes. Adding those numbers counts shared buffers repeatedly, whereas private memory identifies pages belonging to only one process.

How to work through it

1. Separate private and shared memory in the inspection, using the operating system's available accounting tools. Compare process figures with total host pressure, swap and filesystem-backed memory.
2. Take another sample after an interval to distinguish a growing footprint from stable shared buffers. Check the process role before deciding whether its memory is expected or idle waste.

The solution

Use private memory for process attribution and shared-page-aware totals for the host view. Keep scratch files in the investigation too, because RAM-backed files may not be obvious in one process's listing.

How to check it

For an illustrative comparison, inspect several connections to the same database. Their listed resident sizes should not be treated as independent private allocations of the shared buffer pool.

Where this stops

Private memory alone does not describe every resource a process causes the system to retain. Permissions, kernel accounting and swapped pages can limit what a single snapshot reveals.

The problem

A helper can exit while its temporary files keep consuming a large share of host memory. Looking only at running processes can miss the scratch space left behind.

Why it happens

On a tmpfs filesystem, file data uses RAM and may use swap. A directory called /tmp does not necessarily live on disk, and its contents can outlast the process that created them.

How to work through it

1. Check the mount type and available space on the actual host before choosing a scratch directory. Give each run an identifiable directory and a bounded expected footprint.
2. Clean up completed run directories at the workflow boundary, keeping only the artifacts needed for diagnosis. Confirm ownership and retention before removing anything that another process may still need.

The solution

Treat RAM-backed scratch as a budgeted shared resource. Fast temporary I/O is useful only while its memory cost leaves room for the rest of the machine.

How to check it

Compare scratch usage before a representative run and after cleanup. Expected temporary growth should return near the starting level, with required outputs preserved elsewhere.

Where this stops

Hosts differ: /tmp can be disk-backed on one machine and tmpfs on another. RAM scratch also changes persistence and timing behaviour, so disk-specific correctness checks still need a disk path.

The problem

Blank gaps in terminal output can look like lost words or a broken locale. First separate a problem identifying the character from a problem drawing a valid character.

Why it happens

Encoding determines which character the bytes represent; font coverage and selection determine whether it can be drawn. Readable ordinary text alongside a missing symbol is a reason to investigate rendering, rather than change the locale at random.

How to work through it

1. Identify the affected character and check coverage for its code point with fc-list :charset=<hex>. Compare a working character with the missing one; the related code-point lesson explains this probe in more detail.
2. If coverage is absent, consider a suitable user-local fallback font. If coverage exists, inspect font selection and terminal behaviour before installing more fonts or rewriting locale settings.

The solution

Let the evidence choose the layer to investigate. A valid character with no covering font points to a glyph gap; a coverage result by itself still leaves font selection and terminal rendering to check.

How to check it

Render a short line containing ordinary text and the affected symbols in the original terminal. After any cache refresh or restart, check both the symbols and neighbouring spacing rather than relying on a font preview.

Where this stops

The source records a recommended installation, not a verified fix. This is the same font-coverage case used in the related code-point lesson, viewed as a diagnosis across layers. The two historical log dates do not establish separate observed failures.

The problem

A shared remote desktop can appear frozen even while its server is alive. Restarting that server may destroy unrelated long-running work without identifying what blocked the display.

Why it happens

A forwarded GUI depends on a relay chain between the tool and display. A stalled client holding an input grab is a plausible cause of a wider freeze, but that explanation needs a test.

How to work through it

1. Inspect the display server, forwarded jobs and resource pressure before restarting anything. A normal display query answering does not by itself rule out an input grab.
2. Identify the suspect client and arrange a controlled way to disconnect or stop that job. Compare the desktop before and after, keeping other running work intact.

The solution

Reduce the shared failure domain by placing an interactive tool in its own job desktop and viewing that desktop remotely. This removes the forwarding chain from the main session.

How to check it

The diagnostic expectation is that removing the suspect client clears the freeze. If it does not, keep the cause open and investigate other display or session failures.

Where this stops

The original grab explanation is unproven. Per-job desktops still need working node software, access controls and an idle policy; changing login hosts alone does not test the relay hypothesis.

The problem

A batch launch can unexpectedly start a second copy of user services on a compute node.

Why it happens

A shared home exposes the same user-unit files on several hosts. When a scheduler prolog enables linger and starts the user manager, enabled units can start wherever that manager runs.

How to work through it

1. Inventory enabled services and timers in the shared home.
2. Confine host-specific units with an explicit host condition.
3. Guard newly enabled units with the same rule and inspect for existing duplicate processes.

The solution

Illustrative: a tunnel intended for one home host can start again on a farm node simply because both hosts read the same enabled unit tree. Host conditions express where that service belongs.

How to check it

Check the effective units on an allowed host and a disallowed host. The intended host should run them; the other should skip them without launching duplicate work.

Where this stops

This is host placement, not an authorization barrier. A skipped condition is appropriate for optional placement but should not be mistaken for a failed prerequisite guard on a protected service.

The problem

A batch allocation can be counted several times if each srun step is treated as another submitted job.

Why it happens

Inside an existing allocation, srun launches a step under that job. Step metrics and job-level metrics describe different scopes, so mixing them can duplicate counts or overwrite the job's peak with a partial observation.

How to work through it

1. Retain both job identity and step identity.
2. Define which scope each aggregate counts.
3. Combine memory evidence without replacing a known peak with a smaller partial step value.

The solution

Illustrative: one batch script with three srun steps is one allocation with several steps. Reporting four independent jobs changes the denominator without adding four independent allocations.

How to check it

Inspect representative scheduler records and reconcile job counts with step counts separately. Replaying a partial accounting window should not reduce a previously established high-water value.

Where this stops

srun outside an allocation can obtain its own allocation, and nested submission can create child jobs. Inspect the actual scheduler relationship rather than classifying by the command name alone.

The problem

A background task can vanish when the oneshot service that launched it exits.

Why it happens

An ampersand changes shell waiting behavior, but it does not give the child an independent service owner. Under the normal unit cleanup behavior, remaining processes can be terminated when the unit stops.

How to work through it

1. Keep finite work in the foreground so the service owns its completion.
2. If the task must outlive the launcher, give it a separate appropriate service unit.
3. Inspect unit status and task output together.

The solution

Illustrative: a launcher that starts a long copy with & may exit successfully before the copy finishes. Running the copy as the unit's tracked work makes completion and failure observable.

How to check it

Rehearse with a disposable delayed task. Expect the service to remain responsible until completion and to report a failing task as a failure.

Where this stops

Unit type, KillMode and lifecycle settings affect cleanup. Changing those settings to detach work can create orphan-management problems; model ownership deliberately rather than relying on shell backgrounding.

The problem

A scheduler reason field can create fictitious wait time if its meaning is assumed to be the same in every state.

Why it happens

The field is contextual. A reason attached to a running or terminal job does not necessarily describe time that job spent pending, so summing records without a state rule changes the metric.

How to work through it

1. Define which job states count as waiting.
2. Count intervals only while the job meets that state definition.
3. Keep transitions and missing observations explicit when deriving duration.

The solution

Illustrative: a reason value repeated on a running job should not add another poll interval to pending time simply because the string still exists.

How to check it

Inspect a sample job's state timeline alongside the calculated wait. The measured intervals should lie inside its actual pending periods and reconcile with the chosen time window.

Where this stops

Periodic observations cannot always locate transition instants exactly. Report the sampling boundary and do not mix scheduler wait with a tool's later runtime licence wait.

The problem

A database stored under a temporary directory can lose old relation files while recently accessed tables keep working.

Why it happens

Temporary-file cleanup follows directory policy and file age, not the operator's idea of important data. Cold database files can meet the cleanup rule before frequently touched files do.

How to work through it

1. Inspect the cleanup policy for the actual data path.
2. Move durable data into a location with an explicit persistence contract.
3. Verify backups and recovery independently of whether the database currently answers queries.

The solution

Illustrative: a rarely read table can disappear before a busy table, making the early symptoms look isolated. A healthy query against the busy table does not establish intact storage.

How to check it

Confirm the data directory is outside unintended cleanup scope, inspect required relations and rehearse restoration on a separate instance.

Where this stops

A keepalive touch timer is a stopgap with its own failure modes. Persistence also depends on filesystem durability, permissions, backup retention and tested recovery, not just the directory name.

The problem

A memory history can suggest generous headroom while missing the short peak that actually determines whether a job survives.

Why it happens

A periodic sampler sees only its sampling instants. Shared-page accounting can also inflate summed process memory, so the reported peak can err in either direction.

How to work through it

1. Record the source and interval of each memory metric.
2. Capture the job's cgroup high-water mark where available.
3. Keep sampled curves for timing and shape, while using the appropriate peak for sizing.

The solution

Illustrative: a ten-second burst between two thirty-second samples can disappear from the curve. A continuous high-water counter can retain that maximum after the burst ends.

How to check it

Compare sampled values and the cgroup peak for a representative multiprocess job. Preserve disagreements as different measurement semantics rather than silently overwriting one with the other.

Where this stops

A cgroup peak is only as useful as its scope and collection lifetime. A missing capture is unknown; it cannot be reconstructed exactly from a flat sampled curve.

The problem

A responsive collector can appear healthy while the dataset it owns stops advancing.

Why it happens

Liveness checks show that a process answers. They do not prove that polling, parsing and committing new source data still work, so a broken downstream path can remain invisible.

How to work through it

1. Keep a separate liveness signal.
2. Measure the most recent successfully delivered data for each source.
3. Compare that age with the source's expected cadence and expose a stale state.

The solution

Illustrative: an HTTP health route may return promptly while the latest stored sample is hours old. A freshness check makes the missing delivery visible instead of accepting process uptime as evidence.

How to check it

Stop delivery in a disposable setup while leaving the process running. Liveness should stay healthy and freshness should move to stale after the configured allowance.

Where this stops

A naturally quiet source needs a suitable heartbeat or delivery contract. Do not treat 'no new business event' as a collection failure without considering how that source behaves.

The problem

A service can start even though a supposed prerequisite guard never ran successfully.

Why it happens

A false ConditionPathExists skips a unit rather than failing it. Before= establishes order only; it does not make another unit depend on successful validation.

How to work through it

1. Make the guard perform a real check that exits nonzero on failure.
2. Make the protected unit require the guard and order itself after that guard.
3. Exercise the missing-input path before trusting the dependency graph.

The solution

Illustrative: a missing policy file should fail an explicit validator. Requiring that validator and ordering startup after it makes the intended gate visible in the unit configuration.

How to check it

Start a disposable protected service with valid and missing guard inputs. The valid case should start; a failed validator should prevent protected startup and leave an observable failure.

Where this stops

Requires= alone does not turn a skipped condition into a failure. Startup gating also does not continuously enforce a policy after the protected service has already started.

The problem

A shell check can find the expected text and still make a deployment script stop. That makes a working host look broken.

Why it happens

With pipefail enabled, every process contributes to pipeline status. grep -q exits after its first match; a producer still writing can receive SIGPIPE when the reader closes.

How to work through it

1. Keep pipefail enabled and inspect each pipeline status.
2. Replace the early-exit reader with one that consumes the full stream, such as grep -c, and interpret its result deliberately.
3. Keep a genuine producer error distinct from a missing match.

The solution

Illustrative: a long package listing piped to grep -q can fail only when the match appears early. Consuming the whole listing removes that timing-dependent broken pipe without hiding real errors.

How to check it

Exercise a matching input, a nonmatching input and a producer failure. Expect three correctly interpreted results, including success for the first case.

Where this stops

The failure depends on output size and timing, so a small local example may not reproduce it. Do not fix it by ignoring every nonzero pipeline status.

The problem

An audit record expected to contain one terminal identifier can split across lines during automation.

Why it happens

tty writes a diagnostic to standard output when no terminal exists, then exits nonzero. A fallback echo adds another value without removing the first output.

How to work through it

1. Capture the terminal name only on success.
2. Suppress or discard tty's unsuccessful standard output before supplying a single fallback.
3. Exercise the audit writer through a noninteractive invocation as well as a terminal.

The solution

Illustrative: assigning the output of tty followed by an echo fallback can capture both 'not a tty' and 'none'. Branching on success gives the field exactly one value.

How to check it

Inspect raw records from both invocation modes. Each event should remain one parseable record, and the terminal field should contain either a real terminal name or the agreed missing value.

Where this stops

Suppress only the expected terminal probe output. Other audit failures still need to be visible, and a missing terminal must not be confused with a missing event.

The problem

A long-running session can keep reading an old policy file after deployment has replaced the file visible to new sessions.

Why it happens

A file bind mount can keep referring to the mounted object while an atomic replacement changes what the original pathname resolves to. Updating the pathname does not necessarily replace that existing mount.

How to work through it

1. After changing a mounted policy, inspect the content from inside an existing session as well as through the deployment path.
2. Recreate the affected sessions or their mounts through the supported lifecycle, then check the content visible inside the new session.

The solution

Treat session refresh as part of deployment when file replacement changes policy. A current host-side file is not proof that every already-running session sees the current bytes.

How to check it

In a disposable setup, compare digests before replacement, after replacement in the old session, and in a fresh session. The final session should read the intended deployed version.

Where this stops

An in-place write differs from replacing a file object, and a process may cache configuration separately. Check both mount identity and application reload behaviour before choosing the refresh operation.

The problem

A host can contain several installations of the same tool, making a rule tied to one executable path incomplete.

Why it happens

Installation method and policy scope are separate choices. If supported clients read a higher-precedence managed configuration, that shared boundary can govern their behaviour without controlling every package manager.

How to work through it

1. Identify the documented managed-policy path and precedence for the client versions you support.
2. Deploy the protected configuration there, then test each installation route, including editor-bundled clients, against a policy action that should be denied.

The solution

Keep one supported host policy and a minimum-version rule where needed. Record which clients actually consume it rather than assuming a shared filename makes all binaries compliant.

How to check it

For each supported client, inspect effective settings and attempt a harmless denied action in a test environment. Expect the same denial and useful diagnostics across installation methods.

Where this stops

This is governance for clients that honour the policy. Unsupported versions, bypass modes or users who can alter the enforcement layer need separate controls; a configuration file alone is not a hostile-user boundary.

The problem

A resource limit can appear in a service manager’s configuration while doing nothing on the host where it was deployed.

Why it happens

The manager accepting a property does not establish kernel support, the active scheduler or that the constrained process sits in the intended group.

How to work through it

1. Check the host’s cgroup version, controller availability and scheduler support before selecting a resource-control setting.
2. After deployment, confirm the target process’s scope and use a bounded disposable workload to observe whether the intended control changes behaviour.

The solution

For a legacy I/O weight setting, inspect the scheduler it depends on instead of treating `systemctl show` as proof of protection. Remove or replace a control that cannot take effect on the supported host.

How to check it

Read-back should match the intended configuration, and the behavioural check should demonstrate the specific resource effect. Record those as separate observations.

Where this stops

The source’s RHEL 8/cgroup v1 finding is version-specific. It does not establish that every CPU or I/O controller is ineffective everywhere, and a behavioural test needs a safe scope on shared machines.

The problem

A failing check can print an error while the shell reports success after its output is piped through a display command.

Why it happens

Without pipefail, a shell normally returns the last command’s status for a pipeline. A successful tail can therefore hide a failed command before it.

How to work through it

1. For a Bash gate, enable `set -o pipefail` in the shell that actually runs the pipeline, or avoid piping the gate command.
2. Keep log trimming separate from the command’s captured status when the surrounding runner needs precise failure reporting.

The solution

As an illustrative check, `set -o pipefail; false | tail -n 5` should return non-zero. The failure has no output, which makes the exit status the essential signal.

How to check it

Test a failing producer and a successful producer through the real wrapper. Expect failure to remain non-zero after logging and success to remain successful.

Where this stops

The unchanged title’s “always” is too broad: tail can fail too. Early-closing consumers can also cause SIGPIPE under pipefail, so inspect expected pipeline behaviour rather than blindly enabling it.

The problem

A remote editor connected to a jump host may not support opening a second remote session inside the same window to reach the final machine.

Why it happens

The editor’s remote environment is attached to one target. Nesting another editor transport inside it confuses that model, while SSH already has a way to route through an intermediate host.

How to work through it

1. Create an SSH host entry for the destination with its real HostName and a ProxyJump pointing to the approved jump-host entry.
2. Test the destination alias with SSH locally, then ask the editor’s remote extension to open that same alias.

The solution

For example, a destination entry can contain `HostName target.example` and `ProxyJump jump`. The editor connects to the destination, while SSH handles the intermediate hop.

How to check it

On connection, confirm the terminal hostname and opened filesystem belong to the final target. Verify host keys for both hops and test file access needed by the project.

Where this stops

The example names are placeholders. ProxyJump still requires network reachability, credentials and server policy at both hops; it does not make files mounted only on another host appear on the target.

The problem

A fleet can be built from one standard image process and still run different software versions because its machines were launched from different image generations.

Why it happens

An image captures installed packages and pins at bake time. Without an update or replacement policy, older instances retain those captured choices even when a newer image exists.

How to work through it

1. Record the image version and important package versions with each instance, then inventory the running fleet.
2. Set a rebuild and rollout cadence, including how existing instances move to the new image rather than only changing future launches.

The solution

Treat the image as a versioned release. When a pin changes, decide whether existing machines are replaced or updated and keep that choice observable.

How to check it

Compare a newly launched instance with an older one against the desired release. Check the actual binary versions as well as image labels, because later in-place changes can separate the two.

Where this stops

Instance age is a warning signal, not a version identifier. Patching or replacement can break the age/version relationship, and rollout timing depends on service availability needs.

The problem

A deployment health check can run cleanly while inspecting only a small local subset of the accounts the host actually uses.

Why it happens

Reading a local user file assumes that identity is local. On a directory-backed host, that assumption can omit the people whose setup the deployment is meant to verify.

How to work through it

1. State the host assumptions and use the system’s supported identity lookup rather than treating the local file as the whole population.
2. Run the deployment and its health check on a representative host, comparing discovered accounts and paths with the intended scope.

The solution

Include a directory-backed account in the acceptance case. A script that audits local accounts successfully still needs to demonstrate that the real identity source reaches the audit.

How to check it

Check which accounts were examined, not only the exit code. Exercise a known bad setup for an in-scope directory account and expect the health check to report it.

Where this stops

A live run covers that host and version, not an entire fleet. Keep differences such as directory services, home-directory links and security policy explicit when expanding deployment.

The problem

A long-lived model service can show substantial swap usage after sitting idle. That number alone is not enough to diagnose a memory leak.

Why it happens

The kernel can move infrequently used pages out of RAM. Model weights remain part of the process even when those pages are not resident, so swap records placement rather than proving continued allocation.

How to work through it

1. Observe the process’s resident memory and swap over time, together with its workload and idle periods.
2. Compare repeated equivalent requests and subsequent idle periods; look for continuing growth rather than judging one snapshot.

The solution

Separate retained model data from unbounded growth. If an idle service is unnecessary, an idle timeout may address resource use without calling normal paging a leak.

How to check it

The useful evidence is a time series: does memory settle after comparable work, or keep growing? Include response latency after idle, because paging back in can still affect users.

Where this stops

Swapping can coexist with a real leak or host pressure. A stable footprint does not establish acceptable performance, and a single observation cannot settle either question.

The problem

A shell script can fail before starting a background process, then fail again in its cleanup handler and leave other resources behind.

Why it happens

With set -u, expanding an unset variable is an error. A trap that assumes a process id has already been assigned is unsafe on earlier exit paths.

How to work through it

1. Before registering the trap, initialise every variable it reads, using an empty value for a resource that does not yet exist.
2. Guard each cleanup action with a check that the resource was created; update its variable immediately after creation.

The solution

For example, initialise tpid before installing a trap and only signal it when it contains the tunnel process id. Make partial setup a normal case the cleanup code handles.

How to check it

Force an exit before resource creation and another after it in a disposable run. Both should preserve the original failure while avoiding an unbound-variable error; the later case should remove the created resource.

Where this stops

Initialisation prevents this failure, but cleanup can still fail for other reasons. Avoid signalling an unrelated process and preserve useful diagnostics from the original error.

The problem

A watcher can react immediately to local edits and still miss a configuration file changed by someone on another machine.

Why it happens

inotify reports events through the local kernel. A write made through another NFS client need not generate the local event on which the watcher relies.

How to work through it

1. Keep the local event watcher for quick feedback, but add a bounded polling interval for shared files.
2. On each poll compare the relevant content or version, and send changes through the same validation and apply path as local events.

The solution

Use events for speed and polling for eventual detection across hosts. Record the maximum expected delay so a quiet watcher is not mistaken for proof that nothing changed.

How to check it

On a disposable shared file, edit from a second client and expect detection within the polling bound. Also check that an unchanged poll does not reapply the configuration.

Where this stops

Polling adds delay and filesystem work. NFS caching may affect visibility, so measure the supported arrangement rather than promising instant cross-host notification.

The problem

A long test or data job can make everyone else on a shared machine wait. The command may be correct while the place it runs is the wrong choice.

Why it happens

A shared login or desktop host has other users whose work is invisible to the current shell. Starting all available workers treats shared capacity as if it belonged to one task.

How to work through it

1. Estimate the job from previous runs: CPU, peak memory and duration. When there is no previous run, start with a bounded sample rather than assuming the largest allocation is needed.
2. Check whether the machine is personal or shared and whether a scheduler is available. Apply the configured resource policy to choose a local run or a submitted job.
3. Request the measured resources when submitting. With no scheduler on a shared host, bound the workers and lower priority where appropriate, then observe the effect.

The solution

Make placement part of starting the job. A short note saying where it will run and why is easier to review than a slow host after the fact.

How to check it

Check the actual job allocation and peak use after the run. Use that evidence to size the next run instead of repeating the original guess.

Where this stops

A lower process priority does not limit memory, and an idle-looking host may still be shared. Local convenience is not evidence that a heavy job belongs there.

The problem

A program can fail under a memory limit before its resident memory becomes large.

Why it happens

RLIMIT_AS constrains virtual address space, including reserved mappings. Runtimes such as Node, Java and PyTorch may reserve large ranges that do not yet occupy equivalent physical memory.

How to work through it

1. Distinguish virtual size, resident memory and the limit being applied.
2. Check the runtime’s allocation strategy before selecting a cap.
3. Prefer a supported process-group or cgroup policy when the goal is to bound actual memory consumption across workers.

The solution

For example, a large reserved heap can exceed an address-space cap while resident use remains modest. Raising or lowering that cap without inspecting the mapping confuses reservation with consumption.

How to check it

Run a representative bounded fixture and inspect both virtual and resident measurements. Confirm what happens to child processes and any memory-backed temporary storage under the chosen enforcement.

Where this stops

Cgroup accounting and runtime heap settings also have specific semantics. The cited source leaves cluster enforcement unverified, so this draft does not claim a tested alternative cap on that environment.

The problem

A deployment probe can report healthy because something answers on the port, even when the intended model server never started.

Why it happens

A generic health endpoint identifies availability only as precisely as its response contract. Another service can own the same port and return an equally plausible OK.

How to work through it

1. Check the process and listening port, then query the service’s identity-bearing endpoint.
2. Confirm that the returned model list contains the expected served name.
3. Send a small representative inference request and inspect the returned content.

The solution

For example, /v1/models provides more evidence about a model-serving endpoint than /health alone. A completion request adds another layer by exercising actual inference rather than just metadata routing.

How to check it

Test the probe against a wrong service and against a server with the wrong model alias. Both should fail before the deployment is called ready.

Where this stops

An expected alias can still point to incorrect weights, and one completion proves only a narrow path. Keep checks proportional to the required deployment guarantee, and do not treat a CPU rehearsal as GPU validation.

The problem

A user can be listed in a new Unix group while a shell they already had open still gets permission denied.

Why it happens

A process carries a set of group credentials inherited at launch. Updating the account database does not normally rewrite those credentials in existing processes and their children.

How to work through it

1. Check the account’s configured membership and the current shell’s effective groups separately.
2. Start a fresh login session after the group change, or use an appropriate newgrp session.
3. Retry the intended operation from that refreshed context.

The solution

For example, a newly opened login may read a group-protected file while an old terminal cannot. Changing file permissions again would treat the symptom without addressing the stale process credentials.

How to check it

Compare the effective group list in the old and new sessions, then perform the actual allowed operation. Check the GID rather than trusting a similar group name.

Where this stops

Directory-service caches and host-local group configuration can add delays or differences. newgrp starts a changed shell context; it does not retroactively update every existing application or service.

The problem

A running job can approach its memory ceiling while an administrative request change gives the false impression that more memory has become available.

Why it happens

The scheduler installs the enforced resource limit when the job step starts. In the environment described by the source, changing the request after launch did not resize that running cgroup.

How to work through it

1. Estimate and request memory before submission using representative peak measurements.
2. Check the effective limit on the running step rather than only the scheduler’s requested value.
3. If more memory is needed, use a supported restart or requeue plan with checkpoints.

The solution

For example, a long EDA flow can save a stage-boundary database and resume a later stage in a newly sized allocation. A mid-stage memory peak without a checkpoint may require repeating that stage.

How to check it

On an authorized test job, compare the submitted request with the effective cgroup limit. Verify that the chosen restart path restores the expected state before relying on it.

Where this stops

This reflects the cited cluster behavior, not every scheduler version or administrative configuration. Accounting samples can miss short memory peaks, and a pending request change is not proof of effective enforcement.

The problem

Installing a full desktop editor on a headless host can fail on graphical libraries the remote editing workflow never needed.

Why it happens

A remote development setup separates the interface from the execution host. The local machine runs the UI, while a smaller server on the remote host provides files, terminals and language services.

How to work through it

1. Identify whether the task needs a remote server, a browser service or a full desktop application.
2. Choose the supported server package for the host’s architecture and runtime libraries.
3. Keep the UI on the client machine and connect through the approved transport.

The solution

For example, a headless build host can run an editor’s remote server without opening a graphical session. That avoids introducing a desktop stack just to edit files and run tools near the source.

How to check it

Connect from the intended client, open a fixture file and run a small remote command. Verify where extensions and processes execute rather than assuming everything stays local.

Where this stops

Remote servers still have operating-system and library requirements. This is deployment guidance, not a claim that every desktop package is wrong or that the legacy installation was reproduced.

The problem

A large prompt can fail before the target program starts when it is placed in one command-line argument.

Why it happens

Linux limits individual argument strings as well as the combined argument and environment space. On common 4 KiB-page systems, the per-string ceiling is about 128 KiB, including its terminator.

How to work through it

1. Check whether the CLI accepts stdin or a prompt file.
2. Use that supported input path for large content rather than constructing one huge argv element.
3. Keep environment and total invocation size bounded too.

The solution

For example, a prompt assembled from several documents can cross the limit even though the model accepts a much larger context. Moving the bytes to stdin addresses the launch boundary, not the model’s own input limit.

How to check it

Try a controlled fixture around the expected boundary, then confirm the program receives the complete stdin or file content. Report launch failure separately from model rejection.

Where this stops

The exact per-string limit depends on platform and page size; 131,072 bytes is not universal. File permissions and retention also matter when prompts contain sensitive content.

The problem

Installing a new executable does not prove that a long-running service is executing the new version.

Why it happens

A running process retains its loaded code and mappings. Replacing the file used for the next launch does not generally replace code already loaded into an existing daemon.

How to work through it

1. Check the installed version and the active process separately.
2. Use the service’s supported restart or reload procedure after the upgrade.
3. Inspect the restarted process and exercise the behavior the upgrade was meant to change.

The solution

For example, a versioned launcher can point at a new build while an old background session keeps running. A fresh shell proves the launcher’s choice, but says little about that existing session.

How to check it

Confirm a new process start and obtain version information from the active service when possible. Follow with a small request that reaches the running instance.

Where this stops

Some services explicitly support live reload or re-exec, and interpreted programs may load files later. Use the application’s update contract rather than assuming every replaced file behaves identically.

The problem

A supervisor can keep reporting a live process while the server no longer answers useful requests.

Why it happens

Restart policies react to exits. A hung process has not exited, and a process that fails only after the start-limit window may restart indefinitely without crossing that short-window threshold.

How to work through it

1. Probe useful service behavior with a timeout, not just process existence.
2. Choose a restart budget and observation window that cover delayed failures.
3. Record repeated failure reasons so memory exhaustion is not hidden by automatic recovery.

The solution

For example, a model server may start successfully and run out of GPU memory later. A bounded request probe can expose that failure; a startup-only check cannot. Recovery should include a limit or escalation path.

How to check it

Rehearse a nonresponsive fixture and a delayed-exit fixture. Confirm that each becomes unhealthy and that the recovery policy eventually stops or escalates repeated failures.

Where this stops

Probe failure can also reflect network or dependency problems, so avoid restarting blindly on every timeout. CPU-only control tests do not verify real GPU memory behavior.

The problem

A launcher can overcommit a GPU if it treats a failed utilization query as an empty card.

Why it happens

A missing tool, hung driver or timed-out command provides no measurement. Converting that absence to zero turns an observation failure into permission to start work.

How to work through it

1. Give the device query a bounded execution time.
2. Represent valid readings and unavailable readings separately.
3. Start new work only after the required resource checks return usable evidence.

The solution

If nvidia-smi times out, report GPU state as unknown and stop the admission decision. Do not fill in zero used memory simply because no number was parsed.

How to check it

In a safe fixture, simulate a missing executable, timeout and malformed response. Expect each to remain unknown, while a valid idle reading follows the normal admission path.

Where this stops

A fresh reading still is not a reservation. Another process can allocate memory after the check, so concurrent launchers need coordination in addition to honest state reporting.

The problem

A scheduler’s documented behavior can differ from what an installed configuration actually does. Routing work based only on the manual can put jobs in the wrong partition.

Why it happens

The source records a controlled case where the first listed eligible partition won when both had room, despite the manual’s statement that order was ignored.

How to work through it

1. Inspect the live scheduler configuration and submission wrapper.
2. Submit small equivalent jobs with the partition list in each order.
3. Use accounting records to compare the partition actually chosen.

The solution

For an illustrative pair, submit to interactive,standard and then standard,interactive under comparable availability. Changing only the list order makes the routing result easier to interpret.

How to check it

Check the recorded partition and resource conditions for each job. Repeat enough to separate list-order behavior from one partition simply lacking capacity at the time.

Where this stops

This is evidence about the tested environment, not a universal Slurm scheduling guarantee. Recheck after priority, configuration or version changes before relying on the same routing rule.

The problem

A memory-control setting can kill work when the operator intended only to slow it down, or have little effect because it was copied to the wrong cgroup version.

Why it happens

MemoryHigh on cgroup v2 is a reclaim and throttling threshold. MemoryMax is the hard cap; the older MemoryLimit setting serves the hard-limit role on cgroup v1.

How to work through it

1. Identify the host’s cgroup version and supported systemd controls.
2. Choose soft pressure and hard-stop limits according to the intended behavior.
3. Inspect the effective unit settings and monitor how the service behaves under controlled pressure.

The solution

For a shared service, a soft ceiling can reduce its impact before a hard backstop is reached. Copying a v1 hard-limit directive as if it were a v2 throttle changes that operational contract.

How to check it

Use a safe bounded test workload and inspect reclaim, throttling and termination signals. A setting present in a file does not establish that the running host enforces it.

Where this stops

A soft threshold is not an absolute memory ceiling. A throttled service can look hung, and limits do not repair a leak; pair them with health checks and investigation.

The problem

A model-usage chart can show the total token trend while failing to answer which person made the calls.

Why it happens

The Bedrock metric dimensions in this setup identify model-level usage, not the caller needed for per-person attribution. Aggregation has already discarded that distinction.

How to work through it

1. Use metrics for overall trends and operational signals.
2. Use invocation metadata or an identity-aware gateway for caller-level records.
3. Keep the attribution source and any pricing conversion visible in reports.

The solution

If a daily chart rises, it can tell an operator that usage increased. It cannot divide the increase between two people unless another record preserves their identities and relevant usage.

How to check it

Trace one controlled request from the caller to its invocation record and then to the aggregate metric. Confirm which fields identify the person and which are merely client-supplied labels.

Where this stops

The title refers to this metric setup, not every possible custom metric system. Assumed roles, missing identity fields and price-table assumptions can all limit a per-person cost report.

The problem

A small model-serving setup can acquire more orchestration machinery than the machines themselves need. That adds another system for the team to understand and maintain.

Why it happens

For a small fixed deployment, process lifecycle and per-model configuration may be the main requirements. systemd already provides those host-level building blocks.

How to work through it

1. Use one service template for the serving process.
2. Give each model its own configuration and template instance.
3. Group the instances under a target and validate shared GPU limits before startup.

The solution

Run two models as two instances of the same unit instead of placing several ExecStart commands in a simple service. Each instance can restart independently while the target represents the group.

How to check it

Check start, stop, restart and reboot behavior for the actual host arrangement. Also check the combined memory budget; valid individual files can still overcommit one card.

Where this stops

The source’s concrete judgment concerns a very small GPU setup. Multi-host scheduling, elastic placement or stronger fleet requirements may justify Kubernetes or another orchestrator.

The problem

Clicking or moving the mouse in a shell can print escape sequences or shift the display after a full-screen terminal program was suspended.

Why it happens

A program can enable terminal mouse reporting and then stop before restoring the terminal mode. The shell receives those input codes even though it does not interpret them as mouse events.

How to work through it

1. Check whether a full-screen program was suspended rather than exited.
2. Restore the affected terminal’s modes with an appropriate terminal reset.
3. Resume or exit the program cleanly when that is safe for the session.

The solution

Repair the terminal state before assuming the shell or remote session is corrupt. The important distinction is between the program’s process state and the input modes it left behind.

How to check it

After resetting, click and move the mouse in the same terminal and check that no mouse-reporting codes reach the prompt. This is an expected check, not a confirmed outcome of the historical diagnosis.

Where this stops

The source records a hypothesis and a reset attempt without confirmation. Mouse-mode leakage is one explanation; a persistent problem still needs investigation of the terminal and running programs.

The problem

A tiny job can have terrible memory efficiency and still waste very little capacity. Alerting on the percentage alone makes it compete with genuinely costly jobs.

Why it happens

Efficiency is a ratio. The cost of unused reserved memory also depends on how much memory was held and for how long.

How to work through it

1. Estimate unused held memory over the job’s duration.
2. Rank or gate alerts by wasted gigabyte-hours.
3. Use efficiency as supporting context and explain the sizing issue in a clear sentence.

The solution

For illustration, leaving 0.9 GB unused for an hour wastes 0.9 GB·h. Leaving 300 GB unused for the same hour wastes 300 GB·h. Both may look poor as percentages, but their capacity impact is very different.

How to check it

Check the units and the held-memory definition, then compare representative small and large jobs. The larger resource loss should rise in priority even if its efficiency percentage looks less dramatic.

Where this stops

Peak usage, grouping and shared allocations can make estimates approximate. Keep those limits visible instead of presenting a precise-looking total as an exact bill.

The problem

An Electron app can black out a remote desktop or hang during startup even though its UI code works on a local display.

Why it happens

Chromium’s GPU compositing process can fail on forwarded or virtual X displays. The rendering environment is different from a normal local GPU-backed session.

How to work through it

1. Identify whether the app is running through SSH X forwarding, VNC or a virtual display.
2. Disable hardware acceleration in the Electron shell before startup.
3. Launch again in the same remote environment and inspect both the window and the session.

The solution

Use software rendering as the safe default for the affected remote-display setup. This puts the rendering choice in the desktop shell, where it belongs, rather than changing unrelated page code.

How to check it

Expect the window to render and startup to complete without taking down the X session. A local-machine launch alone does not check this failure mode.

Where this stops

Software rendering may cost more CPU and reduce graphical performance. Keep the setting tied to the supported environment, and investigate separately if the same failure occurs without GPU compositing.

The problem

A background task can be marked complete even though the command it wrapped failed. Reading only the task status turns a scheduling result into a false success claim.

Why it happens

The wrapper reports that it finished managing the task. Its status may not preserve the child command’s real exit code or the contents of the result file.

How to work through it

1. Locate the run’s actual output record.
2. Read the real exit code and the produced artifact.
3. Report completion only at the level those records establish.

The solution

If a wrapper exits cleanly but the verification file records exit 1, the verification failed. An echoed command or a status field containing the word “success” does not override that record.

How to check it

Compare the wrapper status, child exit and artifact for a deliberately failing command in a safe fixture. Expect the reporting path to preserve the failure rather than promote wrapper completion to a pass.

Where this stops

An output file can also be stale or belong to another run. Match it to the current run identifier and inputs before treating it as evidence.

The problem

A budget action can stop production because a filter or threshold is wrong. The affected users may only see access disappear.

Why it happens

The failure costs are uneven: a late cap permits extra spend, while a wrong automatic deny interrupts service. Delayed cost data makes an automatic decision less current than it appears.

How to work through it

1. Identify the cost of a late action and the cost of a false stop.
2. Check the budget scope and the principal that an action would deny.
3. For this risk balance, notify an operator and require review before attaching the deny.

The solution

Use the alert as a decision point. The operator can inspect whether the right tier actually exceeded its budget before changing access, rather than letting a mis-scoped total stop unrelated work.

How to check it

Check the proposed target and budget filter with a safe, controlled case. Verify that an alert alone does not attach a production deny and that an approved action has a documented reversal.

Where this stops

Manual review can be too slow for a strict financial ceiling. This is a trade-off for the stated service risk, not a universal ban on automatic caps.

The problem

An agent run under a user's account can reach more authority than the task prompt mentions. Existing credentials may let a routine helper authenticate to services without any new key being supplied.

Why it happens

The operating-system identity carries access to files, sockets, agent sessions and configured credential providers. A passphraseless key or plaintext token can therefore become an available capability by inheritance.

How to work through it

1. Inventory credential channels by type and scope without printing their values. Include SSH access, Git credential helpers, cloud profiles, active sessions and tokens embedded in settings files.
2. Decide which capabilities the task needs and restrict the execution identity or credential delivery accordingly. Remove secrets from broad configuration and rotate any credential whose exposure is established.

The solution

Review what the process can actually use, rather than only the permissions described in the agent interface. Keep the inventory current when account setup or execution hosts change.

How to check it

Use harmless test identities to verify an allowed operation and a denied one. The expected boundary is that the task works with its intended access while unrelated service authority remains unavailable.

Where this stops

An environment allowlist alone does not remove readable credential files or live agent sockets. Encryption at rest also offers limited separation when an authenticated session has already unlocked the credential.

The problem

A list of blocked download commands does not control where an agent can send data. The same network request may be made through a library, another executable or a child process.

Why it happens

Tool rules see particular invocations. A firewall or mandatory proxy sees traffic at a lower boundary, so policy can apply regardless of which program initiated the connection.

How to work through it

1. Map the required destinations and protocols, including non-HTTP services that ordinary tools need. Place the policy at an enforced network boundary and make its ownership clear.
2. Ensure workloads cannot simply bypass the proxy through a direct route. Log allowed and denied destinations at that boundary, then keep tool-level restrictions as additional guards.

The solution

Give one network control responsibility for outbound access. This reduces the chance that several partial domain lists drift while each appears to cover the whole system.

How to check it

From a controlled test workload, try an allowed destination and a harmless disallowed one through more than one client. The policy should hold across those execution paths.

Where this stops

A proxy controls only traffic forced through it, and domain rules can be difficult for raw TCP services. The historical egress control was still a prerequisite, not a completed protection.

The problem

Text exported from a model can become executable spreadsheet syntax when a person opens the file. An answer that was harmless in a chat window may be interpreted differently in a CSV cell.

Why it happens

Spreadsheet applications recognise leading characters such as =, +, - and @ as formula starts. CSV quoting handles delimiters; it does not reliably tell the spreadsheet to treat those contents as plain text.

How to work through it

1. Apply a spreadsheet-safe transformation at the export boundary to every untrusted text cell. Prefix formula-shaped values with an apostrophe using the policy tested for the target application.
2. Use the same protection for CSV downloads and copy-to-spreadsheet features. Leave stored source text unchanged so display and other exports do not accumulate spreadsheet-specific escaping.

The solution

Treat the export format as a new trust boundary. Model output, user text and retrieved content all need the same handling when they become cells.

How to check it

Open a harmless test export containing each trigger character in the supported spreadsheet application. The values should display as text, while ordinary text and multiline cells remain usable.

Where this stops

Different spreadsheet applications and import modes handle escaping differently. Consider whitespace and control characters in the policy, and test numeric fields separately rather than changing legitimate numbers blindly.

The problem

Rejecting a symlink at the final filename does not stop a path from passing through a symlinked parent directory. A check on the string path can also become stale before the file is opened.

Why it happens

O_NOFOLLOW applies to the component being opened. Earlier components may already have redirected the lookup, so a safe boundary needs control over the whole walk from a trusted directory.

How to work through it

1. Open the trusted root and walk each child directory relative to its parent's file descriptor, refusing symlinks at each step. Reject unsupported names and keep descriptors scoped for cleanup.
2. Open the final entry relative to the held parent, then use fstat to verify the opened object is a regular file. Avoid blocking on pipes or devices while performing the check.

The solution

Bind validation to the object actually opened rather than a path checked earlier. The directory descriptors keep the traversal anchored as names change elsewhere.

How to check it

In an isolated fixture, replace a parent with a symlink and try a special-file leaf. Both should be refused, ordinary files should remain readable, and repeated failures should not leak descriptors.

Where this stops

Platform flags differ, and trusted-root selection is part of the security design. A descriptor walk does not by itself establish file ownership, acceptable permissions or content safety.

The problem

A subprocess often inherits credentials it never needed. A helper that prints its environment or forwards it to another tool can then leak a parent process's secrets by accident.

Why it happens

Environment inheritance is automatic unless the launcher supplies an explicit replacement. Starting with an empty mapping and adding permitted names makes every inherited capability a deliberate choice.

How to work through it

1. Define the variable names the child needs, such as the executable search path and required locale settings. Keep secret values out of the allowlist configuration itself.
2. Build the child environment from that list at the shared launch boundary. Review explicit exceptions and make direct unfiltered subprocess launches fail a structural check where practical.

The solution

Use the allowlist to reduce accidental exposure and make inheritance reviewable. It is a useful default even for helpers that are otherwise trusted.

How to check it

Launch a test helper with canary secrets in the parent environment. Required variables should arrive, and the unlisted canaries should be absent from the child's environment.

Where this stops

This is not process isolation. A same-user child may still read credential files or process metadata, and it may have network access; stronger threats require controls beyond environment filtering.

The problem

A secret removed from the current branch can still be read from an older commit. A clean checkout therefore gives a misleading sense that the exposure has ended.

Why it happens

A force-push moves references. It does not make every copy of the old Git objects disappear, and the hosting service may still serve an object by its identifier.

How to work through it

1. Revoke or rotate the exposed credential first, then check which services and users need the replacement. Keep secret values out of the incident notes.
2. Rehearse the history cleanup in a scratch copy, retain a protected backup and coordinate other clones so they cannot merge the old history back.

The solution

Treat revocation as the control that stops use of the credential, and history cleanup as a separate task that reduces further disclosure.

How to check it

Check that the old credential is rejected and the replacement works only where intended. Scan the rewritten history and ask the host about retained objects when needed.

Where this stops

A rewritten repository cannot recall forks, clones or downloaded copies. Rotation addresses credentials; a leaked name or document needs a separate response.

The problem

Passing a token as a command-line option can expose it to another user inspecting processes on a shared host. Removing it from application logs leaves that separate channel open.

Why it happens

The operating system keeps the argument vector, often called argv, as process metadata. On hosts without process-visibility restrictions, peers can read that metadata while the process runs.

How to work through it

1. Check whether the receiving program supports standard input or a protected credential channel. Use a harmless canary value when testing the launch path.
2. Send the value through that channel and keep it out of the argument list, shell tracing and error messages. Limit who can read any credential file used along the way.

The solution

Keep sensitive launch data separate from the command that identifies the program. This makes process listings useful without turning them into a credential directory.

How to check it

Inspect the running canary process and its captured launch arguments. The canary should reach the receiver while remaining absent from process arguments and diagnostic output.

Where this stops

Process visibility varies by host. Standard input is not a full sandbox, and another process running as the same user may have broader access.

The problem

A budget alert can arrive after spending has already crossed the amount an operator meant to enforce.

Why it happens

Provider budgets use delayed billing information, while token rate limits constrain throughput rather than dollars. A gateway can enforce its own allowance only on requests that actually pass through it.

How to work through it

1. Inventory every invocation path and its identity.
2. Separate delayed budget backstops from request-time gateway enforcement.
3. Check for direct routes that bypass the gateway's accounting and admission rule.

The solution

Illustrative: a harness behind a gateway can be stopped while a separate direct client continues spending. Calling the gateway limit a whole-account cap would overstate its reach.

How to check it

Use a controlled allowance to test admitted and denied gateway requests, then verify that the report states which routes were covered. Record reporting lag separately from enforcement behavior.

Where this stops

Concurrent in-flight calls, price estimation and billing adjustments can affect exact totals. Current provider features and gateway settings must be checked before making a hard-dollar guarantee.

The problem

A local peer-identity lookup can attribute a browser request to the wrong process if it matches only part of a connection.

Why it happens

Closed sockets can remain in TIME_WAIT, and several connections can share an address. A partial match can select stale or unrelated state rather than the live request's peer.

How to work through it

1. Match source address, source port, destination address and destination port.
2. Require the live ESTABLISHED state for this lookup.
3. Resolve ownership only from an unambiguous supported match and refuse otherwise.

The solution

Illustrative: a stale socket with the same address should not compete with the request's full connection tuple. The ports and state identify which connection is actually being examined.

How to check it

Exercise a live connection alongside stale and ambiguous recorded entries. Only the intended established tuple should resolve, while missing or conflicting evidence should fail visibly.

Where this stops

This mechanism is scoped to a supported local topology and visible kernel state. Proxies, network clients and connection races can break the assumption; use an appropriate authenticated identity mechanism for those paths.

The problem

A command-line client can reject an intercepted TLS connection after the same certificate works in a browser.

Why it happens

AWS CLI v2 uses its own certificate configuration rather than automatically inheriting every operating-system trust change. The client can therefore remain unaware of an approved proxy root.

How to work through it

1. Identify the CA bundle the CLI actually uses.
2. Configure ca_bundle or AWS_CA_BUNDLE to include the approved trust chain.
3. Check each intended profile and the noninteractive environment that will run the command.

The solution

Illustrative: updating the operating-system store may repair one client while leaving an AWS login unchanged. Configuring the CLI's explicit bundle addresses that client-specific trust boundary.

How to check it

Repeat a small HTTPS-backed AWS operation with certificate verification enabled. Expect the trust error to disappear before judging credentials or service permissions.

Where this stops

A valid bundle does not grant access or repair an expired session. Keep the approved root source and bundle maintenance clear, since a stale custom bundle can become another source of failures.

The problem

A repository can expose a deleted credential on its first push even when its current files look clean.

Why it happens

Git history retains earlier snapshots. Removing a value from the working tree does not remove it from reachable commits, and sending those commits sends the earlier value too.

How to work through it

1. Scan the working tree and the history that will be published.
2. Check that the scanner actually ran and selected the intended commits.
3. If a credential is found, handle revocation and history cleanup as separate tasks before publishing.

The solution

Illustrative: a token added in one commit and removed in the next still appears when the first commit is inspected. A head-only scan misses it.

How to check it

Use a harmless planted fixture to confirm that the history gate catches an added-then-deleted secret pattern and does not silently skip when its scanner is unavailable.

Where this stops

A scan is a detector, not proof that no secret exists. Rewriting history does not revoke a credential or erase copies already received elsewhere; those require their own response.

The problem

A low-query service can still produce a material bill while sitting idle.

Why it happens

Serverless describes who manages capacity, not whether reserved or minimum capacity is free. A collection can have a standing hourly floor before query and ingestion charges are counted.

How to work through it

1. Compare minimum billed capacity before per-request prices.
2. Estimate the cost of keeping the collection alive across the expected month.
3. Choose a backend whose standing cost and capabilities match the workload.

The solution

Illustrative: an occasionally used knowledge base may spend more on its idle floor than on all its queries. A usage-priced alternative can change that trade-off even if individual requests cost more.

How to check it

Use the current regional pricing and chosen configuration to calculate idle and active scenarios separately. Check actual billing categories after a controlled pilot.

Where this stops

Prices and minimum capacities change. A lower idle bill does not establish equivalent search features, latency, operational behavior or total cost at sustained load.

The problem

An authentication status command can call a token invalid when its request never reaches the authentication service.

Why it happens

Certificate verification happens before the server can judge credentials. A wrapper may flatten that transport error into a generic authentication failure, sending troubleshooting down the wrong path.

How to work through it

1. Read the underlying error before replacing credentials.
2. Check the certificate chain and the client's actual trust store.
3. Install the approved trust root or configure the supported CA bundle, then retry the same request.

The solution

Illustrative: if HTTPS fails with a certificate error while an independently configured SSH path works, changing the token will not repair the HTTPS trust chain.

How to check it

Expect certificate validation to complete before interpreting the service's authentication response. Keep transport success and credential acceptance as separate observations.

Where this stops

A TLS fix does not prove the token is valid. Client trust stores differ, and certificate verification should remain enabled while the missing trust configuration is corrected.

The problem

A query tool can allow writes even though its text begins with SELECT.

Why it happens

A leading keyword is not a semantic security boundary. SELECT INTO and called functions complicate a string-based allow rule, and expensive reads can still exhaust resources.

How to work through it

1. Use a least-privilege database role.
2. Execute each request inside a read-only transaction with a local statement timeout.
3. Retain explicit limits for result size and operations that read-only mode does not constrain.

The solution

Illustrative: a query-shaped string should still be rejected by the database if it attempts a persistent write. The text filter becomes an additional check rather than the only protection.

How to check it

On a disposable database, try an allowed read, a forbidden write-shaped statement and an overlong query. Expect the read to succeed and the other cases to fail within their stated limits.

Where this stops

Read-only transactions do not forbid every side effect, such as advisory locks. Review the permitted functions and connection behavior as well as transaction mode.

The problem

An authorization policy can run out of room as its list of model resources grows.

Why it happens

Enumerating resource identifiers makes policy size grow with the catalogue. A resource-tag condition expresses the tier rule once, keeping the authorization rule separate from the model count.

How to work through it

1. Choose a tier tag and assign it to the supported resources.
2. Restrict invocation through a condition matching the intended tier.
3. Check who can change the tag as part of the same authorization design.

The solution

Illustrative: adding another model to a tier should require correctly tagging its application profile, rather than adding another resource string to every caller's policy.

How to check it

Probe a permitted tier resource, another tier and an untagged resource. Expect only the intended tier to be available through the tested role.

Where this stops

Tag conditions depend on service and action support. Constant policy size does not mean unlimited service capacity, and another unrestricted invocation path can defeat the intended isolation.

The problem

A policy can be well designed and still provide little practical coverage across a fleet.

Why it happens

The configuration, hook scripts and log locations must reach the hosts where work runs. A pilot host proves that one installation exists; shared user data can remain exposed from another ungoverned host.

How to work through it

1. List the hosts and entry points that must be covered.
2. Deploy the complete policy artifact set through the managed installation path.
3. Check the installed files and behavior on the actual working hosts.

The solution

Illustrative: validating a policy on a test workstation does not establish coverage on a desktop used every day. Include both in the inventory and record the observed deployment state separately.

How to check it

For a representative host in each supported class, inspect policy presence, hook availability and an approved behavior check. Missing coverage should remain visible in the report.

Where this stops

One successful sample is not fleet-wide proof. Managed application settings also have a limited enforcement boundary and do not replace operating-system or network controls.

The problem

A private installation can let someone test a newer binary while making fleet version reports incomplete.

Why it happens

A package pin governs its package channel. It does not automatically select every executable a user can launch, even when broader managed policy still applies.

How to work through it

1. Inventory the executable each launcher actually selects.
2. Separate approved evaluation builds from the normal fleet path.
3. Check both the package pin and the managed version floor when changing releases.

The solution

Illustrative: a user-local native binary may run ahead of the managed package. Record that evaluation path explicitly rather than reporting the installed package version as proof of the running version.

How to check it

Launch each supported entry point and inspect its resolved executable and version. Compare those results with the intended fleet and evaluation policies.

Where this stops

A private binary is not proof that all security controls disappear. Managed configuration may still apply across installation methods; the gap must be described at the control it actually bypasses.

The problem

An agent can load apparently useful project instructions that another user has changed on a shared filesystem.

Why it happens

A skill is executable influence over future decisions. Anyone who can alter its loaded bytes can change what the agent is told to read, run or trust, even without changing the main task.

How to work through it

1. Inventory loaded skill paths and determine who can write their files and parent directories.
2. Restrict trusted skill sources to controlled releases with protected permissions, or disable loading from paths that untrusted users can modify.

The solution

Keep reviewed instruction releases distinct from a collaborative working folder. Check the resolved path as well as its displayed name, because a protected-looking entry can point into writable storage.

How to check it

Using a harmless test instruction, confirm that untrusted shared content is not loaded. Verify trusted release identity and how existing sessions pick up or retain a policy update.

Where this stops

Filesystem protection reduces this injection route; it does not make every instruction trustworthy. Review content and other loading paths, and distinguish agent governance from controls a determined local user can bypass.

The problem

A failed lookup of a short hostname can be mistaken for a missing machine when the fully qualified name resolves normally.

Why it happens

The short form depends on the client’s search domains and resolver configuration. A resolver that does not know the relevant private zone cannot infer the intended full name.

How to work through it

1. Inspect the configured search domain and try the expected fully qualified hostname with the active resolver.
2. If it resolves, inspect the address’s reverse record as a consistency check and use the confirmed full name for the next connection test.

The solution

For a placeholder host called build, compare build with build.example.internal before changing DNS or declaring the host absent. Keep the exact name used in diagnostics.

How to check it

Record forward lookup results for both names and any reverse mapping. Then test the intended service separately: successful DNS resolution is only a name-to-address result.

Where this stops

Reverse DNS can be absent or stale, so a PTR record is supporting evidence rather than proof of identity or authorization. Split DNS and VPN routing may also change what each resolver knows.

The problem

Git over HTTPS can stop working under certificate inspection while an SSH remote remains usable on the same host.

Why it happens

The two transports use different trust systems. HTTPS verifies an X.509 certificate chain against trusted authorities; ordinary SSH verifies a server host key against a trusted key record.

How to work through it

1. Identify whether the failing remote uses HTTPS or SSH, and diagnose that transport’s actual trust failure.
2. If SSH is an approved available route, verify the server’s host key through a trusted channel before configuring the remote and client key.

The solution

Use the supported SSH route when its independent trust and access requirements are satisfied. Keep the HTTPS CA problem recorded, because changing the Git remote does not repair other HTTPS tools.

How to check it

Check the effective remote URL and establish an authenticated SSH connection to the intended server. A changed host-key warning should remain an error to investigate, not something the workaround disables.

Where this stops

SSH can still be blocked by routing, host policy or missing authorization. It avoids this particular TLS trust-store dependency; it does not bypass security controls or guarantee connectivity.

The problem

A deployment step intended only to ensure a directory exists can also change the permissions of a directory that was already deliberately private.

Why it happens

install -d applies requested directory attributes to existing directories too. Existence and ownership or mode enforcement are different operations hidden behind one convenient command.

How to work through it

1. Decide whether deployment owns the directory’s policy or merely needs to use an existing parent.
2. If preserving an existing directory is the requirement, avoid applying a broad install mode to it; inspect the parent and create only the missing path under an explicit policy.

The solution

Do not use an unconditional `install -d -m 0755` as a harmless existence check for a directory expected to remain 0700. Keep permission changes deliberate and reviewable.

How to check it

In a temporary tree, prepare a 0700 directory and run the proposed deployment step. Read its mode back and expect preservation if that is the contract; also test initial creation.

Where this stops

A check-then-create sequence can race. Security-sensitive directories may need stronger creation and ownership checks, and deployment should fail when an existing path violates its required policy.

The problem

Several unrelated HTTPS clients can fail certificate verification on the same inspected network, even while the browser connects successfully.

Why it happens

An inspecting proxy presents certificates signed by its own certificate authority. A client that does not trust that root cannot verify the substituted chain; different clients may read different bundles.

How to work through it

1. Inspect the presented chain and obtain the approved inspection root through the organisation’s trusted distribution route.
2. Add it to the relevant trust stores and configure each affected client or container to use the intended bundle.

The solution

Fix trust at the client’s real boundary. A host trust-store update does not necessarily reach a Python package bundle or a container image, so trace each failing path.

How to check it

Retry the original verified HTTPS operation for each affected tool. Check that it succeeds with verification enabled and that the chain still terminates at the intended trusted authority.

Where this stops

One missing root is a useful shared explanation, not a diagnosis for every TLS failure. Expired certificates, hostname mismatches and blocked protocol upgrades require different fixes.

The problem

A smoke test can pass while the shipped HTTP client still fails TLS verification on the same machine and network.

Why it happens

The test may use the operating system’s certificates while the application uses its own bundle. Those are different trust decisions even when both requests target the same URL.

How to work through it

1. Identify the client, bundle and proxy settings used by the real application path.
2. Run the smoke test through that same client configuration, with an explicit supported override for operators who need a different approved bundle.

The solution

Prefer one deliberate trust configuration over a test-only workaround. For example, a successful curl request does not validate an application that silently selects certifi instead of the OS store.

How to check it

Exercise a verified request through the production client constructor and record which bundle it selected. Keep a failure visible when the required inspection root is missing.

Where this stops

Equivalent trust configuration only establishes this part of the path. Authentication, request headers and streaming behaviour may still differ, so keep the smoke test’s claim narrow.

The problem

A machine can answer ping and accept a TCP connection while interactive services remain unusable. Treating those probes as a full health check hides the failure users see.

Why it happens

The kernel can handle network responses while the userspace process that should send a service banner is stalled. The connection and the application response exercise different layers.

How to work through it

1. Measure connection establishment separately from the time to an SSH or other expected service banner.
2. If the banner stalls, gather process state and kernel diagnostics through an available console or monitoring route before deciding the cause.

The solution

Put an application-level response into the health check, with a timeout meaningful to users. Keep ping as evidence of network reachability rather than evidence of a usable login service.

How to check it

The probe should distinguish “cannot connect” from “connected but no banner.” Compare those observations with process waits and kernel messages when diagnosing memory pressure or blocked storage.

Where this stops

A missing banner does not uniquely prove userspace starvation. Service configuration, filtering and a failed daemon can look similar; the deeper evidence determines the diagnosis.

The problem

Giving each maintainer a broad personal privilege makes service access hard to review and harder to remove when responsibilities change.

Why it happens

The process identity and the right to manage it are separate concerns. Mixing both with a person’s account ties the service’s lifetime and authority to that person.

How to work through it

1. Run the service under a dedicated service account and list the exact management actions its maintainers need.
2. Grant those actions to a purpose-made group, then manage access through membership and review the writable configuration that group controls.

The solution

A maintainer role might need to restart one service and edit its validated configuration. It does not automatically need permission to manage every service or launch an arbitrary privileged command.

How to check it

Test an allowed action as a group member and an unrelated action that must be denied. Remove a test member and confirm how revocation takes effect for new and existing sessions.

Where this stops

Some policy tools cannot distinguish every verb. Writable service configuration can itself grant substantial power, so the allowed settings need careful review.

The problem

An append-only table can reject ordinary updates and deletes while still allowing a caller to replace a row. The public API may look safe even though a raw database connection can change its history.

Why it happens

SQLite's replacement path is not an ordinary update. A conflicting insert can remove an existing row and create another one, so guards built around the obvious write statements do not cover every mutation route.

How to work through it

1. Write down what append-only means for the table: existing records survive, their values remain unchanged and new records are added without replacing them.
2. Test conflicting inserts and replacement forms through a raw connection, including conflicts involving the hidden row identity. Do not rely only on the application wrapper.
3. Make the database-level protection refuse those replacement paths, then retain the failing examples as regressions alongside ordinary update and delete tests.

The solution

Reject conflicts before insertion: use a BEFORE INSERT guard that raises ABORT when the new row collides with any existing unique key or explicitly supplied rowid. Keep UPDATE and DELETE guards too; test all conflict keys through a raw connection.

How to check it

Insert an original record, attempt each forbidden mutation and read it back. The original values and row identity should remain intact, while legitimate new inserts still work.

Where this stops

Replacement can be useful for a disposable cache. The problem is using it where the contract promises that earlier records cannot be changed.

The problem

A certificate repair can be hard to judge when the next request fails for an unrelated authentication reason.

Why it happens

TLS verification happens before the application’s authorization response. If a verified connection receives the expected service’s normal access-denied result, the request progressed beyond the certificate problem on that path.

How to work through it

1. Keep certificate verification enabled while applying the intended trust bundle.
2. Send a minimal request that reaches the actual service without exposing secrets.
3. Classify the returned error by layer before changing credentials or application configuration.

The solution

For example, a normal unauthenticated service error is useful progress after a certificate-verification failure. It shows that the client accepted the certificate chain and exchanged application traffic.

How to check it

Confirm the response came from the intended endpoint over a verified TLS connection. Then test authorized access separately if that is within scope.

Where this stops

A proxy can generate its own denial, and disabled verification proves no trust repair. This observation is narrow evidence for that client and connection, not proof that every tool uses the same CA store.

The problem

A WebSocket client can fall back to another transport because the upgrade request received a redirect rather than the expected protocol switch.

Why it happens

A successful HTTP/1.1 WebSocket upgrade requires status 101 and the appropriate handshake headers. A 303 carrying a gateway’s HTML page indicates that some HTTP intermediary or endpoint answered instead.

How to work through it

1. Capture the handshake status, headers and a safe description of the response body.
2. Compare the response with the intended application and known network intermediaries.
3. Check an approved alternate path or transport before changing application code.

The solution

For example, branded filtering HTML is evidence to investigate the proxy path. It explains why the client did not establish a WebSocket even if an ordinary HTTPS request works.

How to check it

Confirm that the intended path returns a valid upgrade and sustains a small connection. Record fallback separately from success on the preferred transport.

Where this stops

A redirect alone does not prove interception; an application or login service can also redirect. Attribute the cause using the response and network evidence, and distinguish HTTP/1.1 upgrade from other WebSocket transports.

The problem

Removing a sensitive document from a search system does not prove that nobody already retrieved or copied it.

Why it happens

Indexing creates an access path and may create derived chunks, embeddings, caches or generated answers. Deletion can close future paths but cannot recall information already delivered.

How to work through it

1. Decide which content and audiences are permitted before indexing.
2. Verify that retrieval enforces the intended per-user or per-group scope.
3. Plan deletion across original documents and derived stores, while keeping exposure uncertainty explicit.

The solution

For example, a connector reading as one application may index documents more broadly than an individual user could read. Search must preserve the required permission boundary instead of assuming source permissions automatically follow.

How to check it

Test an allowed and a forbidden retrieval with representative identities. Inspect what the indexing account can see and what each search user can receive.

Where this stops

A retrieval test proves the cases exercised, not that past exposure never happened. Audit logs can help investigate, but incomplete logs should remain an uncertainty rather than a claim of no access.

The problem

A network policy that works for HTTPS tools can block a hardware workflow that needs a license server or scheduler connection.

Why it happens

An HTTP proxy understands HTTP requests and hostnames carried through that protocol. A client speaking a different raw TCP protocol cannot automatically use the same domain-filtering path.

How to work through it

1. Inventory each required connection by protocol as well as destination.
2. Check which connections the sandbox’s proxy can actually transport.
3. Use an approved network-level path for required non-HTTP traffic with explicit destination limits.

The solution

For example, a license checkout is not a web request merely because its server has a DNS name. The design must account for the protocol and any additional ports the service requires.

How to check it

Test a minimal real operation for each protocol through the intended restricted environment. Empty output or an unstarted wrapper should be recorded as inconclusive, not a network-policy pass.

Where this stops

Some proxies support CONNECT or other tunneling, and some products offer protocol-specific gateways. The limitation concerns the configured HTTP-only filter, not a claim that raw TCP can never be controlled.

The problem

A path restriction can be bypassed in effect when an allowed child program loads the same credential through its own configuration.

Why it happens

Tool rules often match the operation presented to the agent interface. Programs such as Git can resolve credentials independently, so denying one named file read does not limit what an already available token can authorize.

How to work through it

1. Give automation the narrowest practical credential scope.
2. Keep credentials outside the caller’s reach when a broker must enforce per-user policy.
3. Test the actual child programs and credential helpers, not only direct file tools.

The solution

For example, a token limited to one repository remains limited when another client reads it. That is a property of server-side authorization, rather than the spelling of the local command used to obtain it.

How to check it

Use controlled allowed and disallowed operations to confirm the credential’s scope. Also check that alternative authentication paths do not silently supply a broader credential.

Where this stops

Limited credentials can still expose or modify data within their scope. Combine them with operating-system and network controls; a token scope is one boundary, not complete isolation.

The problem

A PostgreSQL connection can leave a disruptive advisory lock behind even when its work ran inside a read-only transaction.

Why it happens

READ ONLY restricts particular database writes, but session-level advisory locks are a separate mechanism. A session lock can outlive the transaction and remain attached to a pooled connection.

How to work through it

1. Use a least-privilege role and an explicit allowed-operation boundary.
2. Exclude unwanted advisory-lock functions from any query interface.
3. Bound query time and clean pooled session state as part of connection handling.

The solution

For example, rolling back a read-only transaction does not make a session-level lock safe to ignore. The next borrower of that connection may inherit its consequences even though no table row changed.

How to check it

In an isolated database fixture, try the forbidden lock operation through the actual read-only entry point. Confirm it is refused and that returning a connection leaves no unexpected session lock.

Where this stops

Transaction-scoped and session-scoped locks have different lifetimes. A read-only fence also needs resource limits for long queries and sleeps; preventing row changes alone does not make arbitrary SQL harmless.

The problem

A local broker cannot safely authorize callers from a username they type into the request.

Why it happens

On Linux, SO_PEERCRED exposes credentials associated with a connected Unix-domain peer. The kernel supplies this identity, so it is stronger than request text or environment variables the caller can choose.

How to work through it

1. Read the peer credentials from the connected local socket.
2. Map the uid to an allowed identity and refuse unknown callers.
3. Have the client verify the expected server uid as well as the server checking the client.

The solution

For example, a credential-holding broker can receive a request from a local user while keeping the service key unreadable to that user. Authorization then uses the peer identity, not an arbitrary act-as field.

How to check it

Connect as an allowed and an unmapped local user, and try a socket served by the wrong account in an isolated fixture. Check that each side rejects the wrong peer.

Where this stops

This mechanism applies to supported local Unix sockets, not remote HTTP requests. Shared accounts still collapse identity, and peer authentication does not replace operation-level authorization.

The problem

A wrapper can know the requesting person while the API behind it still sees one broadly privileged service key.

Why it happens

Authentication answers who is calling at a particular boundary. If the destination authenticates only the key, per-person authorization must be enforced before the request uses that key.

How to work through it

1. Bind the request to identity from a trusted authentication layer.
2. Define the allowed tools, arguments and records for that identity.
3. Check ownership or other scope rules using current server-side data before protected access.

The solution

For example, a caller asking for a ticket id does not establish ownership of that ticket. A service using a broad key must check the relationship rather than trusting a user-supplied email or the model’s description.

How to check it

Test the same operation with an owned and an unowned record, then alter identity-like request fields. Only the authenticated identity should control authorization.

Where this stops

An input hook cannot remove sensitive fields from a response it never inspects. Some broad API operations may need to be excluded or placed behind a stronger filtering service.

The problem

A model can follow a scope instruction most of the time while still making an out-of-scope call when context changes or injected text directs it elsewhere.

Why it happens

A prompt influences the decision maker; it is not an independent enforcement layer. The same model choosing arguments cannot also provide the final guarantee that those arguments are authorized.

How to work through it

1. Keep authentication and authorization in code outside the model.
2. Allow only known tools and argument shapes, then check live record ownership.
3. Refuse unknown tools, missing identity and failed ownership lookups rather than guessing.

The solution

For example, a hook can compare a requested record with the authenticated caller before allowing a service-key operation. The model may explain its request, but its explanation cannot override that check.

How to check it

Try cross-user identifiers, extra scope-changing fields and a failed ownership lookup. Each should stop at the enforcing layer, even with persuasive instructions in the model context.

Where this stops

The boundary also depends on protecting the credential and blocking alternate routes. If callers can read the shared key directly, the hook is a workflow guard rather than complete access control.

The problem

Deleting a credential file can leave a valid secret in snapshots, replicas or other copies outside the client’s control.

Why it happens

Overwriting a file assumes the overwritten bytes are the only readable copy. Network storage can preserve earlier versions, so local overwrite success cannot prove that the credential disappeared everywhere.

How to work through it

1. Identify the credential and the authority that can revoke it.
2. Revoke or rotate it, then update the consumers that still need access.
3. Remove obsolete local copies and inspect known background dependencies.

The solution

Revocation changes what the secret can do, even if old bytes survive. For example, a copied token becomes much less useful once the server refuses it; overwriting one local copy does not provide that protection.

How to check it

Confirm that the retired credential is rejected through an approved check and that intended replacement access works. Record any consumer whose migration remains unknown.

Where this stops

Revocation does not erase previously accessed data, and storage cleanup still matters. Avoid claiming secure physical erasure from a client-side command on a managed or snapshotting filesystem.

The problem

A successful local send can be mistaken for a delivered message, leaving an alert path unproven.

Why it happens

The sendmail command hands mail to a local transfer agent. A receiver later evaluates sender authorization and domain policy, and can quarantine a message after that local handoff succeeds.

How to work through it

1. Separate local acceptance, remote acceptance and inbox arrival in the status report.
2. Use an authorized sender domain and inspect its SPF, DKIM and DMARC setup.
3. Send a controlled test to an inbox you can inspect, including its spam folder.

The solution

Keep the queue id and inspect receiver headers when available. SPF lists permitted senders; DKIM signs the message; DMARC checks alignment with the visible From domain. These checks happen beyond the local command’s exit status.

How to check it

Treat an observed test message and its headers as end-to-end evidence. If only local queue acceptance is known, report delivery as unconfirmed.

Where this stops

Quarantine was a likely diagnosis in the source, not a verified receiver finding. Removing a domain’s protection is not a durable delivery fix, and no test mail was sent for this rewrite.

The problem

A tool restriction can look like a security boundary while leaving the user free to launch the same operation another way.

Why it happens

A cooperative agent follows its tool rules. A hostile caller can choose a different process or client unless the operating system or network enforces the restriction independently.

How to work through it

1. Name the asset and the caller being constrained before choosing the control.
2. Put egress policy in a firewall or controlled proxy, and restrict credentials separately.
3. Try an ordinary child process as well as the agent’s named tools when checking coverage.

The solution

Use agent sandboxing to reduce accidental reach, and enforce the required boundary at the layer the caller cannot edit. For example, blocking a file-reading tool does not by itself stop another program from using an accessible credential.

How to check it

A useful check attempts the prohibited action through both allowed and alternate clients, and confirms that the enforcing layer records a refusal.

Where this stops

The headline is shorthand: hardened sandboxes can resist hostile code too. The warning concerns editable tool policy and incomplete sandbox coverage, not every sandbox design.

The problem

Audit records can name the process account correctly and still fail to identify the person responsible for a request.

Why it happens

When several people reach a service through one Unix account or API key, that shared identity is all the downstream system sees. A typed username does not restore authentication.

How to work through it

1. Authenticate the person before the request reaches the shared service.
2. Carry a stable caller id through authorization and audit records.
3. Record the executing agent or service separately from the person who authorized its action.

The solution

Keep both identities. In an illustrative record, a service executes a change while a separately authenticated user authorizes it. This preserves useful attribution without pretending the service credential belongs to one person.

How to check it

Send two requests as different authenticated users through the same service account. Their audit records should retain distinct callers while showing the same executor.

Where this stops

Attribution depends on the front layer being trustworthy and hard to bypass. Kernel uid alone cannot distinguish people who already share the same account.

The problem

A bot can retrieve a document for one authorized requester and then expose its contents to everyone else in a group conversation.

Why it happens

The answer’s audience is larger than the person asking. Retrieval authorization must account for the recipients who can read the reply, including guests.

How to work through it

1. Establish the audience and its access rules before retrieval.
2. Restrict the evidence to material that may be shared with that whole audience.
3. If that scope cannot answer the question, refuse or move the question to an appropriately authorized private context.

The solution

For example, a private engineering document should not enter a channel answer merely because the person mentioning the bot can read it. The channel’s guest rules need enforcement in code.

How to check it

Test a group with a lower-privilege participant and a document that only some members can read. Expect the group reply to exclude that document and any details derived from it.

Where this stops

This is a design rule, not a claim that the source’s agent already implemented it. Membership changes, cached answers and quoted history also need a clear policy.

The problem

A user can belong to the right Unix group and still receive an NFS permission error. A local group listing does not prove the server received that group.

Why it happens

With AUTH_SYS, the request carries a limited set of group IDs, conventionally at most sixteen. A required group can be absent from the transmitted credentials.

How to work through it

1. Confirm the mount’s authentication method.
2. Check the groups used by the actual request and the server’s group-handling configuration.
3. Design and verify the required access path with the filesystem administrator rather than relying only on a long local membership list.

The solution

For a shared control file, test access as another intended team member on the real mount. A successful owner write does not establish that the group-based operating path works for colleagues.

How to check it

Compare an allowed and denied user under the real NFS setup. Inspect which identity and groups the server uses before changing the file modes.

Where this stops

Some servers resolve supplementary groups themselves, and stronger NFS authentication changes the picture. The source’s team access path was not verified as a second user, so no successful rollout is claimed here.

The problem

A role-assumption request can fail after a trust-template edit because the caller is still missing permission or the existing role never received the update.

Why it happens

The intended cross-role path needs caller permission and role trust. Source templates are not the live IAM policies until deployment reconciles them.

How to work through it

1. Check the caller identity and its permission for the target role.
2. Check the target role’s deployed trust policy.
3. Deploy the intended update to existing roles, then retry using the real caller.

The solution

If the application role is trusted but its caller cannot request sts:AssumeRole for that target, changing trust again will not fix the missing permission. Inspect both sides of the path.

How to check it

Use a controlled assumption request and inspect the returned identity. Also read the deployed policies to confirm that the successful request used the intended role relationship.

Where this stops

Other controls, such as explicit denies, boundaries and organization policies, can still block access. Same-account resource-policy grants have nuances; use the exact IAM evaluation rules for the relationship in question.

The problem

Two budget tiers can appear separate in configuration while the first tier to hit its cap blocks both.

Why it happens

The deny policy attaches to an IAM principal. If both workloads assume the same role, they share the permission change regardless of which budget triggered it.

How to work through it

1. Map each workload to the role it actually assumes.
2. Give independently controlled tiers separate roles.
3. Align budget scope and action targets with those roles, then test isolation.

The solution

A personal assistant and a central application harness need separate role targets if one must remain available when the other is capped. Separate budget names alone do not create that boundary.

How to check it

In a controlled drill, deny one tier and confirm that the other role can still perform its allowed request. Check the actual assumed identities, since a guardrail on an unused role protects nothing.

Where this stops

Separate roles isolate the deny action, not automatically the cost calculation. A broadly filtered budget can still count both tiers, so attribution and budget filtering need their own checks.

The problem

Callers become tied to model releases when each configuration names a concrete version. Upgrading then requires finding and editing every consumer.

Why it happens

The version-selection rule is duplicated across callers instead of owned by one model registry or resolver.

How to work through it

1. Let callers request a stable family name.
2. Resolve that name centrally to an approved, working version for the required environment.
3. Reject unresolved aliases clearly and retain the resolved version in run records.

The solution

A caller can ask for an “opus” family while the registry maps it to the newest working member found by the platform sweep. The caller’s intent stays stable and the upgrade rule changes in one place.

How to check it

Check that all consumers resolve through the same mapping and that an unknown family fails visibly. Record the concrete model used so a past run can still be explained after the mapping moves.

Where this stops

A family alias favors maintained behavior over exact replay. Use a recorded concrete version where reproducibility requires it, and do not promote an untested release merely because it is newer.

The problem

A service with a broadly named read capability can bypass the file modes that were supposed to isolate users’ data.

Why it happens

CAP_DAC_READ_SEARCH bypasses normal discretionary read and directory-search permission checks on reachable local files. It is much broader than access to one intended log directory.

How to work through it

1. Inspect the effective capabilities of the actual service process.
2. Remove the broad read capability when file modes are part of the isolation design.
3. Grant the narrow required path through ownership, groups, ACLs or a dedicated collection location.

The solution

If a collector only needs usage records, put those records in a controlled shared directory. Giving it a capability to traverse private homes expands its authority far beyond the data it needs.

How to check it

Check access as the running service identity, including its effective capabilities. A mode-bit review alone cannot establish isolation when the process can bypass those bits.

Where this stops

This capability does not override every security mechanism. NFS server decisions, mandatory access controls and other boundaries still matter; its broad local scope is the reason to avoid treating it as a small grant.

The problem

Usage logging can make private prompts and responses readable to anyone who has log access. Cost visibility does not require exposing those contents.

Why it happens

Invocation records can contain both operational metadata and message bodies. Enabling them together turns a metering decision into a separate data-access decision.

How to work through it

1. List the fields needed for attribution and cost reporting.
2. Disable content-delivery fields unless there is a separately justified need.
3. Set access and retention for the remaining metadata and verify a real record.

The solution

Keep identity, model, time and token usage where the service supports them, while leaving prompt text out. This gives an operator a way to understand usage without storing the engineering conversation itself.

How to check it

Inspect an emitted record for both required metadata and absent message content. Configuration flags alone are weaker evidence than the record the service actually wrote.

Where this stops

Metadata can still be sensitive and may not provide complete identity in every service. Content capture for agent improvement is a separate decision with its own scope and access controls.

The problem

A plugin may look like a collection of instructions while also launching executable hooks or tool-server processes.

Why it happens

The bundle name hides several different mechanisms. Text can guide an agent; executable components act through the host permissions and process-launch path.

How to work through it

1. Inventory the instructions, commands, hooks and tool servers in the bundle.
2. Review executable parts under the host’s process and network controls.
3. Check actual launch behavior rather than assuming a tool-exposure allowlist prevents execution.

The solution

A tool server can be absent from the exposed tool list yet still be launched by the host. Treat “the agent cannot call this tool” and “this process cannot start” as separate claims.

How to check it

Inspect launch records in a controlled setup and compare them with the intended policy. Verify the specific boundary being claimed instead of inferring it from a configuration label.

Where this stops

Instruction text can still influence decisions and trigger allowed actions. The distinction is about enforcement mechanisms, not a claim that words are harmless or that every plugin has the same capabilities.

The problem

Adding embedding replicas does not help if most requests still reach the same server. A shared DNS name can make this look balanced when it is not.

Why it happens

The clients in the source resolved the name and reused the chosen address. DNS round-robin affected resolution, not the distribution of later requests.

How to work through it