Explainers on how AI inference really works, from the transistor to the token. Plus chip design, tooling, and the lessons I pick up building with agents.
Move across the lattice. Click to send a pulse.
Long reads with diagrams, one way into each subject. Turn the page.
All field guides →inference16 Jun 202613 min
Prefill and decode, why one decode is memory-bound, precision and the KV cache, speculative decoding, the memory hierarchy, and where the CPU re-enters the picture.
Read the explainer →prefill: all prompt tokens at oncedecode: one per step
silicon16 Jun 202614 min
Why GPUs are inefficient for LLM inference, how ASICs go from RTL to silicon, and what Cerebras, Fractile, Groq, Etched, and Tenstorrent are doing differently to overcome the memory wall.
Read the explainer →a systolic array, 6 × 6one cell per clock
agents24 Oct 202630 min
The agents I trust aren't the clever ones. They're the ones whose every decision a script could make has been taken away from them.
Read the explainer →one job walks the graph, station by stationthe lit station is the one at work
One thing I learned, most days. Point at a day in the field, or slide through them, newest first.
261 lessons in 16 weeks
Deployment
A test harness that injects its own tools may bypass the installation users actually run—or fail before it starts. Read the host policy and make the test exercise the supported entry point.
Evaluation
A search test is easier when its questions repeat the documents it searches. Ask in the reader’s words and score the facts a useful answer must contain.
Reviews
A finding can be serious without belonging to this change. Decide both how harmful it is and whether it makes the change’s promise false, then give it a clear next action.
Writing
My casual messages show how I explain a problem. They also contain grammar I do not want carried into a finished post.
Debugging
Brief a worker on one failing value and it may repair only that value. Brief it on the broken rule and its input dimensions so the next sibling failure is part of the same job.
Quality gates
A passing number is weak evidence when the person producing the result can replace the script that measures it. The measuring rule needs an owner as well as a threshold.
Data integrity
If two records share an id but differ in fields the store promises never to change, an ordinary second write becomes a collision. Decide what the identity owns before hashing it.
Telemetry
Adding every usage field in a transcript can produce a neat total from incomplete counters. Cross-check one complete session before using that total to price a change.
Writing
My October 24 ruling allows I found or I built for work an agent did on my assignment, while the source names the run that did it.
Testing
The tests installed a fake aws command, but the launcher had already selected a real binary through AWS_CLI_BIN.
Tooling
Every make command became a small guide: explain the fix when it fails, or show the follow-on commands when it succeeds.
Bash
The personal statusline appeared, but the default appeared too. The handover test only checked that the personal output existed.
Testing
A clean worktree exposed a bug that the main checkout had been hiding with an untracked symlink.
Testing
The suite had tested the refusal against fakes. A supervised host pilot checked the behaviour where the update would actually run.
frontend
Character counts guess. The rendered width decides whether a column wraps or overflows.
docs
A few key-value fields at the top let a person or an agent identify a file before reading the rest.
eda
Predicting from the size of the input files made forecasts worse. The tool's own log of design size was the signal.
process
My work stalled while I slept. Now a reversible call is made and logged as what, why and cost if wrong; only irreversible ones wait.
art
Code that draws can be valid and still ugly. A figure is finished when the audit passes and a person has seen the render.
slurm
Runs stopping at the partition ceiling were hung, not done. Raising the limit to unlimited only let them hang longer.
agents
The description decides when it fires. Put the steps in the body; steps in the description only blur the trigger.
security
Passphraseless SSH keys, plaintext git credentials, cloud profiles and tokens left in settings files all come along with the user's session.
css
Components come from Radix, shadcn or your own set. Borrow their structure and behaviour, but colour, type and spacing should still come from your tokens.
ci
Then every CI run tests the real bootstrap on a fresh machine, and the setup steps can't quietly rot.
process
People only record mistakes if the record can't be used against them. A captured mistake should make the team ask why, not who.
process
A lessons-learnt review at the end is short and tidied by hindsight. Record the fix or the slip while the detail is still there.
frontend
The old design on screen was a cached bundle. Compare the served file to the build before debugging the code.
process
Decisions made in conversation get lost or overruled by older drafts. Write each one where the work queue lives.
design
Take every feature it must carry and ask the hard questions: ten times the input, a second user, a failure nobody notices. Gaps found now cost an edit.
design
A design choice shows a requirement is vague or too costly; a requirement shows a missing block. Write them together, in short loops.
systems
A timed-out request is retried while the first attempt is still running downstream. Under load, that doubles traffic exactly when it hurts.
design
A feature is robust when repeated whys end at a recorded decision or policy. If they don't, you've found a decision nobody made.
linux
Listings count shared buffers once per process, so a database looks enormous. Private memory shows what each process really holds.
security
An allowed-domains list inside a sandbox drifts and is hard to audit. The proxy already sees every connection.
linux
Files in tmpfs count against RAM and can dwarf a process's own footprint. Clean up scratch after every slice.
planning
Before you build is the cheap window. Lay everything out while being wrong still costs an edit; once a project ramps up, every mistake costs more and the team ends up firefighting.
ledgers
One writer checking itself is the opposite of what makes a blockchain trustworthy. The writer needs independent checkers, and a mirror it can't rewrite.
reporting
A block with no regression results isn't passing. No evidence means unknown, never green.
code
Think of code as a tree of roads. Every dead end should close with a clear status and a message saying why.
agents
Rules written for imagined problems pile up and get ignored. Write the rule after the mistake, and say which one.
linux
Blank gaps in the terminal were font coverage, not locale. fc-list :charset=<hex> shows which fonts have the character.
security
Cells starting with = + - or @ execute in Excel. Prefix them with an apostrophe, on the export path only.
linux
A symlink earlier in the path is still followed. Safe walks open each directory relative to its parent's descriptor, then fstat the final file.
llm
One chat API fronts many vendors, yet one model family rejected max tokens below 16 while others accepted 1. Test every model.
build
EDA regressions spend their time in single long tool runs, and caching only helps once every input is declared.
typography
Name the exact glyph (say U+23FA), check which installed fonts cover it, then add a fallback font in your user font folder. No root needed.
build
When every file includes a hub header, touching it invalidates everything. Skip-based tools like Bazel or Snakemake save little until such hub files are split.
riscv
The common path rarely breaks. Compliance is judged on corner cases, so enumerate them explicitly during analysis instead of hoping tests stumble on them.
testing
A mocked page test shows the UI asks the server the right thing. Only live data shows the server answers it correctly.
security
Stripping variables cuts accidents. It doesn't stop a process that goes looking for another one's environment.
llm
Most of its cost is the agent writing its own long system prompt to cache. Tiny requests carry a fixed overhead.
process
A ticket says 'done' until someone edits it. Evidence tied to a hash goes stale by itself when an input changes.
python
After an in-place edit within the same second, Python reused the cached bytecode. Run mutation checks with python -B.
git
A trailing slash in .gitignore matches only directories, so a symlink named .venv slips into 'git add -A'. Write '.venv'.
building
A missing 10% is rarely a reason to rebuild the other 90%. Wrap it, extend it, or write a plugin.
testing
In a simulator, time is exact. On a busy machine, a sleep before the next step is a flaky test that only passes on a quiet day.
evals
Constrained random works in verification because a reference model knows the right answer. For LLM output, that means an eval set with known answers, or a judge.
interfaces
So version them, and let both sides cope with the old shape for a while.
eda
A GUI tool in a detached session keeps its licence until a cut-off or time limit, long after the user walked away.
frontend
Rename a backend field and the frontend still compiles. Generate types from one schema so a mismatch goes red.
rtl
Text search loses hierarchy and connectivity. A proper SystemVerilog parser that builds a syntax tree gives you the design's actual structure to reason about.
linux
One stalled hop in a forwarded display chain can freeze a whole shared desktop. Remove the forwarding; moving users doesn't help.
design
One screenshot gave the wrong read on my own site's style. Sample widely before calling a pattern a pattern.
design
Strict ledgers and gates are taste for systems people rely on. The customer should feel the reliability, not carry the rules.
art
Watercolour or oil textures need raster generation. Vectors suit line and low-poly diagrams, and they diff and review like code.
frontend
A route that returns 200 with the wrong shape passes a status check. Checking the shape is the same test, extended.
aws
Budget tag filters need the user: prefix. Get it wrong and there's no error, just a budget that never moves. Prove it with real spend.
git
They stay reachable by ID until the host garbage-collects. Rotate the leaked secret; rewriting history is cleanup, not the fix.
frontend
Highlights are alarms. Keep them under a rate ceiling, or a permanently lit row trains people to look away.
security
Every user on the host can read argv. Pipe secrets and sensitive names in through stdin.
css
Plain CSS on custom-property tokens, and no utility framework beside it. Two systems drift, and nothing tells a new page which one to use.
protocols
A slow peripheral holds PREADY low to stall the access phase, so logic that assumes every APB transfer takes two cycles breaks.
docs
Nothing checks the prose of a hand doc, so it drifts. Generated docs are fluent, cited and sometimes false. Both need a drift test.
aws
Only token rate limits, and budgets lag by hours. The only hard stop is a gateway in front, and only for traffic that goes through it.
networking
Matching a local connection to a user by address alone can pick a stale closed socket. Match the full four-tuple and the ESTABLISHED state.
ai
Output quality follows input quality, so the real work is the grounding and orchestration around the model.
python
A probe can hang forever unless you set one. Also catch HTTPError, which urllib raises for any non-2xx response.
linux
With linger on and an NFS home, every batch job started my user services on the compute node too.
tls
It ships its own CA bundle, so a proxy root added to the OS does nothing. Set ca_bundle or AWS_CA_BUNDLE.
rag
Adding a reranker made retrieval worse, and navigation chunks caused refusals. Grounding is curation.
frontend
Restate the decision in terms of what the reader came to do. The answer usually picks the option.
frontend
One sentence first, then figures, then tables. Make readers dig and they will quote the wrong number.
frontend
Phone-width overflow only showed up in live viewport checks. Unit tests don't render anything.
reviews
Each round finds small issues in the last round's fixes. Without a severity cap, a two-line fix grows into thousands of lines.
security
Before the first push, check every commit. A secret deleted last week is still in the repo.
eda
A step that combines several stages keeps each earlier stage's licence to the end. Splitting the step frees them sooner.
eda
An lmgrd up/down check stays green while the vendor daemons that actually serve the features have failed. Check health at daemon level.
data
Scraping logs to guess which job used what is fragile. Have the flow record the job id and its parent when it runs.
postgres
Types, upserts and time zones differ. A test on the real engine found migrations that couldn't rebuild the live database.
teams
Building stopped being the slow part. Agreeing on direction is, so talk it through together and early.
rag
Route its writes through an inbox and a review gate, with provenance tags, before they become things it answers from.
eda
Two identical runs give different files because of the header. Hash a copy with the header stripped.
eda
Tools that need several licence features at once saturate together, and the rarest one caps everyone.
thoughts
AI is the hands: fast, tireless, good at patterns. The human is the slower brain that holds the taste and makes the call.
data
Peak-only history makes a short spike look like constant saturation. Time spent at full use is what should drive a capacity decision.
rtl
When a vector unit runs apart from the scalar core, dispatch, hazard checks and retirement must keep architectural state in program order, or software sees results out of order.
slurm
Only sbatch makes a child job. Count steps as jobs, or miss their peaks, and the accounting skews.
design
If the architecture can be described without naming components, swapping one is cheap. I swap them often.
postgres
The parent's statistics stay empty until an explicit ANALYZE, which can hide planner and counting bugs for weeks.
data
Mark the items in the ledger first, then build the message from exactly those, so a retry can't send twice.
data
An empty result makes you look. A plausible wrong match looks like it works and quietly poisons everything downstream.
linux
fcntl locks belong to the process, so opening and closing the same file anywhere else in your code quietly releases the lock.
python
A subprocess or sync DB call inside `async def` freezes the event loop, health checks included. Push it to a thread.
python
'exit_code or "not recorded"' turned a clean exit into missing data. Test for None, not for falsy.
frontend
Metrics the database worked out and the page threw away were wasted effort. A lot of value was already there, unshown.
thoughts
I don't need every detail, but I need the mental model before I trust a system. A true black box is a liability.
linux
When the service's main process exits, systemd cleans up the unit's processes, so work launched with & silently never finishes.
slurm
A scheduler's reason code explains why a pending job waits, but on a running job it means something else. Summing it as wait time is wrong.
postgres
The next request on that connection inherits it. Use SET LOCAL inside a transaction instead.
eda
A job gets its node, then waits on a licence the scheduler never knew about, holding memory and cores the whole time.
learning
With something new I start from a working example and take it apart until I understand each piece.
aws
OpenSearch Serverless bills capacity for as long as the collection exists. Compare standing hourly floors before per-query prices.
linux
systemd-tmpfiles ages out files under /tmp and /var/tmp. A database kept there was deleted, starting with the tables nobody read, so it looked harmless at first.
tls
gh auth status reports any failure as a bad token. Check for a certificate error before you log in again.
rag
With several versions in one index, the top hits were drafts 0.5, 0.4 and 0.2 instead of 1.1. Clean out old versions before loading.
debugging
I couldn't prove why a host wedged, but I could prove my service wasn't the reason. Label what you measured apart from what you inferred.
python
On a path you're not allowed to read, it re-raises the permission error. One unreadable file killed whole poll cycles.
frontend
The #fragment never leaves the browser in requests or server logs. The query string ends up in both.
frontend
The UI showed durations like 1.5h and the filter rejected decimals. Whatever the app displays, its inputs should take back.
slurm
It is sampled every 30 seconds, so short peaks vanish, and multi-process jobs can read up to 2x high. The cgroup high-water mark is the true peak.
security
A blocklist that trusts anything starting with SELECT lets writes through. Use a read-only transaction and a statement timeout instead.
frontend
It builds Vite with configFile:false, so proxy and allowedHosts settings silently drop. Use the Astro server block, or astro dev.
frontend
A missing value rendered as 0.0%. Keep null and zero apart all the way to the pixel.
python
An unknown query parameter is dropped and the call still returns 200, so one table quietly ignored a filter its sibling honoured.
typography
A spec like 'Inter 18pt' is honoured through the font's optical-size axis and leading, not by literally blowing the text up.
postgres
The plain view recomputed everything on every read. Its materialised twin cached the work and turned max() into an index seek: 12 s down to 21 ms.
postgres
GREATEST(NULL - x, 0) returns 0, so a missing value reported a confident zero waste. Handle NULL before you compare.
sql
Sorting on a column with few distinct values let the same rows land on two pages. A unique tiebreaker makes paging stable.
postgres
Check a dump by listing what's inside it, not by its exit code.
frontend
A client compiled cleanly while calling four endpoints the API never defined. Only a test against the server's real routes catches that.
css
Every colour, gap and radius comes from a token. One place to change the theme, and no one-off values creeping in.
aws
Listing each model in an IAM policy hits the size limit at around fifty. Matching a tier tag scales to any number.
design
One loud colour, one focal point per view. Use it everywhere and nothing stands out.
eda
Timing and physical checks hit peak memory in phases that can't be checkpointed, so an OOM means starting over. Size from a comparable run.
ops
A collector passed every liveness check while no new data arrived. Check that fresh rows landed, not just that the process answered.
frontend
Wide tables get merged cells and an 'all columns' view, not hidden data. A pretty table that drops a column is a wrong table.
design
Owning a file doesn't make editing it safe. Count its consumers; a file with many readers needs care whoever owns it.
security
The design was never the weak part. The strong policy simply never reached the machines people actually worked on, so distribution is the control.
data
Host-local summer time put the same wait on two different dates in one report. Labelled UTC keeps day buckets stable.
typography
Small, dense type reads calmer. Weight, case and spacing do the ranking without shouting.
eda
The coordinator mostly hands scenarios to worker jobs, so its memory request skews right-sizing. Its own launch log, not job shape, shows which workers it spawned.
git
A pure `git mv` commit keeps `git log --follow` working. Mixing content edits into the move can break rename detection.
typography
Mono for labels, ids and code. A proportional face for long text. Pick type by role, not by mood.
security
Same fact, seen twice: whatever lets one user run ahead of the fleet also runs outside the fleet's policy.
docs
It goes stale the day the pin moves and nothing fails. Point to the file that owns the version instead.
agents
Show an LLM a raw 429 and it invents its own retry plan and narrates it. Rate limits are a platform job.
systemd
A false ConditionPathExists skips the unit and reports success, and Before= only sets order. A fail-closed guard needs Requires=.
shell
grep exits as soon as it matches, the writer gets SIGPIPE, and pipefail reports the whole pipe as failed.
git
A mechanical reformat kept as its own commit protects git blame. Squash it into the feature and blame points at the formatter again.
shell
tty prints 'not a tty' to stdout and still exits non-zero, so you get two strings. It only shows up in non-interactive runs.
config
An older pinned binary silently ignored the newer form of a setting. The docs describe the latest; your fleet runs something else.
design
On my pages, prose gets paper and ink, and code gets a terminal frame. The frame tells you what kind of text you're reading.
data
A reader can't tell a live writer from a dead one, so only the writer should recover a crashed store.
agents
Multi-agent works when each agent gets a bounded context and a clean handoff. Without that discipline, one agent does better.
security
Anyone who can write to that folder can write instructions your agent will follow. Lock the folder or don't load from it.
postgres
It executes the query to time it, so wrap writes in a transaction and roll back. Add BUFFERS to see page counts, not just time.
agents
Across audits, requirements backed by a check stayed true and prose-only governance didn't. Move rules into checks.
teams
AI multiplies a small team once linters, rule decks and checklists are in place. Without them it multiplies chaos.
linux
Sessions that mounted a file keep seeing that version even after it is replaced on disk. Restart them after an update.
ops
You don't have to win the argument about how people install a tool if a managed policy file applies to every copy on the host.
thoughts
The AI ecosystem pulls everything to Python today. For systems-level infrastructure, where speed starts to matter, Rust is worth a serious look.
process
If the job wrapper adds the metadata every job needs, nobody below has to remember it, and the data comes out clean.
data
A storage layout is cheapest to change while nothing is committed and no corpus exists. After that, every split is a migration.
linux
BlockIOWeight on RHEL 8 needs the CFQ scheduler, which was removed, so it reads as protection while enforcing nothing. Reading the value back proves acceptance, not effect.
data
A AND B OR C means (A AND B) OR C in most query languages. That silently dropped a date filter. Brackets make the intent explicit.
networking
A short name can fail while the full name resolves. A reverse lookup is a cheap way to confirm.
agents
Agents should check their own output and correct against the error, the way a controller corrects against a sensor. Open-loop agents drift.
docs
A handover full of 'then we tried' is history. The next maintainer needs the current design and how to run it.
shell
Pipe a failing check into tail and the gate goes green. Use set -o pipefail, or don't pipe the gate.
tooling
Remote editors open one remote per window. To reach a host behind a jump box, put ProxyJump in your SSH config.
docs
The doc bugs I traced came from one fact living in two places and ageing at different speeds. Give each fact one home and link to it.
networking
SSH trusts host keys, not X.509 certificates, so git over SSH keeps working when HTTPS tools fail under TLS interception.
docs
The five Ws, the exact commands, the steps in order. The reader who needs a doc most has the least context.
linux
`sudo cat /root-only/*.log` fails because your own user can't list that directory. Wrap the command in `sudo sh -c`.
ops
A machine built from an old image runs old pins forever. In a baked-image world, how old a host is decides which version it runs.
evals
Word recall was 1.0 while list nesting and emphasis were broken. A word count can't see structure; structure needs its own check.
data
Resolution times have long tails, so the mean misleads. Give n, coverage and method alongside any figure you report.
ops
A health check read the local user file while real accounts lived in the directory service, so it audited almost nobody. Only the live run showed it.
yaml
Bare no, yes, on and off become booleans, and an unquoted id: 42 becomes an int. Identity keys need a type, not a guess.
teams
With few programmers, the next reader of your code is often an agent. Written standards make its reading guaranteed rather than lucky.
python
A byte-order mark survives a plain utf-8 read and breaks the parser on line one. One codec name saves a confusing SyntaxError.
linux
It applies mode and owner even when the directory already exists, so a deploy step quietly loosened a 0700 directory to 0755.
data
Two lists kept by hand drift apart. Derive one from the other and the question goes away.
llm
The whole context is re-sent on each turn, which raises cost and degrades answers. Default to a modest window and opt in to the huge one.
data
Backfill with NULL or UNKNOWN, so missing stays distinct from zero and later readers can tell the two apart.
art
I like drawings where you can see the strokes: line art, low-poly, diagrams that look drawn by hand. A diagram that looks drawn reads as a diagram.
docs
In CommonMark `# Drain procedure #` renders as 'Drain procedure', so a trailing # in a title is silently lost.
design
Pure white on a screen glares. A cream background with dark ink reads like a page.
design
A graph view is a projection of a store. Hand-drawn edges go stale; edges aggregated from the data can't.
agents
Paging through raw records burns tokens and context. A tool that answers 'how many, grouped by what' directly is cheaper and harder to misuse.
tooling
The format target existed, but nothing ran it in verify or CI. A tool that isn't wired into a gate isn't enforcing anything.
data
Split a vocabulary too finely and tags scatter; leave it open and synonyms pile up. Both break grouping, so keep the list closed and short.
agents
'Sample too small' flags do nothing if the agent stops at the headline number. Sometimes you have to force a second step.
tls
An inspecting proxy re-signs traffic. Install its root once and point each tool's bundle at it, and git, pip and containers all recover.
tls
The smoke test passed with the OS certificates while the real path used its own bundle and failed behind the proxy. Test the path you ship.
process
Tool-specific waiver files scattered around hide who accepted which risk and why. A real waiver has an owner, a reason and a date.
eda
Engineers copy the last project's scan scripts and edit them, so nobody owns the flow and a fix has nowhere to live.
build
Headers pulled in through an include directory were left out of the key, so a stale result was served as current.
networking
The kernel answers ICMP and completes TCP handshakes even when userspace is wedged. Only SSH or VNC banner timing showed the host was stuck.
inference
The same vLLM setting gave different KV cache sizes on different starts. Pin the cache in bytes when you need a fixed footprint.
security
A service account runs the process. A purpose-made group may start, stop and edit its config, and nothing else. People join and leave the group.
integrity
Permissions say who may write. Only a content hash tells you whether anything actually changed.
linux
A long-lived model server with nothing to do gets its weights paged out. It looks like a leak and is usually just the kernel tidying.
frontend
curl and server-to-server calls ignore CORS. And an error response without CORS headers shows up in the console as a CORS failure.
shell
Under set -u, a trap that reads an unset variable fails itself, and the cleanup never runs.
design
If the output can't be checked and fed back to the input, you can't tell whether it works. That loop is my test for 'ready to build'.
rag
It scores query and candidate together, but if the right chunk wasn't in the first-stage top-N, no amount of reranking brings it back.
aws
The source profile worked behind the proxy and the role profile failed TLS. Each profile needs the setting.
accessibility
A red dot means nothing to a colour-blind reader or on a grey printout. Add a shape, a pattern or a word.
agents
A plain SQL aggregation shouldn't fail. When it does, the cause is usually missing schema or bad orchestration, not the model.
testing
Exit non-zero catches ignored errors. Exit zero while doing nothing catches code that treats an exit status as proof of state.
linux
On NFS, a file changed from another client fires no event. Watching across hosts needs a poller.
tooling
People trust stale data. Put the truth next to the work so updating it is a side effect, not a chore people skip.
testing
If the test was written from the same wrong reading of the spec, it passes with the code. Check what a test assumes, not just that it's green.
sqlite
WAL coordinates writers through shared memory on one host, which a network filesystem can't provide. On NFS, pin the rollback journal mode.
inference
Some models close their thinking with </think> but never open it. The server needs the matching parser to split the thought from the reply.
testing
A test that quietly skips on a missing dependency can stay unrun for the life of the project. Make skips loud, or count them as failures.
make
Each recipe line in a Makefile runs in its own shell, so an early exit on one line doesn't stop the next.
build
Compiling won't catch a module importing a name another change removed. The gate has to import every module.
protocols
If the sender waits for ready and the receiver waits for valid, both stall forever. The source must raise valid on its own and hold it steady.
rag
Ask something outside the corpus. The system must say 'not found' rather than invent, because the grounding layer is where quality is won.
ops
Size the job and send heavy work to the scheduler. With no farm, run gently. Only on your own machine do you just run it.
design
AI implements a design well and invents one badly. I set the direction and the templates; the agent fills them in.
agents
Work that can't be traced to a spec or a requirement can't be trusted, however good it looks.
shell
It returns the link's folder, not the script's. Use readlink -f to find where the script really lives.
sql
SQLite deletes the clashing row and inserts a new one, so a guard that blocks UPDATE and DELETE can still be bypassed.
docs
Content inside a list item can start a new block, so the text '- - -' turns into a thematic break and the bullet deletes itself.
building
A stub 'for later' is a ghost: it looks like a feature and does nothing. Half-built is worse than missing.
sta
The cause can be the RTL, the constraints, the library or the layout. Working out which is the real job.
docs
To make it sound like you, give it your own writing samples and rules a reviewer can check, not adjectives like 'sound human'.
git
With file modes ignored, a chmod +x never reaches the commit, and the script fails on the next clone.
synthesis
A synthesis run with simple SDC says little about real timing. Results are only as real as the constraints and libraries you feed in.
postgres
DROP and CREATE make a new object with default privileges. Re-grant in the same script, and test reads, not just denials.
frontend
Tooltips need a pointer. Put the hint in the placeholder or a help line people can reach on any screen.
eda
An interactive synthesis or lint session can't be repeated. A script can, and a script is something an agent can run too.
tooling
A template update reported success while silently keeping the template's side of conflicted files. Diff the result; don't trust the exit code.
tls
If an unauthenticated request gets a plain 'access denied' back, the handshake succeeded, so the certificate problem is fixed.
thoughts
The complexity is real. The gatekeeping isn't. More people in means more ideas and faster progress.
agents
An agent should finish a task from the handoff alone. Missing context isn't the agent's failure, and it multiplies token cost.
linux
Node, Java and PyTorch reserve big address ranges up front, so a tight cap kills them before they use anything.
lint
Hardware integration is slow, so a mistake found there costs days. Lint and static formal move the check to where it is cheap.
design
A showcase page can have theatre. A page someone works in for hours gets one short ease and nothing else. Every page needs a reduced-motion fallback.
design
The design that makes a cleaner block diagram is the one you will still understand in a year.
inference
An unrelated service holding the port returns OK. Check /v1/models for the expected model name, then send a real completion.
data
Reordering slides only changes sldIdLst in presentation.xml; slide1.xml keeps its name. Resolve each r:id, and shuffle fixture order so tests can catch it.
networking
Upgrades only accept 101. A 303 with a branded HTML page came from an inspecting proxy, not the server.
linux
Add yourself to a group and existing sessions still lack it. Log in again, or use newgrp.
protocols
On AXI, data moves only when both are high. Assuming the other side is ready 'by now' is a bug waiting for a slower clock.
docs
Docs split into runsheets, references and registers, plus frozen decision records for why. Mix them and every one gets worse.
security
You can't know who already searched it. Decide what goes in up front, because removing it later doesn't undo the exposure.
systemverilog
SystemVerilog is two languages: a small synthesisable subset and a big verification one. A construct can pass simulation and fail synthesis.
systemverilog
Flops in always_ff, next-state logic in always_comb. Mixing them is how latches and sim-versus-synthesis mismatches sneak in.
networking
Domain filtering works through an HTTP proxy. Licence servers, schedulers and SSH speak raw TCP, so they get blocked.
security
Deny rules match text, and child programs like git read their own config. A narrowly scoped token stays narrow whatever reads it.
verification
An SVA fires on the cycle a rule breaks, not three blocks later where the bad data finally shows up.
slurm
The cgroup limit is fixed when the step launches, so even an admin edit after start doesn't change what is enforced. Get the request right before submitting.
ai
People ask it to do their whole job. It earns its keep as one bounded step inside a larger, mostly deterministic process.
tooling
Remote editing needs only a small server on the host; the UI stays on your own machine.
llm
Raw line breaks and invalid escapes are common in model output. Tolerant parsing followed by strict validation keeps good answers without trusting bad ones.
postgres
Session-level advisory locks are permitted and outlive the transaction on a pooled connection, so a read-only fence must deny them by name.
python
Its number highlighter split '$0.04' with ANSI codes. Turn highlighting off for text that has to stay intact.
security
SO_PEERCRED gives the peer's uid, stamped by the kernel, so a local service can identify users without a login. Clients should check the server's uid too.
llm
Later context dilutes early instructions. Inject the rule at the point where the decision is made, not only at the start of the prompt.
reviews
For a merge, use reviewers who didn't write the code and ask for zero blockers. Self-review only finds what you already looked for.
verification
A tracker status is someone's claim. A trace computed from the repo and test markers can be checked, and fails on any requirement no test names.
eda
Strings like V-2023.12 and T-2022.03-SP1 have no total order, so compare them for equality, never with >=.
make
AS, CC and CXX are GNU Make built-ins and are never empty, so '--as $(AS)' quietly became '--as as'.
linux
Linux refuses a single argv element over 131,072 bytes, so a big prompt passed as an argument has a hard ceiling. Use stdin or a file.
agents
A protocol step loses to the main task unless it is written as a commitment, with a named trigger and stop behaviour.
evals
If the checker and the converter use the same extraction, their mistakes cancel and the score reads perfect.
security
When an API authenticates the key and not the person, it can't scope by user. The layer in front has to own identity.
security
Scope has to be enforced by code the model can't talk its way past, like a hook that re-checks arguments and live ownership and fails closed.
testing
When a checker dies before finishing, its findings are a third state. Scoring them as false quietly drops real bugs.
config
It means no one thought about what might change. Put it in config, and let the code and the docs both read it from there.
inference
TorchInductor compiles kernels at runtime, and an old gcc 8.5 can't build them. Disabling Dynamo works, but compare eager output against CI before trusting it.
agents
Distribution matters more than how good the rule is. Put it where the agent already reads, or it doesn't exist.
testing
Most tests should be fast and mocked. A few unmocked tests on storage, auth and timing prove the mocks aren't lying.
rtl
Parity and duplicated logic for functional safety shape the whole block. They are hard to bolt on later, so they belong in the first draft.
frontend
A new client sending requests the old server rejects breaks for everyone during the rollout. Server first, then client.
agents
Humans talk to agents in prose. Agents should talk to each other in schemas, because prose degrades with every hop.
dft
Pushing ATPG coverage up meant chasing each uncovered fault class one by one. The percentage says how far you are; the holes say what to do next.
inference
Feeding 300 DPI pages to a vision retriever trained on small fixed-size images explodes patch count, time and VRAM. Resize first and batch to bound memory.
docs
Bit ranges like [31:0] look like Markdown link text. A converter that treats every bracket as a link mangles every register table.
linux
Long-running processes execute the unlinked old file until they restart, so an upgrade isn't live just because the file changed.
design
A wrong choice is fine if the module can be swapped out. Coupling is what turns a wrong choice into a rewrite.
build
Skipping unchanged steps saves minutes for a compiler and real money for an LLM step, where every rerun is billed.
python
oa|oasis matched '.oa' and truncated the file extension. Put longer options first or add a boundary lookahead.
verification
Constrained random stimulus covers what you didn't think of; directed cases pin down the edges you did. A test plan needs both.
frontend
Sharing a filter control pulled a whole page's code into another chunk, which grew six-fold. Check chunk sizes when you share components.
llm
Agent sessions resend the same long prompt every turn. Cached reads cost a small fraction of fresh input, so most of the bill falls away.
testing
The seed replays any failure, one snapshot keeps a race from looking like a bug, and a simple evaluator that shares no code says what's right.
unicode
It folds ligatures, superscripts and full-width forms, which is great for matching extracted text and lossy for content you keep.
agents
Agents inside a mostly scripted workflow are predictable. The boring version, deterministic steps with AI in the loop, beats the fully autonomous one.
building
A gateway that moves requests but not identity leaves the real problem where it was. A thin wrapper you own can do the one thing that matters.
eda
Before an agent can help with chip design, the tools need clean CLIs and structured output. Making the flow scriptable is step one, not a nice-to-have.
python
A duplicate path silently shadows the later handler. Nothing warns you, so test that each route reaches its own code.
inference
systemd only reacts to process exit, so a hung server needs a probe. A model that runs out of GPU memory minutes after start never trips the start limit and restarts forever.
python
os.kill targets the whole process, so the kernel may deliver it to any thread that hasn't blocked it. signal.pthread_kill aims at the thread you mean.
career
Broad enough to design the whole system, deep enough in one area that people come to you. In a fast-moving field, breadth is underrated.
build
Content addressing proves you fed the same things in. Outputs repeat only if the tool is deterministic, and LLM steps often aren't.
security
The filer may keep copies and snapshots. Revoking the secret is what actually helps.
evals
Whatever a checker doesn't compare can regress in silence. Point a check at the grader too, or its blind spots become yours.
python
In SystemVerilog the ports say what goes in and out before anything runs. Type hints do the same for a Python function, and a checker can read them.
lint
Teams wrote RTL their own way and paid for it at integration. A shared rule set that runs on every block fixed what review kept missing.
design
A system should fit on a block diagram, and the code should be laid out like the diagram, so a reader knows where to look without grep.
python
The name added to your namespace is the top package. Any tool that resolves or patches imports has to follow that rule.
build
Reruns with no code change still differed, because released IP, flow scripts and tool versions weren't declared. Declare them before trusting a cache.
The local mail server accepted it. SPF or DMARC checks at the receiver can still quarantine it silently, with no bounce.
security
It constrains a well-meaning agent. Stopping a hostile user is the network layer's job, at the firewall or proxy.
security
When everyone runs as one account, the log can't say who did what. Per-person identity has to be kept above it.
process
Asking 'what could go wrong' gets polite answers. Declaring that the project failed makes finding reasons the task, and makes doubt safe to say.
rtl
In RTL, logic with no state is checked by its truth table. Software that keeps state at the edges and pure functions in the middle gets the same gift.
llm
It spends the budget thinking and has no answer left. The empty content looks like success; the reasoning sits in a separate field.
llm
LLMs summarise well and count badly. Do the heavy aggregation in code and hand the model only the part that needs judgment.
design
Load-testing a diagram shows what saturates first. It never runs your code, so it can't find a wrong answer.
eda
People forget updates and skip steps. Tools should be deterministic recipe runners, with AI only judging, converting or fixing around them.
testing
A 'my change does nothing' result was the test harness cloning the wrong tree. Harness faults look exactly like tool limits.
testing
If test values collapse to equal numbers, a bug that swaps two fields is invisible. Use distinct values like 33.3 and 50.0.
testing
Several gates passed over real damage before anyone checked they could go red. Inject the fault, watch it fail, and record what you injected.
gpu
A hung driver or a missing tool says nothing about the card. Treating silence as free lets you start work on a busy GPU.
slurm
An experiment showed the first listed partition wins when both have room. Test scheduler behaviour; don't just read it.
security
Its answer is visible to the whole channel, so retrieval must be limited to what every participant is allowed to read.
process
The customer keeps the chip; the company keeps only how it was delivered. So capture lessons on the spot, not in a hindsight review.
systemd
On cgroup v2, MemoryHigh slows a service down. The older v1 MemoryLimit is a hard OOM kill. Know which one your host has.
rag
On small documents the metadata header can be most of the tokens, so the vector ends up describing the header, not the content.
design
A muted ink at about 3.3:1 sits below the 4.5:1 that small text needs. Check the dimmest token, not just the body colour.
llm
Most companies can't fine-tune on their own data. Retrieving the right document into context gets most of the benefit.
ops
Token metrics show trend by model but carry no caller. Per-person spend needs the invocation log or a per-key gateway.
linux
AUTH_SYS sends at most 16 group IDs per request, so permissions granted through any group beyond those fail silently.
postgres
Postgres checks the CREATE permission before existence, so the 'safe' statement fails for a role without it, even when the schema is already there.
protocols
The spec forbids it, so an address generator has to split any burst that would run past a 4KB line.
postgres
Postgres read naive datetime parameters in the session time zone, so a summer-time shift moved every window by an hour. Pass time-zone-aware values.
sql
`state='FAILED' AND exit_signal=0` is NULL when the signal is missing, not true, so the warning badge never renders.
testing
A AND B must sit inside A. count(A) plus count(NOT A) must equal the total. Relations like these catch parser bugs cheaply.
inference
Each model's memory fraction looked fine on its own; together they asked for 1.67 cards. Validate invariants across files before anything starts.
integrity
Name a file by the hash of its bytes and the name tells you where it's stored and whether it changed. Git and build caches both work this way.
art
Thin ink lines, visible characters and one filled accent, never a polished photo-sharp finish. Code-drawn SVG keeps how the picture was made on show.
frontend
Drawing a chat window is easy. Owning sign-in and conversation storage is the real work, and the real risk, of a custom front end.
aws
The caller's identity policy and the role's trust policy must both allow it. Editing a trust template does nothing until deploy updates existing roles.
git
Linux sees two files; a case-insensitive checkout sees a collision. Check case-folded uniqueness wherever filenames are generated.
git
Group, setgid and other mode intent is lost on clone, so it has to live in a package or install script.
frontend
Scrolling the element into view hides what a real first visit shows. Test at the real screen size, from the top.
aws
A budget deny attaches to a principal. Two tiers sharing a role means the first to overspend blocks the other.
ops
Kubernetes pays off at fleet scale. For a few servers, one unit template and one config file per model are simpler to run and to explain.
linux
Suspend a program that turned mouse tracking on, and the shell receives escape codes every time you click. A terminal reset fixes it.
typography
Each weight is another file on every page load. Fonts from a CDN also fail on locked-down networks, so self-host when needed.
knowledge
One giant graph is hard to navigate and many loose ones drift. Keep one per source and connect them like netlists at their ports.
testing
Run the suite serially, with four and sixteen workers, and shuffled. Hidden order dependence only shows when the order changes.
postgres
The server backend keeps running after the client dies. Set statement_timeout on ad-hoc queries instead.
design
A reading page can be the same theme turned down: same palette and fonts, fewer colours, almost no motion.
aws
Callers that use a family alias never need a config edit when a new version ships. The mapping moves in one place.
linux
CAP_DAC_READ_SEARCH skips read and directory-search checks everywhere, so isolation by file modes only holds against processes without it.
process
Mix what it will do with what it does now and you'll be issuing corrections. Two columns keep the claim honest.
frontend
An impossible filter returns no rows, which reads as 'nothing matched'. Reject it on both sides with a message.
security
Prompts are private, and anyone with log access could read them. Metadata alone still attributes usage and cost to each person, cheaply.
security
An instruction file can't widen permissions; a hook or tool server can run anything. Lock each down by its own risk.
agents
In scope, out of scope, reject when. A skill without boundaries does a little of everything and nothing reliably.
data
A tiny job at 10% efficiency wastes nothing worth a ping. Gigabyte-hours wasted is what costs money, so alert on that.
rag
A wrong answer about a PDF table came from corruption at parse time. Swapping the LLM would never have fixed it.
testing
Each rung catches what the one below can't. A green mock says the logic holds; only the top rung says the world agrees.
sta
At 2.6 GHz the period is about 385 ps, shared by PLL jitter, clock-tree skew, variation and setup time. When one share grows, another must shrink.
rtl
Core and accelerator can each pass their own checks while the handoff between them breaks. List the boundary cases early and synchronise retirement across both units.
rag
Two changes passed synthetic tests and broke on a real datasheet, whose PDF outline was flattened to one level. Keep a real-document soak test.
testing
Hundreds of tests checked error wording, so every copy edit broke them. Assert on a code or a field instead.
linux
Chromium's GPU process can crash under ssh -X, VNC or xvfb and black out the whole X session. Disable hardware acceleration there.
git
A reviewer can read 500 lines line by line. Past that, review turns into skimming. Generated files don't count toward it.
networking
Python clients resolved the name once and stuck to one replica. A real load balancer in front spread the load.
growth
Holding a position is a strength until it outlasts the evidence. Repeating without new support is the tell.
ops
A background run reported success because its wrapper exited cleanly. The file it was meant to write said otherwise.
debugging
ddmin guarantees that dropping any one difference stops the failure, not that no smaller set exists. Both ends must reproduce before you start.
inference
vLLM must fit one full max-length request (length times per-token KV) or it refuses to start. Longer context means fewer requests fit at once.
games
Plan the move, predict the response, adapt. Thousands of hours of that loop turned out to be debugging practice.
inference
A fixed byte budget skips memory profiling and makes the footprint repeatable. How many tokens those bytes hold changes between versions, so re-read it after upgrades.
ops
Lateness is bounded, visible and recoverable. A misconfigured automatic stop cuts people off with no human in the loop and no explanation.
music
Grade 8 piano: themes, variations, tension and resolution. The same shapes show up in how I lay out a system.
gpu
A write that completed earlier is still not visible to a reader unless the two sync through an acquire-release pair. Clock order is not memory order.
llm
Cost was visible only for runs that logged their tokens. A run that doesn't report spend is spend you find out about later.
process
'All the error handling everywhere' is too broad to test and easy to miss. 'Error handling and logging for one part' has a clear edge and a clear test.
testing
A find-and-replace over real names rewrote the fixtures too, and tests failed for no real reason.
evals
A deterministic scorer gives the same answer twice. A model judge can flip on a rerun, and a gate that flips isn't a gate.
design
Bright lime works as a button fill but is unreadable as small text on white, so the text role gets a darker ink in light mode.
testing
Tests written after the code tend to describe the code. Written from the plan, they describe the intent and can't be bent to fit.
testing
Callers must tell 'the answer is no' from 'I couldn't ask'. One exit code for both hides a broken check.
security
'No git here, so nothing to check' let changes through. When a guard can't find its subject, that's an error, not a pass.
eda
A user's own directory is the least reproducible input. Local config tweaks are fine, but SDC, UPF and netlists need a controlled, traceable source.
inference
They generate nothing, but the model still stores keys and values for every input token, so an embedder with too little cache won't start.
design
If adding a feature means changing the core pipeline, the integration point is wrong. Fix the seam first.
config
A config value forwarded as raw options lets whoever edits the config run anything the tool allows. Name the options you accept.
frontend
A green bundle says nothing about type errors. Run `tsc --noEmit` as its own gate.
gpu
Tracking every reader's clock costs too much memory, so only a sample is kept. Races with old reads slip by, and a wrapped clock buffer reports races that never happened.
data
Date deadlines go overdue on paper whenever a schedule slips, which trains reviewers to clear the flag instead of the debt.
testing
A live eval where every case was 'not run' scored a pass. A check that selected nothing has proven nothing.
learning
So I rewrite borrowed code until it fits my mental model. That costs time, and sometimes the original was fine.
git
The release order that works: changelog, commit, tag locally, run the full gate, then push. It was found by running it, not by planning it.
linux
root_squash maps root to an anonymous user, so even uid 0 or CAP_DAC_READ_SEARCH can't read other users' private NFS homes. The server decides.
thoughts
It comes from having seen enough good and bad work. AI can copy preferences; it doesn't build taste.
rag
Embed page images too, as patch vectors, so charts, tables and layouts that text extraction loses can still be found.
debugging
Grepping its output for clues is guessing. Two runs that differ by a single input show the cause directly.
Each tile is a day. A lesson stands it up as a column, taller for more; today is outlined and the lesson you are reading is lit.
See all lessons →Nov 1, 2026: Evaluate the installed path, including its managed policy.
Newest first. Every cover is a live drawing: point at it and it answers.
An agent reads the release notes every week and writes up what moved, so I don’t keep twenty tabs open.