Skip to content

Research and experimental engineering

Many more experiments than results, and the expensive part is remembering which ones were already tried.

Research and experimental engineering

Who this is

A team whose unit of work is an attempt rather than a feature. Most attempts are discarded, which is normal and correct; the failure mode is discarding the knowledge along with the attempt.

How they work

Wide and shallow: many parallel probes, frequent model and provider switches, aggressive abandonment. Very little of it is worth keeping, and identifying which little is the whole skill.

What goes wrong

The same dead end explored three times because the first two were never recorded as dead ends. A result that cannot be reproduced because nothing captured which model, which prompt, which repository state produced it. A promising branch abandoned mid-experiment and found months later with no way to tell whether it worked.

10 moments

Every moment that names this one. Derived from the moments; this page keeps no list.

  1. watched the strongest model, at maximum effort, rename a variableCapability, reasoning depth, context size and verbosity collapse into one habit — everything on — because nothing ever named them separately. cost: mediumconstant Why, and what to do →
  2. asked what the workers did last night and had only transcripts to grepA conversation log records what was said, not what happened, so the operational question has to be answered by a person reading prose. cost: highcommon Why, and what to do →
  3. spent an hour discovering something the repository already knew, twiceA worker learns something expensive about the codebase, uses it once, and it dies with the session because nothing turned it into a durable record. cost: mediumcommon Why, and what to do →
  4. found finished work on a branch that nobody ever mergedWorkers produce far more than integration absorbs, and nothing distinguishes work that is done from work that is done and landed. cost: highcommon Why, and what to do →
  5. let a fast, cheap session make a decision the whole codebase now lives withCapability is dialled down for cost and nothing distinguishes the tasks where that is prudent from the ones where it is expensive. cost: highoccasional Why, and what to do →
  6. could not say why one provider was used for that work and another for thisProvider and model selection is made implicitly, per session, and recorded nowhere, so it cannot be reviewed or repeated. cost: mediumcommon Why, and what to do →
  7. looked at the bill and could not say which of it produced anythingConsumption is measured per account and outcomes are recorded per person, so the two can never be joined. cost: mediumcommon Why, and what to do →
  8. lost a failure a worker had already reproduced, because the session endedA failure observed inside a session is described in that session and nowhere else, so the next one starts from the report rather than the evidence. cost: mediumcommon Why, and what to do →
  9. watched a worker spend a day on an approach that had already been ruled outNegative results are the majority of experimental output and the part nobody records, so the search space is re-explored. cost: mediumcommon Why, and what to do →
  10. could not reproduce a result because nothing recorded the state it came fromThe result is recorded and the conditions that produced it are not, so it is an anecdote rather than a measurement. cost: highoccasional Why, and what to do →