# Intelligence Design

URL: https://mechanisticmindset.com/wiki/intelligence-design
Tags: philosophy, speculative, research

# Intelligence Design

#philosophy #speculative #research

## A Note on This Article

These are speculative ideas about agent architecture, developed while building agent systems. The heuristics and mental models remain works in progress requiring empirical validation. They provide a way to reason about design, rather than proven laws.

## The Wrong Mental Model

Suppose a generator produces ten candidate answers and one is bad. A later evaluator can discard it. If a router sends a request to the wrong specialist, however, every later step can perform the wrong task correctly. Improving either component by tweaking its prompt misses the difference between their roles in the system.

The familiar input → function → output model encourages treating an agent as a deterministic executor: give the instruction, obtain the result, and revise the instruction if the result is wrong. An LLM instead samples a probability distribution over outputs. The same prompt can produce different results. Hallucination, drift and misunderstanding are properties of this substrate that the architecture must accommodate.

| Deterministic view | Probabilistic view |
|---|---|
| An agent executes an instruction. | An agent is a noisy channel. |
| A prompt produces the correct result. | A prompt conditions the probability of a correct result. |
| Failure means the prompt was wrong. | Failure means the signal fell below the required threshold. |
| The instruction needs repair. | The output distribution needs to change. |
| Reliability comes from one perfect call. | Reliability comes from multiple calls, filtering and feedback. |

The design task is to obtain reliable outcomes from unreliable components. That requires control over how calls are arranged and assessed, as well as what each prompt says.

## The Universal Pattern

Intelligence reliably produces good outcomes under uncertainty. Its common operation is to generate possibilities and then filter for good ones. Every intelligent system runs a version of that loop:

| Domain | What is generated | What selects |
|---|---|---|
| Evolution | Random mutations | Selection pressure |
| Brain | Neuronal noise and candidate actions | Prediction error and reward |
| Science | Hypotheses | Experiments |
| Markets | Ventures | Profit and loss |
| LLM training | Token samples | Training signals such as RLHF |
| Agent systems | Multiple outputs | Automated evaluation |

Evolution obtains organisms through variation and selection. Science tests hypotheses instead of requiring a correct theory on the first attempt. Markets fund ventures and use profit and loss to select among them. The material differs across these systems, but repeated generation and filtering produce the useful result.

## Agency in System Design

[Agency](/wiki/agency) means producing effects rather than only responding to events. An agent designer can respond to each failure with another prompt revision, leaving reliability dependent on the next sample. Or the designer can arrange a pipeline that detects, rejects and learns from failed samples.

The second approach changes the probability of a correct result across the system. [Probability Space Bending](/wiki/probability-space-bending) describes this intervention in a distribution: build conditions that move probability toward the desired outcome instead of trying to predict which individual attempt will succeed.

## The Signal Metaphor

A signal passing through a noisy channel has to be reconstructed at the receiving end. In an LLM call, the intent is the signal and the model is the channel. Several errors can corrupt it:

- Hallucination adds false content.
- Drift loses the thread during a long context.
- Misinterpretation substitutes a different meaning for the intended one.
- A format error violates the expected structure.
- A knowledge gap leaves the model without required information.

The result contains signal and noise. A system can improve their ratio across several stages even when no individual call becomes completely reliable.

## Why Single Attempts Fail

A detection threshold is the minimum signal strength distinguishable from background noise. One LLM call supplies one sample, regardless of the effort spent crafting it. Trying to make that single attempt exceptional is an intensity strategy. It fails when unusually good prompts cannot be produced reliably, when model variance creates a high noise floor, or when a nonzero hallucination rate leaves attempts below the threshold.

The reliability comparison assigns these outcomes to a component with P(correct) = 0.7:

| Arrangement | Success |
|---|---|
| One call | 70% |
| Three calls and majority vote | 93% |
| Five calls and majority vote | 97% |

Prompting improves the component's 0.7 probability. The architectural intervention combines components to obtain the higher system-level probability. Independence and correlated error matter to this comparison, as discussed under amplification strategies.

## The Core Primitive: Amplification

Repeated attempts make a low-probability success more likely to appear somewhere in the set. If each attempt has P(success) = 0.02:

| Attempts | Probability of at least one success |
|---|---|
| 1 | 0.02 |
| 50 | 0.64 |
| 100 | 0.87 |
| 200 | 0.98 |

The probability compounds rather than increasing linearly. The system must then recognize the successful attempt:

```
Generate N → Auto-evaluate → Select best
```

Several techniques arrange this operation differently:

| Technique | Arrangement |
|---|---|
| Self-consistency | Generate reasoning paths and vote on the answer. |
| Best-of-N | Score several outputs and choose the highest. |
| AlphaCode | Generate millions of programs and filter with tests. |
| Tree of Thoughts | Evaluate branches and expand the best. |
| Rejection sampling | Continue generating until a sample passes the filter. |
| Beam search | Score candidates, retain the top-k and repeat. |

The shared mechanism is generating alternatives and applying selection, although the candidates and selection procedure differ.

## The Structural Requirement

Filtering provides leverage only when evaluation is cheaper and more reliable than generation. Deterministic tests or schemas make it work perfectly. An LLM evaluator can work with some noise when checking the answer is easier than creating it. If evaluation is just as hard as generation, the extra call doubles computation without resolving the original problem.

| Domain | What makes verification available |
|---|---|
| Code | Deterministic tests provide ground truth. |
| Math | Computation can be checked step by step. |
| Factual work | Sources can be consulted. |
| Format | A schema specifies valid structure. |
| Extraction | The source document contains what must be extracted. |

Other tasks lack that asymmetry. Creative writing has subjective judgments and no ground truth. Open-ended reasoning can take as much work to validate as to perform. A novel problem has no known answer to compare against, and taste can make the scoring function as uncertain as the generator. These differences give code generation much more filtering leverage than creative writing.

## Distribution Control Variables

The distribution changes through concrete design choices:

| Variable | What changes | Available adjustment |
|---|---|---|
| Temperature | Spread | Lower values tighten the distribution; higher values diversify it. |
| Prompt structure | Center | A clearer prompt brings the center closer to the desired output. |
| Few-shot examples | Shape | Examples pull output toward their pattern. |
| Output constraints | Permitted region | JSON mode and function calling cut off invalid regions. |
| Model | Base distribution | Different models bring different priors. |
| Context | Conditional distribution | The supplied context changes P(output). |
| Decomposition | Task distribution | Smaller tasks give each step a tighter distribution. |
| Number of samples | Coverage | Additional samples explore more of the distribution. |
| Scoring function | Selection pressure | Filtering changes which outputs survive. |

A pipeline combines these controls. More samples are useful only if the selected result is better; a narrower task is useful only if its output can be incorporated into the larger one.

## Amplification Strategies

| Strategy | Situation | Operation |
|---|---|---|
| Temporal retries | One channel needs greater reliability. | Repeat the call N times and vote. |
| Spatial parallelism | Several approaches are available. | Run different prompts or models and combine results. |
| Validation | Ground truth exists. | Generate, validate and retain passing outputs. |
| Diversity | Errors are correlated. | Vary the prompt, temperature or model to reach different regions. |

The five-call, 0.97 comparison assumes independent calls. Calls to the same model with the same prompt instead share systematic biases, so repeating them does not supply independent evidence.

Diversity addresses the shared error. Self-consistency changes the reasoning path rather than merely rerunning an identical route. AlphaCode generates structurally different programs. When errors are correlated, sampling different regions is more useful than taking more samples from the same one.

## Signal Function Taxonomy

A call's role determines where failure propagates and how much reliability it needs.

### Source Functions (Generate Signal)

A **generator** produces candidates, drafts or options. Individual reliability can be low because downstream filtering removes bad outputs: twenty proposed approaches might yield three worth pursuing. A **planner** decomposes an intention into executable steps. Its reliability must be medium-high because later work depends on that structure; the plan needs validation and room for revision before execution.

### Routing Functions (Direct Signal)

A **router or classifier** chooses the next path. Incorrect routing can corrupt all later work, so reliability must be very high. Constrained outputs, explicit categories and fallbacks limit that risk. An **orchestrator** coordinates execution across agents and needs the same reliability because it controls the whole flow. Simple logic and deterministic operations reduce the amount entrusted to an LLM.

### Transformation Functions (Modify Signal)

A **specialist** performs one bounded transformation. Medium reliability is acceptable when attempts can be retried and filtered; scope should be clear. A **translator** changes representations, such as natural language to SQL or prose to structured data. A **compressor** summarizes or distills while preserving the essential information. An **extractor** isolates entities or other requested information from noisy input. A **synthesizer** combines several sources into one coherent result.

### Filtering Functions (Reduce Noise)

A **validator or evaluator** checks criteria and supplies a direction for correction. Its reliability must be high because bad feedback produces bad learning. Multiple validators, explicit rubrics and cross-checks support it. A **critic** reviews generated content for errors before use. A **recovery** function classifies failures, adjusts parameters and chooses a fallback.

### Memory Functions (Persist Signal)

**Memory** stores and retrieves information across sessions. Corruption persists and propagates, making reliability important. Structured storage and validation at the point of writing protect later uses of the record.

## Composition Patterns

### Pattern 1: Reliable Output from Unreliable Source

```
Generator(n=10) → Evaluator → Filter(threshold) → Output
```

When verification is cheap, ten noisy candidates can be evaluated and filtered to produce one reliable answer. The generator does not need to avoid every mistake because the later stages can recover from them.

### Pattern 2: Domain-Appropriate Processing

```
Router → Specialist[domain] → Validator → Output
```

Different inputs require different specialists. The router chooses the domain, the specialist performs the work and the validator checks it. An incorrect route can make every subsequent stage solve the wrong problem, so reliability is concentrated at that first choice.

### Pattern 3: Iterative Refinement

```
Generator → Critic → Refiner → Critic → ... → Output
```

This arrangement helps when a reliable critic can identify a useful change. Each pass removes noise or adds signal, and the revised output returns for another assessment.

### Pattern 4: Parallel Decomposition

```
Planner → [Specialist × N in parallel] → Synthesizer → Output
```

Independent subtasks can run at the same time. The planner defines the separation, specialists perform the pieces, and the synthesizer recombines them. The independence supplies the opportunity for large-scale parallelism.

### Pattern 5: Generate-Test-Refine Loop

```
Generator(n) → Tester → [passing] → Select best
                     → [failing] → Analyzer → Generator(n, with feedback)
```

Tests separate passing outputs from failures. Failure analysis supplies feedback to the next generation round rather than ending the attempt. This is the generate-test-refine arrangement used by AlphaCode.

## Foundational Observations

### Prompting Is Necessary But Not Sufficient

Each node still needs a good prompt: it shifts the distribution toward the desired result. Single-call improvement has a ceiling, however. Volume, filtering and decomposition can take the system beyond that ceiling. Prompting designs the component; amplification and filtering determine how the components work together.

### Reliability Requirements Vary by Function

A router's wrong decision corrupts downstream work, while one bad generator output among ten can simply be discarded. Where reliability is critical, deterministic fallbacks, constraints, repeated verification and limited LLM dependence are appropriate. Where filtering can recover, more freedom and greater candidate volume can be useful.

### Noise Budget Is Finite

Every stage can add error. If a router directs three agents to the wrong task, all three efforts are wasted. If nineteen of twenty generated candidates fail a filter, the twentieth can still make the system succeed. The noise budget belongs where failure is recoverable; routing and orchestration need to minimize it because their errors cascade.

### Evals Measure Distributions

One successful test does not establish production reliability, and one failure does not establish universal failure. A prompt change can also appear helpful or harmful depending on which sample is observed. An evaluation uses repeated runs—100 in this example—to obtain P(correct), then measures whether architectural changes move that probability past the required threshold.

### The Framework Applies Where Asymmetry Is Largest

Code, structured extraction, factual work and format compliance have tests, sources or schemas. These are also the domains where most production agent use cases live. Their available checks give generation and filtering its largest advantage.

## Evidence

Published results support the generation-and-filtering pattern:

| Method | Mechanism | Reported improvement |
|---|---|---|
| Self-consistency, Wang et al. | Sample reasoning chains and vote. | +10–20% on reasoning benchmarks |
| AlphaCode, DeepMind | Generate millions of programs and test them. | Competitive with humans, top 54% |
| Best-of-N sampling | Generate, score and select. | Consistent gains across tasks |
| Constitutional AI, Anthropic | Generate, critique and revise. | Fewer harmful outputs |
| Tree of Thoughts, Yao et al. | Generate, evaluate and select branches. | +20–30% on planning tasks |
| Verifier models, Cobbe et al. | Score solutions with a separate model. | +15% on math word problems |

These methods use volume and filtering instead of relying on a better single-shot prompt. They do not establish the proposed signal-function taxonomy, its reliability requirements, the best compositions for each domain or generalization to every agent task. Those claims still require empirical measurement.

## The Meta-Principle

The same intervention appears in [agency](/wiki/agency) and [forcing functions](/wiki/forcing-functions). Requiring [willpower](/wiki/willpower) for each gym visit acts on individual instances. Changing the [surrounding architecture](/wiki/prevention-architecture) changes P(gym) across future instances. Agent architecture similarly changes P(correct) across outputs.

The engineering is identical across these substrates: the intervention changes the generator of outcomes rather than one outcome. [Level 4 agency](/wiki/ladder-of-agency) calls this engineering the probability distribution. A weather forecaster describes a distribution while a climate engineer changes it; an intelligence designer changes what an agent is likely to produce.

## Related Concepts

- [Agency](/wiki/agency) concerns causing effects through system design.
- [Probability Space Bending](/wiki/probability-space-bending) changes distributions of outcomes.
- [Ladder of Agency](/wiki/ladder-of-agency) places distribution engineering at Level 4.
- [Forcing Functions](/wiki/forcing-functions) changes probability through structure.
- [Prevention Architecture](/wiki/prevention-architecture) makes failure paths unavailable.
- [Signal Boosting](/wiki/signal-boosting) amplifies weak signals through volume and filtering.
- [Effective AI Usage](/wiki/effective-ai-usage) develops practical AI workflows.
- [AI as Accelerator](/wiki/ai-as-accelerator) examines the collapse of implementation complexity.
- [Cybernetics](/wiki/cybernetics) supplies feedback and control.
- [Statistical Mechanics](/wiki/statistical-mechanics) reasons about distributions and microstates.