Information Theory

What It Is

A customer interview can rule out explanations for why a product is not being used. Two weeks spent polishing the product may leave exactly the same explanations unresolved. Both activities consume resources, but only the first has changed what the team knows about the decision it needs to make.

Claude Shannon formalized information theory in 1948. It measures information as a reduction in uncertainty, in bits. One bit halves uncertainty about a system's state. Acquiring information takes energy and resources; its decision value depends on whether the reduction in uncertainty changes what happens next.

An interesting fact can therefore have no value for the current action. A startup searching with limited runway and a control loop both need information that improves a consequential decision before its acquisition consumes the resources available to act.

Shannon's entropy formula measures uncertainty:

$$H(X) = -\sum P(x_i) \times \log_2(P(x_i))$$

Where:
  H = entropy (uncertainty in bits)
  X = random variable (system state)
  P(x_i) = probability of state i

Higher entropy means more uncertainty. Information removes possibilities and reduces entropy. The value assigned to that reduction is multiplied by the value of making the correct decision.

The article's decision-making application asks what different answers would cause you to do, alongside what the information costs to obtain.

Read this diagram

Name the decision → Find the uncertainty → Acquire relevant evidence → Update the choice.

Information Cost vs Information Value

The direct cost of a measurement is only part of what it consumes. It can also displace another experiment, delay the next iteration or require measurement infrastructure:

Value = (Uncertainty_before - Uncertainty_after) × Value_of_correct_decision

Information has value when:
  Value > Cost_direct + Cost_opportunity + Cost_delay

Suppose an interview begins with eight equally likely explanations for the severity of a customer's problem: 3 bits of uncertainty. Behavioral evidence leaves two explanations, or 1 bit. The interview has supplied 2 bits of information.

If choosing the right feature has a revenue impact of $100,000 and the interview costs $500 in time and runway, the calculation assigns value to those 2 bits multiplied by $100,000, weighted by the probability of using the information. The value exceeds the cost, so the interview is worth running.

Now consider two weeks of feature polish. Uncertainty about retention is 2 bits before the work and 2 bits afterward because the polish tests none of the possible retention drivers. It provides 0 bits at a cost of two weeks' runway. Its value as an information-gathering activity is negative, so the polish should be skipped.

Information Acquisition Strategies

High Information/Cost Ratio Activities

Watching users act reveals preferences through behavior rather than a statement about hypothetical behavior. Testing payment gives those preferences a real price. Retention measurement waits to see whether the value survives a first encounter.

Low Information/Cost Ratio Activities

A hypothetical answer leaves uncertainty about whether the person will act. Building an entire feature before testing it can reveal whether the package works while leaving its components unexplained. A long private beta adds delay and can overrepresent users unlike the eventual public audience.

Shannon's Channel Capacity and Feedback Loops

Shannon's channel capacity theorem gives the maximum transmission rate through a noisy channel:

C = B × log₂(1 + S/N)

Where:
  C = channel capacity (bits per second)
  B = bandwidth (measurements per unit time)
  S/N = signal-to-noise ratio (sensor accuracy)

Applied to feedback, bandwidth becomes the frequency of measurement and signal-to-noise ratio becomes sensor accuracy. A faster measurement cycle and more reliable sensors, including multiple sensors used together, raise the amount of information available to guide corrections.

Higher bandwidth and signal quality increase channel capacity, accelerating learning and optimization. A daily loop with calibrated sensors therefore outperforms a monthly review based on vague impressions.

Mutual Information and Sensor Correlation

Mutual information measures how much learning Y reduces uncertainty about X:

I(X;Y) = H(X) - H(X|Y)

Where:
  H(X) = entropy before measurement
  H(X|Y) = entropy after knowing Y

For startup measurements, the pairs differ in how much one observation reveals about the other:

High mutual information allows correlated sensors to be combined for greater accuracy. Low mutual information means the sensors measure independent dimensions, so both need separate monitoring. A statement of interest cannot replace a measurement of use when it predicts use poorly.

Information Acquisition vs Action Execution

The same resource budget has to fund learning and acting:

Total_runway = E_information_acquisition + E_execution

Optimal split when:
  Marginal_value(more info) = Marginal_value(more execution)

Spending 90% of runway on research and 10% on execution can leave a team confident in its direction but unable to capitalize on it. Reversing those proportions can fund extensive work in the wrong direction. The first pattern is analysis paralysis; the second is premature commitment.

The split depends on the initial uncertainty:

The 30-day pattern makes a similar transition. Days 1–7 emphasize discovering what works; days 8–30 emphasize repeating the validated behavior. The value of further information falls as the pattern becomes established, while the value of executing it rises.

The Value of Information Formula

Decision theory's expected value of perfect information compares the value of a decision made with perfect information to the value available with current information:

$$EVPI = E[\text{Value}_{\text{perfect}}] - E[\text{Value}_{\text{current}}]$$

Where E[Value] = ∑ P(state) × Value(optimal decision | state)

The acquisition rule is:

EVPI > Cost_acquisition

Consider whether to spend two weeks building an analytics dashboard:

Value if you have perfect usage data:
  Build high-value features = +$200K expected value

Value with current vague intuition:
  Random feature work = +$50K expected value

$EVPI = \$200K - \$50K = \$150K$
Cost = 2 weeks runway ≈ $40K

EVPI > Cost → Yes, build the dashboard

Perfect usage data raises expected value from $50,000 to $200,000, a difference of $150,000. The two weeks of runway cost approximately $40,000. Because the difference exceeds that cost, the calculation favors building the dashboard. Measurement infrastructure earns its return by improving the subsequent decisions at scale.

Integration with Mechanistic Framework

Questions initiate searches that spend cognitive energy. A useful question produces enough uncertainty reduction to justify that computation. An expensive question that leaves the decision unchanged has a poor information-to-cost ratio.

Tracking pays setup and maintenance costs to reduce uncertainty about behavioral patterns. External records allow a decision to use observed data instead of the current mood. Its net value is positive when Decision_frequency × Decision_impact > Setup_cost.

The Braindump spends ten minutes daily to externalize working-memory state. Processing that record reveals patterns, reduces internal uncertainty and removes ambiguity about the next action.

Related Concepts

• Startup as a Bug allocates information-gathering resources before they run out.
• Cybernetics uses sensors and feedback to steer a system.
• Optimal Foraging Theory compares search strategies under energy limits.
• Expected Value examines decisions made with incomplete information.
• Tracking builds the measurement record.
• Question Theory examines the computation required by an inquiry.
• The Braindump makes information available for processing.

Key Principle

Information acquisition is worthwhile when the decisions it enables are worth more than its cost. Maximizing (Value_decisions_enabled - Cost_acquisition) calls for experiments that resolve consequential uncertainty, reliable behavioral or payment signals, and feedback frequent enough to guide the next action. A mosquito needs enough information to reach blood before its energy runs out. A perfect map of every possible source would be useless if obtaining it consumed the energy needed to fly there.