Before intelligence.
A better way to learn.
Craig Teroh compared his engineers’ learning strategies. You can run a small version of that idea here.
Train nine candidates. Select using validation examples. Then measure the winner on entirely new tasks. Every score below comes from actual predictions.
The useful distinction: training learns an answer within a task. This experiment selects a learning strategy across tasks. It does not create the fictional Meta, consciousness, or general intelligence.
The experiment
Each candidate gets 8 development tasks and 250 training steps per task. Select by validation error; test once on 12 fresh tasks. Fixed seeds make comparisons repeatable. Everything runs locally after you press Run.
Ready. No models have been trained yet.
| Degree | Learning rate | Validation error |
|---|
What should you notice?
Keep training and testing separate. A configuration that wins on familiar examples may fail on a new kind of problem. Try curves first, then switch only the unseen tasks to sine waves. The selected configuration will stay the same: the final test never chooses the winner.
Do not confuse a score with a mind. These candidates learn polynomial coefficients using gradient descent. The outer loop compares degree and learning rate. This is a bounded hyperparameter search across tasks, not a new general-purpose learning algorithm.
The data is synthetic. Test examples and task seeds are held out, but all tasks share this generator’s assumptions. Repeatedly designing the experiment around the displayed test score would also turn that score into development feedback; a real study needs a final untouched evaluation.
Nothing hidden behind the result.
Browser experiment · JavaScript source
Python companion · standalone standard-library version
This companion uses its own seeded data generator and 300 training steps. Its numerical results differ from the browser experiment; it demonstrates the same train / select / unseen-test design.
"""Protocol 10: a small, reproducible learning-strategy selection experiment.
Inspired by Maxwell Jann's Runaway AI. Python 3; no dependencies.
This is bounded model selection across tasks, not AGI or self-modifying AI.
"""
from dataclasses import dataclass
from random import Random
from statistics import mean
import math
@dataclass(frozen=True)
class Strategy:
degree: int
rate: float
steps: int = 300
def task(seed):
"""Independent noisy quadratic tasks with disjoint train/validation/test samples."""
rng = Random(seed)
coefficients = [rng.uniform(-2, 2) for _ in range(3)]
def sample(n):
rows = []
for _ in range(n):
x = rng.uniform(-1, 1)
y = sum(c * x**j for j, c in enumerate(coefficients)) + rng.gauss(0, 0.05)
rows.append((x, y))
return rows
return sample(32), sample(32), sample(128)
def predict(weights, x):
return sum(w * x**j for j, w in enumerate(weights))
def mse(weights, rows):
return mean((predict(weights, x) - y)**2 for x, y in rows)
def fit(strategy, rows):
weights = [0.0] * (strategy.degree + 1)
for _ in range(strategy.steps):
gradients = [0.0] * len(weights)
for x, y in rows:
error = predict(weights, x) - y
for j in range(len(weights)):
gradients[j] += 2 * error * x**j / len(rows)
weights = [w - strategy.rate * g for w, g in zip(weights, gradients)]
if not all(math.isfinite(w) for w in weights):
raise ValueError('Numerical instability: this is a failed training run, not intelligence.')
return weights
def select_strategy(tasks, candidates):
"""Select using validation data only; lower mean squared error is better."""
scores = []
for strategy in candidates:
score = mean(mse(fit(strategy, train), validation)
for train, validation, _ in tasks)
scores.append((score, strategy))
scores.sort(key=lambda item: item[0])
return scores[0][1], scores
def evaluate_unseen(strategy, seeds):
selected, baseline = [], []
for seed in seeds:
train, _, test = task(seed)
selected.append(mse(fit(strategy, train), test))
baseline.append(mse([mean(y for _, y in train)], test))
return mean(selected), mean(baseline)
def main():
# Fictional Protocol 10 pools evidence across engineers' experiments.
# Here, every candidate gets the same development tasks and training budget.
candidates = [Strategy(degree, rate) for degree in (1, 2, 3)
for rate in (0.01, 0.05, 0.1)]
development_tasks = [task(seed) for seed in range(8)]
selected, scores = select_strategy(development_tasks, candidates)
print('PROTOCOL 10 — candidate learning strategies')
for score, strategy in scores:
print(f'{strategy}: validation MSE={score:.6f}')
# Fresh task identities; no test result is used to select a strategy.
test_mse, baseline = evaluate_unseen(selected, range(1000, 1012))
print(f'\nSelected: {selected}')
print(f'Unseen-task test MSE: {test_mse:.6f}')
print(f'Train-mean baseline MSE: {baseline:.6f}')
print('Scope: noisy quadratic regression only. No evidence of general intelligence.')
return selected, test_mse, baseline
if __name__ == '__main__':
main()