A.A. Fel'dbaum's early-1960s dual control theory was the first mathematical formalization of the exploration-exploitation tradeoff

feldbaumdual-control-theoryreinforcement-learningexploration-exploitationcontrol-theorycross-domain-bridge

A 2026 control-theory survey by Tomas Meijer and Anders Rantzer (Lund University) credits A.A. Fel'dbaum with the origin point of a lineage running directly into modern reinforcement learning: "The first researcher to formulate a mathematical problem treating the exploration–exploitation tradeoff in its full generality was Feldbaum. He introduced the term dual control in the early 1960s... In the following years, this idea propagated into a wide variety of subject areas in engineering, including adaptive control, reinforcement learning, and Bayesian optimization." The problem Fel'dbaum posed — a controller must simultaneously probe a system to learn its dynamics and act on it to regulate it, and these two goals trade off — is the same tension named today as exploration vs. exploitation in RL's epsilon-greedy policies and bandit algorithms. The paper is directly quotable on the mechanism too: "Feldbaum emphasized that learning often needs to be active: Without probing, you will not learn how the system responds."

This is a distinct lineage from the vault's existing optimal-control-to-backpropagation thread (moc-backpropagation-origins) — Fel'dbaum's problem is about learning while controlling, not about computing gradients through a control sequence — though both trace back to the same mid-century Soviet/Western control-theory moment that also produced Pontryagin and Kalman's results. Fel'dbaum taught Song Jian in Moscow (claim-song-jian-studied-under-feldbaum-in-moscow), placing one teacher at the root of two of this vault's separately-tracked clusters.

A direct read of Fel'dbaum's own 1960 papers confirms this claim's core in his own words — see claim-feldbaum-1960-papers-define-dual-control-as-additive-risk-of-action-and-risk-of-study. His formalism's relationship to "RL's epsilon-greedy policies and bandit algorithms" specifically, named earlier in this note, is narrower than that sentence implies — see the correction below.

Correction history.

• 2026-09-15 — This note's body named "RL's epsilon-greedy policies and bandit algorithms" in the same breath as Fel'dbaum's formalism, as if one lineage ran directly from the other. A direct read of the underlying survey's own structure (2026-09-15 capture, 10-inbox/raw/2026-09-15-what-do-aa-feldbaums-own-1960-dual-control.md) shows the survey traces mainstream RL's bandit-and-regret exploration algorithms to an independent 1933 Thompson-sampling lineage, and names Fel'dbaum nowhere in its own bandit section. Fel'dbaum's formalism maps cleanly onto the narrower, later subfield of Bayesian reinforcement learning specifically, per Klenske & Hennig 2016. See claim-mainstream-rl-exploration-traces-to-1933-bandit-lineage-not-feldbaum-dual-control for the full correction. The core claim in this note's title — that Fel'dbaum formalized the tradeoff — is unweakened and is now also confirmed against his own 1960 text, not only this survey's account (see claim-feldbaum-1960-papers-define-dual-control-as-additive-risk-of-action-and-risk-of-study).