Researchers test AI self-play training without human data

The approach starts with two models at random initialization and trains them together.[1] One model proposes programs, which a universal Turing machine executes to produce byte sequences; the second model learns to predict those sequences autoregressively.[1] The learner is trained using cross-entropy, while reinforcement learning trains the generator to produce sequences at the edge of what the learner can handle.[1] The researchers describe the generator’s choices as an adaptive curriculum for the learner.[1] They say this could provide an effectively unbounded supply of training data, constrained by compute rather than human knowledge.[1]

The authors report that zero-shot loss scaled predictably with compute across several natural datasets.[1] They also report in-context learning and the discovery of recognizable mathematical sequences during training.[1] A separate experiment, according to Ynet, found that self-generated pretraining helped a model learn faster after exposure to real-world data.[3] Ynet also reports that, during testing, the model could infer rules from examples without additional training.[3] These findings describe reported experiments, not evidence that self-play alone gives a model knowledge of real-world facts.[2]

The experiments were confined to models with fewer than 25 million parameters and a 4,000-token context, according to the authors as reported by OnTheWire.[2] TPS describes the study model as more than 10,000 times smaller than today’s largest AI systems.[4] Yoav Levine told TPS that self-play can teach general patterns but not real-world facts such as who wrote Hamlet.[4] The researchers also say the method cannot recover facts about a particular real-world setting without information from that world.[2] The preprint has not been peer reviewed.[2]

Levine told TPS that self-play could be useful where high-quality training data becomes a bottleneck, particularly in fields where data is scarce or expensive.[4] The researchers’ next milestone is to test whether the method works with models containing billions of parameters and can replace a measurable amount of conventional training data.[4] That makes scale and measurable data substitution the key tests to watch; the reported proof-of-concept does not yet establish either.[1][2][4]