Tain Research

We are a research organisation focused on automating the design of neural networks end-to-end, with particular interest in AutoML-Zero and self-referential meta-learning.

A large part of modern machine learning still depends on human choices about model families, training procedures, search spaces, and optimization settings. Existing work in AutoML and meta-learning has shown that some parts of this design process can themselves be learned or searched over. We are interested in pushing that line of work further, especially in settings where current systems still rely heavily on hand-specified priors.

One current direction is the bootstrap problem for self-referential neural networks. In principle, a model that can modify parts of its own update process or parameterization may be able to represent richer forms of adaptation than standard fixed-weight systems. In practice, these systems are difficult to initialize and train because useful self-modifying behavior has to emerge before it can be exploited. A central question is how to construct training procedures and inductive biases that make this class of models experimentally workable without collapsing back into ordinary externally specified optimization.

A second direction is improving search efficiency in evolutionary approaches related to AutoML-Zero. Existing search methods can discover nontrivial update rules and algorithmic components, but the search process is still expensive and heavily constrained by the choice of representation, mutation operators, and evaluation procedure. The aim here is to study whether that search can be made materially more sample-efficient while preserving the ability to discover algorithmic structure rather than only tuning within a narrow hand-designed family.

Recommended reading

  1. E. Real et al. (2020), AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
  2. K. Irie, I. Schlag, R. Csordás, J. Schmidhuber (2022), A Modern Self-Referential Weight Matrix That Learns to Modify Itself
  3. L. Kirsch, J. Schmidhuber (2022), Eliminating Meta Optimization Through Self-Referential Meta Learning
  4. K. O. Stanley, R. Miikkulainen (2002), Evolving Neural Networks Through Augmenting Topologies
  5. K. O. Stanley, D. D'Ambrosio, J. Gauci (2009), A Hypercube-Based Indirect Encoding for Evolving Large-Scale Neural Networks
  6. M. Andrychowicz et al. (2016), Learning to learn by gradient descent by gradient descent
  7. O. Wichrowska et al. (2017), Learned Optimizers that Scale and Generalize
  8. C. Finn, P. Abbeel, S. Levine (2017), Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
  9. B. Zoph, Q. V. Le (2016), Neural Architecture Search with Reinforcement Learning
  10. E. Real et al. (2018), Regularized Evolution for Image Classifier Architecture Search
  11. H. Liu, K. Simonyan, Y. Yang (2018), DARTS: Differentiable Architecture Search

If these problems are already part of your mental landscape, write to chris@tainresearch.com.