AutoML相關論文

本文為Awesome-AutoML-Papers的譯文。

1、AutoML簡介

Machine Learning幾年來取得的不少可觀的成績，越來越多的學科都依賴於它。然而，這些成果都很大程度上取決於人類機器學習專家來完成如下工作：

• 數據預處理 Preprocess the data
• 選擇合適的特征 Select appropriate features
• 選擇合適的模型族 Select an appropriate model family
• 優化模型參數 Optimize model hyperparameters
• 模型后處理 Postprocess machine learning models
• 分析結果 Critically analyze the results obtained

隨着大多數任務的復雜度都遠超非機器學習專家的能力范疇，機器學習應用的不斷增長使得人們對現成的機器學習方法有了極大的需求。因為這些現成的機器學習方法使用簡單，並且不需要專業知識。我們將由此產生的研究領域稱為機器學習的逐步自動化。

AutoML借鑒了機器學習的很多知識，主要包括：

• 貝葉斯優化 Bayesian optimization
• 結構化數據的大數據的回歸模型 Regression models for structured data and big data
• 元學習 Meta learning
• 遷移學習 Transfer learning
• 組合優化 Combinatorial optimization.

2、目錄

• Papers
  • Automated Feature Engineering
    • Expand Reduce
    • Hierarchical Organization of Transformations
    • Meta Learning
    • Reinforcement Learning
  • Architecture Search
    • Evolutionary Algorithms
    • Local Search
    • Meta Learning
    • Reinforcement Learning
    • Transfer Learning
  • Hyperparameter Optimization
    • Bayesian Optimization
    • Evolutionary Algorithms
    • Lipschitz Functions
    • Local Search
    • Meta Learning
    • Particle Swarm Optimization
    • Random Search
    • Transfer Learning
  • Performance Prediction
    • Performance Prediction
  • Frameworks
  • Miscellaneous
• Tutorials
  • Bayesian Optimization
  • Meta Learning
• Articles
  • Bayesian Optimization
  • Meta Learning
• Slides
  • Bayesian Optimization
• Books
  • Meta Learning
• Projects
• Prominent Researchers

Papers

Automated Feature Engineering

• Expand Reduce

  • 2017 | AutoLearn — Automated Feature Generation and Selection | Ambika Kaul, et al. | ICDM | PDF
  • 2017 | One button machine for automating feature engineering in relational databases | Hoang Thanh Lam, et al. | arXiv | PDF
  • 2016 | Automating Feature Engineering | Udayan Khurana, et al. | NIPS | PDF
  • 2016 | ExploreKit: Automatic Feature Generation and Selection | Gilad Katz, et al. | ICDM | PDF
  • 2015 | Deep Feature Synthesis: Towards Automating Data Science Endeavors | James Max Kanter, Kalyan Veeramachaneni | DSAA | PDF

• Hierarchical Organization of Transformations

  • 2016 | Cognito: Automated Feature Engineering for Supervised Learning | Udayan Khurana, et al. | ICDMW | PDF

• Meta Learning

  • 2017 | Learning Feature Engineering for Classification | Fatemeh Nargesian, et al. | IJCAI | PDF

• Reinforcement Learning

  • 2017 | Feature Engineering for Predictive Modeling using Reinforcement Learning | Udayan Khurana, et al. | arXiv | PDF
  • 2010 | Feature Selection as a One-Player Game | Romaric Gaudel, Michele Sebag | ICML | PDF

Architecture Search

• Evolutionary Algorithms

  • 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | PDF
  • 2002 | Evolving Neural Networks through Augmenting Topologies | Kenneth O.Stanley, Risto Miikkulainen | Evolutionary Computation | PDF

• Local Search

  • 2017 | Simple and Efficient Architecture Search for Convolutional Neural Networks | Thomoas Elsken, et al. | ICLR | PDF

• Meta Learning

  • 2016 | Learning to Optimize | Ke Li, Jitendra Malik | arXiv | PDF

• Reinforcement Learning

  • 2018 | Efficient Neural Architecture Search via Parameter Sharing | Hieu Pham, et al. | arXiv | PDF
  • 2017 | Neural Architecture Search with Reinforcement Learning | Barret Zoph, Quoc V. Le | ICLR | PDF

• Transfer Learning

  • 2017 | Learning Transferable Architectures for Scalable Image Recognition | Barret Zoph, et al. | arXiv | PDF

Frameworks

• 2017 | Google Vizier: A Service for Black-Box Optimization | Daniel Golovin, et al. | KDD |PDF
• 2017 | ATM: A Distributed, Collaborative, Scalable System for Automated Machine Learning | T. Swearingen, et al. | IEEE | PDF
• 2015 | AutoCompete: A Framework for Machine Learning Competitions | Abhishek Thakur, et al. | ICML | PDF

Hyperparameter Optimization

• Bayesian Optimization

  • 2016 | Bayesian Optimization with Robust Bayesian Neural Networks | Jost Tobias Springenberg， et al. | NIPS | PDF
  • 2016 | Scalable Hyperparameter Optimization with Products of Gaussian Process Experts | Nicolas Schilling, et al. | PKDD | PDF
  • 2016 | Taking the Human Out of the Loop: A Review of Bayesian Optimization | Bobak Shahriari, et al. | IEEE | PDF
  • 2016 | Towards Automatically-Tuned Neural Networks | Hector Mendoza, et al. | JMLR | PDF
  • 2016 | Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization | Martin Wistuba, et al. | PKDD | PDF
  • 2015 | Efficient and Robust Automated Machine Learning | PDF
  • 2015 | Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | PKDD | PDF
  • 2015 | Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization | Martin Wistua, et al. | PDF
  • 2015 | Joint Model Choice and Hyperparameter Optimization with Factorized Multilayer Perceptrons | Nicolas Schilling, et al. | ICTAI | PDF
  • 2015 | Learning Hyperparameter Optimization Initializations | Martin Wistuba, et al. | DSAA | PDF
  • 2015 | Scalable Bayesian optimization using deep neural networks | Jasper Snoek, et al. | ACM | PDF
  • 2015 | Sequential Model-free Hyperparameter Tuning | Martin Wistuba, et al. | ICDM | PDF
  • 2013 | Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms | PDF
  • 2013 | Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures | J. Bergstra | JMLR | PDF
  • 2012 | Practical Bayesian Optimization of Machine Learning Algorithms | PDF
  • 2011 | Sequential Model-Based Optimization for General Algorithm Configuration(extended version) | PDF

• Evolutionary Algorithms

  • 2018 | Autostacker: A Compositional Evolutionary Learning System | Boyuan Chen, et al. | arXiv | PDF
  • 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | PDF

• Lipschitz Functions

  • 2017 | Global Optimization of Lipschitz functions | C´edric Malherbe, Nicolas Vayatis | arXiv | PDF

• Local Search

  • 2009 | ParamILS: An Automatic Algorithm Configuration Framework | Frank Hutter, et al. | JAIR | PDF

• Meta Learning

  • 2008 | Cross-Disciplinary Perspectives on Meta-Learning for Algorithm Selection | PDF

• Particle Swarm Optimization

  • 2017 | Particle Swarm Optimization for Hyper-parameter Selection in Deep Neural Networks | Pablo Ribalta Lorenzo, et al. | GECCO | PDF
  • 2008 | Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines | Shih-Wei Lin, et al. | Expert Systems with Applications | PDF

• Random Search

  • 2016 | Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization | Lisha Li, et al. | arXiv | PDF
  • 2012 | Random Search for Hyper-Parameter Optimization | James Bergstra, Yoshua Bengio | JMLR | PDF
  • 2011 | Algorithms for Hyper-parameter Optimization | James Bergstra, et al. | NIPS | PDF

• Transfer Learning

  • 2016 | Efficient Transfer Learning Method for Automatic Hyperparameter Tuning | Dani Yogatama, Gideon Mann | JMLR | PDF
  • 2016 | Flexible Transfer Learning Framework for Bayesian Optimisation | Tinu Theckel Joy, et al. | PAKDD | PDF
  • 2016 | Hyperparameter Optimization Machines | Martin Wistuba, et al. | DSAA | PDF
  • 2013 | Collaborative Hyperparameter Tuning | R´emi Bardenet, et al. | ICML | PDF

Miscellaneous

• 2018 | Accelerating Neural Architecture Search using Performance Prediction | Bowen Baker, et al. | ICLR | PDF
• 2017 | Automatic Frankensteining: Creating Complex Ensembles Autonomously | Martin Wistuba, et al. | SIAM | PDF

Tutorials

Bayesian Optimization

• 2010 | A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning | PDF

Meta Learning

• 2008 | Metalearning - A Tutorial | PDF

Articles

Bayesian Optimization

• 2016 | Bayesian Optimization for Hyperparameter Tuning | Link

Meta Learning

• 2017 | Why Meta-learning is Crucial for Further Advances of Artificial Intelligence? | Link
• 2017 | Learning to learn | Link

Slides

Automated Feature Engineering

• Automated Feature Engineering for Predictive Modeling | Udyan Khurana, etc al. | PDF

Hyperparameter Optimization

Bayesian Optimization

• Bayesian Optimisation | PDF
• A Tutorial on Bayesian Optimization for Machine Learning | PDF

Books

Meta Learning

• 2009 | Metalearning - Applications to Data Mining | Springer | PDF

Projects

• Advisor | Python | Open Source | Code
• auto-sklearn | Python | Open Source | Code
• Auto-WEKA | Java | Open Source | Code
• Hyperopt | Python | Open Source | Code
• Hyperopt-sklearn | Python | Open Source | Code
• SigOpt | Python | Commercial | Link
• SMAC3 | Python | Open Source | Code
• RoBO | Python | Open Source | Code
• BayesianOptimization | Python | Open Source | Code
• Scikit-Optimize | Python | Open Source | Code
• HyperBand | Python | Open Source | Code
• BayesOpt | C++ | Open Source | Code
• Optunity | Python | Open Source | Code
• TPOT | Python | Open Source | Code
• ATM | Python | Open Source | Code
• Cloud AutoML | Python | Commercial| Link
• H2O | Python | Commercial | Link
• DataRobot | Python | Commercial | Link
• MLJAR | Python | Commercial | Link
• MateLabs | Python | Commercial | Link

MARSGGBO♥原創





2018-7-14