From 3bfdd5ec7f279451012f8947c10c3b8dea19d3f5 Mon Sep 17 00:00:00 2001 From: Miles Olson Date: Wed, 19 Feb 2025 10:41:25 -0800 Subject: [PATCH] Ax for AutoML (#3386) Summary: Pull Request resolved: https://github.com/facebook/Ax/pull/3386 Tutorial demonstrating using Ax to tune a sklearn model. Uses many of Ax's advanced features. Differential Revision: D69810535 --- tutorials/automl/automl.ipynb | 673 ++++++++++++++++++++++++++++++++++ website/tutorials.json | 4 + 2 files changed, 677 insertions(+) create mode 100644 tutorials/automl/automl.ipynb diff --git a/tutorials/automl/automl.ipynb b/tutorials/automl/automl.ipynb new file mode 100644 index 00000000000..bf1b7e8f026 --- /dev/null +++ b/tutorials/automl/automl.ipynb @@ -0,0 +1,673 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "6889e8ed-572e-432c-aafe-23d791980a3b", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "# Ax for AutoML\n", + "AutoML, short for automated machine learning, refers to a large class of problems related to automating time-consuming and labor-intensive parts of developing machine learning models.\n", + "Adaptive experimentation is a natural fit for solving many AutoML tasks, which are often iterative in nature and can involve many expensive trial evaluations.\n", + "\n", + "In this tutorial we will use Ax for hyperparameter optimization (HPO), a common AutoML task in which a model's hyperparameters are adjusted in order to improve model performance.\n", + "A model's hyperparameters are parameters that are set before a model is trained or fit rather than being learned from data.\n", + "Machine learning engineers must choose good hyperparameters using either by their domain knowledge and intuition or by running experiments in which many models are trained and evaluated with different hyperparameter combinations.\n", + "As the number of hyperparameters grows and as models become more expensive to train and evaluate sample efficient aproaches to experimentation like Bayesian optimization become increasingly valuable.\n", + "\n", + "In this tutorial we will train an `SGDClassifier` from the popular [scikit-learn](https://scikit-learn.org/) library to recognize handwritten digits and tune the model's hyperparameters to improve its performance.\n", + "You can read more about the `SGDClassifier` model in their example [here](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html), which this tutorial is largely based on.\n", + "This tutorial will incorporate many advanced features in Ax to demonstrate how they can be applied on complex engineering challenges in a real-world setting.\n", + "\n", + "## Learning Objectives\n", + "- Understand how Ax can be used for HPO tasks\n", + "- Use complex optimization configurations like multiple objectives and outcome constraints to achieve nuanced real-world goals\n", + "- Use early stopping to save experimentation resources\n", + "- Analyze the results of the optimization\n", + "\n", + "## Prerequisites\n", + "- Familiarity with [scikit-learn](https://scikit-learn.org/) and basic machine learning concepts\n", + "- Understanding of [adaptive experimentation](#) and [Bayesian optimization](#)\n", + "- [Ask-tell Optimization of Python Functions with early stopping](#)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "74d17b49-d048-4cd5-b727-75c3d597d019", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 1: Import Necessary Modules\n", + "\n", + "First, ensure you have all the necessary imports:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customOutput": null, + "executionStartTime": 1739485077830, + "executionStopTime": 1739485077965, + "isAgentGenerated": false, + "language": "python", + "originalKey": "5c09dddd-6087-4d3a-acd5-8acb069d8389", + "outputsInitialized": false, + "requestMsgId": "5c09dddd-6087-4d3a-acd5-8acb069d8389", + "serverExecutionDuration": 1.667891163379 + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "import matplotlib.pyplot as plt\n", + "\n", + "import sklearn.datasets\n", + "import sklearn.linear_model\n", + "import sklearn.model_selection\n", + "from ax.preview.api.client import Client\n", + "from ax.preview.api.configs import (\n", + " ChoiceParameterConfig,\n", + " ExperimentConfig,\n", + " GenerationMethod,\n", + " GenerationStrategyConfig,\n", + " ParameterScaling,\n", + " ParameterType,\n", + " RangeParameterConfig,\n", + ")\n", + "from pyre_extensions import assert_is_instance" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "b94a9e80-f1d4-40ee-8fe9-2c9b4f0e7ddd", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 1.1: Understanding the baseline performance of `SGDClassifier`\n", + "Before we begin HPO, let's understand the task and the performance of `SGDClassifier` with its default hyperparameters.\n", + "The following code is largely adapted from the example on scikit-learn's webiste [here](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html).\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485077968, + "executionStopTime": 1739485078623, + "isAgentGenerated": false, + "language": "python", + "originalKey": "9d3eeb8a-03fb-4e20-b795-d06c3cd99b68", + "outputsInitialized": true, + "requestMsgId": "9d3eeb8a-03fb-4e20-b795-d06c3cd99b68", + "serverExecutionDuration": 185.13395590708, + "showInput": true + }, + "outputs": [], + "source": [ + "# Load the digits dataset and display the first 4 images to demonstrate\n", + "digits = sklearn.datasets.load_digits()\n", + "classes = list(set(digits.target))\n", + "\n", + "_, axes = plt.subplots(nrows=1, ncols=4, figsize=(10, 3))\n", + "for ax, image, label in zip(axes, digits.images, digits.target):\n", + " ax.set_axis_off()\n", + " ax.imshow(image, cmap=plt.cm.gray_r, interpolation=\"nearest\")\n", + " ax.set_title(\"Training: %i\" % label)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485078629, + "executionStopTime": 1739485079003, + "isAgentGenerated": false, + "language": "python", + "originalKey": "199de951-1af5-4a57-b341-e67f823c9e69", + "outputsInitialized": true, + "requestMsgId": "199de951-1af5-4a57-b341-e67f823c9e69", + "serverExecutionDuration": 33.779789227992, + "showInput": true + }, + "outputs": [], + "source": [ + "# Instantiate a SGDClassifier with default hyperparameters\n", + "clf = sklearn.linear_model.SGDClassifier()\n", + "\n", + "# Split the data into a training set and a validation set\n", + "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n", + " digits.data, digits.target, test_size=0.20, random_state=0\n", + ")\n", + "\n", + "# Train the classifier on the training set using 10 batches\n", + "#Also time the training.\n", + "batch_size = len(train_x) // 10\n", + "start_time = time.time()\n", + "for i in range(10):\n", + " start_idx = i * batch_size\n", + " end_idx = (i + 1) * batch_size\n", + "\n", + " # Use partial fit to update the model on the current batch\n", + " clf.partial_fit(\n", + " train_x[start_idx:end_idx], train_y[start_idx:end_idx], classes=classes\n", + " )\n", + "\n", + "training_time = time.time() - start_time\n", + "\n", + "# Evaluate the classifier on the validation set\n", + "score = clf.score(valid_x, valid_y)\n", + "score, training_time" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "ca935c9c-23c1-444b-aecc-09eac0452e44", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "The model performs well, but let's see if we can improve performance by tuning the hyperparameters." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "6caa7b9a-17bd-4e0b-9e45-9ded2d452a45", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 2: Initialize the Client\n", + "As always, the first step in running our adaptive experiment with Ax is to create an instance of the `Client` to manage the state of your experiment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485079013, + "executionStopTime": 1739485079217, + "isAgentGenerated": false, + "language": "python", + "originalKey": "f71bb5ed-6ff4-433c-9ca1-6d00df0a9f17", + "outputsInitialized": false, + "requestMsgId": "f71bb5ed-6ff4-433c-9ca1-6d00df0a9f17", + "serverExecutionDuration": 1.6501909121871, + "showInput": true + }, + "outputs": [], + "source": [ + "client = Client()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "2eb882e0-4b3c-4308-acd4-f2563a421b5f", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 3: Configure the Experiment\n", + "\n", + "The `Client` expects a series of `Config`s which define how the experiment will be run.\n", + "We'll set this up the same way as we did in our previous tutorial.\n", + "\n", + "Our current task is to tune the hyperparameters of an scikit-learn's [SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html).\n", + "These parameters control aspects of the model's training process and configuring them can have dramatic effects on the model's ability to correctly classify inputs.\n", + "A full list of this model's hyperparameters and appropriate values are available in the library's documentation.\n", + "In this tutorial we will tune the following hyperparameters:\n", + "* **loss:** The loss function to be used\n", + "* **penalty:** The penalty (aka regularization term) to be used\n", + "* **learning_rate:** The learning rate schedule\n", + "* **alpha:** Constant that multiplies the regularization term. The higher the value, the stronger the regularization\n", + "* **eta0**: The learning rate for training. In this example we will use a constant learning rate schedule\n", + "* **batch_size**: A training parameter which controls how many examples are shown during a single epoch. We will use all samples in the dataset for each model training, so a smaller batch size will translate to more epochs and vice versa.\n", + " \n", + "You will notice hyperparameters are continuous ranges, some are discrete ranges, and some are categorical choices; Ax is able to handle all of these types of parameters via its `RangeParameterConfig` and `ChoiceParameterConfig` classes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "customOutput": null, + "executionStartTime": 1739485079221, + "executionStopTime": 1739485079350, + "isAgentGenerated": false, + "language": "python", + "originalKey": "ba904311-af2e-491a-a25e-157602e0e906", + "outputsInitialized": true, + "requestMsgId": "ba904311-af2e-491a-a25e-157602e0e906", + "serverExecutionDuration": 2.7205082587898, + "showInput": true + }, + "outputs": [], + "source": [ + "# Create an experiment configuration\n", + "experiment_config = ExperimentConfig(\n", + " name=\"SGDClassifier_hpo\",\n", + " parameters=[\n", + " ChoiceParameterConfig(\n", + " name=\"loss\",\n", + " parameter_type=ParameterType.STRING,\n", + " values=[\n", + " \"hinge\",\n", + " \"log_loss\",\n", + " \"squared_hinge\",\n", + " \"modified_huber\",\n", + " \"perceptron\",\n", + " ],\n", + " is_ordered=False,\n", + " ),\n", + " ChoiceParameterConfig(\n", + " name=\"penalty\",\n", + " parameter_type=ParameterType.STRING,\n", + " values=[\"l1\", \"l2\", \"elasticnet\"],\n", + " is_ordered=False,\n", + " ),\n", + " ChoiceParameterConfig(\n", + " name=\"learning_rate\",\n", + " parameter_type=ParameterType.STRING,\n", + " values=[\"constant\", \"optimal\", \"invscaling\", \"adaptive\"],\n", + " is_ordered=False,\n", + " ),\n", + " RangeParameterConfig(\n", + " name=\"alpha\",\n", + " bounds=(1e-8, 100),\n", + " parameter_type=ParameterType.FLOAT,\n", + " scaling=ParameterScaling.LOG,\n", + " ),\n", + " RangeParameterConfig(\n", + " name=\"eta0\",\n", + " bounds=(1e-8, 1),\n", + " parameter_type=ParameterType.FLOAT,\n", + " scaling=ParameterScaling.LOG,\n", + " ),\n", + " RangeParameterConfig(\n", + " name=\"batch_size\",\n", + " bounds=(5, 500),\n", + " parameter_type=ParameterType.INT,\n", + " ),\n", + " ],\n", + " # The following arguments are optional\n", + " description=\"Optimization of SGDClassifier for digits dataset\",\n", + " owner=\"developer\",\n", + ")\n", + "\n", + "# Apply the experiment configuration to the client\n", + "client.configure_experiment(experiment_config=experiment_config)\n", + "\n", + "client.configure_generation_strategy(\n", + " GenerationStrategyConfig(method=GenerationMethod.FAST)\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "59cd82c7-81a4-4411-bc4a-2f7db3effbf1", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 4: Configure Optimization\n", + "Now, we must set up the optimization objective in `Client`, where `objective` is a string that specifies which metric we would like to optimize and which direction we consider optimal.\n", + "\n", + "scikit-learn models use function called `score` to report mean accuracy of the model on a given test set and labels -- we will of course want to maximize this.\n", + "However, these hyperparameters affect a number of other metrics as well.\n", + "For example, different configurations may cause the model's training loop to take more or less time.\n", + "Since model training can be a very expensive process, especially for large models, this can represent a significant cost.\n", + "\n", + "Let's configure Ax to maximize score while minimizing training time.\n", + "We call this a multi-objective optimization, and rather than returning a single best parameterization we return a Pareto frontier of points which represent optimal tradeoffs between all metrics present.\n", + "This is useful for competing metrics where a gain in one metric may represent a regression in the other.\n", + "\n", + "In these settings we can also specify outcome constraints, which tell Ax that below a certain threshold we are not interested in any result, regardless of the wins observed in any other metric.\n", + "For a concrete example, we could imagine Ax finding a parameterization that trains in no time at all but has an score no better than if the model were guessing at random.\n", + "\n", + "For this toy example let's configure Ax to maximize score and minimize training time, but avoid any hyperparameter configurations that result in a mean accuracy score of less than 75% or a training time greater than 1 second." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485079353, + "executionStopTime": 1739485079475, + "isAgentGenerated": false, + "language": "python", + "originalKey": "86e8826b-77ee-482c-8bc6-de3af8246a05", + "outputsInitialized": false, + "requestMsgId": "86e8826b-77ee-482c-8bc6-de3af8246a05", + "serverExecutionDuration": 3.385323099792, + "showInput": true + }, + "outputs": [], + "source": [ + "client.configure_optimization(\n", + " objective=\"score, -training_time\",\n", + " outcome_constraints=[\"score >= 0.85\", \"training_time <= 1\"],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "2b2645c0-1d38-4c9c-b083-3f095a7f8a73", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 5: Run Trials with early stopping\n", + "Before we begin our Bayesian optimization loop, we can attach the data from the default trial we ran to the experiment.\n", + "This will give us a head start on our optimization by providing a (likely good) datapoint to our surrogate model.\n", + "It is always advantageous to attach any existing data to an experiment to improve performance.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485079478, + "executionStopTime": 1739485079664, + "isAgentGenerated": false, + "language": "python", + "originalKey": "58b3a8e5-4ab9-4877-bde7-6461b5c2d3af", + "outputsInitialized": true, + "requestMsgId": "58b3a8e5-4ab9-4877-bde7-6461b5c2d3af", + "serverExecutionDuration": 16.574951820076, + "showInput": true + }, + "outputs": [], + "source": [ + "trial_index = client.attach_baseline(\n", + " parameters={\n", + " \"loss\": clf.loss,\n", + " \"penalty\": clf.penalty,\n", + " \"alpha\": clf.alpha,\n", + " \"learning_rate\": clf.learning_rate,\n", + " \"eta0\": clf.eta0\n", + " + 1e-8, # Default eta is 0.0, so add a small value to avoid division by zero\n", + " \"batch_size\": batch_size,\n", + " }\n", + ")\n", + "\n", + "client.complete_trial(\n", + " trial_index=trial_index,\n", + " raw_data={\"score\": score, \"training_time\": training_time},\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "8100a8fa-ccd9-4f52-bc7f-b4c66494ad5f", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "After attaching the initial trial, will begin the experimentation loop by writing a for loop to execute our full experimentation budget of 30 trials.\n", + "In each iteration we will ask Ax for the next trials (in this case just one), then instantiate an `SGDClassifier` with the suggested hyperparameters.\n", + "We will then split the data into train and test sets.\n", + "Next we will define an inner loop to perform minibatch training, in which we divide the train set into a number of smaller batches and train one epoch of stochastic gradient descent at a time.\n", + "After each epoch we will report the score and the time.\n", + "\n", + "Because training machine learning models is expensive, we will utilize Ax's early stopping functionality to kill trials unlikely to produce optimal results before they have been completed.\n", + "After data has been attached we will ask the `Client` whether or not we should stop the trial, and if it advises us to do so we will report it early stopped and exit out of the training loop." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "customOutput": null, + "executionStartTime": 1739485079666, + "executionStopTime": 1739485314259, + "isAgentGenerated": false, + "language": "python", + "originalKey": "4c64f747-e167-41fb-9066-95d10287bf3a", + "outputsInitialized": false, + "requestMsgId": "4c64f747-e167-41fb-9066-95d10287bf3a", + "serverExecutionDuration": 234449.878457, + "showInput": true + }, + "outputs": [], + "source": [ + "for _ in range(30):\n", + " trials = client.get_next_trials(maximum_trials=1)\n", + " for trial_index, parameters in trials.items():\n", + " clf = sklearn.linear_model.SGDClassifier(\n", + " loss=parameters[\"loss\"],\n", + " penalty=parameters[\"penalty\"],\n", + " alpha=parameters[\"alpha\"],\n", + " learning_rate=parameters[\"learning_rate\"],\n", + " eta0=parameters[\"eta0\"],\n", + " )\n", + "\n", + " train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n", + " digits.data,\n", + " digits.target,\n", + " test_size=0.20,\n", + " )\n", + "\n", + " batch_size = assert_is_instance(parameters[\"batch_size\"], int)\n", + "\n", + " start_time = time.time()\n", + " is_early_stopped = False # Flag to indicate if the trial was early stopped\n", + " for i in range(0, len(train_x) // batch_size):\n", + " start_idx = i * batch_size\n", + " end_idx = (i + 1) * batch_size\n", + "\n", + " # Use partial fit to update the model on the current batch\n", + " clf.partial_fit(\n", + " train_x[start_idx:end_idx], train_y[start_idx:end_idx], classes=classes\n", + " )\n", + "\n", + " raw_data = {\n", + " \"score\": clf.score(valid_x, valid_y),\n", + " \"training_time\": time.time() - start_time,\n", + " }\n", + " client.attach_data(\n", + " trial_index=trial_index,\n", + " raw_data=raw_data,\n", + " progression=end_idx, # Use the index of the last example in the batch as the progression value\n", + " )\n", + "\n", + " # If the trial is underperforming, stop it\n", + " if client.should_stop_trial_early(trial_index=trial_index):\n", + " client.mark_trial_early_stopped(\n", + " trial_index=trial_index,\n", + " raw_data=raw_data,\n", + " progression=end_idx,\n", + " )\n", + " is_early_stopped = True\n", + " break\n", + "\n", + " if not is_early_stopped:\n", + " # If the trial was not early stopped, mark it as completed\n", + " client.complete_trial(trial_index=trial_index)\n", + " print(f\"Completed trial {trial_index} with {parameters=}, {raw_data=}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "4d100826-3658-418c-ad0a-eff9fa2d2cc8", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 6: Analyze Results\n", + "\n", + "After running trials, you can analyze the results.\n", + "Most commonly this means extracting the parameterization from the best performing trial you conducted.\n", + "\n", + "Since we are optimizing multiple objectives, rather than a single best point we want to get the Pareto frontier -- the set of points that presents optimal tradeoffs between maximizing score and minimizing training time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485317774, + "executionStopTime": 1739485318269, + "isAgentGenerated": false, + "language": "python", + "originalKey": "e920d76d-d2bd-4fac-bb3d-854047a25d75", + "outputsInitialized": true, + "requestMsgId": "e920d76d-d2bd-4fac-bb3d-854047a25d75", + "serverExecutionDuration": 27.325752191246, + "showInput": true + }, + "outputs": [], + "source": [ + "frontier = client.get_pareto_frontier()\n", + "\n", + "# Frontier is a list of tuples, where each tuple contains the parameters, the metric readings, the trial index, and the arm name for a point on the Pareto frontier\n", + "for parameters, metrics, trial_index, arm_name in frontier:\n", + " print(f\"Trial {trial_index} with {parameters=} and {metrics=}\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "b8319cba-e144-4703-9d49-22943710ed2d", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 7: Compute Analyses\n", + "\n", + "Ax can also produce a number of analyses to help interpret the results of the experiment via `client.compute_analyses`.\n", + "Users can manually select which analyses to run, or can allow Ax to select which would be most relevant.\n", + "In this case Ax selects the following:\n", + "* **Scatter Plot** shows a plane with each objective on its own axis and a point for each observation. In multi-objective optimizations like ours it also draws a line through the Pareto frontier, indicating which points represent optimal tradeoffs between our objectives.\n", + "* **Interaction Analysis Plot** shows which parameters have the largest affect on the function and plots the most important parameters as 1 or 2 dimensional surfaces\n", + "* **Summary** lists all trials generated along with their parameterizations, observations, and miscellaneous metadata" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "customOutput": null, + "executionStartTime": 1739485320847, + "executionStopTime": 1739485321404, + "isAgentGenerated": false, + "language": "python", + "originalKey": "32f20264-24f5-410e-a869-af0b27eecc46", + "outputsInitialized": true, + "requestMsgId": "32f20264-24f5-410e-a869-af0b27eecc46", + "serverExecutionDuration": 378.66530800238, + "showInput": true + }, + "outputs": [], + "source": [ + "client.compute_analyses(display=True) # By default Ax will display the AnalysisCards produced by compute_analyses" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "06787f89-119a-4571-9db7-4c338374d5d9", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Conclusion\n", + "\n", + "This tutorial demonstates Ax's ability to solve AutoML tasks with in a resource efficient manor.\n", + "We configured a complex optimization which captures the nuanced goals of the experiment and utilized early stopping to save resources by killing training runs unlikely to produce optimal results.\n", + "\n", + "While this tutorial shows how to use Ax for HPO on an `SGDClassifier`, the same techniques can be used for many different AutoML tasks such as feature selection, architecture search, and more." + ] + } + ], + "metadata": { + "fileHeader": "", + "fileUid": "0e96c438-339b-4292-bbb7-b2747fc93475", + "isAdHoc": false, + "kernelspec": { + "display_name": "python3", + "isCinder": true, + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3" + } + } +} diff --git a/website/tutorials.json b/website/tutorials.json index 877ea6405e4..d7bffb94cbd 100644 --- a/website/tutorials.json +++ b/website/tutorials.json @@ -11,6 +11,10 @@ { "id": "closed_loop", "title": "Closed-loop Optimization with Ax" + }, + { + "id": "automl", + "title": "Ax for AutoML" } ], "Advanced features": [