diff --git a/tutorials/automl/automl.ipynb b/tutorials/automl/automl.ipynb new file mode 100644 index 00000000000..8bf46dfa5dd --- /dev/null +++ b/tutorials/automl/automl.ipynb @@ -0,0 +1,676 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "6889e8ed-572e-432c-aafe-23d791980a3b", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "# Ax for AutoML\n", + "Automated machine learning (AutoML) encompasses a large class of problems related to automating time-consuming and labor-intensive aspects of developing ML models.\n", + "Adaptive experimentation is a natural fit for solving many AutoML tasks, which are often iterative in nature and can involve many expensive trial evaluations.\n", + "\n", + "In this tutorial we will use Ax for hyperparameter optimization (HPO), a common AutoML task in which a model's hyperparameters are adjusted to improve model performance.\n", + "Hyperparameters refer to the parameters which are set prior to model training or fitting, rather than parameters being learned from data.\n", + "Traditionally, ML engineers use a combination of domain knowledge, intuition, and manual experimentation comparing many models with different hyperparameter configurations to determine good hyperparameters.\n", + "As the number of hyperparameters grows and as models become more expensive to train and evaluate sample efficient aproaches to experimentation like Bayesian optimization become increasingly valuable.\n", + "\n", + "In this tutorial we will train an `SGDClassifier` from the popular [scikit-learn](https://scikit-learn.org/) library to recognize handwritten digits and tune the model's hyperparameters to improve its performance.\n", + "You can read more about the `SGDClassifier` model in their example [here](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html), which this tutorial is largely based on.\n", + "This tutorial will incorporate many advanced features in Ax to demonstrate how they can be applied on complex engineering challenges in a real-world setting.\n", + "\n", + "## Learning Objectives\n", + "- Understand how Ax can be used for HPO tasks\n", + "- Use complex optimization configurations like multiple objectives and outcome constraints to achieve nuanced real-world goals\n", + "- Use early stopping to save experimentation resources\n", + "- Analyze the results of the optimization\n", + "\n", + "## Prerequisites\n", + "- Familiarity with [scikit-learn](https://scikit-learn.org/) and basic machine learning concepts\n", + "- Understanding of [adaptive experimentation](#) and [Bayesian optimization](#)\n", + "- [Ask-tell Optimization of Python Functions with early stopping](#)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "74d17b49-d048-4cd5-b727-75c3d597d019", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 1: Import Necessary Modules\n", + "\n", + "First, ensure you have all the necessary imports:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customOutput": null, + "executionStartTime": 1739485077830, + "executionStopTime": 1739485077965, + "isAgentGenerated": false, + "language": "python", + "originalKey": "5c09dddd-6087-4d3a-acd5-8acb069d8389", + "outputsInitialized": false, + "requestMsgId": "5c09dddd-6087-4d3a-acd5-8acb069d8389", + "serverExecutionDuration": 1.667891163379 + }, + "outputs": [], + "source": [ + "import time\n", + "\n", + "import matplotlib.pyplot as plt\n", + "\n", + "import sklearn.datasets\n", + "import sklearn.linear_model\n", + "import sklearn.model_selection\n", + "from ax.preview.api.client import Client\n", + "from ax.preview.api.configs import (\n", + " ChoiceParameterConfig,\n", + " ExperimentConfig,\n", + " GenerationMethod,\n", + " GenerationStrategyConfig,\n", + " ParameterScaling,\n", + " ParameterType,\n", + " RangeParameterConfig,\n", + ")\n", + "from pyre_extensions import assert_is_instance" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "b94a9e80-f1d4-40ee-8fe9-2c9b4f0e7ddd", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 1.1: Understanding the baseline performance of `SGDClassifier`\n", + "Before we begin HPO, let's understand the task and the performance of `SGDClassifier` with its default hyperparameters.\n", + "The following code is largely adapted from the example on scikit-learn's webiste [here](https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html).\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485077968, + "executionStopTime": 1739485078623, + "isAgentGenerated": false, + "language": "python", + "originalKey": "9d3eeb8a-03fb-4e20-b795-d06c3cd99b68", + "outputsInitialized": true, + "requestMsgId": "9d3eeb8a-03fb-4e20-b795-d06c3cd99b68", + "serverExecutionDuration": 185.13395590708, + "showInput": true + }, + "outputs": [], + "source": [ + "# Load the digits dataset and display the first 4 images to demonstrate\n", + "digits = sklearn.datasets.load_digits()\n", + "classes = list(set(digits.target))\n", + "\n", + "_, axes = plt.subplots(nrows=1, ncols=4, figsize=(10, 3))\n", + "for ax, image, label in zip(axes, digits.images, digits.target):\n", + " ax.set_axis_off()\n", + " ax.imshow(image, cmap=plt.cm.gray_r, interpolation=\"nearest\")\n", + " ax.set_title(\"Training: %i\" % label)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485078629, + "executionStopTime": 1739485079003, + "isAgentGenerated": false, + "language": "python", + "originalKey": "199de951-1af5-4a57-b341-e67f823c9e69", + "outputsInitialized": true, + "requestMsgId": "199de951-1af5-4a57-b341-e67f823c9e69", + "serverExecutionDuration": 33.779789227992, + "showInput": true + }, + "outputs": [], + "source": [ + "# Instantiate a SGDClassifier with default hyperparameters\n", + "clf = sklearn.linear_model.SGDClassifier()\n", + "\n", + "# Split the data into a training set and a validation set\n", + "train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n", + " digits.data, digits.target, test_size=0.20, random_state=0\n", + ")\n", + "\n", + "# Train the classifier on the training set using 10 batches\n", + "#Also time the training.\n", + "batch_size = len(train_x) // 10\n", + "start_time = time.time()\n", + "for i in range(10):\n", + " start_idx = i * batch_size\n", + " end_idx = (i + 1) * batch_size\n", + "\n", + " # Use partial fit to update the model on the current batch\n", + " clf.partial_fit(\n", + " train_x[start_idx:end_idx], train_y[start_idx:end_idx], classes=classes\n", + " )\n", + "\n", + "training_time = time.time() - start_time\n", + "\n", + "# Evaluate the classifier on the validation set\n", + "score = clf.score(valid_x, valid_y)\n", + "score, training_time" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "ca935c9c-23c1-444b-aecc-09eac0452e44", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "The model performs well, but let's see if we can improve performance by tuning the hyperparameters." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "6caa7b9a-17bd-4e0b-9e45-9ded2d452a45", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 2: Initialize the Client\n", + "As always, the first step in running our adaptive experiment with Ax is to create an instance of the `Client` to manage the state of your experiment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485079013, + "executionStopTime": 1739485079217, + "isAgentGenerated": false, + "language": "python", + "originalKey": "f71bb5ed-6ff4-433c-9ca1-6d00df0a9f17", + "outputsInitialized": false, + "requestMsgId": "f71bb5ed-6ff4-433c-9ca1-6d00df0a9f17", + "serverExecutionDuration": 1.6501909121871, + "showInput": true + }, + "outputs": [], + "source": [ + "client = Client()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "2eb882e0-4b3c-4308-acd4-f2563a421b5f", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 3: Configure the Experiment\n", + "\n", + "The `Client` expects a series of `Config`s which define how the experiment will be run.\n", + "We'll set this up the same way as we did in our previous tutorial.\n", + "\n", + "Our current task is to tune the hyperparameters of an scikit-learn's [SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html).\n", + "These parameters control aspects of the model's training process and configuring them can have dramatic effects on the model's ability to correctly classify inputs.\n", + "A full list of this model's hyperparameters and appropriate values are available in the library's documentation.\n", + "In this tutorial we will tune the following hyperparameters:\n", + "* **loss:** The loss function to be used\n", + "* **penalty:** The penalty (aka regularization term) to be used\n", + "* **learning_rate:** The learning rate schedule\n", + "* **alpha:** Constant that multiplies the regularization term. The higher the value, the stronger the regularization\n", + "* **eta0**: The learning rate for training. In this example we will use a constant learning rate schedule\n", + "* **batch_size**: A training parameter which controls how many examples are shown during a single epoch. We will use all samples in the dataset for each model training, so a smaller batch size will translate to more epochs and vice versa.\n", + " \n", + "You will notice some hyperparameters are continuous ranges, some are discrete ranges, and some are categorical choices; Ax is able to handle all of these types of parameters via its `RangeParameterConfig` and `ChoiceParameterConfig` classes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "customOutput": null, + "executionStartTime": 1739485079221, + "executionStopTime": 1739485079350, + "isAgentGenerated": false, + "language": "python", + "originalKey": "ba904311-af2e-491a-a25e-157602e0e906", + "outputsInitialized": true, + "requestMsgId": "ba904311-af2e-491a-a25e-157602e0e906", + "serverExecutionDuration": 2.7205082587898, + "showInput": true + }, + "outputs": [], + "source": [ + "# Create an experiment configuration\n", + "experiment_config = ExperimentConfig(\n", + " name=\"SGDClassifier_hpo\",\n", + " parameters=[\n", + " ChoiceParameterConfig(\n", + " name=\"loss\",\n", + " parameter_type=ParameterType.STRING,\n", + " values=[\n", + " \"hinge\",\n", + " \"log_loss\",\n", + " \"squared_hinge\",\n", + " \"modified_huber\",\n", + " \"perceptron\",\n", + " ],\n", + " is_ordered=False,\n", + " ),\n", + " ChoiceParameterConfig(\n", + " name=\"penalty\",\n", + " parameter_type=ParameterType.STRING,\n", + " values=[\"l1\", \"l2\", \"elasticnet\"],\n", + " is_ordered=False,\n", + " ),\n", + " ChoiceParameterConfig(\n", + " name=\"learning_rate\",\n", + " parameter_type=ParameterType.STRING,\n", + " values=[\"constant\", \"optimal\", \"invscaling\", \"adaptive\"],\n", + " is_ordered=False,\n", + " ),\n", + " RangeParameterConfig(\n", + " name=\"alpha\",\n", + " bounds=(1e-8, 100),\n", + " parameter_type=ParameterType.FLOAT,\n", + " scaling=ParameterScaling.LOG,\n", + " ),\n", + " RangeParameterConfig(\n", + " name=\"eta0\",\n", + " bounds=(1e-8, 1),\n", + " parameter_type=ParameterType.FLOAT,\n", + " scaling=ParameterScaling.LOG,\n", + " ),\n", + " RangeParameterConfig(\n", + " name=\"batch_size\",\n", + " bounds=(5, 500),\n", + " parameter_type=ParameterType.INT,\n", + " ),\n", + " ],\n", + " # The following arguments are optional\n", + " description=\"Optimization of SGDClassifier for digits dataset\",\n", + " owner=\"developer\",\n", + ")\n", + "\n", + "# Apply the experiment configuration to the client\n", + "client.configure_experiment(experiment_config=experiment_config)\n", + "\n", + "client.configure_generation_strategy(\n", + " GenerationStrategyConfig(method=GenerationMethod.FAST)\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "59cd82c7-81a4-4411-bc4a-2f7db3effbf1", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 4: Configure Optimization\n", + "Now, we must set up the optimization objective in `Client`, where `objective` is a string that specifies which metric we would like to optimize and the direction (higher or lower) that is considered optimal.\n", + "\n", + "In our example we want to consider both performance and computational cost implications of hyperparameter modifications.\n", + "`scikit-learn` models use a function called `score` to report the mean accuracy of the model, and in our optimization we should seek to maximize this value.\n", + "Since model training can be a very expensive process, especially for large models, this can represent a significant cost.\n", + "\n", + "Let's configure Ax to maximize score while minimizing training time.\n", + "We call this a multi-objective optimization, and rather than returning a single best parameterization we return a Pareto frontier of points which represent optimal tradeoffs between all metrics present.\n", + "Multi-objective optimization is useful for competing metrics where a gain in one metric may represent a regression in the other.\n", + "\n", + "In these settings we can also specify outcome constraints, which indicate that if a metric result falls outside of the specified threshold we are not interested in any result, regardless of the wins observed in any other metric.\n", + "For a concrete example, imagine Ax finding a parameterization that trains in no time at all but has an score no better than if the model were guessing at random.\n", + "\n", + "For this toy example let's configure Ax to maximize score and minimize training time, but avoid any hyperparameter configurations that result in a mean accuracy score of less than 75% or a training time greater than 1 second." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485079353, + "executionStopTime": 1739485079475, + "isAgentGenerated": false, + "language": "python", + "originalKey": "86e8826b-77ee-482c-8bc6-de3af8246a05", + "outputsInitialized": false, + "requestMsgId": "86e8826b-77ee-482c-8bc6-de3af8246a05", + "serverExecutionDuration": 3.385323099792, + "showInput": true + }, + "outputs": [], + "source": [ + "client.configure_optimization(\n", + " objective=\"score, -training_time\",\n", + " outcome_constraints=[\"score >= 0.85\", \"training_time <= 1\"],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "2b2645c0-1d38-4c9c-b083-3f095a7f8a73", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 5: Run Trials with early stopping\n", + "Before we begin our Bayesian optimization loop, we can attach the data we collected from triaing `SGDClassifier` with default hyperparameters.\n", + "This will give our experiment a head start by providing a datapoint to our surrogate model.\n", + "Because these are the default settings provided by `scikit-learn`, it's likely they will be pretty good and will provide the optimization with a promising start.\n", + "It is always advantageous to attach any existing data to an experiment to improve performance.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485079478, + "executionStopTime": 1739485079664, + "isAgentGenerated": false, + "language": "python", + "originalKey": "58b3a8e5-4ab9-4877-bde7-6461b5c2d3af", + "outputsInitialized": true, + "requestMsgId": "58b3a8e5-4ab9-4877-bde7-6461b5c2d3af", + "serverExecutionDuration": 16.574951820076, + "showInput": true + }, + "outputs": [], + "source": [ + "trial_index = client.attach_baseline(\n", + " parameters={\n", + " \"loss\": clf.loss,\n", + " \"penalty\": clf.penalty,\n", + " \"alpha\": clf.alpha,\n", + " \"learning_rate\": clf.learning_rate,\n", + " \"eta0\": clf.eta0\n", + " + 1e-8, # Default eta is 0.0, so add a small value to avoid division by zero\n", + " \"batch_size\": batch_size,\n", + " }\n", + ")\n", + "\n", + "client.complete_trial(\n", + " trial_index=trial_index,\n", + " raw_data={\"score\": score, \"training_time\": training_time},\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "8100a8fa-ccd9-4f52-bc7f-b4c66494ad5f", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "After attaching the initial trial, we will begin the experimentation loop by writing a for loop to execute our full experimentation budget of 30 trials.\n", + "In each iteration we will ask Ax for the next trials (in this case just one), then instantiate an `SGDClassifier` with the suggested hyperparameters.\n", + "We will then split the data into train and test sets.\n", + "Next we will define an inner loop to perform minibatch training, in which we divide the train set into a number of smaller batches and train one epoch of stochastic gradient descent at a time.\n", + "After each epoch we will report the score and the time.\n", + "\n", + "Because training machine learning models is expensive, we will utilize Ax's early stopping functionality to kill trials unlikely to produce optimal results before they have been completed.\n", + "After data has been attached we will ask the `Client` whether or not we should stop the trial, and if it advises us to do so we will report it early stopped and exit out of the training loop.\n", + "By early stopping, we proactively save compute without regressing optimization performance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "customOutput": null, + "executionStartTime": 1739485079666, + "executionStopTime": 1739485314259, + "isAgentGenerated": false, + "language": "python", + "originalKey": "4c64f747-e167-41fb-9066-95d10287bf3a", + "outputsInitialized": false, + "requestMsgId": "4c64f747-e167-41fb-9066-95d10287bf3a", + "serverExecutionDuration": 234449.878457, + "showInput": true + }, + "outputs": [], + "source": [ + "for _ in range(30):\n", + " trials = client.get_next_trials(maximum_trials=1)\n", + " for trial_index, parameters in trials.items():\n", + " clf = sklearn.linear_model.SGDClassifier(\n", + " loss=parameters[\"loss\"],\n", + " penalty=parameters[\"penalty\"],\n", + " alpha=parameters[\"alpha\"],\n", + " learning_rate=parameters[\"learning_rate\"],\n", + " eta0=parameters[\"eta0\"],\n", + " )\n", + "\n", + " train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n", + " digits.data,\n", + " digits.target,\n", + " test_size=0.20,\n", + " )\n", + "\n", + " batch_size = assert_is_instance(parameters[\"batch_size\"], int)\n", + "\n", + " start_time = time.time()\n", + " is_early_stopped = False # Flag to indicate if the trial was early stopped\n", + " for i in range(0, len(train_x) // batch_size):\n", + " start_idx = i * batch_size\n", + " end_idx = (i + 1) * batch_size\n", + "\n", + " # Use partial fit to update the model on the current batch\n", + " clf.partial_fit(\n", + " train_x[start_idx:end_idx], train_y[start_idx:end_idx], classes=classes\n", + " )\n", + "\n", + " raw_data = {\n", + " \"score\": clf.score(valid_x, valid_y),\n", + " \"training_time\": time.time() - start_time,\n", + " }\n", + " client.attach_data(\n", + " trial_index=trial_index,\n", + " raw_data=raw_data,\n", + " progression=end_idx, # Use the index of the last example in the batch as the progression value\n", + " )\n", + "\n", + " # If the trial is underperforming, stop it\n", + " if client.should_stop_trial_early(trial_index=trial_index):\n", + " client.mark_trial_early_stopped(\n", + " trial_index=trial_index,\n", + " raw_data=raw_data,\n", + " progression=end_idx,\n", + " )\n", + " is_early_stopped = True\n", + " break\n", + "\n", + " if not is_early_stopped:\n", + " # If the trial was not early stopped, mark it as completed\n", + " client.complete_trial(trial_index=trial_index)\n", + " print(f\"Completed trial {trial_index} with {parameters=}, {raw_data=}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "4d100826-3658-418c-ad0a-eff9fa2d2cc8", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 6: Analyze Results\n", + "\n", + "After running trials, you can analyze the results.\n", + "Most commonly this means extracting the parameterization from the best performing trial you conducted.\n", + "\n", + "Since we are optimizing multiple objectives, rather than a single best point we want to get the Pareto frontier -- the set of points that presents optimal tradeoffs between maximizing score and minimizing training time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "executionStartTime": 1739485317774, + "executionStopTime": 1739485318269, + "isAgentGenerated": false, + "language": "python", + "originalKey": "e920d76d-d2bd-4fac-bb3d-854047a25d75", + "outputsInitialized": true, + "requestMsgId": "e920d76d-d2bd-4fac-bb3d-854047a25d75", + "serverExecutionDuration": 27.325752191246, + "showInput": true + }, + "outputs": [], + "source": [ + "frontier = client.get_pareto_frontier()\n", + "\n", + "# Frontier is a list of tuples, where each tuple contains the parameters, the metric readings, the trial index, and the arm name for a point on the Pareto frontier\n", + "for parameters, metrics, trial_index, arm_name in frontier:\n", + " print(f\"Trial {trial_index} with {parameters=} and {metrics=}\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "b8319cba-e144-4703-9d49-22943710ed2d", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Step 7: Compute Analyses\n", + "\n", + "Ax can also produce a number of analyses to help interpret the results of the experiment via `client.compute_analyses`.\n", + "Users can manually select which analyses to run, or can allow Ax to select which would be most relevant.\n", + "In this case Ax selects the following:\n", + "* **Scatter Plot** shows a plane with each objective on its own axis and a point for each observation. In multi-objective optimizations like ours it also draws a line through the Pareto frontier, indicating which points represent optimal tradeoffs between our objectives.\n", + "* **Interaction Analysis Plot** shows which parameters have the largest affect on the function and plots the most important parameters as 1 or 2 dimensional surfaces\n", + "* **Summary** lists all trials generated along with their parameterizations, observations, and miscellaneous metadata" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": false, + "customInput": null, + "customOutput": null, + "executionStartTime": 1739485320847, + "executionStopTime": 1739485321404, + "isAgentGenerated": false, + "language": "python", + "originalKey": "32f20264-24f5-410e-a869-af0b27eecc46", + "outputsInitialized": true, + "requestMsgId": "32f20264-24f5-410e-a869-af0b27eecc46", + "serverExecutionDuration": 378.66530800238, + "showInput": true + }, + "outputs": [], + "source": [ + "client.compute_analyses(display=True) # By default Ax will display the AnalysisCards produced by compute_analyses" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "customInput": null, + "isAgentGenerated": false, + "language": "markdown", + "originalKey": "06787f89-119a-4571-9db7-4c338374d5d9", + "outputsInitialized": false, + "showInput": false + }, + "source": [ + "## Conclusion\n", + "\n", + "This tutorial demonstates Ax's ability to solve AutoML tasks with in a resource efficient manor.\n", + "We configured a complex optimization which captures the nuanced goals of the experiment and utilized early stopping to save resources by killing training runs unlikely to produce optimal results.\n", + "\n", + "While this tutorial shows how to use Ax for HPO on an `SGDClassifier`, the same techniques can be used for many different AutoML tasks such as feature selection, architecture search, and more." + ] + } + ], + "metadata": { + "fileHeader": "", + "fileUid": "0e96c438-339b-4292-bbb7-b2747fc93475", + "isAdHoc": false, + "kernelspec": { + "display_name": "python3", + "isCinder": true, + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/website/tutorials.json b/website/tutorials.json index 877ea6405e4..d7bffb94cbd 100644 --- a/website/tutorials.json +++ b/website/tutorials.json @@ -11,6 +11,10 @@ { "id": "closed_loop", "title": "Closed-loop Optimization with Ax" + }, + { + "id": "automl", + "title": "Ax for AutoML" } ], "Advanced features": [