Merge pull request #1 from dreoporto/load-or-fit-model

Load or fit model

Author: dreoporto

README.md

@@ -2,13 +2,12 @@
 
 ## Summary
 
-**PTMLib** is a set of utilities that I have used with Machine Learning frameworks such as Scikit-Learn and TensorFlow.  The purpose is to eliminate code that I kept repeating in multiple projects.  
+**PTMLib** is a set of utilities for use with Machine Learning frameworks such as Scikit-Learn and TensorFlow.
 
 - **ptmlib.time.Stopwatch** - measure the time it takes to complete a long-running task, with an audio alert for task completion
 - **ptmlib.cpu.CpuCount** - get info on CPUs available, with options to adjust/exclude based on a specific number/percentage.  Useful for setting `n_jobs` in Scikit-Learn tools that support multiple CPUs, such as `RandomForestClassifier`
-- **ptmlib.charts** - render separate line charts for TensorFlow accuracy and loss, with corresponding validation data if available
-
-*This code and documentation were created by Andre Oporto at [Pendragon AI](https://www.pendragonai.com)*
+- **ptmlib.charts** - render separate line charts for TensorFlow metrics such as accuracy and loss, with corresponding validation data if available
+- **ptmlib.model_tools.load_or_fit_model()** - train, save, and reload Tensorflow models and metric charts automatically, making it easier to pick up where you left off
 
 ## ptmlib.time.Stopwatch
 
@@ -133,6 +132,44 @@ TensorFlow History Loss Chart: *loss-20210201-111545.png*
 
 ![TF History Accuracy Chart](ptmlib/examples/loss-20210201-111545.png)
 
+The default file name format for these images is *searchstring-timestamp.png*.  The `file_name_suffix` parameter lets you replace the timestamp with another value, for more predictable filenames to simplify reuse of images in your code.
+
+## ptmlib.model_tools.load_or_fit_model()
+
+The `ptmlib.model_tools.load_or_fit_model()` function makes it easy to train and save a model for later use, in cases where you may need to stop and restart work in Jupyter or your IDE *after* model training has completed.  This can be very helpful when working through a long and detailed notebook with multiple example models, where some models take significant time to train.  You can avoid repeatedly training models you are satisfied with and have completed, and still close and reopen your notebook as needed.
+
+### Example Usage:
+
+```python
+# from examples/computer_vision_caching.py
+
+import ptmlib.model_tools as modt
+
+...
+
+model_file_name = "computer_vision_1"
+
+...
+
+fit_model_function_with_callback = lambda my_model, x, y, validation_data, epochs: my_model.fit(
+    x, y, validation_data, epochs=epochs, callbacks=[early_callback], 
+    validation_split=hp_validation_split)
+
+# if this has previously been executed, we will load the trained/saved model 
+model, history = modt.load_or_fit_model(model, model_file_name, x=training_images, y=training_labels,
+    epochs=hp_epochs, fit_model_function=fit_model_function_with_callback, metrics=["accuracy"])
+
+model.evaluate(test_images, test_labels)
+```
+
+### Example Output:
+
+You will see output similar to the following if you re-run a previously saved notebook where `load_or_fit_model` was used.
+
+![Sample load_or_fit_model Screenshot](ptmlib/media/load_or_fit_model_screenshot.png)
+
+If you wish to retrain a model that has previously been saved, simply delete the model file and related images, which are stored as `h5` and `png` files respectively.
+
 ## Installation
 
 To install `ptmlib` in a virtualenv or conda environment:

ptmlib/charts.py

@@ -28,6 +28,10 @@ def show_history_chart(history: Any, search_string: str, fig_size: (int, int) =
 
     filtered_hist = {k: v for (k, v) in history.history.items() if search_string in k}
 
+    if len(filtered_hist.keys()) == 0:
+        print('No data to plot for search_string:', search_string)
+        return
+
     pd.DataFrame(filtered_hist).plot(figsize=fig_size)
     plt.grid(True, which='major')
     plt.grid(True, which='minor', alpha=0.3, linestyle='--')

ptmlib/examples/computer_vision_caching.py

@@ -0,0 +1,102 @@
+# THIS CODE IS A MODULARIZED AND UPDATED VERSION OF CODE FROM THE "DEEPLEARNING.AI TENSORFLOW DEVELOPER" COURSE
+# SOURCE:
+# https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%204%20-%20Lesson%202%20-%20Notebook.ipynb
+
+import matplotlib.pyplot as plt
+import numpy as np
+import tensorflow as tf
+from tensorflow import keras
+from tensorflow.keras import layers
+
+import ptmlib.model_tools as modt
+
+
+class MyCallback(keras.callbacks.Callback):
+
+    def __init__(self, target):
+        super().__init__()
+        self.target = target
+
+    def on_epoch_end(self, _, logs=None):
+
+        if logs is None:
+            logs = {}
+        if logs.get("accuracy") > self.target:
+            print(f"\nReached {self.target * 100}% accuracy so cancelling training!")
+            self.model.stop_training = True
+
+
+def print_diagnostics() -> None:
+    print('TF VERSION:', tf.__version__)
+    print('KERAS VERSION:', keras.__version__)
+
+
+def get_data():
+    mnist = keras.datasets.fashion_mnist
+    (training_images, training_labels), (test_images, test_labels) = mnist.load_data()
+
+    np.set_printoptions(linewidth=200)
+    plt.imshow(training_images[0])
+    plt.show()
+
+    print(training_labels[0])
+
+    # normalize image data to values between 0 and 1
+    training_images = training_images / 255.0
+    test_images = test_images / 255.0
+
+    return (training_images, training_labels), (test_images, test_labels)
+
+
+def get_model() -> keras.models.Sequential:
+    model = keras.models.Sequential([
+        layers.Flatten(input_shape=(28, 28)),
+        layers.Dropout(0.2),
+        layers.Dense(512, activation=tf.nn.relu),
+        layers.Dense(10, activation=tf.nn.softmax)
+    ])
+
+    model.summary()
+
+    model.compile(
+        optimizer=tf.optimizers.Adam(),
+        loss="sparse_categorical_crossentropy",
+        metrics=["accuracy"]
+    )
+
+    return model
+
+
+def main():
+
+    # HYPER PARAMS, CONSTANTS, ETC
+    hp_epochs = 50
+    hp_target = 0.91
+    hp_validation_split = 0.2
+    model_file_name = "computer_vision_1"
+
+    print_diagnostics()
+
+    (training_images, training_labels), (test_images, test_labels) = get_data()
+
+    model = get_model()
+
+    early_callback = MyCallback(target=hp_target)
+
+    fit_model_function_with_callback = lambda my_model, x, y, validation_data, epochs: my_model.fit(
+        x, y, validation_data, epochs=epochs, callbacks=[early_callback], validation_split=hp_validation_split)
+
+    model, history = modt.load_or_fit_model(model, model_file_name, x=training_images, y=training_labels,
+                                            epochs=hp_epochs, fit_model_function=fit_model_function_with_callback,
+                                            metrics=["accuracy"])
+
+    model.evaluate(test_images, test_labels)
+
+    classifications = model.predict(test_images)
+    print(classifications[0])
+    print(test_labels[0])
+    print(max(classifications[0]))
+
+
+if __name__ == '__main__':
+    main()

ptmlib/media/load_or_fit_model_screenshot.png

@@ -0,0 +1,67 @@
+import os
+from typing import Any, List
+
+import matplotlib.image as mpimg
+import matplotlib.pyplot as plt
+from tensorflow import keras
+
+import ptmlib.charts as pch
+from ptmlib.time import Stopwatch
+
+
+def default_load_model_function(model_file_name: str):
+    return keras.models.load_model(f'{model_file_name}.h5')
+
+
+def _default_fit_model_function(model: Any, x: Any, y: Any = None, validation_data: Any = None, epochs: int = 1):
+    return model.fit(x, y, validation_data=validation_data, epochs=epochs)
+
+
+def load_or_fit_model(model: Any, model_file_name: str, x: Any, y: Any = None, validation_data: Any = None,
+                      epochs: int = 1, metrics: List[str] = None, images_enabled=True, fig_size: (int, int) = (10, 6),
+                      load_model_function=default_load_model_function,
+                      fit_model_function=_default_fit_model_function):
+    history = None
+
+    if os.path.exists(f'{model_file_name}.h5'):
+        print(f'Loading existing model file: {model_file_name}.h5')
+        model = load_model_function(model_file_name)
+        if images_enabled:
+            _show_saved_images(metrics, model_file_name, fig_size)
+    else:
+        stopwatch = Stopwatch()
+        stopwatch.start()
+        history = fit_model_function(model, x, y, validation_data, epochs)
+        stopwatch.stop()
+        print(f'Saving new model file: {model_file_name}.h5')
+        model.save(f'{model_file_name}.h5')
+        if images_enabled:
+            _show_new_images(history, model_file_name, metrics)
+
+    return model, history
+
+
+def _show_new_images(history: Any, model_file_name: str, metrics: List[str]):
+    if metrics is not None:
+        for metric in metrics:
+            pch.show_history_chart(history, metric, save_fig_enabled=True, file_name_suffix=model_file_name)
+    pch.show_history_chart(history, "loss", save_fig_enabled=True, file_name_suffix=model_file_name)
+
+
+def _show_saved_images(metrics: List[str], model_file_name: str, fig_size: (int, int) = (10, 6)):
+    if metrics is not None:
+        for metric in metrics:
+            if os.path.exists(f'{metric}-{model_file_name}.png'):
+                _show_saved_image(f'{metric}-{model_file_name}.png', fig_size)
+    if os.path.exists(f'loss-{model_file_name}.png'):
+        _show_saved_image(f'loss-{model_file_name}.png', fig_size)
+
+
+def _show_saved_image(filename: str, fig_size: (int, int) = (10, 6)):
+    image_data = mpimg.imread(filename)
+    fig = plt.figure(figsize=fig_size)
+    ax = plt.Axes(fig, [0., 0., 1., 1.])
+    fig.add_axes(ax)
+    plt.axis('off')
+    plt.imshow(image_data)
+    plt.show()