Modernize Ludwig for v0.11 release

.github/ISSUE_TEMPLATE/bug_report.md

@@ -28,8 +28,8 @@ If applicable, add screenshots to help explain your problem.
 
 **Environment (please complete the following information):**
 
-- OS: \[e.g. iOS\]
-- Version \[e.g. 22\]
+- OS: [e.g. iOS]
+- Version [e.g. 22]
 - Python version
 - Ludwig version
 

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,7 +7,7 @@ assignees: ''
 ---
 
 **Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when \[...\]
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
 
 **Describe the use case**
 A clear and concise description of what the use case for this feature is.

.github/workflows/upload-pypi.yml

@@ -18,17 +18,16 @@ jobs:
       # IMPORTANT: this permission is mandatory for trusted publishing
       id-token: write
     steps:
-      # retrieve your distributions here
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
         with:
           submodules: "recursive"
 
-      - uses: actions/setup-python@v2
+      - uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: "3.12"
 
-      - name: Build and upload to PyPI
+      - name: Build source distribution
         run: |
           python setup.py sdist
 

.gitignore

@@ -140,3 +140,15 @@ examples/*/visualizations/
 
 # benchmarking configs
 ludwig/benchmarking/configs/
+
+# Aim tracking
+.aim/
+
+# Comet
+.comet.config
+
+# Test-generated artifacts (image/audio features)
+*.png
+*.wav
+generated_audio/
+generated_images/

.pre-commit-config.yaml

@@ -14,10 +14,10 @@ ci:
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v6.0.0
     hooks:
       - id: check-ast
-      - id: check-byte-order-marker
+      - id: fix-byte-order-marker
       - id: check-case-conflict
       - id: check-executables-have-shebangs
       - id: check-json
@@ -29,43 +29,43 @@ repos:
       - id: trailing-whitespace
       - id: mixed-line-ending
   - repo: https://github.com/asottile/pyupgrade
-    rev: v3.3.1
+    rev: v3.21.2
     hooks:
       - id: pyupgrade
-        args: [--py36-plus]
+        args: [--py310-plus]
   - repo: https://github.com/PyCQA/docformatter
-    rev: v1.5.1
+    rev: v1.7.7
     hooks:
       - id: docformatter
         args: [--in-place, --wrap-summaries=115, --wrap-descriptions=120]
   - repo: https://github.com/PyCQA/isort
-    rev: 5.12.0
+    rev: 8.0.0
     hooks:
       - id: isort
         name: Format imports
   - repo: https://github.com/pycqa/flake8
-    rev: 6.0.0
+    rev: 7.3.0
     hooks:
       - id: flake8
   - repo: https://github.com/psf/black
-    rev: 23.3.0
+    rev: 26.1.0
     hooks:
       - id: black
         name: Format code
   - repo: https://github.com/asottile/blacken-docs
-    rev: 1.13.0
+    rev: 1.20.0
     hooks:
       - id: blacken-docs
         args: [--line-length=120]
-  - repo: https://github.com/executablebooks/mdformat
-    rev: 0.7.16
+  - repo: https://github.com/hukkin/mdformat
+    rev: 1.0.0
     hooks:
       - id: mdformat
         additional_dependencies:
-          - mdformat-gfm
-          - mdformat_frontmatter
+          - mdformat-gfm==1.0.0
+          - mdformat_frontmatter==2.0.10
         exclude: CHANGELOG.md
   - repo: https://github.com/yoheimuta/protolint
-    rev: v0.42.2
+    rev: v0.56.4
     hooks:
       - id: protolint

CONTRIBUTING.md

@@ -49,7 +49,7 @@ Work on your self-assigned issue and eventually create a Pull Request.
    pip install -e .
    ```
 
-   The above command will install only the packages in "requirements.txt" in the developer mode.  If you would like to
+   The above command will install only the packages in "requirements.txt" in the developer mode. If you would like to
    be able to potentially make changes to the overall Ludwig codebase, then use the following command:
 
    ```bash
@@ -93,9 +93,9 @@ Work on your self-assigned issue and eventually create a Pull Request.
 
 1. Finally, if during the installation of `horovod`, the exception `ModuleNotFoundError: No module named 'packaging'`
    occurs, you may need to disable `horovod` (that means you will need train with another backend, just not `horovod`).
-   To do that, edit the file `requirements_extra.txt` and comment out the line that begins with `horovod`.  After that,
-   please execute the long `pip install` command given in the previous step.  With these work-around provisions, your
-   installation should run to completion successfully.  If you are still having difficulty, please reach out with the
+   To do that, edit the file `requirements_extra.txt` and comment out the line that begins with `horovod`. After that,
+   please execute the long `pip install` command given in the previous step. With these work-around provisions, your
+   installation should run to completion successfully. If you are still having difficulty, please reach out with the
    specifics of your environment in the Ludwig Community [Discord](https://discord.gg/CBgdrGnZjy).
 
 1. Develop features on your branch.

README.md

@@ -17,7 +17,7 @@ _Declarative deep learning framework built for scale and efficiency._
 
 </div>
 
-> \[!IMPORTANT\]
+> [!IMPORTANT]
 > Our community has moved to [Discord](https://discord.gg/CBgdrGnZjy) -- please join us there!
 
 # 📖 What is Ludwig?
@@ -39,7 +39,7 @@ Ludwig is hosted by the
 
 # 💾 Installation
 
-Install from PyPi. Be aware that Ludwig requires Python 3.8+.
+Install from PyPi. Be aware that Ludwig requires Python 3.10+.
 
 ```shell
 pip install ludwig
@@ -55,7 +55,7 @@ Please see [contributing](https://github.com/ludwig-ai/ludwig/blob/master/CONTRI
 
 # 🚂 Getting Started
 
-Want to take a quick peek at some of the Ludwig 0.8 features? Check out this Colab Notebook 🚀 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lB4ALmEyvcMycE3Mlnsd7I3bc0zxvk39)
+Want to take a quick peek at some of Ludwig's features? Check out this Colab Notebook 🚀 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lB4ALmEyvcMycE3Mlnsd7I3bc0zxvk39)
 
 Looking to fine-tune Llama-2 or Mistral? Check out these notebooks:
 
@@ -335,6 +335,6 @@ more accessible and feature rich framework for everyone to use!
 # 👋 Getting Involved
 
 - [Discord](https://discord.gg/CBgdrGnZjy)
-- [X](https://twitter.com/ludwig_ai)
+- [X (Twitter)](https://twitter.com/ludwig_ai)
 - [Medium](https://medium.com/ludwig-ai)
 - [GitHub Issues](https://github.com/ludwig-ai/ludwig/issues)

README_KR.md

@@ -59,15 +59,14 @@ pip install ludwig
 - `ludwig[audio]` for audio and speech dependencies.
 - `ludwig[image]` for image dependencies.
 - `ludwig[hyperopt]` for hyperparameter optimization dependencies.
-- `ludwig[horovod]` for distributed training dependencies.
+- `ludwig[distributed]` for distributed training dependencies.
 - `ludwig[serve]` for serving dependencies.
 - `ludwig[viz]` for visualization dependencies.
 - `ludwig[test]` for dependencies needed for testing.
 
-[Horovod](https://github.com/horovod/horovod)를 통해 분산 학습이 지원되며, `pip install ludwig[horovod]` 또는 `HOROVOD_GPU_OPERATIONS=NCCL pip install ludwig[horovod]` 와 같이 GPU 환경에서 설치가 가능합니다.
-설치 가능한 옵션들을 더 확인하고 싶으시다면 Horovod's [installation guide](https://horovod.readthedocs.io/en/stable/install_include.html) 를 참고하시기 바랍니다.
+[Ray](https://www.ray.io/)를 통해 분산 학습이 지원되며, `pip install ludwig[distributed]`와 같이 설치가 가능합니다.
 
-추가하려는 package들은  `pip install ludwig[extra1,extra2,...]` 의 명령어를 통해 설치가 가능합니다. 예를 들어, `pip install ludwig[text,viz]` 와 같은 조합으로 설치가 가능합니다. 모든 파일들을 한 번에 설치하려면 `pip install ludwig[full]`을 사용하면 됩니다.
+추가하려는 package들은 `pip install ludwig[extra1,extra2,...]` 의 명령어를 통해 설치가 가능합니다. 예를 들어, `pip install ludwig[text,viz]` 와 같은 조합으로 설치가 가능합니다. 모든 파일들을 한 번에 설치하려면 `pip install ludwig[full]`을 사용하면 됩니다.
 
 소스코드를 repository에서 build하려는 개발자들은 아래와 같은 방법을 사용하면 됩니다.
 
@@ -177,7 +176,7 @@ ludwig visualize --visualization learning_curves --training_statistics path/to/t
 
 ## Distributed Training
 
-사용자는 [Horovod](https://github.com/horovod/horovod)를 통해 사용자가 훈련시킨 모델을 배포할 수 있고 여러 GPU가 있는 단일 기계 및 여러 GPU가 있는 다중 기계를 통해 학습하는 것을 허용합니다. 더 자세한 정보를 알고 싶으시다면 [User Guide](https://ludwig-ai.github.io/ludwig-docs/latest/user_guide/distributed_training/)를 확인해 주시기 바랍니다.
+사용자는 [Ray](https://www.ray.io/)를 통해 사용자가 훈련시킨 모델을 배포할 수 있고 여러 GPU가 있는 단일 기계 및 여러 GPU가 있는 다중 기계를 통해 학습하는 것을 허용합니다. 더 자세한 정보를 알고 싶으시다면 [User Guide](https://ludwig-ai.github.io/ludwig-docs/latest/user_guide/distributed_training/)를 확인해 주시기 바랍니다.
 
 ## Prediction and Evaluation
 

docker/ludwig-gpu/Dockerfile

@@ -9,17 +9,7 @@
 #   model serving
 #
 
-FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel
-
-# https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212771
-RUN sh -c 'echo "APT { Get { AllowUnauthenticated \"1\"; }; };" > /etc/apt/apt.conf.d/99allow_unauth' && \
-    apt -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true update && \
-    apt-get install -y curl wget && \
-    apt-key del 7fa2af80 && \
-    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
-    dpkg -i cuda-keyring_1.0-1_all.deb && \
-    rm -f /etc/apt/sources.list.d/cuda.list /etc/apt/apt.conf.d/99allow_unauth cuda-keyring_1.0-1_all.deb && \
-    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A4B469963BF863CC F60F4B3D7FA2AF80
+FROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel
 
 RUN apt-get -y update && DEBIAN_FRONTEND="noninteractive" apt-get -y install \
     git \
@@ -33,8 +23,7 @@ RUN pip install -U pip
 WORKDIR /ludwig
 
 COPY . .
-RUN pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cu118
-RUN ludwig check_install
+RUN pip install --no-cache-dir '.[full]'
 
 WORKDIR /data
 

docker/ludwig-ray-gpu/Dockerfile

@@ -1,5 +1,5 @@
 #
-# Ludwig Docker image with Ray nightly support and full dependencies including:
+# Ludwig Docker image with Ray support and full dependencies including:
 #   text features
 #   image features
 #   audio features
@@ -9,22 +9,7 @@
 #   model serving
 #
 
-FROM rayproject/ray:2.3.1-py38-cu118
-
-# Fix kubernetes package repositories
-# https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/
-RUN sudo mkdir /etc/apt/keyrings
-RUN echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
-RUN wget https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key
-RUN cat Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
-# Done
-
-# https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212771
-RUN sudo apt-key del 7fa2af80 && \
-    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
-    sudo dpkg -i cuda-keyring_1.0-1_all.deb && \
-    sudo rm -f /etc/apt/sources.list.d/cuda.list /etc/apt/apt.conf.d/99allow_unauth cuda-keyring_1.0-1_all.deb && \
-    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A4B469963BF863CC F60F4B3D7FA2AF80
+FROM rayproject/ray:2.9.0-py311-cu121
 
 
 # Upgrade to GCC-9 from GCC-7.5
@@ -53,4 +38,4 @@ WORKDIR /ludwig
 RUN pip install --no-cache-dir torch==2.1.0 torchtext torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118
 
 COPY . .
-RUN pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cu118
+RUN pip install --no-cache-dir '.[full]'

docker/ludwig-ray/Dockerfile

@@ -1,5 +1,5 @@
 #
-# Ludwig Docker image with Ray nightly support and full dependencies including:
+# Ludwig Docker image with Ray support and full dependencies including:
 #   text features
 #   image features
 #   audio features
@@ -9,15 +9,7 @@
 #   model serving
 #
 
-FROM rayproject/ray:2.3.1-py38
-
-# Fix kubernetes package repositories
-# https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/
-RUN sudo mkdir /etc/apt/keyrings
-RUN echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
-RUN wget https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key
-RUN cat Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
-# Done
+FROM rayproject/ray:2.9.0-py311
 
 RUN sudo apt-get update && DEBIAN_FRONTEND="noninteractive" sudo apt-get install -y \
 	build-essential \

docker/ludwig/Dockerfile

@@ -9,7 +9,7 @@
 #   model serving
 #
 
-FROM python:3.8.13-slim
+FROM python:3.11-slim
 
 RUN apt-get -y update && apt-get -y install \
     git \

examples/README.md

@@ -2,11 +2,11 @@
 
 This directory contains example programs demonstrating Ludwig's Python APIs.
 
-| Directory       | Examples Provided                                                                                                                                                                                                                                                                                                                 |
-| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| hyperopt        | Demonstrates Ludwig's to hyper-parameter optimization capability.                                                                                                                                                                                                                                                                 |
-| kfold_cv        | Provides two examples for performing a k-fold cross validation analysis.  One example uses the `ludwig experiment` cli.  The other example uses the `ludwig.experiment.kfold_cross_validate()` api function.                                                                                                                      |
-| mnist           | Creates a model config data structure from a yaml file and trains a model.  Programmatically modify the model config data structure to evaluate several different neural network architectures.  Jupyter notebook demonstrates using a hold-out test data set to visualize model performance for alternative model architectures. |
-| titanic         | Trains a simple model with model config contained in a yaml file.  Trains multiple models from yaml files and generate visualizations to compare training results.  Jupyter notebook demonstrating how to programmatically create visualizations.                                                                                 |
-| serve           | Demonstrates running Ludwig http model server.  A sample Python program illustrates how to invoke the REST API to get predictions from input features.                                                                                                                                                                            |
-| class_imbalance | Demonstrates using our class balancing feature to over-sample an imbalanced dataset.                                                                                                                                                                                                                                              |
+| Directory       | Examples Provided                                                                                                                                                                                                                                                                                                               |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| hyperopt        | Demonstrates Ludwig's to hyper-parameter optimization capability.                                                                                                                                                                                                                                                               |
+| kfold_cv        | Provides two examples for performing a k-fold cross validation analysis. One example uses the `ludwig experiment` cli. The other example uses the `ludwig.experiment.kfold_cross_validate()` api function.                                                                                                                      |
+| mnist           | Creates a model config data structure from a yaml file and trains a model. Programmatically modify the model config data structure to evaluate several different neural network architectures. Jupyter notebook demonstrates using a hold-out test data set to visualize model performance for alternative model architectures. |
+| titanic         | Trains a simple model with model config contained in a yaml file. Trains multiple models from yaml files and generate visualizations to compare training results. Jupyter notebook demonstrating how to programmatically create visualizations.                                                                                 |
+| serve           | Demonstrates running Ludwig http model server. A sample Python program illustrates how to invoke the REST API to get predictions from input features.                                                                                                                                                                           |
+| class_imbalance | Demonstrates using our class balancing feature to over-sample an imbalanced dataset.                                                                                                                                                                                                                                            |

examples/hyperopt/README.md

@@ -5,7 +5,7 @@ Demonstrates hyperparameter optimization using Ludwig's in-built capabilities.
 ### Preparatory Steps
 
 - Create `data` directory
-- Download [Kaggle wine quality data set](https://www.kaggle.com/rajyellow46/wine-quality) into the `data` directory.  Directory should
+- Download [Kaggle wine quality data set](https://www.kaggle.com/rajyellow46/wine-quality) into the `data` directory. Directory should
   appear as follows:
 
 ```

examples/kfold_cv/README.md

@@ -64,10 +64,10 @@ retrieving results from  results
 
 ## Regression Example
 
-This illustrates using the Ludwig API to run the K-fold cross validation analysis.  To run the example, open the jupyter notebook `regression_example.ipynb`.  Following steps are performed:
+This illustrates using the Ludwig API to run the K-fold cross validation analysis. To run the example, open the jupyter notebook `regression_example.ipynb`. Following steps are performed:
 
 - Download and prepare data for training and create a Ludwig config data structure from a pandas dataframe structure
-- Use `ludwig.api.kfold_cross_validate()` function  to run the 5-fold cross validation
+- Use `ludwig.api.kfold_cross_validate()` function to run the 5-fold cross validation
 - Display results from the 5-fold cross validation analysis
 
 Expected output from running the example: