Add AutoEncoder as a Feature Selection Method

[superichmann]

Is your feature request related to a problem? Please describe.
No.
Currently the only practical option for feature selection in ML.net is MI. maybe PFI (post fit post transform) and maybe PCA (dimensionality reduction).

Describe the solution you'd like
My online research shows that autoencoders for feature selection can be highly accurate and add another layer to the feature selection catalog.

Describe alternatives you've considered
Using already implemented feature importance methods.

Additional context
Research:
https://hex.tech/blog/autoencoders-for-feature-selection/
https://deepai.org/publication/autoencoder-feature-selector
Relates to issues:
4254
5777

Note: Perhaps utilizing autoencoders is actually possible in ml.net with importing external models or something like that, if it does I would love a to see how :) Thanks for everything guys!

[superichmann]

?

[luisquintanilla]

Hi @superichmann,

Thanks for the suggestion.

Can you expand some more on what your scenario / use-case for this is?

While not directly intended for feature selection / dimensionality reduction, for images we have DnnFeaturizeImage transforms which as the name suggests, leverages neural net models to convert an image into a feature vector.

Here's a general writeup on the transform.

[superichmann]

@luisquintanilla The use case is to provide an extra accurate layer of feature selection other then lgbm/ff default ones for datasets with more than 10000 features.

[superichmann]

@luisquintanilla :(

[luisquintanilla]

Thanks for that additional information. There are no plans to add this to ML.NET at this time.

As an alternative, you might consider using TorchSharp to implement AEFS. From the paper (note some code may be wrong since it was AI-generated).

using System;
using TorchSharp;
using TorchSharp.Modules;
using TorchSharp.Tensor;

class AutoEncoderFeatureSelector : Module
{
    private readonly Linear encoder;
    private readonly Linear decoder;
    private readonly float alpha;
    private readonly float beta;

    // Constructor to initialize the autoencoder with input and hidden dimensions, and regularization parameters
    public AutoEncoderFeatureSelector(int inputDim, int hiddenDim, float alpha = 1.0f, float beta = 1.0f) : base("AutoEncoderFeatureSelector")
    {
        this.alpha = alpha;
        this.beta = beta;
        encoder = nn.Linear(inputDim, hiddenDim); // Encoder layer
        decoder = nn.Linear(hiddenDim, inputDim); // Decoder layer
        RegisterComponents(); // Register the components
    }

    // Forward pass through the autoencoder
    public override Tensor forward(Tensor x)
    {
        var encoded = encoder.forward(x).sigmoid(); // Encode input and apply sigmoid activation
        var decoded = decoder.forward(encoded).sigmoid(); // Decode encoded input and apply sigmoid activation
        return decoded;
    }

    // Group Lasso regularization term
    public Tensor GroupLasso(Tensor W)
    {
        return W.pow(2).sum(1).sqrt().sum(); // Compute the group lasso regularization
    }

    // Static method to train the autoencoder model
    public static void TrainModel(AutoEncoderFeatureSelector model, Tensor data, int epochs = 100, float learningRate = 0.01f)
    {
        var criterion = nn.MSELoss(); // Mean Squared Error loss function
        var optimizer = optim.Adam(model.parameters(), learningRate); // Adam optimizer

        for (int epoch = 0; epoch < epochs; epoch++)
        {
            optimizer.zero_grad(); // Zero the gradients
            var outputs = model.forward(data); // Forward pass
            var loss = criterion.forward(outputs, data); // Compute reconstruction loss
            var lassoLoss = model.alpha * model.GroupLasso(model.encoder.weight); // Compute group lasso loss
            var weightDecay = model.beta * (model.encoder.weight.pow(2).sum() + model.decoder.weight.pow(2).sum()); // Compute weight decay
            var totalLoss = loss + lassoLoss + weightDecay; // Total loss
            totalLoss.backward(); // Backward pass
            optimizer.step(); // Update weights
            Console.WriteLine($"Epoch {epoch + 1}/{epochs}, Loss: {totalLoss.item<float>()}"); // Print loss
        }
    }

    // Static method to select top-k features based on the row norms of the encoder weights
    public static Tensor SelectFeatures(AutoEncoderFeatureSelector model, Tensor data, int k)
    {
        using (torch.no_grad()) // Disable gradient calculation
        {
            var weights = model.encoder.weight; // Get encoder weights
            var rowNorms = weights.norm(2, 1); // Compute row norms
            var topKIndices = rowNorms.topk(k, largest: true).indices; // Get indices of top-k features
            return data.index_select(1, topKIndices); // Select top-k features from data
        }
    }
}

// Example usage
class Program
{
    static void Main()
    {
        torch.manual_seed(0); // Set random seed for reproducibility
        var X = torch.rand(200, 3); // Generate synthetic data
        X.index_put_(new[] { torch.tensor(2) }, torch.sin(X.index_select(1, torch.tensor(0))) + torch.cos(X.index_select(1, torch.tensor(1)))); // Create nonlinear relationship

        var aefs = new AutoEncoderFeatureSelector(inputDim: 3, hiddenDim: 2, alpha: 0.1f, beta: 0.01f); // Initialize AEFS
        AutoEncoderFeatureSelector.TrainModel(aefs, X, epochs: 100, learningRate: 0.01f); // Train AEFS

        var X_transformed = AutoEncoderFeatureSelector.SelectFeatures(aefs, X, k: 2); // Select top-2 features
        Console.WriteLine("Selected features:\n" + X_transformed); // Print selected features
    }
}