Is there a non linear regression / fit method in Math.Net?

[dotChris90]

Hello Team,

I am just curious if there is a Special "non-linear" generalized Regression method in Math.Net so user can do fits with any nonlinear function / model?

In the examples I saw Fit.Line(x,y) but I would like to know if there is sth like "Fit.Curve(x,y,f)" ?
In other words - user give the method a function f and method determines Parameter.

In MATLAB and other Regression Tools this is often done by Gradient free Algorithmen.
Mostly done by nelder mead algorithm.

And in case it is not implemented - you accept pull requests?

[fafairuz]

Usually you can do linear transformation like log or 1/x to use linear regression.

[dotChris90]

Yeah that's true.

But I just saw that many people I met in engineering area like easy stuff, e.g. in MATLAB.
MATLAB is very popular in our area because keep stuff easy. There is no need for a Transforming.

Just curious if nobody else think like that?

[cdrnet]

For non-linear regression have a look at the MathNet.Numerics.Optimization namespace, which among others also implements gradient-free algorithms like the Nelder-Mead simplex algorithm. See also FindMinimum in the root namespace where some of them are exposed for simple use cases.

[cdrnet]

We do accept pull requests - although I've been a bit slow to accept them lately.

[cdrnet]

I've added a new Fit.Curve function:

/// <summary>
/// Non-linear least-squares fitting the points (x,y) to an arbitrary function y : x -> f(p, x),
/// returning its best fitting parameter p.
/// </summary>
public static double Curve(double[] x, double[] y, Func<double, double, double> f, double initialGuess, double tolerance = 1e-8, int maxIterations = 1000)
{
    return FindMinimum.OfScalarFunction(p => Distance.Euclidean(Generate.Map(x, t => f(p, t)), y), initialGuess, tolerance, maxIterations);
}

Is this what you're looking for - except that this is scalar only and would need to be expanded to multi-dimensional functions and multi-dimensional parameters?

[dotChris90]

Wow - fantastic!

And yes thats what i was thinking about. If you accept Pull requests - that's fine. The multi-dimensional i can try myself ;)
It is great.

Honestly spoken - I know there is always a way you can create such Things by yourself but "easy to use methods" in a lib makes the lib good. ;)
Just I worked many years in Simulation area and there People use MATLAB for what reason? because toolboxes are easy to use. And honestly MATLAB is a horrible language but has a lot of easy to use toolboxes.

Thanks for anwsering and adding already the Fit.Curve. have nice Weekend.

[diluculo]

Currently, the Numerics supports some of numerical optimization or non-linear least square regression algorithms:

• Nelder-Mead Simplex
• Newton
• Broyden–Fletcher–Goldfarb–Shanno (BFGS)
• Broyden–Fletcher–Goldfarb–Shanno Bounded (BFGS-B)
• Levenberg–Marquardt (LEVM) : optimization-3 branch
• Trust-region or Trust-region-dogleg (TRDL)

I'd like to know about the progress of the Optimization project

---

One more, the general curve fit function, I think, has the followings:

• Inputs

  • function f - calculate f(p0, p1, ...) or f(x; p0, p1, ...)
  • function f' - (optional) calculate df/dp
  • double[] x - (optional) observed x
  • double[] y - observed y
  • double[] w - weights for observation y
  • double[] p - initial parameters
  • double[] lb - (optional) lower bound of parameters
  • double[] ub - (optional) upper bound of parameters
  • bool[] fix - {optional) set fix or free of parameters
  • tolerance options
  • termination conditions
  • object option - (optional) user-defined
  • algorithm - used to find p minimizing sum of residuals. LEVM, TRDL, ...

• Outputs

  • double[] y - best fitted
  • double[] p - best fitted parameters
  • double[,] C - matrix of covariances
  • GoodnessOfFit - Chi-squared, R-square, ...

[eriove]

I did some work in the optimization project, but what is left is above my knowledge in numerical methods. Note that there is a bug in the Nelder-Mead Simplex (#501) algorithm as well. There was a pull request to fix this, but it was abandoned (it fixed the immediate problem, but made other tests fail).

[diluculo]

What about adding more generalized Curve method to the Fit class?

public static MinimizationResult Curve(double[] x, double[] y, Func<Vector<double>, double, double> f, double[] initialGuess, double tolerance = 1e-8, int maxIterations = 1000)
{
        return OfFunction((p) => SquareEuclidean(Generate.Map(x, t => f(p, t)), y), Vector<double>.Build.DenseOfArray(initialGuess), tolerance, maxIterations);
 }

public static MinimizationResult OfFunction(Func<Vector<double>, double> function, Vector<double> initialGuess, double tolerance = 1e-8, int maxIterations = 1000)
{
       var objective = ObjectiveFunction.Value(v => function(v));
       var result = NelderMeadSimplex.Minimum(objective, initialGuess, tolerance, maxIterations);
       return result;
}

// calculate the residual sum of squares, RSS = Sum of (a[i] - b[i])^2
private static double SquareEuclidean(double[] a, double[] b)
{
       var A = Vector<double>.Build.DenseOfArray(a);
       var B = Vector<double>.Build.DenseOfArray(b);
       var norm = (A - B).L2Norm(); // or (A - B).DotProduct(A - B)
       return norm * norm;
}

Example for NIST rat43 data,

 var ret = Curve(xData, yData, (p, x) => p[0] / Math.Pow(1 + Math.Exp(p[1] - p[2] * x), 1.0 / p[3]), initialGuess);

In the sense of data fitting, I think that the objective function F(p) is ||y - f(x; p)||^2, not ||y - f(x; p)||.

If a user-defined Jacobian function is provided, other minimization algorithms can be used.

One more, the NelderMeadSimplex throws an exception when exceeding MaxIterations. I think it's better just to stop the iteration with ExceedIterations ExitCondition.

[grantyuc]

I notice that there might be some inconsistency in namespace MathNet.Numerics.Optimization.
The method ObjectiveFunction.NonlinearModel accepts multi-dimensional model f(p; x), especially x a datum in vector form. However, it only take one-dimensional ObservedX as input; this means I cannot create a model with N vectors of data x_1, x_2, .., x_N but only one-dimensional data.
That all the four models in MathNet.Numerics.UnitTests.OptimizationTests.NonLinearCurveFittingTests only accept x in scalar form also approves my guess.

Is there any support for multivariate regression in MathNet.Numerics?

Updated:
I opened a discussion here: #868