Infinite Loop: Weibull Estimate

[denning-s]

There are certain conditions that can cause the Weibull.Estimate() method to enter an infinite loop. This seems to occur with relatively large data sets that contain one or more outliers. It can be replicated by passing the following list of values into the method:

var samples = new List<double> { 99.6, 97.2, 98.4, 101.5, 99.5, 98.2, 97.8, 98.1, 99.2, 100.1, 96.6, 98.2, 98.8, 95.8, 97.1, 96.7, 98.3, 99.3, 99.4, 96, 96.6, 99.5, 96.4, 96.4, 98.7, 95.1, 96.6, 96.2, 96.2, 93.3, 98.7, 91.8, 96.4, 97.2, 97.3, 93.1, 94.9, 96, 99.7, 96.8, 98.8, 98, 98.3, 94.1, 97.4, 99, 97.7, 98.7, 96.2, 99.5, 101.2, 96.9, 99.7, 96.1, 100.2, 98.8, 98, 101.3, 95.8, 98.3, 96.5, 95.8, 95.4, 94.2, 98.2, 96.9, 98.5, 96.4, 95.7, 96.5, 96, 95.1, 97.6, 96.4, 98.2, 96.6, 98.1, 98.1, 97.9, 94.8, 97.6, 98.5, 95.8, 95.8, 96.7, 101.7, 99.7, 102.7, 103, 101.1, 102.8, 99.4, 98.3, 101.4, 99.4, 98.3, 100.1, 99.2, 98.9, 99.2, 100.7, 98.1, 100.1, 98.4, 98.3, 98.4, 97.3, 97.8, 97.8, 95.1, 98.3, 98.4, 100.6, 95.6, 100.4, 94.6, 98, 98.8, 97.8, 96.4, 95.6, 96.6, 100.2, 95.4, 98, 97.5, 99.8, 93, 95.4, 97, 95.8, 96, 98, 96.7, 100.1, 96.6, 101.4, 98.9, 100.7, 97.1, 97.2, 98, 96, 97.5, 97, 98.8, 100, 95.5, 99, 99.8, 97.5, 96.3, 96.9, 96.9, 96.8, 96.2, 98.8, 96.4, 95.9, 100.9, 98.8, 95.5, 96.9, 96.7, 97.7, 96.5, 98.6, 99, 100, 96.4, 99.9, 98.3, 96.5, 97.2, 98, 98.1, 99.3, 98.3, 100.4, 97.7, 99.2, 94.9, 97.2, 97.7, 99.2, 96.7, 97.7, 96.8, 95.4, 99.6, 96.7, 95.4, 98.3, 97.1, 96.6, 99.2, 95.7, 95.2, 95.8, 94.3, 93.1, 94.3, 95.8, 93.4, 95.2, 89.2, 94.7, 92.8, 93.9, 92.8, 130.8 };

The final value in this list is causing the issue. It is significantly higher than other values, but it is representative of real data and should not be excluded.

Any help on how to address this issue would be appreciated.

Thanks.

[guneyozsan]

I checked the data provided and noticed that the control statement while (Math.Abs(c - previousC) >= 0.0001) still converges as the reference article suggests but not to 0 as in the implementation, which causes the infinite loop.

Normally checking if the dif of dif converge to 0 resolves the infinite loop issue but we need to be sure if it corresponds to the mathematically correct estimation. (it does not)

I can go through the article and correct the implementation. However the article is paid, so do we have any access to it for purposes of this project?

Edit: Updated the reference article link to the correct one mentioned in the source code.

[guneyozsan]

I was able to find the estimation method provided in another paper of the same authors which was publicly available (page 174, Section 2. Two parameter Weibull model).

I can confirm that the implementation looks correct.

However, for the given problematic data set, instead of converging to a single number as expected, c converges to a narrow tunnel of two close values, 14.076950346587 and 14.3034894520983, infinitely oscillating between these two numbers in each iteration, causing the infinite loop.

Since Theorem 1 in original paper suggests that:
"The Simple Iterative Procedure always converges for any positive starting point, that is, c[k] generated by SIP converges to the unique fixed point c* of q(c) as k -> oo, for any starting point c[0] > 0." we are sure that method converges. This means that it may be a result of a precision error as s2 and s3 are around E+30 when outlier (last data point) is included and around E+100 when omitted. But I'm not sure if this is relevant or not.

Such a narrow tunnel case may be handled by interpolating the oscillating numbers. However I should mention that c converges to a significantly different value of 48.7923530410799 when the outlier is omitted (last data point). So, such an oscillating condition may not be representative of the actual distribution.

@denning-s I know this is an old issue but being familiar with the data set, do you have any comments regarding this?

At the very least, it is easy to detect such a double convergence condition and exit the infinite loop with a warning or error. @cdrnet what is the approach of mathnet in such conditions?