Capture and report additional metadata upon cholesky factorisation error

[jhaasdyk-au]

Brief description
Please report more details concerning the station(s) and/or measurement(s) involved in errors such as Cholesky factorisation.

Background
Cholesky factorisation errors currently occur with an error message similar to:
- Error: Cannot compute the least squares estimates. Reason:
choleskyinverse_mkl(): Cholesky factorisation failed.
Phased adjustment terminated whilst processing block 13

On large adjustments this can be difficult to troubleshoot.
If failure occurs during Iteration1 there may be little to no direct output concerning the block which has failed.

It has been suggested to me that the MKL library may be able to return useful metadata.

Basic requirements
Where possible, report details to screen and error log which identify station(s) and/or measurement(s) which precipitate an error such as Cholesky factorisation.

Priority
Suggest: priority 2 (medium), or priority 3 (low)

[daleroberts]

@jhaasdyk-au Would you be able to test the latest 1.2.9 version? I'm hoping this is now fixed as we've done a lot of work to make it more stable with large networks.

Prebuilt binaries are now available at:

https://github.com/GeoscienceAustralia/DynAdjust/releases

[nicgowans]

Hi @daleroberts,

I've just today run DynAdjust v1.2.9 (both Windows OpenBLAS and MKL) on the full NSW Survey Network adjustment (~190,000 stns, ~1.2M msrs) and unfortunately neither run the adjustment to completion.

OpenBLAS terminates without providing any error messaging while MKL provides the following output after encountering a Cholesky factorisation failure:

+---------------------------------------------------------------------------
+ Title:        adjust - DynAdjust 2025.
+ Description:  Geodetic network adjustment software
+ Version:      1.2.9, Release (64-bit) with Intel MKL
+ Build:        Oct  3 2025, 05:24:52 - MSVC++ 17.14, VS2022
+ Copyright:    (C) 2025 Geoscience Australia.
                This software is released under the Apache License.
+ Contact:      geodesy@ga.gov.au
+---------------------------------------------------------------------------

+ Options:
  Network name:                    20250928_NSW_GDA2020
  Input folder:                    .
  Output folder:                   .
  Associated station file:         .\20250928_NSW_GDA2020.asl
  Associated measurement file:     .\20250928_NSW_GDA2020.aml
  Binary station file:             .\20250928_NSW_GDA2020.bst
  Binary measurement file:         .\20250928_NSW_GDA2020.bms
  Segmentation file:               .\20250928_NSW_GDA2020.seg
  Adjustment output file:          .\20250928_NSW_GDA2020.phased-stage.adj
  Coordinate output file:          .\20250928_NSW_GDA2020.phased-stage.xyz
  Corrections output file:         .\20250928_NSW_GDA2020.phased-stage.cor
  Stage using hard disk:           yes
  Recreate mapped stage files:     yes
  Purge mapped stage files:        yes
  Reference frame:                 GDA2020
  Epoch:                           01.01.2020
  Geoid model:                     C:\Users\GOWANSN2\NSWGOV\DCS Survey Operations - Documents\NSW_SCN\ref_files\geoids\AUSGeoid2020_20180201.gsb

+ Rigorous sequential phased adjustment mode (staged)

+ Preparing for a 116 block adjustment...  done.
+ Adjusting network...
  Iteration  1, block     9 (forward)
+ Error: Cannot compute the least squares estimates.  Reason:
  cholesky_inverse(): Cholesky factorisation failed.

System Information:
  sizeof(lapack_int): 4 bytes
  sizeof(MKL_INT): 4 bytes
  BLAS/LAPACK: Intel MKL
  MKL Version: 2025.0 Update 0
  MKL Build: 20241009
  Interface: LP64 (32-bit integers)
  Matrix dimensions: 13827 x 13827
  Triangle processed: uplo='L' (LOWER_IS_CLEARED=false)
  Diagonal range: [5.1233e-13, 7.52164e+08]
  Condition estimate: 1.46812e+21

Scaling Information:
  Matrix was scaled to improve conditioning
   Scale factor applied: 0.0196305

Error Details:
  dpotrf info = 108
  Meaning: The leading minor of order 108 is not positive definite

Matrix Diagnostics (Original Matrix):
  Note: Matrix was triangular (uplo='L'), filled to symmetric for analysis
  Trace: 3.10706e+09
  Min diagonal: 5.1233e-13
  Max diagonal: 7.52164e+08
  Negative diagonal elements: 0
  NaN count: 0
  Inf count: 0
  First diagonal elements: 25062.3, 77165.6, 29612.5, 157746, 143220, 106338, 27012.5, 23450.6, 29943.7, 17476.5, ...

Leading Minor Analysis (k=108):
  Leading 108×108 block:
      2.506e+04  2.813e+04 -8.442e+03  0.000e+00  0.000e+00  0.000e+00 -3.810e-23 -8.206e-23 -1.663e-23  2.532e-23 ...
      2.813e+04  7.717e+04  2.546e+04  0.000e+00  0.000e+00  0.000e+00 -1.266e-22 -2.728e-22 -5.525e-23  8.417e-23 ...
     -8.442e+03  2.546e+04  2.961e+04  0.000e+00  0.000e+00  0.000e+00 -5.143e-23 -1.108e-22 -2.244e-23  3.419e-23 ...
      0.000e+00  0.000e+00  0.000e+00  1.577e+05  1.249e+05 -1.140e+05  0.000e+00  0.000e+00  0.000e+00  0.000e+00 ...
      0.000e+00  0.000e+00  0.000e+00  1.249e+05  1.432e+05 -5.766e+04  0.000e+00  0.000e+00  0.000e+00  0.000e+00 ...
      0.000e+00  0.000e+00  0.000e+00 -1.140e+05 -5.766e+04  1.063e+05  0.000e+00  0.000e+00  0.000e+00  0.000e+00 ...
     -3.810e-23 -1.266e-22 -5.143e-23  0.000e+00  0.000e+00  0.000e+00  2.701e+04  1.189e+04 -2.303e+04 -1.637e+03 ...
     -8.206e-23 -2.728e-22 -1.108e-22  0.000e+00  0.000e+00  0.000e+00  1.189e+04  2.345e+04  3.563e+03  1.763e+03 ...
     -1.663e-23 -5.525e-23 -2.244e-23  0.000e+00  0.000e+00  0.000e+00 -2.303e+04  3.563e+03  2.994e+04  3.264e+03 ...
      2.532e-23  8.417e-23  3.419e-23  0.000e+00  0.000e+00  0.000e+00 -1.637e+03  1.763e+03  3.264e+03  1.748e+04 ...
    ... (98 more rows)

  Gershgorin Analysis:
    Lower bound for eigenvalues: -1.963433e+05
    Critical row: 44
    => Matrix is NOT positive definite (Gershgorin bound <= 0)

  Eigenvalue Analysis (smallest 10 eigenvalues):
    Successfully computed 10 smallest eigenvalues:
      ╬╗[1] = 3.250e-11
      ╬╗[2] = 3.250e-11
      ╬╗[3] = 1.912e-10
      ╬╗[4] = 1.622e-08
      ╬╗[5] = 2.347e-07
      ╬╗[6] = 8.606e-07
      ╬╗[7] = 2.186e-06
      ╬╗[8] = 4.698e-06
      ╬╗[9] = 6.208e-06
      ╬╗[10] = 7.176e-06
    => All computed eigenvalues are positive
    Note: Only computed first 10 of 108 eigenvalues

  Triangle Comparison (first 3×3 of leading 108×108):
    Upper triangle:
        2.506228e+04  2.812861e+04 -8.442204e+03
        7.716562e+04  2.546104e+04
        2.961247e+04
    Lower triangle:
        2.506228e+04
        2.812861e+04  7.716562e+04
       -8.442204e+03  2.546104e+04  2.961247e+04

  Phased adjustment terminated whilst processing block 9

While this is significantly more detailed output than was previously provided in this situation it doesn't really address the request from @jhaasdyk-au to "report more details concerning the station(s) and/or measurement(s) involved". It would be ideal if DynAdjust could report the stations names and/or measurement database ids that are contributing to the Cholesky factorisation failure to allow the user to quickly identify and resolve the problem.

Please note:

• This adjustment runs to completion without issue on DynAdjust v1.2.8. Identical input data, segmentation and output options.
• I have separately tested a much smaller (simultaneous) traverse network on DynAdjust v1.2.9 MKL which runs without issue.

[daleroberts]

Hi @nicgowans,

The difficulty is that once the Cholesky factorisation (dpotrf) has failed, it's hard to recover and work backwards to find the bad data for a number of reasons.

• On failure, dpotrf provides a number 'k' which means that "the leading principal minor of order k is not positive definite"
• So it tells you the index k of the step where the Cholesky process first encounters a nonpositive pivot, it doesn't tell you which entry of the matrix caused it.
• Perhaps we can say that the problem is localised in the top-left k x k block because that is the one that ceases to be positive definite. This is why I now print out the "Leading Minor Analysis (k=108) block" and try to find eigenvalues to check positive definiteness of the top-left k x k block.
• Annoyingly, the dpotrf function operates in-place on the matrix, so we would need to keep a full copy of the matrix before passing it to dpotrf if we want to do more analysis but that would blow out memory requirements by two.

From the debug output above, I can see:

Diagonal range: [5.1233e-13, 7.52164e+08]
Condition estimate: 1.46812e+21

This tells me that you have a matrix that has very large values and very small values and this is making the matrix have a very high condition number, which can lead to factorisation failing. Sometimes scaling can help (--scale-normals-to-unity option) but it seems from the debug output above that you are already doing that.

IMHO, the only two solutions are:

• Designing an algorithm that detects bad measurements before we start solving.
• Having a more robust mathematical solving algorithm available that might not choke on datasets like this.

I am working on and thinking about both those cases, but this requires major surgery to the core of dnaadjust so we can plug-in different algorithms with the ability to switch between them or turn on / off (through options at command line) while still maintaining backward compatibility.

[ibaran73]

Hi Robert, I remember setting up the state network for the first time er did GDA2020 and encountering the Cholesky decomposition error. It was quite tricky to pinpoint the cause. My approach was to divide the network into two separate subnetworks and attempt to solve each one individually. When one of those failed, I split it again into halves and continued this process until I could successfully analyse the data. In most cases, the was either no standard deviation provided (set to zero) or an anomalous observation. I recall one situation where an observation could yield two different outcomes. For example, a height difference of 0.002 with a standard deviation of ±0.005 caused issues in the solution. The problem was resolved only after I manually set the standard deviation to 0.001, which helped keep the height difference on the positive side (+0.002). Just a few thoughts from the user side. You're doing amazing work. Thank you, Regards, Irek Irek Baran | Principal Geodesist Land Boundaries, Location Services, Landgate 1 Midland Square, Midland WA 6056 t: +61 (0)8 9273 7135 e: ***@***.******@***.***> w: landgate.wa.gov.au www.linkedin.com/in/irek-baran<http://www.linkedin.com/in/irek-baran> [A picture containing art, graphic design, graphics, poster Description automatically generated] Work Location: Mon Tues Wed Thu Fri Office Office Office/Home Home Home From: Dale Roberts ***@***.***> Sent: Thursday, 9 October 2025 8:08 AM To: GeoscienceAustralia/DynAdjust ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [GeoscienceAustralia/DynAdjust] Capture and report additional metadata upon cholesky factorisation error (Issue #228) [https://avatars.githubusercontent.com/u/582436?s=20&v=4]daleroberts left a comment (GeoscienceAustralia/DynAdjust#228)<#228 (comment)> Hi @nicgowans<https://github.com/nicgowans>, The difficulty is that once the Cholesky factorisation (dpotrf) has failed, it's hard to recover and work backwards to find the bad data for a number of reasons. * On failure, dpotrf provides a number 'k' which means that "the leading principal minor of order k is not positive definite" * So it tells you the index k of the step where the Cholesky process first encounters a nonpositive pivot, it doesn't tell you which entry of the matrix caused it. * Perhaps we can say that the problem is localised in the top-left k x k block because that is the one that ceases to be positive definite. This is why I now print out the "Leading Minor Analysis (k=108) block" and try to find eigenvalues to check positive definiteness of the top-left k x k block. * Annoyingly, the dpotrf function operates in-place on the matrix, so we would need to keep a full copy of the matrix before passing it to dpotrf if we want to do more analysis but that would blow out memory requirements by two. From the debug output above, I can see: Diagonal range: [5.1233e-13, 7.52164e+08] Condition estimate: 1.46812e+21 This tells me that you have a matrix that has very large values and very small values and this is making the matrix have a very high condition number, which can lead to factorisation failing. Sometimes scaling can help (--scale-normals-to-unity option) but it seems from the debug output above that you are already doing that. IMHO, the only two solutions are: * Designing an algorithm that detects bad measurements before we start solving. * Having a more robust mathematical solving algorithm available that might not choke on datasets like this. I am working on and thinking about both those cases, but this requires major surgery to the core of dnaadjust so we can plug-in different algorithms with the ability to switch between them or turn on / off (through options at command line) while still maintaining backward compatibility. — Reply to this email directly, view it on GitHub<#228 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALBJ335PAVQ6YJOUQEESI3T3WWRM7AVCNFSM6AAAAACISB4I3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOBTGYYDOOJYGE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

[nicgowans]

Thanks for your reply, @daleroberts, I expected it would be fairly difficult to work backwards so appreciate your efforts to date.

Where I've run into this problem on smaller adjustments, I've started with minimal measurements to run DynAdjust successfully and then iteratively added more measurements until Cholesky failure is encountered to identify which are causing the problem - but this brute-force strategy isn't viable for large networks.

The more concerning issue in the above is that both MKL and OpenBLAS versions of 1.2.9 fail on this dataset but 1.2.8 runs to completion without any problems. I've run again with 1.2.9 MKL this time passing in --scale-normals-to-unity (which was not supplied in the initial run). The adjustment still fails but I get a slightly different error message:

+---------------------------------------------------------------------------
+ Title:        adjust - DynAdjust 2025.
+ Description:  Geodetic network adjustment software
+ Version:      1.2.9, Release (64-bit) with Intel MKL
+ Build:        Oct  3 2025, 05:24:52 - MSVC++ 17.14, VS2022
+ Copyright:    (C) 2025 Geoscience Australia.
                This software is released under the Apache License.
+ Contact:      geodesy@ga.gov.au
+---------------------------------------------------------------------------

+ Options:
  Network name:                    20250928_NSW_GDA2020
  Input folder:                    .
  Output folder:                   .
  Associated station file:         .\20250928_NSW_GDA2020.asl
  Associated measurement file:     .\20250928_NSW_GDA2020.aml
  Binary station file:             .\20250928_NSW_GDA2020.bst
  Binary measurement file:         .\20250928_NSW_GDA2020.bms
  Segmentation file:               .\20250928_NSW_GDA2020.seg
  Adjustment output file:          .\20250928_NSW_GDA2020.phased-stage.adj
  Coordinate output file:          .\20250928_NSW_GDA2020.phased-stage.xyz
  Corrections output file:         .\20250928_NSW_GDA2020.phased-stage.cor
  Stage using hard disk:           yes
  Recreate mapped stage files:     yes
  Purge mapped stage files:        yes
  Reference frame:                 GDA2020
  Epoch:                           01.01.2020
  Geoid model:                     C:\Users\GOWANSN2\NSWGOV\DCS Survey Operations - Documents\NSW_SCN\ref_files\geoids\AUSGeoid2020_20180201.gsb
  Scale normals to unity:          yes

+ Rigorous sequential phased adjustment mode (staged)

+ Preparing for a 116 block adjustment...  done.
+ Adjusting network...
  Iteration  1, block     3 (forward)
- Error: Cannot compute the least squares estimates.  Reason:
  cholesky_inverse(): Triangle validation failed!
  Expected: Upper triangle cleared (LOWER_IS_CLEARED = false)
  Found non-zero elements in upper triangle:
    [0,1] = -4.97422e+11
    [0,2] = 6.1592e+11
    [0,3] = 31808
    [0,4] = 27472.1
    [0,5] = -32341.3
    [0,6] = 17719.9
    [0,7] = 23255.6
    [0,8] = -27147.2
    [0,9] = 31506.9
    [0,10] = 46734.3
  ... (more violations exist, stopped checking after 10)

Symmetry check:
  Matrix appears to be nearly symmetric with 10 asymmetric elements
  Max asymmetry: 0.00195312 at [1584,1590]
[1584,1590] = 1.41964e+13, [1590,1584] = 1.41964e+13
  This might be a symmetric matrix with numerical errors

This error typically occurs when:
  - The matrix data is stored in the wrong triangle
  - The LOWER_IS_CLEARED parameter is incorrect
  - The matrix hasn't been properly initialised
  - A symmetric matrix has numerical precision issues

Thanks again for your work here.

[daleroberts]

Ok interesting, would it be possible to get a copy of the dataset causing the problem? Then I could try to debug what is wrong, or even try to come up with a strategy to detect these types of problems.

[daleroberts]

I remember setting up the state network for the first time er did GDA2020 and encountering the Cholesky decomposition error. It was quite tricky to pinpoint the cause. My approach was to divide the network into two separate subnetworks and attempt to solve each one individually. When one of those failed, I split it again into halves and continued this process until I could successfully analyse the data.

Yes, maybe an automatic bisection algorithm could work to try to pinpoint the problem.

[ibaran73]

Unfortunately, not. It was long time ago and I am not able today to recreate this dataset. Thank you. Regards Irek Irek Baran | Principal Geodesist Land Boundaries, Location Services, Landgate 1 Midland Square, Midland WA 6056 t: +61 (0)8 9273 7135 e: ***@***.******@***.***> w: landgate.wa.gov.au www.linkedin.com/in/irek-baran<http://www.linkedin.com/in/irek-baran> [A picture containing art, graphic design, graphics, poster Description automatically generated] Work Location: Mon Tues Wed Thu Fri Office Office Office/Home Home Home From: Dale Roberts ***@***.***> Sent: Thursday, 9 October 2025 1:39 PM To: GeoscienceAustralia/DynAdjust ***@***.***> Cc: Irek Baran ***@***.***>; Comment ***@***.***> Subject: Re: [GeoscienceAustralia/DynAdjust] Capture and report additional metadata upon cholesky factorisation error (Issue #228) [https://avatars.githubusercontent.com/u/582436?s=20&v=4]daleroberts left a comment (GeoscienceAustralia/DynAdjust#228)<#228 (comment)> I remember setting up the state network for the first time er did GDA2020 and encountering the Cholesky decomposition error. It was quite tricky to pinpoint the cause. My approach was to divide the network into two separate subnetworks and attempt to solve each one individually. When one of those failed, I split it again into halves and continued this process until I could successfully analyse the data. Yes, maybe an automatic bisection algorithm could work to try to pinpoint the problem. — Reply to this email directly, view it on GitHub<#228 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALBJ33YYJBE24JPITF7EDFT3WXYGLAVCNFSM6AAAAACISB4I3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOBUGIYDKOBYG4>. You are receiving this because you commented.Message ID: ***@***.******@***.***>> Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

[nicgowans]

Ok interesting, would it be possible to get a copy of the dataset causing the problem? Then I could try to debug what is wrong, or even try to come up with a strategy to detect these types of problems.

Of course. I'll email a link through.

Much appreciated.

[daleroberts]

Thank you @nicgowans, I've got a copy of the data and I'm trying to spin it up on the NCI now.

[daleroberts]

I've been working on some functionality to help diagnose problematic measurements. Since you know your dataset a bit better, could you have a look at the output below and tell me if it helps you work out what could be wrong? Also, open to suggestions for what else to show in this diagnostic.

====================================================================================
MEASUREMENTS MOST LIKELY CAUSING INVERSION FAILURE
====================================================================================

Showing top 38 of 38 measurements involving problematic stations.

Score explanation:
  - Based on eigenvector analysis of near-zero eigenvalues
  - Higher scores indicate stronger association with singularity
  - Scores are normalised (1.0 = highest contribution)
  - Consider removing or reweighting measurements with high scores

Suspicious measurement indicators:
  - StdDev = 0.0000: Zero-variance measurements cause singularity
  - Value = 0.0000: May indicate data errors or missing measurements
  - High scores on direction (D) measurements: Check for datum defects
  - Measurements between poorly connected stations
  - Investigate stations appearing in many high-scoring measurements

Column descriptions:
  Score:     Normalised contribution to singularity (0.0 to 1.0)
  Type:      Measurement type (e.g., D=Direction, E=Ellipsoid dist, H=Height)
  FileOrd:   Original file order number for identifying measurement in data
  Value:     Measured value (term1)
  StdDev:    Measurement standard deviation (sqrt of variance)
  Constr1/2: Station constraint types (F=Free, C=Constrained)
             Format: XYZ where X=Lat/X, Y=Lon/Y, Z=Hgt/Z
             Example: FFF=all free, CCC=fully constrained, FFC=height constrained

Key diagnostics: Look for StdDev=0.0000 (zero variance) and FFF constraints.

Score  Type  FileOrd Station1            Station2            Station3                   Value      StdDev  Constr1  Constr2
----------------------------------------------------------------------------------------------------------------------------------
1.000  E     630759  (BM6DPY)_229890     (BM6)_229890        -                         8.7290      0.0036      FFF      FFF
1.000  E     630747  PM50435             (BM6)_229890        -                       656.9017      0.0067      FFF      FFF
1.000  E     630746  PM50435             (BM6)_229890        -                       656.8997      0.0067      FFF      FFF
1.000  E     630744  PM50434             (BM6)_229890        -                         5.6790      0.0036      FFF      FFF
1.000  E     630742  PM50433             (BM6)_229890        -                      1062.9983      0.0079      FFF      FFF
1.000  D     153889  (BM6)_229890        TS2196              -                         0.0000      0.0000      FFF      FFF
1.000  D     153884  (BM6)_229890        PM50435             -                         0.0000      0.0000      FFF      FFF
1.000  H     1282674 (BM6)_229890        -                   -                       673.0392      1.0000      FFF        -
1.000  D     153795  PM50434             (BM6)_229890        -                         0.0000      0.0007      FFF      FFF
0.994  E     549659  SS21395-1           (21395)_229434      -                         7.6870      0.0050      FFF      FFF
0.994  H     1250507 (21395)_229434      -                   -                       249.3763      1.0000      FFF        -
0.994  E     549668  SS8978              (21395)_229434      -                       211.5840      0.0060      FFF      FFF
0.994  E     549658  (21395)_229434      SS21395             -                         8.1040      0.0050      FFF      FFF
0.994  E     549656  (21395)_229434      SS8978              -                       211.5850      0.0060      FFF      FFF
0.994  E     549657  (21395)_229434      SS21395-1           -                         7.6870      0.0050      FFF      FFF
0.994  E     549655  (21395)_229434      SS21396             -                       190.6320      0.0060      FFF      FFF
0.994  E     549654  SS21396             (21395)_229434      -                       190.6320      0.0060      FFF      FFF
0.994  D     42607   (21395)_229434      SS21396             -                         0.0000      0.0000      FFF      FFF
0.994  D     42620   SS8978              (21395)_229434      -                         0.0000      0.0000      FFF      FFF
0.994  D     42611   SS21395-1           (21395)_229434      -                         0.0000      0.0007      FFF      FFF
0.990  H     1329400 (GIP18)_233237      -                   -                       109.9711      1.0000      FFF        -
0.990  E     754312  SS53870             (GIP18)_233237      -                       144.1317      0.0059      FFF      FFF
0.990  D     311523  (GIP17)_233237      (GIP18)_233237      -                         0.0000      0.0000      FFF      FFF
0.990  D     311525  (GIP18)_233237      SS53870             -                         0.0000      0.0000      FFF      FFF
0.990  E     754311  (GIP17)_233237      (GIP18)_233237      -                       836.0960      0.0072      FFF      FFF
0.000  H     1372615 (0027TR)_219634     -                   -                        57.6919      1.0000      FFF        -
0.000  E     872668  (0027TR)_219634     PM61747             -                       166.5586      0.0058      FFF      FFF
0.000  E     872667  (0027TR)_219634     TS10887             -                       407.8370      0.0059      FFF      FFF
0.000  D     516327  (0027TR)_219634     TS10887             -                         0.0000      0.0000      FFF      FFF
0.000  D     311527  SS53870             TS7284-3            -                         0.0000      0.0000      FFF      FFF
0.000  D     153790  PM50433             PM50439             -                         0.0000      0.0000      FFF      FFF
0.000  D     153810  PM50435             PM60414             -                         0.0000      0.0000      FFF      FFF
0.000  D     153803  PM50435             PM50436             -                         0.0000      0.0000      FFF      FFF
0.000  D     153797  PM50434             PM50435             -                         0.0000      0.0000      FFF      FFF
0.000  D     516292  TS10887             TS7284-3            -                         0.0000      0.0000      FFF      FFF
0.000  D     153787  PM50433             PM50432             -                         0.0000      0.0000      FFF      FFF
0.000  D     42605   SS21396             (21397BOLT_229434   -                         0.0000      0.0000      FFF      FFF
0.000  D     516307  PM61747             TS7090              -                         0.0000      0.0000      FFF      FFF

====================================================================================

[nicgowans]

Nice work, @daleroberts!

This looks excellent. From the above, I'd read (BM6)_229890 to be the worst offender - so I'm now running another test to see what happens when this station is excluded.

Also, open to suggestions for what else to show in this diagnostic.

Only one suggestion thus far, and that would be to report measurement database id (where available). Using the FileOrd index isn't really practical for very large text files so I've had to use other methods to match the measurements reported here.

Could I ask for clarification on the following?

• Are the StdDev units for direction measurements in radians? The input data has non-zero standard deviations so there's a slight mismatch/misreporting there.
• I also note that all reported type D measurements have a value of 0.0, which in this dataset means they are the reference direction. Should I interpret this report/score to say the reference direction in particular is the flagged measurement, or that it is simply somewhere within that direction set?

I also have a much smaller dataset available that fails with the Cholesky issue, and for which I have identified the offending measurements. Happy to provide this for further testing/refinement if you'd find it helpful.

Thanks again!