Description
Describe the bug 🐞
I have been using NonlinearSolve to solve nonlinear least squares problems in an algorithm I have been developing, but when I upgraded to version v4.x.x the Levenberg Marquardt solver started giving poor results. When I pin the package back at v3.15.1 all my issues resolve, and all the v4.x.x versions I have tried have the behavior below. I can recreate the behavior on a linear least squares problem, I saved the input matrices at the bottom of this issue.
Expected behavior
When I run this code in v3.15.1, the linear least squares problem is solved to a high level of accuracy:
Status `~/nolinearsolveissue/Project.toml`
[8bb1440f] DelimitedFiles v1.9.1
⌃ [8913a72c] NonlinearSolve v3.15.1 ⚲
Info Packages marked with ⌃ have new versions available and may be upgradable.
[ Info: Relative Error 4.280111795651271e-9
Minimal Reproducible Example 👇
## Load dependencies
using NonlinearSolve: NonlinearLeastSquaresProblem, NonlinearFunction, LevenbergMarquardt, solve
using DelimitedFiles: readdlm
using Pkg
Pkg.status()
## Build linear least square problem for testing
A = readdlm(joinpath(@__DIR__,"A.csv"),',',Float64)
b = readdlm(joinpath(@__DIR__,"b.csv"),',',Float64)[:]
M,N = size(A)
function r!(r,x, ::Any)
r .= A*x - b
nothing
end
function ∇r!(J, p, ::Any)
J .= A
nothing
end
# Run NonlinearSolve.solve
x0 = zeros(N)
sol = solve(
NonlinearLeastSquaresProblem(
NonlinearFunction(
r!;
jac=∇r!,
resid_prototype=zeros(M)
),
x0
),
LevenbergMarquardt()
)
xhat = sol.u
x = A \ b
relErr = norm(xhat- x)/norm(x)
@info "Relative Error $relErr"
Error & Stacktrace
Status `~/nolinearsolveissue/Project.toml`
[8bb1440f] DelimitedFiles v1.9.1
[8913a72c] NonlinearSolve v4.3.0
[ Info: Relative Error 1.0
Environment (please complete the following information):
- Output of
using Pkg; Pkg.status()
Status `~/scratch/wendy/_wendyTriage/nolinearsolveissue/Project.toml`
[8bb1440f] DelimitedFiles v1.9.1
[8913a72c] NonlinearSolve v4.3.0
- Output of
using Pkg; Pkg.status(; mode = PKGMODE_MANIFEST)
Status `~/scratch/wendy/_wendyTriage/nolinearsolveissue/Manifest.toml`
[47edcb42] ADTypes v1.12.1
[7d9f7c33] Accessors v0.1.41
[79e6a3ab] Adapt v4.1.1
[4fba245c] ArrayInterface v7.18.0
[4c555306] ArrayLayouts v1.11.0
[62783981] BitTwiddlingConvenienceFunctions v0.1.6
[70df07ce] BracketingNonlinearSolve v1.1.0
[2a0fbf3d] CPUSummary v0.2.6
[d360d2e6] ChainRulesCore v1.25.1
[fb6a15b2] CloseOpenIntervals v0.1.13
[38540f10] CommonSolve v0.2.4
[bbf7d656] CommonSubexpressions v0.3.1
[f70d9fcc] CommonWorldInvalidations v1.0.0
[34da2185] Compat v4.16.0
[a33af91c] CompositionsBase v0.1.2
[2569d6c7] ConcreteStructs v0.2.3
[187b0558] ConstructionBase v1.5.8
[adafc99b] CpuId v0.3.1
[9a962f9c] DataAPI v1.16.0
[864edb3b] DataStructures v0.18.20
[e2d170a0] DataValueInterfaces v1.0.0
[8bb1440f] DelimitedFiles v1.9.1
[2b5f629d] DiffEqBase v6.161.0
[163ba53b] DiffResults v1.1.0
[b552c78f] DiffRules v1.15.1
[a0c0ee7d] DifferentiationInterface v0.6.39
[ffbed154] DocStringExtensions v0.9.3
[4e289a0a] EnumX v1.0.4
[f151be2c] EnzymeCore v0.8.8
[e2ba6199] ExprTools v0.1.10
⌅ [6b7a57c9] Expronicon v0.8.5
[7034ab61] FastBroadcast v0.3.5
[9aa1b823] FastClosures v0.3.2
[29a986be] FastLapackInterface v2.0.4
[a4df4552] FastPower v1.1.1
[1a297f60] FillArrays v1.13.0
[6a86dc24] FiniteDiff v2.27.0
[f6369f11] ForwardDiff v0.10.38
[069b7b12] FunctionWrappers v1.1.3
[77dc65aa] FunctionWrappersWrappers v0.1.3
[46192b85] GPUArraysCore v0.2.0
[3e5b6fbb] HostCPUFeatures v0.1.17
[615f187c] IfElse v0.1.1
[3587e190] InverseFunctions v0.1.17
[92d709cd] IrrationalConstants v0.2.4
[82899510] IteratorInterfaceExtensions v1.0.0
[692b3bcd] JLLWrappers v1.7.0
[ef3ab10e] KLU v0.6.0
[ba0b0d4f] Krylov v0.9.9
[10f19ff3] LayoutPointers v0.1.17
[5078a376] LazyArrays v2.5.1
[87fe0de2] LineSearch v0.1.4
⌅ [7ed4a6bd] LinearSolve v2.39.0
[2ab3a3ac] LogExpFunctions v0.3.29
[bdcacae8] LoopVectorization v0.12.171
[d8e11817] MLStyle v0.4.17
[1914dd2f] MacroTools v0.5.15
[d125e4d3] ManualMemory v0.1.8
[bb5d69b7] MaybeInplace v0.1.4
[46d2c3a1] MuladdMacro v0.2.4
[77ba4419] NaNMath v1.1.2
[8913a72c] NonlinearSolve v4.3.0
[be0214bd] NonlinearSolveBase v1.4.0
[5959db7a] NonlinearSolveFirstOrder v1.2.0
[9a2c21bd] NonlinearSolveQuasiNewton v1.1.0
[26075421] NonlinearSolveSpectralMethods v1.1.0
[6fe1bfb0] OffsetArrays v1.15.0
[bac558e1] OrderedCollections v1.8.0
[d96e819e] Parameters v0.12.3
[f517fe37] Polyester v0.7.16
[1d0040c9] PolyesterWeave v0.2.2
[d236fae5] PreallocationTools v0.4.24
[aea7be01] PrecompileTools v1.2.1
[21216c6a] Preferences v1.4.3
[3cdcf5f2] RecipesBase v1.3.4
[731186ca] RecursiveArrayTools v3.29.0
[f2c3362d] RecursiveFactorization v0.2.23
[189a3867] Reexport v1.2.2
[ae029012] Requires v1.3.0
[7e49a35a] RuntimeGeneratedFunctions v0.5.13
[94e857df] SIMDTypes v0.1.0
[476501e8] SLEEFPirates v0.6.43
[0bca4576] SciMLBase v2.72.2
[19f34311] SciMLJacobianOperators v0.1.1
[c0aeaf25] SciMLOperators v0.3.12
[53ae85a6] SciMLStructures v1.6.1
[efcf1570] Setfield v1.1.1
[727e6d20] SimpleNonlinearSolve v2.1.0
[0a514795] SparseMatrixColorings v0.4.12
[e56a9233] Sparspak v0.3.9
[276daf66] SpecialFunctions v2.5.0
[aedffcd0] Static v1.1.1
[0d7ed370] StaticArrayInterface v1.8.0
[1e83bf80] StaticArraysCore v1.4.3
[10745b16] Statistics v1.11.1
[7792a7ef] StrideArraysCore v0.5.7
[2efcf032] SymbolicIndexingInterface v0.3.37
[3783bdb8] TableTraits v1.0.1
[bd369af6] Tables v1.12.0
[8290d209] ThreadingUtilities v0.5.2
[a759f4b9] TimerOutputs v0.5.26
[d5829a12] TriangularSolve v0.2.1
[781d530d] TruncatedStacktraces v1.4.0
[3a884ed6] UnPack v1.0.2
[3d5dd08c] VectorizationBase v0.21.71
[1d5cc7b8] IntelOpenMP_jll v2025.0.4+0
[856f044c] MKL_jll v2025.0.1+1
[efe28fd5] OpenSpecFun_jll v0.5.6+0
[1317d2d5] oneTBB_jll v2021.12.0+0
[0dad84c5] ArgTools v1.1.2
[56f22d72] Artifacts v1.11.0
[2a0f44e3] Base64 v1.11.0
[ade2ca70] Dates v1.11.0
[8ba89e20] Distributed v1.11.0
[f43a241f] Downloads v1.6.0
[7b1f6079] FileWatching v1.11.0
[9fa8497b] Future v1.11.0
[b77e0a4c] InteractiveUtils v1.11.0
[4af54fe1] LazyArtifacts v1.11.0
[b27032c2] LibCURL v0.6.4
[76f85450] LibGit2 v1.11.0
[8f399da3] Libdl v1.11.0
[37e2e46d] LinearAlgebra v1.11.0
[56ddb016] Logging v1.11.0
[d6f4376e] Markdown v1.11.0
[a63ad114] Mmap v1.11.0
[ca575930] NetworkOptions v1.2.0
[44cfe95a] Pkg v1.11.0
[de0858da] Printf v1.11.0
[9a3f8284] Random v1.11.0
[ea8e919c] SHA v0.7.0
[9e88b42a] Serialization v1.11.0
[6462fe0b] Sockets v1.11.0
[2f01184e] SparseArrays v1.11.0
[fa267f1f] TOML v1.0.3
[a4e569a6] Tar v1.10.0
[8dfed614] Test v1.11.0
[cf7118a7] UUIDs v1.11.0
[4ec0a83e] Unicode v1.11.0
[e66e0078] CompilerSupportLibraries_jll v1.1.1+0
[deac9b47] LibCURL_jll v8.6.0+0
[e37daf67] LibGit2_jll v1.7.2+0
[29816b5a] LibSSH2_jll v1.11.0+1
[c8ffd9c3] MbedTLS_jll v2.28.6+0
[14a3606d] MozillaCACerts_jll v2023.12.12
[4536629a] OpenBLAS_jll v0.3.27+1
[05823500] OpenLibm_jll v0.8.1+2
[bea87d4a] SuiteSparse_jll v7.7.0+0
[83775a58] Zlib_jll v1.2.13+1
[8e850b90] libblastrampoline_jll v5.11.0+0
[8e850ede] nghttp2_jll v1.59.0+0
[3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
- Output of
versioninfo()
Julia Version 1.11.3
Commit d63adeda50d (2025-01-21 19:42 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin24.0.0)
CPU: 12 × Apple M3 Pro
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, apple-m2)
Threads: 16 default, 0 interactive, 6 GC (on 6 virtual cores)
Environment:
JULIA_EDITOR = code
JULIA_NUM_THREADS = 16
Additional context
Here are the CSV files for the example:
A.csv
15.31698920416511,-11.726412549473807
-10.438595789013336,10.97251587049842
4.029921808623694,-5.193570598440186
2.9507936437397126,-2.6653276608823404
-3.0423578706194543,2.8167146033216515
-2.344793176199911,2.571503122253318
-2.073658703845579,2.1093729937977974
1.7172250648421552,-1.6889667376190396
-1.4907004052690813,1.412898364096967
-1.2670801583008657,1.2485470687147258
1.3293545871918862,-1.2498232016825104
-1.0872649961130803,1.193279536912415
1.266932392006042,-1.191047333791235
0.8846856350189316,-0.9543938303387552
-0.9978299502210634,1.0073111207859942
-1.0307159064513747,1.0362056331607332
0.8489041023115367,-0.952428325255705
0.8354742675480029,-0.7838381473504932
-0.7473267661856161,0.6951207205275893
0.6426054360210914,-0.6989607710096313
-0.8345305995522474,0.7177705185389395
-0.5446386042434861,0.5497706285958638
0.5946994625026157,-0.5145400399806652
0.6237520501489061,-0.6205610925231428
-0.7084469917362827,0.7392039463906601
0.6600084713736479,-0.7207641425719827
0.5643730714787103,-0.6308893311190805
-0.6176493670512581,0.612133656243226
0.5371220260780638,-0.5610198166132709
0.504860186077633,-0.4067523551341881
-0.3832360314205804,0.38597803863845004
0.5803529992540006,-0.5674988897379801
-0.6297740593687509,0.7342458540276233
0.5914988693508249,-0.6213261713363094
-0.41826048538403426,0.42453562654534166
-0.4321529715236706,0.31930044022798787
-0.3723508877091608,0.3178238205953371
-0.4629146974821977,0.39592605336406295
-0.43825150690430204,0.4993084394871265
0.5502053877376477,-0.46263330926035146
0.34701035027462623,-0.26876414203792925
0.2894219866399238,-0.27538621312842404
-0.6411814730849736,0.5596384489453076
0.4500419754952206,-0.49437854729919484
-0.36374205048997327,0.33051865335860653
0.4342074287233244,-0.36645671571838184
-0.49138823019393957,0.4844954500305481
0.45700213515820587,-0.4820939504896067
-0.5232299524171138,0.5322162576678684
0.511753599174885,-0.5064833405488872
-0.5282787494442396,0.5994055682529909
0.6149680424469477,-0.6733756418739236
0.614848239903988,-0.6429011746708992
-0.458762548798176,0.5641673748626708
0.6897389246100802,-0.6394532461189
-0.4955468030704978,0.4193999908802304
-0.38712715933803205,0.30537724208853706
-0.5106475493508476,0.49351545980134287
-0.8151472858417373,0.7517145391454767
0.5544951133518521,-0.6159495394107869
-0.6281481711940268,0.6002327169789048
0.7268463308497111,-0.7400609273076515
-0.8068712548646904,0.8138366217270263
0.7027660215868308,-0.7803703010534163
0.8444135838847499,-0.7393034933555336
-0.6155729726994683,0.6871640249504669
0.8769743639169122,-0.7663648549635036
0.6963466366214607,-0.807381884631105
-0.8219522065379572,0.7831442554419158
0.6748132141065819,-0.7171908292098317
0.6229181142981455,-0.6824446992651523
0.7291280320793929,-0.6582159980876617
-0.47561335847588293,0.5982084519490123
0.6225139221674381,-0.6277599148555619
-0.531025263096857,0.6491970369742706
0.5235669361697027,-0.5229423554435827
-0.3405519979854382,0.42589413073862725
0.47451651110013493,-0.414192383846739
-0.353549404766648,0.4352425014209105
0.3971221398033505,-0.4352021114383679
-0.3690934036743908,0.36399050037793973
-0.2512865362695087,0.23262445775538426
-0.222517856094778,0.22844988575105535
-0.33815265952964524,0.3603548280251424
-0.375769012268218,0.4778883329049375
-0.33512084648319096,0.39112960365360805
0.24437506823362373,-0.3109144760432603
-0.3088130422676923,0.2230838559889195
0.1768701069861921,-0.16237888385100951
0.17600579790199497,-0.1353139326857466
0.2669638932925321,-0.17698671536408353
-0.197286453174369,0.2958559178113006
-0.39061200432295845,0.43482993347465065
-0.30871895190516524,0.4193828616137175
0.29391716589635936,-0.27558173506832284
0.0952412646160802,0.00521080731427484
0.09719797092005528,-0.052414887910547325
0.3695741471252669,-0.4041580818277236
0.39425200593170734,-0.5791061170743713
-0.43544619558226344,0.5487152628441256
-0.2591093219173957,0.39784374229386243
-0.29334312099422255,0.2499022118464873
-0.1707500564905986,0.15246347134103388
0.20078574226960666,-0.17046948389749744
-0.3028007982479594,0.32391307292923327
-0.33003344172003724,0.3756023849630652
0.2417483410279664,-0.39408150523295005
0.4600726870282602,-0.35771689310946847
0.04054067038963957,-0.16109639885519372
-0.3136308465626233,0.20126379478031312
-0.28305912364703145,0.3652958252151882
-0.33970591688812524,0.32524446038096905
-0.19752618658290705,0.24241302551282043
-0.2928350153652468,0.2114923449720733
0.22934938329114313,-0.26491724211483036
-0.34787228877671555,0.3472012024139604
0.33715700525797354,-0.4186581456686787
0.34925316613057766,-0.3732134273764104
0.2947436351966891,-0.30481641038628704
-0.31241649487355355,0.3593072610375401
0.43026348794545743,-0.4476552348877719
0.30368657057094167,-0.31834267021284324
-0.270792688841467,0.3137446792353615
0.41987202499185017,-0.39256226324388616
-0.3250903381055521,0.3911090267360291
0.3782895653906675,-0.4027273169838218
0.3960044608179958,-0.47965237413414746
-0.45319231492538736,0.4376552362839835
-0.261927284929387,0.30959778203565147
0.43148051593930203,-0.351921456795581
0.33855091142052124,-0.39675287831132977
0.3800938126919347,-0.2897768566223905
-0.23083515052994472,0.21451405234860282
-0.41902527756073626,0.26684217161325463
-0.25898026008024533,0.37997350314728606
-0.5824241061999049,0.4909040300269799
-0.2947648941614897,0.47881159669277296
-0.5338562673259628,0.47678314397527827
0.4178399385959303,-0.5492828943376231
-0.49277417010962116,0.5913785765952709
0.5251105064509278,-0.622371107343866
-0.387378145996487,0.4808956555969969
-0.36599410177224967,0.3932532251843047
-0.37087539696093713,0.4298187234523071
-0.3856478532324044,0.36198349771368427
-0.2288526733326996,0.2544379728515085
-0.31930085926050716,0.2640091493534826
-0.20901004149537433,0.212390567784223
-0.24219311276781683,0.15143793199504935
-0.12696358492747722,0.06753885741844164
-0.15538267992611615,0.057736374267889454
-0.19696925081155953,0.09623664798971798
-0.11881898250730885,0.08187702321264526
-0.20125646005108744,0.14917126681116763
-0.24158211448652314,0.26248685391945115
-0.2613547504595848,0.37009446395450707
0.34588588850707547,-0.3630759491115456
-0.1159880086405698,0.1681007156178012
-0.19113143531694407,0.18821471263043862
0.2671540056679125,-0.32272087300022545
-0.2656913091127036,0.3503192928841008
-0.1818751295764001,0.24010228705967246
-0.1764208156033733,0.14599032534514292
-0.07842393911029183,0.13072651466545704
-0.2888838456156634,0.2645799582070679
0.18070926491583006,-0.2846203258099826
-0.18984691493100062,0.10812136016120485
0.07429951809227864,-0.09060105350687761
-0.10141129409831318,-0.026170218822731572
0.12824171456245193,-0.09306992326406346
0.08116231658931064,-0.07259306012037121
-0.12344976037579275,0.07739352808266382
0.10810765024821371,-0.12613792851135475
0.14857589227275947,-0.1330471509101291
-0.10586333088399132,0.08488927891808704
-0.0846039352908137,0.11037377131682231
-0.18435256462049182,0.20471224516111908
0.21073682868791627,-0.2641229842555395
-0.14859002544879563,0.22340405204263816
-0.1468027710425927,0.17578666480674826
-0.20401514466438372,0.17153496856471717
0.04543012987518978,-0.09638931641516402
0.13666627874159684,-0.06602719184788636
-0.02883612292517313,-0.027790200675794557
-0.013151555053463306,-0.05960499652081239
0.06553186218406641,0.02843375180140834
-0.05566506891832588,0.019322841100172156
-0.10925935755538066,0.06477200068065167
0.08889806878740567,-0.1458608246643706
0.20294065090504915,-0.18139236116614477
0.07014520930063164,-0.047029004522437676
0.021420424784059366,-0.09756818073071659
0.07777119556202515,0.029350572544840872
-0.025307928162814894,-0.07004357974248605
0.05279324383978711,-0.13528135049612455
0.047659613509575816,0.14765601000348147
-0.08078013448880816,0.009475322475584955
-0.3237615847426172,0.2782254966753789
-0.14222390864356982,0.2229696750124515
-0.030808376787092077,0.0066234274213425835
b.csv
3.838577886612667
0.6418135348071083
-1.237531153203716
0.5567529304284147
-0.2752201369663889
0.10570608703620865
0.1233756312158436
-0.22227232613328907
0.3010552144375698
-0.2880108058386597
0.02550509585443539
-0.3528908531062189
0.04398826360319508
-0.43704597053147376
-0.8913060773068899
0.3645393422730292
-0.33699987806598797
-0.18194478323974272
0.6736294960374588
0.40464232166455094
0.19471743192101876
0.9163388731954898
0.8318473669037703
0.3493791694897719
-0.7288023515814915
-0.8804802791264039
0.07756400841396882
0.5435237540062179
-0.46059300554516547
-0.932339528413995
-0.5082494559625446
1.3635660409265173
0.09905060717582392
-1.4431946526714086
1.4641011669536876
0.49841896295128363
0.24627566645343624
-0.6676445967592713
-0.013811419193756613
0.13026026254849654
-2.523473506445044
2.894603645505611
-0.8869205565263987
-1.6209016704643584
0.025827001445585918
1.491696645213716
-0.2835051512997713
1.0354506944717317
-0.5779333989083049
0.6831265320073575
-0.4530162612002928
1.1934626341103418
-2.5839119323824007
-0.26622090843116686
-0.9121277713920928
3.4822500594537686
0.6504516844672721
-4.062914402672129
-0.19757254642476596
-2.022677661478904
-1.8362968261929153
1.6018667138653249
0.35461791769471684
0.4422715616639463
-0.41881214432731273
-0.3108890986472108
3.193446814457941
-0.9041749796078016
-3.5478734076846066
-2.7963061307876393
4.655914411969542
-1.6664856556412728
-1.7863981590978737
1.2855790893357457
-0.6843864226795082
-1.8732801459945314
-0.4638572540717001
0.8111358513019292
0.7123322793204299
0.7975477235131607
1.7269572207601214
1.5653486327003931
-1.9987608660791636
-2.478701474047271
-0.7176852030985379
2.084409086297714
-0.47905618874320055
1.1620017776952598
-2.2552877682568524
2.188186102334536
1.0016159915449103
-2.4174616035618244
-2.9987213159721007
2.738447827159227
-1.4351678726328687
-4.525220041046816
7.498087087289962
4.195555010016727
3.485880971803544
3.712468693378973
2.1629707111996495
3.6614098247375364
1.3985087016065232
0.4734472183305769
-3.8292433310806206
3.5921452906767746
3.5154289012169
-5.7470690341535535
-3.155911861937013
-3.969732314809608
0.8344057057875189
1.3409589746383488
1.9385653379393426
-1.1196029762918895
0.19048931320922424
-2.9665171069089773
-0.9805467169054851
-0.322732763188027
-1.9419273914151471
-2.025963549527194
-0.6842543914994748
-5.041945330810787
-2.4588481868828347
-0.8284994185616935
0.89090107960673
-0.5866722838998539
1.0923766020477101
5.458934093470433
1.4692477362038399
-0.7456868300193424
-2.073943973277592
-3.669556440006734
-1.2701433548187202
-1.2383938144183997
-4.430889798296054
-4.0980910676832245
2.1341188027226
-7.62120489459804
-4.327912164966708
-6.115326747123527
-6.986092369337415
2.5515529052427977
5.444944539831922
-0.19173153651993768
8.265886598690603
2.6526497843609924
4.302696673966967
3.5936959892754894
3.7613843575519885
3.111360011139528
-1.978658586929187
-0.521894996158929
-1.3966880052454052
-6.912288258659547
-2.1086132023867963
-5.85569668303706
-3.667230306211683
2.929878627454754
-3.7403178818547285
1.9317558410427544
3.7931680102842926
2.6080204509882057
5.219017792922445
-1.766033530760726
-0.2991703762851622
-5.809729370859308
8.416531507550705
6.634316785351898
-6.560022639680682
-3.583071202413106
2.083557069866404
0.7064932544663987
1.8661020494239633
-0.4609842283502312
0.9255864282892972
-1.929304472157051
-4.0704351491411055
-2.2340976089781814
3.3621246296249305
1.2071699622319612
5.846167685532446
-6.9504269056551635
-2.037150901054769
9.286106612063012
-1.6280773087495017
-2.6102254159845213
-2.868980034132153
1.7099793749546595
2.8870219265676482
-3.5953731020114756
-8.530598947217992
3.6047616216212743
1.6969051420122478
7.763292351090271
-4.598360782800665
-1.2289023389420435
12.061787348023385
-9.767669822073955
9.576599949373211
0.5263049796963806