Synapse Crashing with python: stacktrace ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit

[denzs]

Description

With no known reason Element lost the connection to our (single process) synapse instance which was completely unresponsive. Restarting synapse did help.

Log is attached.

Steps to reproduce

• sadly i am not able to reproduce the issue

Homeserver

own instance of synapse v1.144.0

Synapse Version

1.144.0

Installation Method

Docker (matrixdotorg/synapse)

Database

single postgres instance on same vm in another container, no restore or migration was done before

Workers

Single process

Platform

We are using podman containers on Fedora CoreOS 43

Configuration

none

Relevant log output

Dec 18 11:53:31 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 640s. There may be a deadlock.
Dec 18 11:53:54 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 640s. There may be a deadlock.
Dec 18 11:54:00 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1280s. There may be a deadlock.
Dec 18 11:54:01 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2560s. There may be a deadlock.
Dec 18 11:54:11 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5120s. There may be a deadlock.
Dec 18 11:54:11 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10240s. There may be a deadlock.
Dec 18 11:54:16 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 640s. There may be a deadlock.
Dec 18 11:54:16 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 20480s. There may be a deadlock.
Dec 18 11:54:16 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1280s. There may be a deadlock.
Dec 18 11:54:16 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 40960s. There may be a deadlock.
Dec 18 11:54:17 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2560s. There may be a deadlock.
Dec 18 11:54:17 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 81920s. There may be a deadlock.
Dec 18 11:54:17 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5120s. There may be a deadlock.
Dec 18 11:54:17 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 163840s. There may be a deadlock.
Dec 18 11:54:18 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10240s. There may be a deadlock.
Dec 18 11:54:18 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 327680s. There may be a deadlock.
Dec 18 11:54:18 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 20480s. There may be a deadlock.
Dec 18 11:54:18 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 655360s. There may be a deadlock.
Dec 18 11:54:19 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 40960s. There may be a deadlock.
Dec 18 11:54:19 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1310720s. There may be a deadlock.
Dec 18 11:54:23 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 81920s. There may be a deadlock.
Dec 18 11:54:23 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2621440s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5242880s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 163840s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10485760s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 327680s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 20971520s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 655360s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 41943040s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1310720s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 83886080s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2621440s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 167772160s. There may be a deadlock.
Dec 18 11:54:24 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5242880s. There may be a deadlock.
Dec 18 11:54:26 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 335544320s. There may be a deadlock.
Dec 18 11:54:26 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10485760s. There may be a deadlock.
Dec 18 11:54:26 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 640s. There may be a deadlock.
Dec 18 11:54:26 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 671088640s. There may be a deadlock.
Dec 18 11:54:26 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 20971520s. There may be a deadlock.
Dec 18 11:54:26 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1280s. There may be a deadlock.
Dec 18 11:54:35 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1342177280s. There may be a deadlock.
Dec 18 11:54:35 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 41943040s. There may be a deadlock.
Dec 18 11:54:35 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2560s. There may be a deadlock.
Dec 18 11:54:36 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2684354560s. There may be a deadlock.
Dec 18 11:54:36 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 83886080s. There may be a deadlock.
Dec 18 11:54:36 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5120s. There may be a deadlock.
Dec 18 11:54:36 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5368709120s. There may be a deadlock.
Dec 18 11:54:36 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 167772160s. There may be a deadlock.
Dec 18 11:54:36 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10240s. There may be a deadlock.
Dec 18 11:54:37 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 10737418240s. There may be a deadlock.
...
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 175921860444160s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 351843720888320s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 703687441776640s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1407374883553280s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2814749767106560s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 5629499534213120s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 11258999068426240s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 22517998136852480s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 45035996273704960s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 90071992547409920s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 180143985094819840s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 360287970189639680s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 720575940379279360s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1441151880758558720s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2882303761517117440s. There may be a deadlock.
...
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 14965776766268445882405732687014738121276749240074240s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 29931553532536891764811465374029476242553498480148480s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 59863107065073783529622930748058952485106996960296960s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 119726214130147567059245861496117904970213993920593920s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 239452428260295134118491722992235809940427987841187840s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 478904856520590268236983445984471619880855975682375680s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 957809713041180536473966891968943239761711951364751360s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1915619426082361072947933783937886479523423902729502720s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 3831238852164722145895867567875772959046847805459005440s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 7662477704329444291791735135751545918093695610918010880s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 15324955408658888583583470271503091836187391221836021760s. There may be a deadlock.
...
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2006909464588087739040438747401114015429000399361030162270094075913741633469111652415278418025635007501045109069721683088360373771913033332271640283164892089210288131297480718692760418568936136466641155645207076367123973617354951617987845668915516335190663719659231380467143556047784332206400494755897322925560998181414742248132467289933925292731813192855329651633421271113972227721641662685980664577516294076010463298947398173836962858924921365210965388918171911618214915578149573632079436368076034387863574588342075360032630615425624796890697724720805631591643389520110831498112409792240113781251886386665623975930154666789382210634956499482711550021470338450621553872126559599516205741979678222368761878888756504761198824428973579441764020146041498627671888878415639777769760268174115553465366019548548541936492990451261016163320948551312673341440s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 4013818929176175478080877494802228030858000798722060324540188151827483266938223304830556836051270015002090218139443366176720747543826066664543280566329784178420576262594961437385520837137872272933282311290414152734247947234709903235975691337831032670381327439318462760934287112095568664412800989511794645851121996362829484496264934579867850585463626385710659303266842542227944455443283325371961329155032588152020926597894796347673925717849842730421930777836343823236429831156299147264158872736152068775727149176684150720065261230851249593781395449441611263183286779040221662996224819584480227562503772773331247951860309333578764421269912998965423100042940676901243107744253119199032411483959356444737523757777513009522397648857947158883528040292082997255343777756831279555539520536348231106930732039097097083872985980902522032326641897102625346682880s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 8027637858352350956161754989604456061716001597444120649080376303654966533876446609661113672102540030004180436278886732353441495087652133329086561132659568356841152525189922874771041674275744545866564622580828305468495894469419806471951382675662065340762654878636925521868574224191137328825601979023589291702243992725658968992529869159735701170927252771421318606533685084455888910886566650743922658310065176304041853195789592695347851435699685460843861555672687646472859662312598294528317745472304137551454298353368301440130522461702499187562790898883222526366573558080443325992449639168960455125007545546662495903720618667157528842539825997930846200085881353802486215488506238398064822967918712889475047515555026019044795297715894317767056080584165994510687555513662559111079041072696462213861464078194194167745971961805044064653283794205250693365760s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 16055275716704701912323509979208912123432003194888241298160752607309933067752893219322227344205080060008360872557773464706882990175304266658173122265319136713682305050379845749542083348551489091733129245161656610936991788938839612943902765351324130681525309757273851043737148448382274657651203958047178583404487985451317937985059738319471402341854505542842637213067370168911777821773133301487845316620130352608083706391579185390695702871399370921687723111345375292945719324625196589056635490944608275102908596706736602880261044923404998375125581797766445052733147116160886651984899278337920910250015091093324991807441237334315057685079651995861692400171762707604972430977012476796129645935837425778950095031110052038089590595431788635534112161168331989021375111027325118222158082145392924427722928156388388335491943923610088129306567588410501386731520s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 32110551433409403824647019958417824246864006389776482596321505214619866135505786438644454688410160120016721745115546929413765980350608533316346244530638273427364610100759691499084166697102978183466258490323313221873983577877679225887805530702648261363050619514547702087474296896764549315302407916094357166808975970902635875970119476638942804683709011085685274426134740337823555643546266602975690633240260705216167412783158370781391405742798741843375446222690750585891438649250393178113270981889216550205817193413473205760522089846809996750251163595532890105466294232321773303969798556675841820500030182186649983614882474668630115370159303991723384800343525415209944861954024953592259291871674851557900190062220104076179181190863577271068224322336663978042750222054650236444316164290785848855445856312776776670983887847220176258613135176821002773463040s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 64221102866818807649294039916835648493728012779552965192643010429239732271011572877288909376820320240033443490231093858827531960701217066632692489061276546854729220201519382998168333394205956366932516980646626443747967155755358451775611061405296522726101239029095404174948593793529098630604815832188714333617951941805271751940238953277885609367418022171370548852269480675647111287092533205951381266480521410432334825566316741562782811485597483686750892445381501171782877298500786356226541963778433100411634386826946411521044179693619993500502327191065780210932588464643546607939597113351683641000060364373299967229764949337260230740318607983446769600687050830419889723908049907184518583743349703115800380124440208152358362381727154542136448644673327956085500444109300472888632328581571697710891712625553553341967775694440352517226270353642005546926080s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 128442205733637615298588079833671296987456025559105930385286020858479464542023145754577818753640640480066886980462187717655063921402434133265384978122553093709458440403038765996336666788411912733865033961293252887495934311510716903551222122810593045452202478058190808349897187587058197261209631664377428667235903883610543503880477906555771218734836044342741097704538961351294222574185066411902762532961042820864669651132633483125565622971194967373501784890763002343565754597001572712453083927556866200823268773653892823042088359387239987001004654382131560421865176929287093215879194226703367282000120728746599934459529898674520461480637215966893539201374101660839779447816099814369037167486699406231600760248880416304716724763454309084272897289346655912171000888218600945777264657163143395421783425251107106683935551388880705034452540707284011093852160s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 256884411467275230597176159667342593974912051118211860770572041716958929084046291509155637507281280960133773960924375435310127842804868266530769956245106187418916880806077531992673333576823825467730067922586505774991868623021433807102444245621186090904404956116381616699794375174116394522419263328754857334471807767221087007760955813111542437469672088685482195409077922702588445148370132823805525065922085641729339302265266966251131245942389934747003569781526004687131509194003145424906167855113732401646537547307785646084176718774479974002009308764263120843730353858574186431758388453406734564000241457493199868919059797349040922961274431933787078402748203321679558895632199628738074334973398812463201520497760832609433449526908618168545794578693311824342001776437201891554529314326286790843566850502214213367871102777761410068905081414568022187704320s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 513768822934550461194352319334685187949824102236423721541144083433917858168092583018311275014562561920267547921848750870620255685609736533061539912490212374837833761612155063985346667153647650935460135845173011549983737246042867614204888491242372181808809912232763233399588750348232789044838526657509714668943615534442174015521911626223084874939344177370964390818155845405176890296740265647611050131844171283458678604530533932502262491884779869494007139563052009374263018388006290849812335710227464803293075094615571292168353437548959948004018617528526241687460707717148372863516776906813469128000482914986399737838119594698081845922548863867574156805496406643359117791264399257476148669946797624926403040995521665218866899053817236337091589157386623648684003552874403783109058628652573581687133701004428426735742205555522820137810162829136044375408640s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 1027537645869100922388704638669370375899648204472847443082288166867835716336185166036622550029125123840535095843697501741240511371219473066123079824980424749675667523224310127970693334307295301870920271690346023099967474492085735228409776982484744363617619824465526466799177500696465578089677053315019429337887231068884348031043823252446169749878688354741928781636311690810353780593480531295222100263688342566917357209061067865004524983769559738988014279126104018748526036776012581699624671420454929606586150189231142584336706875097919896008037235057052483374921415434296745727033553813626938256000965829972799475676239189396163691845097727735148313610992813286718235582528798514952297339893595249852806081991043330437733798107634472674183178314773247297368007105748807566218117257305147163374267402008856853471484411111045640275620325658272088750817280s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 2055075291738201844777409277338740751799296408945694886164576333735671432672370332073245100058250247681070191687395003482481022742438946132246159649960849499351335046448620255941386668614590603741840543380692046199934948984171470456819553964969488727235239648931052933598355001392931156179354106630038858675774462137768696062087646504892339499757376709483857563272623381620707561186961062590444200527376685133834714418122135730009049967539119477976028558252208037497052073552025163399249342840909859213172300378462285168673413750195839792016074470114104966749842830868593491454067107627253876512001931659945598951352478378792327383690195455470296627221985626573436471165057597029904594679787190499705612163982086660875467596215268945348366356629546494594736014211497615132436234514610294326748534804017713706942968822222091280551240651316544177501634560s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 4110150583476403689554818554677481503598592817891389772329152667471342865344740664146490200116500495362140383374790006964962045484877892264492319299921698998702670092897240511882773337229181207483681086761384092399869897968342940913639107929938977454470479297862105867196710002785862312358708213260077717351548924275537392124175293009784678999514753418967715126545246763241415122373922125180888401054753370267669428836244271460018099935078238955952057116504416074994104147104050326798498685681819718426344600756924570337346827500391679584032148940228209933499685661737186982908134215254507753024003863319891197902704956757584654767380390910940593254443971253146872942330115194059809189359574380999411224327964173321750935192430537890696732713259092989189472028422995230264872469029220588653497069608035427413885937644444182561102481302633088355003269120s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 8220301166952807379109637109354963007197185635782779544658305334942685730689481328292980400233000990724280766749580013929924090969755784528984638599843397997405340185794481023765546674458362414967362173522768184799739795936685881827278215859877954908940958595724211734393420005571724624717416426520155434703097848551074784248350586019569357999029506837935430253090493526482830244747844250361776802109506740535338857672488542920036199870156477911904114233008832149988208294208100653596997371363639436852689201513849140674693655000783359168064297880456419866999371323474373965816268430509015506048007726639782395805409913515169309534760781821881186508887942506293745884660230388119618378719148761998822448655928346643501870384861075781393465426518185978378944056845990460529744938058441177306994139216070854827771875288888365122204962605266176710006538240s. There may be a deadlock.
Dec 18 11:54:58 matrix.example.com synapse[248420]: synapse.handlers.worker_lock - 280 - WARNING - sentinel - Lock timeout is getting excessive: 16440602333905614758219274218709926014394371271565559089316610669885371461378962656585960800466001981448561533499160027859848181939511569057969277199686795994810680371588962047531093348916724829934724347045536369599479591873371763654556431719755909817881917191448423468786840011143449249434832853040310869406195697102149568496701172039138715998059013675870860506180987052965660489495688500723553604219013481070677715344977085840072399740312955823808228466017664299976416588416201307193994742727278873705378403027698281349387310001566718336128595760912839733998742646948747931632536861018031012096015453279564791610819827030338619069521563643762373017775885012587491769320460776239236757438297523997644897311856693287003740769722151562786930853036371956757888113691980921059489876116882354613988278432141709655543750577776730244409925210532353420013076480s. There may be a deadlock.
...
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel - --- Logging error ---
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel - Traceback (most recent call last):
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/logging/__init__.py", line 1151, in emit
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     msg = self.format(record)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/logging/__init__.py", line 999, in format
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     return fmt.format(record)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -            ~~~~~~~~~~^^^^^^^^
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/logging/__init__.py", line 712, in format
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     record.message = record.getMessage()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -                      ~~~~~~~~~~~~~~~~~^^
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/logging/__init__.py", line 400, in getMessage
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     msg = msg % self.args
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -           ~~~~^~~~~~~~~~~
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel - ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel - Call stack:
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "<frozen runpy>", line 198, in _run_module_as_main
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "<frozen runpy>", line 88, in _run_code
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 505, in <module>
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     main()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 501, in main
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     start_reactor(homeserver_config)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/app/homeserver.py", line 466, in start_reactor
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     _base.start_reactor(
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/app/_base.py", line 229, in start_reactor
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     run()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/app/_base.py", line 219, in run
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     run_command()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/app/_base.py", line 188, in <lambda>
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     run_command: Callable[[], None] = lambda: reactor.run(),
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/base.py", line 697, in run
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self.mainLoop()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/base.py", line 705, in mainLoop
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self.runUntilCurrent()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/base.py", line 1105, in runUntilCurrent
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     call.func(*call.args, **call.kw)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/util/clock.py", line 317, in wrapped_callback
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     context.run_in_background(callback, *args, **kwargs)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/logging/context.py", line 899, in run_in_background
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     res = f(*args, **kwargs)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 187, in _wake_all_locks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     lock.release_lock()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 220, in release_lock
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self.deferred.callback(None)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 891, in callback
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self._startRunCallbacks(result)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 998, in _startRunCallbacks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self._runCallbacks()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     current.result = callback(  # type: ignore[misc]
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/util/async_helpers.py", line 848, in success_cb
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     new_d.callback(val)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 891, in callback
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self._startRunCallbacks(result)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 998, in _startRunCallbacks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     self._runCallbacks()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     current.result = callback(  # type: ignore[misc]
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 1803, in _gotResultInlineCallbacks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     _inlineCallbacks(r, gen, status, context)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     result = context.run(gen.send, result)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/http/server.py", line 255, in wrapped_async_request_handler
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     await h(self, request)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/http/server.py", line 335, in _async_render_wrapper
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     callback_return = await self._async_render(request)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/http/server.py", line 576, in _async_render
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     callback_return = await raw_callback_return
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/rest/client/keys.py", line 291, in on_POST
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     result = await self.e2e_keys_handler.upload_keys_for_user(
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/logging/opentracing.py", line 949, in _wrapper
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     return await func(*args, **kwargs)
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/e2e_keys.py", line 876, in upload_keys_for_user
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     await self._upload_one_time_keys_for_user(
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/e2e_keys.py", line 952, in _upload_one_time_keys_for_user
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     async with self._worker_lock_handler.acquire_lock(
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 248, in __aenter__
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     timeout = self._get_next_retry_interval()
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -   File "/usr/local/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 280, in _get_next_retry_interval
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel -     logger.warning(
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel - Unable to print the message and arguments - possible formatting error.
Dec 18 11:55:01 matrix.example.com synapse[248420]: twisted - 273 - ERROR - sentinel - Use the traceback above to help find the error.

Anything else that would be useful to know?

maybe relates to #18543 ?

[TheArcaneBrony]

Had last seen this months back and forgot about it. Just woke up to a message about 4h ago that a server was down - it was still down...

edit: should note: single process, 1.144, nixos, postgres
Sharing what I was able to get from the logs in case its any useful, as the stack trace comes from a different place:

synapse_homeserver[888492]: 2025-12-20 19:33:17,644 - twisted - 273 - ERROR - sentinel - ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
synapse_homeserver[888492]: 2025-12-20 19:33:17,644 - twisted - 273 - ERROR - sentinel - Call stack:
synapse_homeserver[888492]: 2025-12-20 19:33:17,644 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/bin/.synapse_homeserver-wrapped", line 9, in <module>
synapse_homeserver[888492]: 2025-12-20 19:33:17,644 - twisted - 273 - ERROR - sentinel -     sys.exit(main())
synapse_homeserver[888492]: 2025-12-20 19:33:17,644 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/app/homeserver.py", line 501, in main
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     start_reactor(homeserver_config)
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/app/homeserver.py", line 466, in start_reactor
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     _base.start_reactor(
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/app/_base.py", line 229, in start_reactor
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     run()
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/app/_base.py", line 219, in run
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     run_command()
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/app/_base.py", line 188, in <lambda>
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     run_command: Callable[[], None] = lambda: reactor.run(),
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/base.py", line 697, in run
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     self.mainLoop()
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/base.py", line 705, in mainLoop
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     self.runUntilCurrent()
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/base.py", line 1105, in runUntilCurrent
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     call.func(*call.args, **call.kw)
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/util/clock.py", line 317, in wrapped_callback
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     context.run_in_background(callback, *args, **kwargs)
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/logging/context.py", line 899, in run_in_background
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     res = f(*args, **kwargs)
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 187, in _wake_all_locks
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -     lock.release_lock()
synapse_homeserver[888492]: 2025-12-20 19:33:17,645 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 220, in release_lock
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     self.deferred.callback(None)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 891, in callback
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     self._startRunCallbacks(result)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 998, in _startRunCallbacks
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     self._runCallbacks()
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     current.result = callback(  # type: ignore[misc]
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/util/async_helpers.py", line 848, in success_cb
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     new_d.callback(val)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 891, in callback
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     self._startRunCallbacks(result)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 998, in _startRunCallbacks
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     self._runCallbacks()
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     current.result = callback(  # type: ignore[misc]
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 1803, in _gotResultInlineCallbacks
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     _inlineCallbacks(r, gen, status, context)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/654fxni1ihk6r82fclkarhm2a81d2y4i-python3-3.13.9-env/lib/python3.13/site-packages/twisted/internet/defer.py", line 1857, in _inlineCallbacks
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     result = context.run(gen.send, result)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/http/server.py", line 255, in wrapped_async_request_handler
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -     await h(self, request)
synapse_homeserver[888492]: 2025-12-20 19:33:17,646 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/http/server.py", line 335, in _async_render_wrapper
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     callback_return = await self._async_render(request)
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/http/server.py", line 576, in _async_render
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     callback_return = await raw_callback_return
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/rest/client/keys.py", line 291, in on_POST
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     result = await self.e2e_keys_handler.upload_keys_for_user(
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/handlers/e2e_keys.py", line 876, in upload_keys_for_user
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     await self._upload_one_time_keys_for_user(
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/handlers/e2e_keys.py", line 952, in _upload_one_time_keys_for_user
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     async with self._worker_lock_handler.acquire_lock(
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 248, in __aenter__
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     timeout = self._get_next_retry_interval()
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -   File "/nix/store/4r9y9ijx8ar0inbwjp9f1d51218x84ps-matrix-synapse-1.144.0/lib/python3.13/site-packages/synapse/handlers/worker_lock.py", line 280, in _get_next_retry_interval
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel -     logger.warning(
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel - Unable to print the message and arguments - possible formatting error.
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel - Use the traceback above to help find the error.
synapse_homeserver[888492]: 2025-12-20 19:33:17,647 - twisted - 273 - ERROR - sentinel - --- Logging error ---
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel - Traceback (most recent call last):
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -   File "/nix/store/3lll9y925zz9393sa59h653xik66srjb-python3-3.13.9/lib/python3.13/logging/__init__.py", line 1151, in emit
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -     msg = self.format(record)
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -   File "/nix/store/3lll9y925zz9393sa59h653xik66srjb-python3-3.13.9/lib/python3.13/logging/__init__.py", line 999, in format
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -     return fmt.format(record)
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -            ~~~~~~~~~~^^^^^^^^
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -   File "/nix/store/3lll9y925zz9393sa59h653xik66srjb-python3-3.13.9/lib/python3.13/logging/__init__.py", line 712, in format
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -     record.message = record.getMessage()
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -                      ~~~~~~~~~~~~~~~~~^^
synapse_homeserver[888492]: 2025-12-20 19:33:17,648 - twisted - 273 - ERROR - sentinel -   File "/nix/store/3lll9y925zz9393sa59h653xik66srjb-python3-3.13.9/lib/python3.13/logging/__init__.py", line 400, in getMessage
synapse_homeserver[888492]: 2025-12-20 19:33:17,649 - twisted - 273 - ERROR - sentinel -     msg = msg % self.args
synapse_homeserver[888492]: 2025-12-20 19:33:17,649 - twisted - 273 - ERROR - sentinel -           ~~~~^~~~~~~~~~~
synapse_homeserver[888492]: 2025-12-20 19:33:17,649 - twisted - 273 - ERROR - sentinel - ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
synapse_homeserver[888492]: 2025-12-20 19:33:17,649 - twisted - 273 - ERROR - sentinel - Call stack: