[QUERY] Azure Communication Services Chat slow/long response times

[ErikAndreas]

Library name and version

Azure.Communication.Chat 1.3.1

Query/Question

We're constantly seeing slower and slower response times from various chat sdk endpoints (using chat sdk from azure functions). Most endpoints are in +200 ms, but sometimes we get timeouts (code 499?)

This makes e.g. listing chat threads (GetAccessToken, GetMessages + GetReadReceipts) and some info about their contents, a rather "standard" use-case, almost unbearable from a ux perspective when each call takes such a long time.

InvocationId f04bd15e-a07b-4bf1-8c5b-5c3c43f9b1f5 makes 11 request to Chat SDK endpoints with response times in the range of 101 - 352 ms

We're also seeing cases where GetAccessToken (POST /identities/<ACS UserId>/:issueAccessToken) completely times out:
b5b1845e-fa3c-4000-8632-315256c01552
3206e6ee-1623-482c-8f8c-609fafc80d80
3a121576-8b85-4a35-a3c9-598e95e67f9e

Environment

Azure Functions .net 8 iwp

[ErikAndreas]

stats for dependency calls to (acsInstanceName).europe.communication.azure.com last 24hrs

[gabbsmo]

Also seeing similar behaviour with Communication Services Email. Connections closing or timing out around 1 minute at peak load that should be well within our sending quota.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @abhishesingh-msft @abhiucandoit @adamtuck-msft @adriansynal-msft @AikoBB @ajaych-ms @ajitht-msft @akania @alcail @alexandra142 @AliRafiee @alkwa-msft @allchiang-msft @allenplusplus @AlonsoMondal @alvinjiang-msft @amagginetti @amandaong @amariwest-msft @amatukmolina @AmeliaHeMS @AMollis @andboyer @andrewjhopkins @angelcaz-msft @angellan-msft @angiurgiu @anjulgarg @ankeshni @ankitamm @ankitarorabit @anmolbohra97 @ansrin @antonsamson-msft @anujb-msft @anujissarMS @arifibrahim4 @arifsaikat-microsoft @armandliv @armansabaa @arupdutta-msft @ashwinder @BalajiUmmadisetti @beltr0n @besh2014 @bga-msft @bharat-kalyan-namburi @bobserr @boris-bazilevskiy @brpiment @bterlson @calvinkwtang @carlosalm-msft @carocao-msft @cemateia @chengyuanlai-msft @cheukchuen @CHILIU-MSFT @chriswhilar @chrwhit @claireoberg-msft @cn0151 @cochi2 @csandjon @dailam-msft @danielav7 @danielgerlag @danielortega-msft @dassabya15 @DaybreakQuip @ddouglas-msft @devwolf1 @dinazavyr @dmceachernmsft @DominikMe @dvillasenor-msft @dzeliar @ealejandrootalvaro @edwardlee-msft @elavarasidc @emlynmac @enguerrandb-msft @enricohuang @ericasp16 @eriwang-msft @fangchen0601 @fanruisun @FarhadJabiyev @FerOsorio @fhaghbin-msft @fizampou @fuyan2024 @gaobob-msft @garchiro7 @Gemakk @genevievetok @gfeitosa-msft @glorialimicrosoft @GoWang @grangasamy-msft @grigoryk @guoqing2023 @hansung-msft @hrazi @huachuandeng @HuangXin-MS @iaulakh @ihuseynov-msft @ikumarapeli-msft @ilyapaliakou-msft @imikemo @jadacampbell @jakublehotsky @JamesBurnside @jamescadd @jannovak-msft @jdebroin-msft @jethier-msft @JianpingChen @jimchou-dev @jiminwen-msft @jiriburant @jirisofka @jjsanchezms @Joeleniqs @jorge-beauregard @josecomboni @JoshuaLai @jowang-msft @jpeng-ms @jrathor @jsaurezlee-msft @juancamilor @juntuchen-msft @jutik0 @kagbakpem @khannavikas @kieraniles-msft @ktimofejev-msft @kurtzeborn @Leah-Xia-Microsoft @lei-msft @lisaleehan @LoadLibrary @lsundaralingam @LuChen-Microsoft @lucianopa-msft @magesh-ms @mannyovena @marche0133 @mariusu-msft @matthohn-msft @maximrytych-ms @mayssamm @megheshpatil @mgamis-msft @miguhern @mikehang-msft @MilanKaur-01 @minnieliu @minwoolee-msft @mjafferi-msft @mmpowers-mi @msft-qifan @MSFTFox @namratasimha-msft @natekimball-msft @NathanJBennett @navali-msft @nemofei11 @nikithauc @nikuklic @ninikasharma @nmaredia @nmurav @nostojicMs @osaghaso @palatter @paolamvhz @Paresh-Arvind-Patil @paveldostalms @pavelprystinka @pereiralex @pgrandhi @phermanov-msft @pkestikar @pohtsng @poorva-MSFT @prabhjot-msft @prasadker @priyankaaprakash @rajasekaran2003 @rajat-rastogi @rajuanitha88 @ralphgabrinao @ramneettung-msft @raosanat @rasinive @RezaJooyandeh @rkprasad-ks @rorezende @ryturn757 @rzdor @sacheu @Saisang @SamuelSA @sankum-msft @sarkar-rajarshi @satyakonmsft @Scott-Leong @Shamkh @sharifrahaman @ShaunaSong @shirleyqin-msft @shwali-msft @slpavkov @sofiar-msft @soricos85 @sorrego-msft @sphenry @Srinuvasu-Bodepudi @stefang931 @subhasan @swagatmishra2007 @tadam-msft @tariqzafa700 @tomaschladek @tonyliu43361 @tophpalmer @usvoyager @v-durgeshs @v-mbarad-msft @v-pivamshi @v-shazilms @v-vdharmaraj @vaibhavjain-msft @vamoskal @vhuseinova-msft @VikramDhumal @vikrampraveen @viniciusl-msft @vinoPuzzle @vivekmore-msft @vriosrada-msft @wangrui-msft @waynemo @whisper6284 @williamzhao87 @xixian73 @xumo-95 @xxwikkixx @yassirbisteni @yogeshmo @ypradhan-msft @zeyingguo1991 @zhengni-msft @zihzhan-msft.

[ErikAndreas]

GetAccessToken (POST /identities/<ACS UserId>/:issueAccessToken) has very long response times
InvocationId e292e400-e496-4d71-9fbc-8a97d2405960, ClientRequestId 3a4472f4-96db-4c3a-87d6-c45192ab9562 37,2 secs
InvocationId 6140e077-5f38-497e-8abd-a7cf7d101b5c, ClientRequestId 56634e3c-4d46-4c10-9bae-356ef7423627 24,7 secs
InvocationId 04ecde28-b625-4ea5-bb6d-2e8afe9ebb06, ClientRequestId 6fca2aed-07c9-4d97-b95a-51ee28822547, 12,6 secs

this can't be expected or within range of what should be considered normal?

[ErikAndreas]

@gabbsmo cross-posted here: https://learn.microsoft.com/en-us/answers/questions/2128273/azure-communication-services-chat-service-degradat