Classify undeclared exceptions for ServiceRouter

Summary:
`glean.glass`'s ServiceRouter server-fatal SLI is ~99.9% undeclared
`ApplicationException`, and ~94% of those are downstream `glean.query.prod`
failures leaking out of the Glass handler untyped. Two consequences:

1. The SLI measures Glean's throttling, not Glass's availability. The
   `glean.glass fatals` detector fires on "a Glass client exceeded its Glean
   per-client QPS share" — something Glass oncall cannot act on.
2. **ServiceRouter marks down healthy Glass hosts.** Undeclared exceptions get
   `kUndeclaredAppErrorConfig`, which is `logFatal = true` *and*
   `markdown = true`, so a Glean-side throttle shrinks the routable Glass pool
   exactly when load is highest.

For a Thrift error, the following headers are appended to the response:
* 'uex': a short string for the exception type, e.g. ApplicationException
* 'ex': the error code for that exception type
* 'uexw': the complete error message.

hsthrift cannot distinguish undeclared exceptions on its own. The generated `respWriter'`
replaces the escaping exception with a fresh `Thrift.ApplicationException`
before `handlerWrapper` takes `show (typeOf ex)`, so `uex` is uniformly
`ApplicationException` and all triage has to string-match `uexw`.

This diff adds function `Glean.Glass.ErrorClassification.srErrorName`, which maps an escaping
exception to the `uex` name ServiceRouter should classify it under. D116930502 in configerator then defines the behaviour for each new `uex` value. The mapping of escapting exceptions is as follows:

| escaping exception | new `uex` name |
|---|---|
| downstream `THROTTLING_CLIENT_ID_REQUEST` | `GleanClientThrottled` |
| downstream `THROTTLING_ERROR` | `GleanServiceThrottled` |
| downstream `APP_QUEUE_TIMEOUT` | `GleanQueueTimeout` |
| downstream `HOST_OVERLOAD` / `APP_OVERLOAD` | `GleanOverload` |
| GHC allocation limit | `GlassAllocationLimit` |
| Glass's own 30s `withTimeout` | `GlassTimeout` |

`assignHeaders` emits the new `uex` header.

The downstream reason is recovered by matching the `(REASON)` token that
`TServiceRouterException::getExceptionMsg` puts at the front of its message.
String matching is unavoidable today — `HsChannel.h` stringifies the whole
`folly::exception_wrapper` into a `ChannelException Text` and drops the
response headers — but only the enum name is matched, never the prose after
it, and an unrecognised reason returns `Nothing`.

This diff is inert on its own: ServiceRouter falls back to
`kUndeclaredAppErrorConfig` for any name with no `appErrorsMap` entry, so
behaviour is unchanged until the companion `configerator` change adds the
per-name policy. `ex` and `uexw` are deliberately left alone, so the wire-level
record still says "undeclared application exception" and no client sees a
different response.

Reviewed By: bochko

Differential Revision: D116930531

fbshipit-source-id: 9f92d799e9d135d8f73379735244c11ca5e15421

Author: Catherine Gasnier

glean/glass/ARCHITECTURE.md

@@ -117,6 +117,41 @@ client instead of seeing all Glass traffic as a single `glean.glass` client.
 it in from the process's first SMC tier — see
 `fbcode/tupperware/config/glean/glass/glass.tw`.)
 
+### Error classification for ServiceRouter (`uex`)
+
+`assignHeaders` is the `postProcess` hook passed to `runFacebookService'`. For a
+failed request it asks
+[`srErrorName`](Glean/Glass/ErrorClassification.hs) for a name and, when there
+is one, emits it as the `uex` response header. ServiceRouter looks `uex` up in
+`appErrorsMap` in glean.glass's routing config
+(`configerator/raw_configs/fbrpcclient/configs/glean.glass/glean.glass`) and
+takes the fatal/markdown/retry policy from there.
+
+This exists because hsthrift cannot distinguish undeclared exceptions on its
+own: `handlerWrapper` sets `uex` to `show (typeOf ex)` *after* the generated
+`respWriter'` has replaced the original exception with a fresh
+`Thrift.ApplicationException`, so every undeclared failure reaches
+ServiceRouter as the single name `ApplicationException`. ServiceRouter then
+applies `kUndeclaredAppErrorConfig` — `logFatal = true` **and**
+`markdown = true`. In practice most of those failures are downstream Glean
+throttles and queue timeouts, so before this change a Glean-side throttle both
+polluted Glass's server-fatal SLI and marked down healthy Glass hosts.
+
+The override works because `postProcess` headers are appended last in
+`handlerWrapper` and `HaskellProcessor.cpp` applies them with `setHeader`,
+which overwrites. `ex` and `uexw` are deliberately left alone, so the
+wire-level record still says "undeclared application exception"; only
+ServiceRouter's classification policy changes, and no client sees a different
+response.
+
+`srErrorName` recovers the downstream cause by string-matching the
+`(REASON)` token that `TServiceRouterException::getExceptionMsg` puts at the
+front of its message. That is unavoidable today: hsthrift's cpp-channel
+stringifies the whole `folly::exception_wrapper` into a `ChannelException Text`
+and drops the response headers, so the error code never reaches Haskell. Only
+the enum name is matched, never the prose after it. An unrecognised reason
+returns `Nothing`, which keeps ServiceRouter's default behaviour.
+
 ## 4. Glean DB Selection (“best/closest DB”)
 
 This is one of Glass’s key logic layers.

glean/glass/Glean/Glass/ErrorClassification.hs

@@ -0,0 +1,70 @@
+{-
+  Copyright (c) Meta Platforms, Inc. and affiliates.
+  All rights reserved.
+
+  This source code is licensed under the BSD-style license found in the
+  LICENSE file in the root directory of this source tree.
+-}
+
+{-# LANGUAGE OverloadedStrings #-}
+
+module Glean.Glass.ErrorClassification
+  ( srErrorName
+  ) where
+
+import Control.Exception
+  ( AllocationLimitExceeded(..), SomeException, fromException )
+import Data.ByteString (ByteString)
+import Data.Text (Text)
+import qualified Data.Text as Text
+
+import Thrift.Channel (ChannelException(..))
+import Thrift.Protocol.ApplicationException.Types
+  ( ApplicationException(..), ApplicationExceptionType(..) )
+
+-- | The @uex@ name ServiceRouter should classify this failure under, or
+-- 'Nothing' to leave its default undeclared-exception handling (fatal plus
+-- host markdown) in place.
+--
+-- A name only has an effect if glean.glass's routing config gives it an
+-- @appErrorsMap@ entry; without one ServiceRouter falls back to that same
+-- default, so adding a name here is never worse than not classifying at all.
+srErrorName :: SomeException -> Maybe ByteString
+srErrorName e
+  | Just (ChannelException msg) <- fromException e =
+      flip lookup downstreamErrorNames =<< serviceRouterErrorReason msg
+  | Just AllocationLimitExceeded <- fromException e =
+      Just "GlassAllocationLimit"
+  | Just ApplicationException
+      { applicationException_type = ApplicationExceptionType_Timeout }
+      <- fromException e =
+      Just "GlassTimeout"
+  | otherwise = Nothing
+
+-- | The @ErrorReason@ enum name that ServiceRouter reported a downstream failure
+-- under, e.g. APP_QUEUE_TIMEOUT, THROTTLING_CLIENT_ID_REQUEST, etc.
+-- hsthrift's cpp-channel flattens the whole @TServiceRouterException@
+-- to its @what()@ string, so this is the only way to recover it. Match the
+-- enum name only: @TServiceRouterException::getExceptionMsg@ formats it as
+-- @(REASON) @, while the message that follows is prose, not contract.
+serviceRouterErrorReason :: Text -> Maybe Text
+serviceRouterErrorReason msg
+  | Text.null afterReason = Nothing
+  | otherwise = Just reason
+  where
+    (_, atMarker) = Text.breakOn marker msg
+    (reason, afterReason) =
+      Text.breakOn ")" (Text.drop (Text.length marker) atMarker)
+    marker = "TServiceRouterException: ("
+
+-- | ServiceRouter @ErrorReason@ names (@servicerouter/common/error.thrift@)
+-- that Glass reports under a dedicated @uex@ name. Anything absent keeps
+-- ServiceRouter's undeclared-exception default.
+downstreamErrorNames :: [(Text, ByteString)]
+downstreamErrorNames =
+  [ ("THROTTLING_CLIENT_ID_REQUEST", "GleanClientThrottled")
+  , ("THROTTLING_ERROR", "GleanErrorRateShed")
+  , ("APP_QUEUE_TIMEOUT", "GleanQueueTimeout")
+  , ("HOST_OVERLOAD", "GleanOverload")
+  , ("APP_OVERLOAD", "GleanOverload")
+  ]

glean/glass/Glean/Glass/Main.hs

@@ -41,6 +41,7 @@ import Util.Log.Text ( logInfo )
 import Logger.IO (withLogger)
 
 import Control.Exception (SomeException, fromException, throwIO)
+import Data.ByteString (ByteString)
 import Data.Text (Text)
 import qualified Data.Text as Text
 import Data.Text.Encoding (encodeUtf8)
@@ -56,6 +57,7 @@ import Glean.Util.ConfigProvider
 import Glean.Util.Some ( Some(Some) )
 
 import Glean.Glass.RepoMapping -- site-specific
+import Glean.Glass.ErrorClassification (srErrorName)
 import qualified Glean.Glass.Env as Glass
 import Glean.Glass.Repos (withLatestRepos)
 import Glean.Glass.SourceControl (setCallerInfo)
@@ -84,6 +86,12 @@ import JustKnobs (evalKnob)
 kThriftCacheNoCache :: Text
 kThriftCacheNoCache = "nocache"
 
+-- | hsthrift's undeclared-exception name header. ServiceRouter looks the value
+-- up in @appErrorsMap@ to decide fatal/markdown, so overriding it here is how
+-- Glass classifies its own failures.
+kUex :: ByteString
+kUex = "uex"
+
 -- | Ok, go.
 mainWith
   :: Parser (Glass.Config GlassTraceWithId -> Glass.Config GlassTraceWithId)
@@ -164,19 +172,23 @@ runGlass res@Glass.Env{fb303, evp} conf@Glass.Config{..} = do
   runFacebookService' fb303 (glassHandler res) assignHeaders options
 #endif
 
+-- | @postProcess@ hook for 'runFacebookService''. These headers are appended
+-- after the ones hsthrift derives from the escaping exception, and
+-- @HaskellProcessor@ sets rather than adds them, so 'kUex' here wins.
 assignHeaders :: GlassServiceCommand r -> Either SomeException r -> Header
-assignHeaders _ (Left e) | isRevisionNotAvailableException e =
-  [ (encodeUtf8 kThriftCacheNoCache, "1")]
+assignHeaders _ (Right _) = []
+assignHeaders _ (Left e) =
+  [ (encodeUtf8 kThriftCacheNoCache, "1") | isRevisionNotAvailableException ] ++
+  [ (kUex, name) | Just name <- [srErrorName e] ]
   where
-    isRevisionNotAvailableException e = case fromException e of
+    isRevisionNotAvailableException = case fromException e of
       Just GlassException{glassException_reasons} ->
         all isRevisionNotAvailable glassException_reasons
       _ -> False
-    isRevisionNotAvailable e = case e of
+    isRevisionNotAvailable r = case r of
       GlassExceptionReason_exactRevisionNotAvailable{} -> True
       GlassExceptionReason_matchingRevisionNotAvailable{} -> True
       _ -> False
-assignHeaders _ _ = []
 
 #if GLEAN_FACEBOOK
 -- | ModifyFunction (C++ FFI) that installs the shared CAT ServiceInterceptor

glean/glass/test/Glean/Glass/Test/ErrorClassification.hs

@@ -0,0 +1,110 @@
+{-
+  Copyright (c) Meta Platforms, Inc. and affiliates.
+  All rights reserved.
+
+  This source code is licensed under the BSD-style license found in the
+  LICENSE file in the root directory of this source tree.
+-}
+
+{-# LANGUAGE OverloadedStrings #-}
+
+module Glean.Glass.Test.ErrorClassification (main) where
+
+import Control.Exception
+  ( AllocationLimitExceeded(..), SomeException, toException )
+import Data.ByteString (ByteString)
+import Data.Text (Text)
+import Test.HUnit (Test(..), assertEqual)
+
+import Glean.Init (withUnitTest)
+import TestRunner (testRunner)
+import Thrift.Channel (ChannelException(..))
+import Thrift.Protocol.ApplicationException.Types
+  ( ApplicationException(..), ApplicationExceptionType(..) )
+
+import Glean.Glass.ErrorClassification (srErrorName)
+import Glean.Glass.Types
+  ( GlassException(..), GlassExceptionReason(..), Revision(..) )
+
+main :: IO ()
+main = withUnitTest $ testRunner $ TestList unitTests
+
+unitTests :: [Test]
+unitTests =
+  [ TestLabel label $ TestCase $ assertEqual label expected (srErrorName e)
+  | (label, e, expected) <- classifications
+  ]
+
+-- | The downstream messages are shaped after real @error_reason_what@ values
+-- from the @service_router@ dataset for @service = 'glean.glass'@.
+classifications :: [(String, SomeException, Maybe ByteString)]
+classifications =
+  [ ( "per-client Glean throttle"
+    , downstream
+        "(THROTTLING_CLIENT_ID_REQUEST) Throttling request to tier \
+        \'glean.query.prod' with client_id \
+        \'glean.glass~useGleanGlassNavigationForDiffTool:phabricator' \
+        \because global request rate is higher than max configured 100"
+    , Just "GleanClientThrottled"
+    )
+  , ( "Glean error-rate throttle"
+    , downstream
+        "(THROTTLING_ERROR) This request to service 'glean.query.prod' was \
+        \throttled because the error rate exceeded the configured threshold"
+    , Just "GleanErrorRateShed"
+    )
+  , ( "Glean queue timeout, with the trailing help text"
+    , downstream
+        "(APP_QUEUE_TIMEOUT) Server Queue Timeout -- For help debugging \
+        \visit https://fburl.com/wiki/sr_app_queue_timeout"
+    , Just "GleanQueueTimeout"
+    )
+  , ( "Glean host overload"
+    , downstream
+        "(HOST_OVERLOAD) Host is overloaded -- For help debugging visit \
+        \https://fburl.com/wiki/sr_host_overload"
+    , Just "GleanOverload"
+    )
+  , ( "Glean app overload shares the overload name"
+    , downstream
+        "(APP_OVERLOAD) Application is overloaded -- For help debugging \
+        \visit https://fburl.com/wiki/sr_app_overload"
+    , Just "GleanOverload"
+    )
+  , ( "unclassified reason falls through to ServiceRouter's default"
+    , downstream
+        "(RECV_TIMEOUT) Receive timeout after 30000ms -- For help debugging \
+        \visit https://fburl.com/wiki/sr_recv_timeout"
+    , Nothing
+    )
+  , ( "channel failure that is not a ServiceRouter exception"
+    , toException (ChannelException "recvCob: connection closed")
+    , Nothing
+    )
+  , ( "GHC allocation limit"
+    , toException AllocationLimitExceeded
+    , Just "GlassAllocationLimit"
+    )
+  , ( "Glass's own request timeout"
+    , toException $ ApplicationException
+        "glass server timeout" ApplicationExceptionType_Timeout
+    , Just "GlassTimeout"
+    )
+  , ( "other ApplicationExceptions are not Glass timeouts"
+    , toException $ ApplicationException
+        "something else" ApplicationExceptionType_InternalError
+    , Nothing
+    )
+  , ( "declared exceptions keep the classification hsthrift gives them"
+    , toException $ GlassException
+        { glassException_reasons =
+            [GlassExceptionReason_exactRevisionNotAvailable "deadbeef"]
+        , glassException_revisions = [Revision "deadbeef"]
+        }
+    , Nothing
+    )
+  ]
+
+downstream :: Text -> SomeException
+downstream what = toException $ ChannelException $
+  "recvCob: facebook::servicerouter::TServiceRouterException: " <> what