11# Error Handling and Error Backend
22
33The AppSignal Elixir integration provides multiple mechanisms for capturing and reporting errors.
4- This document explains the error handling architecture, the error backend, and why it 's disabled by default.
4+ This document explains the error handling architecture, the error backend, and why that 's disabled by default.
55
66## Error Reporting Overview
77
@@ -25,45 +25,12 @@ The error backend is a Logger backend that intercepts error reports from Erlang'
2525
2626### Architecture
2727
28- The error backend implements the ` :gen_event ` behavior (lib/appsignal/error/backend.ex:9):
29-
30- ``` elixir
31- @behaviour :gen_event
32- ```
33-
34- It attaches to Elixir's Logger as a backend:
35-
36- ``` elixir
37- def attach do
38- case Logger .add_backend (Appsignal .Error .Backend ) do
39- {:error , error} ->
40- Logger .warning (" Appsignal.Error.Backend not attached to Logger: #{ error } " )
41- :error
42-
43- _ ->
44- Appsignal .IntegrationLogger .debug (" Appsignal.Error.Backend attached to Logger" )
45- :ok
46- end
47- end
48- ```
28+ The error backend implements the ` :gen_event ` behavior.
29+ It attaches to Erlang's internal ` :error_logger ` as a backend.
4930
5031### What It Catches
5132
52- The error backend processes error events from the Logger (lib/appsignal/error/backend.ex:25-35):
53-
54- ``` elixir
55- def handle_event ({:error , gl, {_ , _ , _ , metadata}}, state) when node (gl) == node () do
56- metadata
57- |> Enum .into (%{})
58- |> handle_report ()
59-
60- {:ok , state}
61- end
62-
63- def handle_event (_event , state) do
64- {:ok , state}
65- end
66- ```
33+ The error backend processes error events from the Logger.
6734
6835** Event Structure:**
6936
7340
7441### Crash Reason Handling
7542
76- The backend looks for a ` crash_reason ` in the metadata (lib/appsignal/error/backend.ex:37-49):
77-
78- ``` elixir
79- defp handle_report (%{crash_reason: {reason, stacktrace}} = report) do
80- pid = report_pid (report)
81-
82- unless :cowboy in report_domains (report) do
83- pid
84- |> @tracer .lookup ()
85- |> do_handle_report (pid, reason, stacktrace)
86- end
87- end
88-
89- defp handle_report (_ ) do
90- :ok
91- end
92- ```
93-
43+ The backend looks for a ` crash_reason ` in the metadata.
9444The crash reason is a tuple: ` {reason, stacktrace} `
9545
9646### PID Extraction
9747
98- The PID is extracted from different report structures (lib/appsignal/error/backend.ex:51-56):
99-
100- ``` elixir
101- defp report_pid (%{conn: %{owner: pid}}), do: pid # Phoenix/Plug crash with conn
102- defp report_pid (%{pid: pid}), do: pid # Generic crash with PID
103- defp report_pid (_ ), do: nil # No PID available
104- ```
48+ The PID is extracted from different report structures.
10549
10650** Why Different Sources?**
10751
10852- Phoenix/Plug crashes include a ` conn ` struct with the request process's PID in ` conn.owner `
10953- Generic crashes include a ` pid ` field directly
11054- Some crashes don't include any process information
11155
112- ### Filtering Cowboy Errors (dcdac33e, 2023-06-12)
56+ ### Filtering Cowboy Errors
11357
114- Cowboy errors are explicitly filtered out:
115-
116- ``` elixir
117- unless :cowboy in report_domains (report) do
118- # ... process error
119- end
120- ```
58+ Cowboy errors are explicitly filtered out.
12159
12260** Why?**
12361
@@ -130,52 +68,7 @@ Cowboy errors were causing issues:
13068Before this fix, only cowboy errors without a ` conn ` were filtered.
13169The fix ensures ALL cowboy errors are ignored, even those with conns.
13270
133- ** See:** https://github.com/appsignal/appsignal-elixir/issues/850
134-
135- ### Span Lookup and Creation
136-
137- The backend looks up existing spans for the process (lib/appsignal/error/backend.ex:58-72):
138-
139- ``` elixir
140- defp do_handle_report ([{_pid , :ignore }], _ , _ , _ ) do
141- :ok
142- end
143-
144- defp do_handle_report ([], pid, reason, stacktrace) do
145- " background_job"
146- |> @tracer .create_span (nil , pid: pid)
147- |> set_error_data (reason, stacktrace)
148- end
149-
150- defp do_handle_report (spans, _ , reason, stacktrace) when is_list (spans) do
151- {_pid , span} = List .last (spans)
152-
153- set_error_data (span, reason, stacktrace)
154- end
155- ```
156-
157- ** Three Scenarios:**
158-
159- 1 . ** Process is ignored** - Do nothing
160- 2 . ** No existing spans** - Create new "background_job" span for the error
161- 3 . ** Existing spans** - Add error to the last (current) span
162-
163- ### Setting Error Data
164-
165- Error data is added to the span with a special tag (lib/appsignal/error/backend.ex:90-95):
166-
167- ``` elixir
168- defp set_error_data (span, reason, stacktrace) do
169- span
170- |> @span .add_error (:error , reason, stacktrace)
171- |> @span .set_sample_data (" tags" , %{" reported_by" => " error_backend" })
172- |> @tracer .close_span ()
173- end
174- ```
175-
176- The ` "reported_by" => "error_backend" ` tag indicates the error was caught by the backend, not through manual instrumentation.
177-
178- ## Why It's Disabled by Default (cb7c2e9d, 2024-06-04)
71+ ## Why It's Disabled by Default
17972
18073The error backend was changed to be ** disabled by default** in June 2024.
18174
@@ -199,155 +92,8 @@ This makes debugging difficult.
19992** 2. Duplicate Reporting**
20093
20194Modern framework integrations (Phoenix, Oban) already report errors properly with full context.
202- The error backend would catch the same errors again, leading to duplicates.
95+ Although we have error ignoring, sometimes the error backend would catch the same errors again, leading to duplicates.
20396
20497** 3. Confusing Results**
20598
20699Because the errors lack context, they appear in AppSignal without the information needed to understand or fix them.
207-
208- ### When to Enable It
209-
210- Enable the error backend if:
211- - You're using custom GenServers/processes without proper instrumentation
212- - You have background work that isn't covered by framework integrations
213- - You're missing errors that should be reported
214-
215- ``` elixir
216- # config/config.exs
217- config :appsignal , :config ,
218- enable_error_backend: true
219- ```
220-
221- ### Configuration
222-
223- Check if enabled (lib/appsignal/config.ex:188-193):
224-
225- ``` elixir
226- def error_backend_enabled? do
227- case Application .fetch_env (:appsignal , :config ) do
228- {:ok , value} -> !! Access .get (value, :enable_error_backend , false )
229- _ -> false
230- end
231- end
232- ```
233-
234- Attachment happens during application start if enabled (lib/appsignal.ex:28-30):
235-
236- ``` elixir
237- if Config .error_backend_enabled? () do
238- Appsignal .Error .Backend .attach ()
239- end
240- ```
241-
242- ## Integration with Framework Error Handlers
243-
244- The error backend ** does not replace** framework-specific error handling.
245- It's a fallback for errors that aren't caught by other means.
246-
247- ### Phoenix
248-
249- Phoenix's error view and exception tracking work independently of the error backend.
250- The Plug instrumentation catches errors during request processing.
251-
252- ### Oban
253-
254- Oban's exception handler reports errors with full job context.
255- The error backend would only catch errors outside of job execution.
256-
257- ### GenServers
258-
259- Custom GenServers that crash without supervision will be caught by the error backend (if enabled).
260-
261- ## Stacktrace Handling
262-
263- The stacktrace formatter handles edge cases (lib/appsignal/stacktrace.ex):
264-
265- ``` elixir
266- def format (stacktrace) when is_list (stacktrace) do
267- Enum .map (stacktrace, & format_stacktrace_entry/ 1 )
268- end
269-
270- def format (_stacktrace ) do
271- []
272- end
273- ```
274-
275- ** Why Guard for Lists?**
276-
277- In some situations (like cowboy errors), the stacktrace isn't actually a list.
278- It might be a tuple or other structure.
279-
280- Returning an empty list prevents the error handler from crashing.
281-
282- ## Process Isolation
283-
284- The error backend only processes errors from the current node:
285-
286- ``` elixir
287- def handle_event ({:error , gl, {_ , _ , _ , metadata}}, state) when node (gl) == node () do
288- # ... process error
289- end
290- ```
291-
292- ** Why Check the Group Leader's Node?**
293-
294- In distributed Erlang systems, error reports can propagate across nodes.
295- Without this check, the same error could be reported multiple times (once per node).
296-
297- The group leader check ensures each error is only processed on the node where it originated.
298-
299- ## Testing Without the Error Backend
300-
301- In tests, the error backend is typically not attached (unless specifically testing it).
302-
303- Tests use compile-time configuration to inject fake implementations:
304-
305- ``` elixir
306- # config/test.exs
307- config :appsignal , appsignal_tracer, Appsignal .Test .Tracer
308- config :appsignal , appsignal_span, Appsignal .Test .Span
309- ```
310-
311- The error backend respects these test implementations (lib/appsignal/error/backend.ex:6-7):
312-
313- ``` elixir
314- @tracer Application .compile_env (:appsignal , :appsignal_tracer , Appsignal .Tracer )
315- @span Application .compile_env (:appsignal , :appsignal_span , Appsignal .Span )
316- ```
317-
318- ## Key Design Decisions Summary
319-
320- | Decision | Rationale |
321- | ----------| -----------|
322- | Disabled by default | Modern integrations provide better context; reduces confusion |
323- | Filter cowboy errors | Phoenix handles these better; prevents duplicate/bad reports |
324- | Check group leader node | Prevent duplicate reports in distributed systems |
325- | Handle non-list stacktraces | Edge case compatibility; prevents backend crashes |
326- | Tag with "reported_by" | Distinguish backend-caught errors from manually reported ones |
327- | Create "background_job" spans | Provide namespace for errors without existing spans |
328- | Use last span in list | Attach error to current span, not parent spans |
329-
330- ## Migration Guide
331-
332- If you're upgrading and rely on the error backend:
333-
334- 1 . ** Audit your error reports** - Check what errors the backend is catching
335- 2 . ** Add explicit instrumentation** - Use ` Appsignal.send_error/2 ` for important processes
336- 3 . ** Enable if necessary** - Set ` enable_error_backend: true ` if you can't instrument everything
337-
338- If you're seeing errors you expect to be reported but aren't:
339-
340- 1 . ** Check framework integration** - Ensure Ecto/Phoenix/Oban integrations are enabled
341- 2 . ** Add manual reporting** - Use ` set_error ` or ` send_error ` in rescue blocks
342- 3 . ** Enable error backend temporarily** - See what it catches, then instrument those areas
343-
344- ## Future Direction
345-
346- The error backend is expected to be removed in a future major version.
347- The package is moving toward explicit instrumentation through:
348-
349- - Framework integrations (Phoenix, Oban, Ecto)
350- - Telemetry attachments
351- - Manual error reporting (` set_error ` , ` send_error ` )
352-
353- These provide better context and more control than the catch-all error backend approach.
0 commit comments