Robust tracing and monitoring for failures

[duynguyen]

Currently, the solution heavily relies on logs and activation result for tracing issues during ops.

Challenges:

• App Builder logs are not easy to retrieve unless customer provides BYO logging infra (Splunk, NR). This is not the case for majority of users so far
• Activation results are noisy as the poller is run frequently, making it hard to retrieve the problematic ones

Proposed solutions:

• Store vertical metrics (generate HTML failures, unable to preview / publish) in aio-lib-state following a key convention
• It will allow retrieving and browsing errors quickly
• Example: pdp-errors-html-<timestamp>
• Querying for errors can just be like stateLib.list({ match: 'pdp-errors-html-*' })