[0781 PS] Reporting

[lisacapaccioli]

Starting with the date of the launch

Rollout of the new form
In order to support gradually releasing the new 0781 mental health form to a small percentage of users at first, then incrementally increasing the user base over time, we need to monitor performance and identify potential issues.

• Number of claims in progress with new 0781 mental health data/form
• Number of successful submissions with new 0781 mental health data/form
• Number of failed submissions with new 0781 mental health data/form

Primary vs. backup path
In order to identify if the new 0781 mental health is "performing" as expected through the API path.

• Number of submissions with new 0781 mental health data/form down the primary path
• Number of submissions with new 0781 mental health data/form down the secondary/backup path

Compare to old form
We'd like to compare how many claims and/or Veterans are using the old 0781/a online form vs. the new 0781 form. The reporting does not need to create a comparison, just capture the data so that a comparison can be done.
These:

• Number of successful submissions with old 0781/a PTSD data/form
• Number of claims in progress with old 0781/a PTSD data/form

Will be compared to these from above:

• Number of successful submissions with new 0781 mental health data/form
• Number of claims in progress with new 0781 mental health data/form

[freeheeling]

Number of claims in progress with new 0781 mental health data/form

• Is the intent to track each instance a user opts in or uploads a form? Can this be accomplished using Datadog?
• Or is this simply a query that is run at launch? If so, if rollout is incremental, when should this value be determined?

Number of successful submissions with new 0781 mental health data/form
Number of failed submissions with new 0781 mental health data/form

• What is being tracked here if we're also going to compare the "Number of submissions with new 0781 mental health data/form down the primary/backup path"? I'm confused by the "successful"/"failed" differentiator.

In order to identify if the new 0781 mental health form tends to have more submission fails, we want to compare the numbers that go down primary vs. backup path.

• The success rate of Form 526 "should not" be impacted by the presence of 0781 data; form 526 transmission is independent of any ancillary form data. Neither would we necessarily be able to determine that new 0781 data were responsible for 526 failing transmission. We could track and report on the incidence of complete 0781 transmission failure with regards to the form version. However, given that we are uploading a PDF, and not transmitting data, I would suspect a processing issue to happen before arriving at the upload stage.

We'd like to compare how many claims and/or Veterans are using the old 0781/a online form vs. the new 0781 form.

• I'm not clear on how this objective will be informative. Those users that have an IPF at the time the feature is enabled globally will always either already have initiated or be conditionally routed through the legacy 0781 flow.

[lisacapaccioli]

The reason for detecting old vs. new is for the next initiative which is to get users who have an old 0781 in progress to use the new 0781. To do that we need to be able to tell them apart and OCTO would like to know how many are impacted to support the work we have to do.

We also want to see if the new 0781 is performing as well or better than the old (hence the backup path). Although I admit, I don't think tracking the backup is a true case of that. Regardless, we have to track that the new is working in both the primary and backup path and it would be better if we can distinguish old vs. new because they will both be in production at the same time.

Like I mentioned in the refinement meeting, I do not know what DD can or can't do. I just know what we've been asked to track. If we need to break these out into different tickets (DD vs. reporting vs whatever) we can do that.

[lisacapaccioli]

@kylesoskin This needs solutioning so that we can deliver on this for reporting on the canary rollout.

[pammicmac]

• Can you help define “successful” and “failed”? The 0781 data does not affect whether a form goes down the backup path
• Not sure if primary vs backup path is necessary because going down the backup path would not be related to an 0781 issue
• Are there any validations to be aware of? Since the form 0781 hasn’t been added to vets-json-schema for validations, the frontend won’t get any information on validation issues with the new form data. The frontend would only get 422s if the submission doesn’t make it to vets-api.
• Rollout of the new form and Compare to old form are very similar, can you clarify the difference?
• Most of these will need to be tracked either with a query in Argo or we’d have to create a job reporting the numbers to DataDog. Which approach would you like?

[lisacapaccioli]

These are all good questions - hence why refinement is always helpful :-)

1. Can you help define “successful” and “failed”? - Success means the 0781 PDF made it to the eFolder.
2. 0781 data does not affect whether a form goes down the backup, is this necessary? - Correct, backup path is not related to a 0781 issue, but we would like to know if the new form (vs the old) goes down the backup path more often than the primary. If we don't think this is any indication that there might be a problem, then we could consider this was as nice to have vs. need to know.
3. Are there any validations to be aware of? Will have to get back to you on this one. The JSON schema ticket keeps getting pushed out, so until that is done we won't have anything.
4. Rollout of the new form and Compare to old form are very similar, can you clarify the difference? - Technically we just need to capture both and the comparison can happen manually. OCTO and VBA want to be able to determine how many claims are in progress with 0781 PTSD (old flow) since the time of the release to determine the number of claims that might be submitting with the old data. The reason why is the VA is doing a hard stop (currently planned for June) of allowing the old form. If there are a lot claims with 0781 in progress they might want to delay that date.
5. Which approach would you like? Datadog. (For this ticket we are specifically looking forward (from the time of release), so I suspect that Argo should not be necessary).

Generally it takes me a couple of times to refine a ticket because conversations are necessary. Feel free to refine it so that it makes sense to you. What is currently written in the ticket is just the "what" and not the "how". If it needs to be broken out into separate tickets that's fine too. This is only the first draft.

[lisacapaccioli]

If this is easier, here are OCTO BP Standard Form Metrics:

% of successful and failed form submissions

• Overall submission volume Count of unique submissions that contain sub-form flows (0781 and 4142) along with the primary submission (such as 526 or 0995)
• Submission error % - Number and % of submissions that are submitted on VA.gov (user receives a successful confirmation message) but never reach the next system

We can update the AC's with these if that helps. We still need to be able to determine this information from "new" vs. "existing/old" flow.