FAQs
Q: My project is using multiple different CD tools at the same time. How do you consolidate the execution data into one and compute the metrics?
A: To be answered soon.
Q: Why do you synchronize execution history to local? Will this become a security risk?
A: To be answered soon.
Q: Do you include failure deployments when calculating DF?
A: No. DF is supposed to reflect the frequency we ship value to the target environments. A failure deployment does not ship any change to the target environments hence should not be included.
Q: Do you include "empty builds" (i.e. the builds which include no code changes, like a manual build) when calculating DF?
A: Yes we do. A success execution of the pipeline/workflow usually represents a success deployment to the target environment, regardless of having code changes or not.
And, in case you're worried. No, empty builds has no positive/negative impact to the average lead time value at all.
Q: Is the MLT "deployment based" or "commit based"? Do you simply compute lead time for each deployment respectively?
A: Every deployment has different batch size, has different numbers of commits, sometimes 0 commits. Instead of calculating lead time for each deployment and get the mean value, we calculate the lead time for every commit which are included in the evaluation period. This give us a more accurate evaluation in terms of "how long it takes from code commit to code running in the environment".
Q: How do you compute "delayed deployments"? Does "delayed deployments" result in negative impact to the MLT value?
A: When computing MLT value, commits are selected base on the time range in which the deployment occurs instead of the build trigger.
i.e. When deployment occurs, compute its lead time. Otherwise no.
There's one special case though - imagine we have two deployments, where the deployment#1 happens before deployment #2. However if we trace back to their build time respective, It is possible though the time when build#1 occurs is actually after build#2.
Q: How do you classify or deem a failure from the CI? A build failure? A deployment failure? Or .... how do you know the actual deployed release failed to deliver value?
A: To be answered soon.
Q: Is "an aborted build" a failure build? Should it be included in CFR?
A: No. Since for each failure we would also want to calculate its TTR value. An aborted build doesn't necessarily result in a failure for which team wants to apply a fix.
Q: When calculating CFR, except for the failure execution of selected stage/workflow, should we also calculate the ones failed at preceding stages? For example, failed at build stage.
A: No.
Because it doesn't sound reasonable for counting an execution as failure when it did not even attempt to deploy to the target environment. On the other hand, technically it is also cumbersome to identify the order of different stages.
Q: For consecutive failures, do we measure the TTR time for all of them, or the first failure only?
A: For consecutive failures, we measure the TTR value for first failure only.
i.e. the time between first failure, and first fix.