Below are oracle failure scenarios we saw during testing:
Scenario 1: skipped days
Description: Oracle fails to run on a specific day (e.g. due to not enough funds or API failures) and a full day is skipped
Impact: Because the MarketContractProxy.getExpiringMarketContract function uses total number of deployed contracts to find the expiring contract to settle, it will be settling contracts with the delay equal to the number of skipped days. E.g. if 2 days is skipped then every day it would settle contract that expired 2 days ago.
Operational Solution: Before restarting the oracle manually deploy contracts for skipped days using deployContract function. That will correct the length of the marketContracts array. If Oracle has already run, similar solution, although less desirable - deploy multiple contracts for today (number of additional contracts equals to number of skipped days).
Guardrails: Oracle address funds monitoring, Oracle error notifications, better/redundant API
Scenario 2: multiple runs during same day
Description: Oracle runs multiple times on the same day creating more contracts than necessary.
Impact: If nothing is done, it looks like this would cause the dailySettlement to try to settle the contract that's about to expire in the future. E.g. if number of incorrectly deployed contracts is 2 then Oracle will try to settle contract expiring 2 days from now. This will lead to no settlement being triggered on actually expiring contracts, which means no user will be able to settle their contracts.
Operational solution: To restore normal working of the protocol it should be paused for the number of days equivalent to the number of incorrectly deployed contracts. Alternatively, oracle script needs to be changed to do settlement and deployment in two separate call - the atomic dailySettlement function will not work correctly
Guardrails: Oracle script should be modified to gracefully exit if same-day contract has already been deployed
Below are oracle failure scenarios we saw during testing:
Scenario 1: skipped days
Description: Oracle fails to run on a specific day (e.g. due to not enough funds or API failures) and a full day is skipped
Impact: Because the
MarketContractProxy.getExpiringMarketContractfunction uses total number of deployed contracts to find the expiring contract to settle, it will be settling contracts with the delay equal to the number of skipped days. E.g. if 2 days is skipped then every day it would settle contract that expired 2 days ago.Operational Solution: Before restarting the oracle manually deploy contracts for skipped days using
deployContractfunction. That will correct the length of themarketContractsarray. If Oracle has already run, similar solution, although less desirable - deploy multiple contracts for today (number of additional contracts equals to number of skipped days).Guardrails: Oracle address funds monitoring, Oracle error notifications, better/redundant API
Scenario 2: multiple runs during same day
Description: Oracle runs multiple times on the same day creating more contracts than necessary.
Impact: If nothing is done, it looks like this would cause the
dailySettlementto try to settle the contract that's about to expire in the future. E.g. if number of incorrectly deployed contracts is 2 then Oracle will try to settle contract expiring 2 days from now. This will lead to no settlement being triggered on actually expiring contracts, which means no user will be able to settle their contracts.Operational solution: To restore normal working of the protocol it should be paused for the number of days equivalent to the number of incorrectly deployed contracts. Alternatively, oracle script needs to be changed to do settlement and deployment in two separate call - the atomic
dailySettlementfunction will not work correctlyGuardrails: Oracle script should be modified to gracefully exit if same-day contract has already been deployed