Data integrity bugs

[mafaldacf]

Hi,

It appears that TrainTicket may have potential issues that break data integrity and cause inconsistencies during execution. I am currently conducting research on data integrity in microservices, and as part of my work, I would like to share this information to understand if these types of issues are valid in the context of the application, and whether these bugs could be addressed in the future. Thank you in advance.

Absence of Cascading Deletes

The first type of issue concerns the occurrence of delete operations whose effects are not propagated to other microservices, leaving pending references in other entities that are persisted in different databases.

Fig. 1 illustrates one example in which deleting an order may leave pending references in assurance, delivery, consign, and food order records, breaking referential integrity. Records of all entities contain an OrderID field that references the corresponding OrderID of a potentially deleted order.

Below is a list of all 20 issues identified for the absence of cascading deletes, which can be used as steps to reproduct bugs. The method and operation names are chosed to closely follow the original microservice endpoints and database operations.

delete: AdminBasicInfoService.DeleteContact() … ContactsService.Delete()
missing cascade 1: database={order_db}, entity={order}, pending fields={ContactsDocumentNumber, ContactsName, DocumentType}

delete: AdminBasicInfoService.DeletePrice() … PriceService.DeletePriceConfig()
missing cascade 2: database={order_db}, entity={order}, pending fields={Price}

delete: AdminBasicInfoService.DeleteStation() … StationService.DeleteStation()
missing cascade 3: database={consign_db}, entity={consign}, pending fields={FromPlace, ToPlace}
missing cascade 4: database={order_db}, entity={order}, pending fields={FromStation, ToStation}
missing cascade 5: database={delivery_db}, entity={delivery}, pending fields={StationName}
missing cascade 6: database={food_db}, entity={foodOrder}, pending fields={StationName}
missing cascade 7: database={route_db}, entity={route}, pending fields={StartStation, EndStation, Stations}
missing cascade 8: database={travel_db}, entity={trip}, pending fields={StartStationName, TerminalStationName, StationsName}

delete: AdminBasicInfoService.DeleteTrain() … TrainService.DeleteTrain()
missing cascade 9: database={price_db}, entity={priceConfig}, pending fields={TrainType}
missing cascade 10: database={travel_db}, entity={trip}, pending fields={TrainTypeName}

delete: AdminOrderService.DeleteOrder() … OrderService.Delete()
missing cascade 11: database={assurance_db}, entity={assurance}, pending fields={OrderID}
missing cascade 12: database={consign_db}, entity={consignRecord}, pending fields={AccountID, OrderID, TargetDate}
missing cascade 13: database={delivery_db}, entity={delivery}, pending fields={OrderID}
missing cascade 14: database={food_db}, entity={foodOrder}, pending fields={OrderID}

delete: AdminRouteService.DeleteRoute() … RouteService.DeleteRoute()
missing cascade 15: database={price_db}, entity={priceConfig}, pending fields={RouteID}
missing cascade 16: database={travel_db}, entity={trip}, pending fields={RouteID}

delete: AdminTravelService.DeleteTravel() … TravelService.DeleteTrip()
missing cascade 17: database={order_db}, entity={order}, pending fields={TrainNumber}

delete: AdminUserService.DeleteUser() … UserService.DeleteUser()
missing cascade 18: database={consign_db}, entity={consignRecord}, pending fields={AccountID}
missing cascade 19: database={order_db}, entity={order}, pending fields={AccountID}
missing cascade 20: database={contact_db}, entity={contact}, pending fields={AccountID}

Concurrent Operations

The second type of issue concerns operations that can be issued concurrently during execution and leave invalid references, affecting the consistency of application data.

Fig. 2 illustrates one example where two pairs of two concurrent requests (1-2 and 1-3) and occur:

• request 1: deletion of a contact issued to the Admin Basic Info Service by an admin
• request 2: insertion of a new order that uses the former contact, issued to the Admin Order Service by an admin
• request 3: insertion of a new order that uses the former contact, issued to the Preserve Service by a client

In these pairs of requests, even if the deletion of the contact triggers cascading effects in the Order Service, if these effects take place prior to creating the new record, the write operation will still succeed and will remain with an invalid reference to a non-existent contact (e.g., ContactsName in the order record, which references the Name of a contact record), breaking referential integrity.

Below is a list of all 22 issues identified for concurrent operations, which can be used as steps to reproduct bugs. The method and operation names are chosed to closely follow the original microservice endpoints and database operations.

delete: AdminBasicInfoService.DeleteContact() ... ContactsService.Delete()
write 1: AdminOrderService.AddOrder() … OrderService.Create()
database={order_db}, entity={order}, written fields={ContactsDocumentNumber, ContactsName, DocumentType}
write 2: PreserveService.Preserve() … OrderService.Create()
database={order_db}, entity={order}, written fields={ContactsDocumentNumber, ContactsName, DocumentType}

delete: AdminBasicInfoService.DeletePrice() …PriceService.DeletePriceConfig()
write 3: AdminOrderService.AddOrder() … OrderService.Create()
database={order_db}, entity={order}, written fields={Price}
write 4: PreserveService.Preserve() … OrderService.Create()
database={order_db}, entity={order}, written fields={Price}

delete: AdminBasicInfoService.DeleteStation() … StationService.DeleteStation()
write 5: AdminOrderService.AddOrder() … OrderService.Create()
database={order_db}, entity={order}, written fields={FromStation, ToStation}
write 6: PreserveService.Preserve() … OrderService.Create()
database={order_db}, entity={order}, written fields={FromStation, ToStation}
write 7: PreserveService.Preserve() … ConsignService.InsertConsign()
database={consign_db}, entity={consignRecord}, written fields={FromPlace, ToPlace}
write 8: PreserveService.Preserve() … DeliveryService.ProcessQueue()
database={delivery_db}, entity={delivery}, written fields={StationName}
write 9: PreserveService.Preserve() … FoodService.CreateFoodOrder()
database={food_db}, entity={foodOrder}, written fields={StationName}
write 10: AdminRouteService.AddRoute() … RouteService.CreateAndModifyRoute()
database={route_db}, entity={route}, written fields={StartStation, EndStation, Stations}
write 11: AdminTravelService.AddTravel() … TravelService.CreateTrip()
database={travel_db}, entity={trip}, written fields={FromStation, ToStation}

delete: AdminBasicInfoService.DeleteTrain() … TrainService.DeleteTrain()
write 12: AdminBasicInfoService.AddPrice() … PriceService.CreateNewPriceConfig()
database={price_db}, entity={priceConfig}, written fields={RouteID}
write 13: AdminBasicInfoService.AddPrice() … PriceService.CreateNewPriceConfig() (NOTE: find and modify existing record)
database={price_db}, entity={priceConfig}, written fields={RouteID}
write 14: AdminTravelService.CreateTrip() … TravelService.AddTravel()
database={travel_db}, entity={trip}, written fields={TrainTypeName}

delete: AdminRouteService.DeleteRoute() … RouteService.DeleteRoute()
write 15: AdminBasicInfoService.AddPrice() … PriceService.CreateNewPriceConfig()
database={price_db}, entity={priceConfig}, written fields={RouteID}
write 16: AdminBasicInfoService.AddPrice() … PriceService.CreateNewPriceConfig() (NOTE: find and modify existing record)
database={price_db}, entity={priceConfig}, written fields={RouteID}
write 17: AdminTravelService.CreateTrip() … TravelService.AddTravel()
database={travel_db}, entity={trip}, written fields={RouteID}

delete: AdminTravelService.DeleteTravel() … TravelService.DeleteTrip()
write 18: AdminOrderService.AddOrder() … OrderService.Create()
database={order_db}, entity={order}, written fields={TrainNumber}
write 19: PreserveService.Preserve() … OrderService.Create()
database={order_db}, entity={order}, written fields={TrainNumber}

delete: AdminUserService.DeleteUser() … UserService.DeleteUser()
write 20: PreserveService.Preserve() … ConsignService.InsertConsign()
database={consign_db}, entity={consignRecord}, written fields={AccountID}
write 21: AdminOrderService.AddOrder() … OrderService.Create()
database={order_db}, entity={order}, written fields={AccountID}
write 22: PreserveService.Preserve() … OrderService.Create()
database={order_db}, entity={order}, written fields={AccountID}

Uncoordinated Replication

The third type of issue concerns the lack of coordination upon write and read operations when databases are deployed across regions with asynchronous replication.

Fig. 3 illustrates an example in which data integrity can be broken. For instance, when a client books a ticket, the external request to the Preserve Service triggers six write operations for the order, consign, assurance, food order, and delivery. Since the databases individually replicate their own writes, data integrity is not always ensured, potentially resulting in two possible scenarios of inconsistencies, illustrated in the figure.

1. Suppose a user in a secondary region uses the UI dashboard to list their orders. In that case, this triggers two read operations, one to read the order and another to read the corresponding consign using the OrderID. However, if the order is visible before the consign, the Consign Service won't be able to retrieve the expected record since no consign exists for that ID.
2. The second scenario is similar, but instead the client sends requests that include read operations for assurance, food order, and delivery using the OrderID retrieved during the previous read. However, if the order is visible before the assurance, food order, or delivery, the services won't be able to find the expected records, since there are no records for that ID.