Fetching the charge on each order screen is causing excessive `resource_missing (charge id)` errors

[Jinksi]

Describe the bug

On each order screen where WooPayments is the payment method, we fetch the charge to determine if the charge is disputed and render a disputed order notice.

This is contributing to excessive Invalid request error: resource_missing (charge id) errors occurring (see logstash), due to:

• the charge ID stored as order meta may be for a live transaction, and the site is in test mode (or vice versa).
• the charge ID stored as order meta may not exist anymore (I'm unsure if this is a possibility).

To Reproduce

1. Use a WooPayments account that can switch between live/test mode (e.g. onboard with real bank account details).
2. Enable test mode.
3. Create a disputed order in test mode using the card 4000000000000259.
4. View the order. See the dispute notice introduced in 6.2.
5. Switch to live mode.
6. View the order. See request to /charges responds 500 with message

   Error: No such charge: 'ch_abcde'; a similar object exists in test mode, but a live mode key was used to make this request.
   

Additional context

• Slack discussion Aug 26 2023: p1692999889340419-slack-CGGCLBN58
• Slack discussion Jul 24 2023: p1690190397004289-slack-CGGCLBN58
• The disputed order notice was introduced in WooPayments 6.2:
  Epic: WCPay Disputes – Highlight active dispute in Woo order details screen #6354

[zmaglica]

This issue impacts disputes, so assigning to team Helix (based on team responsibilities Pc2DNy-3z-p2) @haszari . Assigning as part of Gamma Triage process PcreKM-yM-p2.

Another reason why I moved it there is that I noticed that it is already in the backlog of the team Helix. @haszari please let me know if you think that this is a mistake and that this issue could be triaged and prioritized accordingly.

[Jinksi]

Moving this to the backlog since a potential solution to this (#7174) has been closed while we gather more information about the problem via the tracks event introduced in #7437.

[Jinksi]

Keep an eye on the solution used for #334 which may provide a method of resolving this, e.g. order meta.

[Jinksi]

We could use the _wcpay_mode order meta to ONLY fetch the charge (dispute) if the current WooPayments test/live mode equals the order meta test/live.

woocommerce-payments/includes/class-wc-payment-gateway-wcpay.php

Line 1210 in 3ff9210

$order->update_meta_data( '_wcpay_mode', WC_Payments::mode()->is_test() ? 'test' : 'prod' );

[haszari]

I'm thinking we can close this. While the code path for getting this data is not ideal, it's not causing problems. We can reopen or action this anytime we like, but it's unlikely to push ahead of other items in the backlog so let's close.

@Jinksi @souravdebnath1986 feel free to reopen (or start a new issue) if there's a specific merchant impact or operational pain point here that we should prioritise over other maintenance.

[Jinksi]

@shendy-a8c FYI since you raised this as an issue to discuss in our last team hang.