Miners return files but synapse drops them

[mccrindlebrian]

Sometimes, when the filesizes for simulations become large, the http protocol for the synapse doesn't seem to carry this information over the wire back to the vali. A result of this is that miners who actually reply get 0 reward. Miners seem to reply with a non-zero response time and 200-level status codes but with empty returns.

Event information: {energies': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'response_times': [0, 1.736046314239502, 1.1655731201171875, 0.36842966079711914, 0, 0, 0, 0, 0, 0], 'response_status_messages': ['Timedout after 45.0 seconds.', 'Success', 'Success', 'Success', 'Service at 162.244.82.108:58406/FoldingSynapse unavailable.', 'Timedout after 45.0 seconds.', 'Timedout after 45.0 seconds.', 'Service at 213.173.105.87:11964/FoldingSynapse unavailable.', 'Service at 80.124.71.190:10954/FoldingSynapse unavailable.', 'Service at 65.108.32.140:2501/FoldingSynapse unavailable.'], 'response_status_codes': ['408', '200', '200', '200', '503', '408', '408', '503', '503', '503'], 'response_returned_files': [[], [], [], [], [], [], [], [], [], []], 'response_returned_files_sizes': [[], [], [], [], [], [], [], [], [], []], 'md_inputs': ['em.gro', 'posre_DNA_chain_A.itp', 'posre_DNA_chain_B.itp', 'topol_DNA_chain_A.itp', 'topol.top', 'topol_DNA_chain_B.itp', 'nvt.mdp', 'npt.mdp', 'md.mdp', 'emin.mdp'], 'md_inputs_sizes': [332365, 3962, 4024, 53526, 1195, 54493, 1950, 2136, 2363, 1195], 'pdb': '181d', 'created_at': Timestamp('2024-06-05 19:21:28'), 'updated_at': Timestamp('2024-06-05 19:21:28'), 'active': True, 'best_loss': inf, 'best_loss_at': NaT, 'best_hotkey': 'nan', 'commit_hash': 'nan', 'gro_hash': 'nan', 'update_interval': Timedelta('0 days 00:10:00'), 'updated_count': 0, 'max_time_no_improvement': Timedelta('0 days 01:00:00'), 'min_updates': 10, 'rewards': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}

[mccrindlebrian]

Another reason for this actually is that miners are running too many jobs, so they will (by design) return an empty synapse. This will show up with the same effect as above. Potentially we need a signal for this to then remove these uids all together from the job