feat: update retry mechanism

[greged93]

closes: #319

[greged93]

Returning a stream which captured the lifetime of the TX caused lifetime issues when trying to return the stream from the database call.

[codspeed-hq]

CodSpeed Performance Report

Merging #358 will improve performances by 34.88%

_{Comparing feat/update-retry (c6cbd77) with main (50b63b2)}

Summary

⚡ 1 improvement
✅ 1 untouched

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	pipeline_derive_in_file_blobs	972.4 ms	721 ms	+34.88%

[frisitano]

Looks great! Added some minor comments inline, primiarly about when we should use the raw methods and when to instantiate a transaction.

[frisitano]

lgtm

[greged93]

@frisitano should we merge this with the old benchmark and fix in another PR?

[frisitano]

@frisitano should we merge this with the old benchmark and fix in another PR?

Yes lets revent back to the old bencmarks, get CI green and merge. We can look at codespeed in another PR as you suggest.

[greged93]

@frisitano regression seems to have came from not having a get_n_block_data_hint. With the added method in 22e5d4c, regression is corrected.

[frisitano]

@frisitano regression seems to have came from not having a get_n_block_data_hint. With the added method in 22e5d4c, regression is corrected.

Can you elaborate on this please? In main we don't have a get_n_block_data_hint so why would this result in a regression?

Also looks like CI isn't passing

edit: Is this because in the derivation pipeline using the new api we would create a new transaction for each block data hint we fetched?

[frisitano]

Looks good. lgtm