Fix integer overflow in RGB addframe byte-size calculation and add re…

[rootvector2]

…gression test

[rootvector2]

Hi @dloebl

I know in my previous PR I made wrong assumptions and did not include a proper test. This time I rechecked everything carefully.

I found an integer overflow in the RGB addframe size calculation. I reproduced it with large dimensions and added a minimal test file so you can verify the behavior directly.

I ran the full test suite and everything passes.

If anything still looks wrong please tell me and I will fix it properly.

[dloebl]

Hey @rootvector2 ,
please read the contributing guidelines before opening any new pull requests or issues.

Passing test suite(required): The tests are still failing for your changes. Please fix them.

AI tooling attribution: If any AI models or tooling were used to produce this PR, please disclose that per our contributing guidelines (CONTRIBUTING.md:, item 5).

[rootvector2]

@dloebl
test suite are paased

[rootvector2]

ci went green after the last push, all seven jobs pass including the 32-bit build where the overflow actually fires.

on the tooling question: i lean on an ai tool now and then for research and to sanity-check my reasoning, but the patch here is mine. the root cause is real, fmtChan * numPixel was computed in uint32_t, so a large rgba frame wraps past UINT32_MAX and undersizes the malloc, then the memcpy runs off the end. i promoted the multiply to size_t and added an overflow guard. found it reading the addframe path while checking the earlier alloc fix. happy to reshape the check if you'd prefer a different style.