Make SigLip implementation less weird

[zucchini-nlp]

What does this PR do?

Fixes #42759. WIP, I will fix more models that copied this pattern from siglip

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: siglip

[vasqu]

iirc @yonigozlan started a few refactors but that was before the weight converter. Might be smart to sync

Clip definitely has a similar issue 😢

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42841&sha=74e9f9

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yonigozlan]

Nice cleanup! As @vasqu mentioned, I started some refactors for vision models but paused them for now. I'll get back to them once the v5 refactoring is more stable.

[zucchini-nlp]

Oh I remember, in that case we can close it and wait for the big refactor imo. The bug mentioned isn't affecting many users, not many reactions under the issue