Add comprehensive architecture documentation

[shruthis4]

This PR adds detailed architecture documentation for the Spark Operator:

• High-level architecture overview with component diagram
• Detailed description of core components (CRDs, Controllers, Webhook)
• SparkApplication and ScheduledSparkApplication lifecycle workflows
• Project structure and development guidelines
• Configuration, deployment, and security considerations
• Performance and scalability insights

This documentation will help new contributors understand the project architecture and accelerate onboarding for developers and operators.

Purpose of this PR

Proposed changes:

• <Change 1>
• <Change 2>
• <Change 3>

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 35187286-b44a-4423-9d24-996b86e41b05

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}