feat: generate an Symlink Manifest for External Engines

[JustinRush80]

Description

Add the GENERATE command that create an symlink manifest.

I have done an very dumb solution for my day job but it definitely not scalable. So creating this draft to first understand the expected behavior of this command and then turn my initial solution to fit the code standard.

Related Issue(s)

• closes deltaTable.generate("symlink_format_manifest") #2375

Documentation

https://books.japila.pl/delta-lake-internals/post-commit-hooks/GenerateSymlinkManifest/

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[codecov]

Codecov Report

❌ Patch coverage is 73.97260% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.75%. Comparing base (b2ca3b3) to head (5269e97).

Files with missing lines	Patch %	Lines
crates/core/src/operations/generate.rs	84.37%	3 Missing and 7 partials ⚠️
python/src/lib.rs	0.00%	6 Missing ⚠️
crates/core/src/operations/mod.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3889      +/-   ##
==========================================
- Coverage   73.76%   73.75%   -0.02%     
==========================================
  Files         151      152       +1     
  Lines       39396    39447      +51     
  Branches    39396    39447      +51     
==========================================
+ Hits        29061    29094      +33     
- Misses       9023     9036      +13     
- Partials     1312     1317       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rtyler]

@JustinRush80 I have finished this functionality but would request that you test the generated manifest with an engine that requires it.

@roeap this is ready to merge once it's green in CI

[JustinRush80]

@JustinRush80 I have finished this functionality but would request that you test the generated manifest with an engine that requires it.

@roeap this is ready to merge once it's green in CI

Sure thing! Is this functionality able to incrementally update the manifest? I am still learning the code base but how does it handle partitions?

[rtyler]

@JustinRush80 I have no idea what partitions are supposed to look like, if there is a specification of what this manifest is supposed to look like, I'm happy to help make this implementation compliant