feat!: Unify X and layers

[flying-sheep]

• Closes Unification of X and layers #244, closes concat when some values of X are None #678
• Tests added
• Release note not necessary because:

TODO:

• decide on dtype arg: just remove? or ignore+warning? (unless we already do that, then remove)
• backed mode: replace with Dask? not yet
• concat when some values of X are None #678

[codecov]

Codecov Report

❌ Patch coverage is 94.85714% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.64%. Comparing base (3dd761b) to head (3d8f4bd).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/anndata/_core/aligned_mapping.py	96.70%	3 Missing ⚠️
src/anndata/_core/raw.py	25.00%	3 Missing ⚠️
src/anndata/_core/anndata.py	95.91%	2 Missing ⚠️
src/anndata/utils.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1707      +/-   ##
==========================================
- Coverage   87.50%   85.64%   -1.87%     
==========================================
  Files          49       49              
  Lines        7725     7766      +41     
==========================================
- Hits         6760     6651     -109     
- Misses        965     1115     +150     

Files with missing lines	Coverage Δ
src/anndata/_core/access.py	72.22% <100.00%> (ø)
src/anndata/_core/file_backing.py	87.30% <100.00%> (ø)
src/anndata/_core/merge.py	84.97% <100.00%> (-8.53%)	⬇️
src/anndata/_io/h5ad.py	93.17% <100.00%> (-0.40%)	⬇️
src/anndata/_io/specs/methods.py	91.03% <100.00%> (-0.30%)	⬇️
...anndata/experimental/multi_files/_anncollection.py	71.37% <100.00%> (-0.21%)	⬇️
src/anndata/tests/helpers.py	84.23% <100.00%> (-8.72%)	⬇️
src/anndata/typing.py	100.00% <ø> (ø)
src/anndata/utils.py	86.49% <83.33%> (-0.95%)	⬇️
src/anndata/_core/anndata.py	86.81% <95.91%> (+0.31%)	⬆️
... and 2 more

... and 4 files with indirect coverage changes

[ilan-gold]

So re: the backed question, I think the answer is not to get rid of it, but to clarify its behavior. To do that, I think we should merge #1927 first, which may include a breaking change. If we stop allowing overriding of on-disk stores (which currently relies on inheritance form scipy.sparse private methods), then this question becomes much more simplified.

The next step after #1927 would probably be to simplify it. From what I can tell, the reaosn we need to track if the file is open or not is because if writing. If everything were purely read-only, I think it would be less of an issue.

My vision there would be in "backed" mode (after the two above points are implemented), we would simply take the X attribute on-disk, and read it in using ad.io.sparse_dataset with mode="r" and call it a day. That's how layers and other elements handle sparse_dataset so X shouldn't be different

[ilan-gold]

My rough plan here:

1. chore: add anndata.settings.copy_on_write_X #2327
2. Turn off the above behavior by simply removing the setting on main (i.e., 0.13) - hopefully this doesn't make merges into 0.12.x impossible
3. feat: support array-api #2071 is in-flight and relevant to setting X so likely would want this to go in first
4. read_lazy kwarg for not using dask #2147 in parallel to 3. will force us to start thinking about managing open-or-closed hdf5 files outside of X which is what will be needed here for backed mode (and backed mode would also probably be superseded by read_lazy kwarg for not using dask #2147 but not until 0.14 at least)
5. Merge this PR, which is in a mostly complete state except for the above

[ilan-gold]

For concat (i.e., #678), the default inner will now have a warning when some X's are None while outer will start to properly do the outer join i.e., add a fill value.

Put that in the release note.