Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2361 commits
Select commit Hold shift + click to select a range
9520bfe
chore: remove peek_next_commit on DeltaTable which has been deprecate…
rtyler May 17, 2025
43d5fd3
chore: refactor some symbols out of table/mod.rs into their own files
rtyler May 17, 2025
e2035bc
docs: add 1.0.0 migration guide
ion-elgreco May 17, 2025
0a2042c
refactor: more specific factory parameter names
roeap May 17, 2025
8e31fc9
feat: expose kernel Engine on LogStore
roeap May 18, 2025
d94bf5a
chore: pr feedback and test fixes
roeap May 18, 2025
a94b404
test: avoid circular dependency with core/test crates
roeap May 19, 2025
9aac7bb
refactor: use LogStore in Snapshot / LogSegment APIs
roeap May 20, 2025
5b8db7c
chore: build default tests with the crate in CI
rtyler May 21, 2025
17ac062
chore: enable the datafusion feature for integration tests which need it
rtyler May 21, 2025
d581c93
chore: annotate tests which require datafusion appropriately
rtyler May 21, 2025
05166aa
ci: add spellchecker to pr tests
roeap May 23, 2025
ff5ca34
chore: mark more tests which require datafusion
rtyler May 23, 2025
8b132d7
refactor: move from pyarrow to arro3
ion-elgreco May 24, 2025
c089e61
chore: pr feedback
ion-elgreco May 24, 2025
2b019d1
refactor: use root store in log processing
roeap May 24, 2025
c72f6d9
fix: use more accurate log path parsing
roeap May 24, 2025
e7235e4
chore: set correct markers
ion-elgreco May 25, 2025
fa89735
chore: update kernel
roeap May 24, 2025
34d6122
chore: update kernel
roeap May 24, 2025
d100f18
fix: remove problematic typos configuration and fix Spellcheck issues
fvaleye May 25, 2025
f58225c
feat: use kernel checkpoint writer
roeap May 24, 2025
4cd67e3
refactor: use kernel log segment for some log inspection
roeap May 25, 2025
b5a7419
chore: remove unused time_utils
roeap May 25, 2025
4a465c9
chore: more typos
roeap May 25, 2025
59e1eed
refactor: remove protocol error
roeap May 26, 2025
4f0fdb0
feat: add table description and name API for Python
fvaleye May 25, 2025
c97c468
feat: add validator crate and use to have update table metadata valid…
fvaleye May 26, 2025
b85468b
chore: remove unused stats parsed field
roeap May 27, 2025
8fe9a8e
fix: arro3 schema conversion logic
ion-elgreco May 28, 2025
4e743e6
chore: update migration docs
ion-elgreco May 28, 2025
0b2da7f
chore: improve wording
ion-elgreco May 28, 2025
1ce4c43
chore: update kernel to 0.11
roeap May 28, 2025
5ad4092
fix: set casting safe param to False
ion-elgreco May 28, 2025
dae9690
chore: add xfail to flaky test
ion-elgreco May 28, 2025
6ce9bbd
fix bullet list formatting
avriiil May 28, 2025
f7a3b5a
refactor!: get transaction versions for specific applications
roeap May 28, 2025
3cf5af4
test: improve storage config testing
roeap May 28, 2025
c7fd06e
chore: exclude Invariants from the default writer v2 feature set
rtyler May 28, 2025
cf46110
refactor!: remove and deprecate some python methods
roeap May 28, 2025
6da9ed5
fix: ensure projecting only columns that exist in new files afte sche…
alexwilcoxson-rel May 28, 2025
8213c53
docs: update link to df
rluvaton May 28, 2025
dc2b52f
chore: update runner
ion-elgreco May 29, 2025
db3b5df
ci: improve coverage collection
roeap May 29, 2025
bc3f69b
chore: prepare for the next python release
rtyler May 29, 2025
2864ed9
chore!: remove get_earliest_version
roeap May 29, 2025
9174966
refactor!: have DeltaTable::version return an Option
roeap May 29, 2025
f601b1f
Revert "chore: add test for handling fields with spaces in constraints"
ion-elgreco May 30, 2025
83c4339
Revert "fix: if field contains space in constraint expression, the ch…
ion-elgreco May 30, 2025
93506c4
fix: spaced columns parsing
ion-elgreco May 30, 2025
996ef09
chore: update tests
ion-elgreco May 30, 2025
653654b
chore: fmt
ion-elgreco May 30, 2025
69df3c4
fix: wrong schema set in table provider
ion-elgreco Jun 1, 2025
5f1c82b
chore: bump version
ion-elgreco Jun 1, 2025
b0ab08b
refactor: move LazyTableProvider into python crate
roeap May 30, 2025
31cf610
feat: add convenience extensions for kernel engine types
roeap Jun 1, 2025
a577d22
refactor: compute stats schema with kernel types
roeap Jun 3, 2025
637c360
chore: set java version to 21
ion-elgreco Jun 6, 2025
47fe31a
chore: disable ansi
ion-elgreco Jun 6, 2025
66a1323
refactor: remove unecessary uses of datafusion subcrates
alamb Jun 5, 2025
3225994
chore: pin aws crates
ion-elgreco Jun 14, 2025
715ec4b
fix: using arg state in execute
gtrawinski Jun 6, 2025
753bc95
chore: removed _ from state arg
gtrawinski Jun 9, 2025
ecda27f
fix: remove forced table update from python writer
ohanf Jun 3, 2025
cd20f3c
feat: make TableConfig accessible
ion-elgreco Jun 4, 2025
9503366
chore: update the minor version to reflect a behavior change
rtyler Jun 18, 2025
07ccac7
chore: update arrow/parquet to 55.2.0
alamb Jun 26, 2025
781c0d3
chore: clean up licenses in python project which are causing build is…
rtyler Jun 28, 2025
e32d534
fix: version binary search (exclude checkpoint.tmp)
aditanase Jun 20, 2025
f0ba904
fix: load_with_datetime max_version
aditanase Jun 17, 2025
b01d2a7
test: time travel for checkpoint_vacuumed tables
aditanase Jun 23, 2025
11020b3
chore: update to DataFusion 48.0.0
alamb Jun 5, 2025
ccb8479
chore: upgrade to delta_kernel 0.12.x
rtyler Jun 29, 2025
eee5fe8
docs: ensure create_checkpoint() is visible in the Python API docs
pythonspeed Jun 30, 2025
e5e9b48
fix: use new DeltaTableState for subsequent commits and return value …
jeromegn Jun 21, 2025
a783536
chore: remove redundant words in comment
shangchenglumetro Jul 4, 2025
df5e5cd
refactor: move schema code to kernel module
roeap Jul 5, 2025
f0225d7
feat: convert partition filters to kernel predicates
roeap Jul 6, 2025
72604f2
chore: latest clippy
roeap Jul 6, 2025
de4be58
chore: remove the deltalake-sql crate
rtyler Jul 8, 2025
7289e0e
write engineInfo
zachschuermann Jul 9, 2025
d2ee730
chore: bump patch versions for another relaese
rtyler Jul 10, 2025
bd2ce8e
Allow specifying specific versions to retain when performing a vacuum…
corwinjoy Jun 13, 2025
4f2ad01
chore: add additional test coverage for with_keep_versions
rtyler Jul 10, 2025
9cd41ff
chore: bump minor version for rust crate
rtyler Jul 11, 2025
5328bdd
feat!: use kernel Metadata action
roeap Jul 7, 2025
0e3f64f
feat!: use kernel Protocol action
roeap Jul 8, 2025
46435f1
chore: add comments docstring and deprecations
roeap Jul 9, 2025
3fe9a05
chore: upgrade to the latest kernel 0.13.0
rtyler Jul 12, 2025
d457264
chore: generate a more recentish updated changelog
rtyler Jul 12, 2025
2c834e0
Gate RetryConfig on 'cloud' feature
zeevm Jul 13, 2025
570f487
feat: utilize StorageConfig for the log store's parse_url_opts
rtyler Jul 14, 2025
5acea52
chore: Exclude typos in the changelog :laughing:
rtyler Jul 14, 2025
f9ca498
feat: remove redundant Handle from parse_url_opts
rtyler Jul 14, 2025
b357427
Creating new DeltaTable uses from_valid_url instead of from_url
smeyerre May 31, 2025
2fed5d2
Add test for attemping to create DeltaTable with invalid path
smeyerre Jun 1, 2025
1598970
fix: fix typo to fix CI typo check
alamb Jul 18, 2025
ef06364
fix: scan time was always 0 for merge metrics
alexwilcoxson-rel Jun 25, 2025
cb3e731
chore: add a validation of scan_time_ms into assert_merge
rtyler Jul 15, 2025
54ea9a3
chore: clean up CODEOWNERS a bit more to make auto-review assignment …
rtyler Jul 15, 2025
238d518
chore: expand visibility of new_metadata for transaction log applicat…
rtyler Jul 15, 2025
846242e
chore: re-export the kernel engine for downstream crates to use
rtyler Jul 15, 2025
9d3fd3d
fix: switch the url schemes for Azure integration tests
rtyler Jul 23, 2025
a68aae1
fix: Allow non-string primitive filter types when converting to pyarr…
smeyerre Jul 22, 2025
8f48ec8
Make match_partitions and new_metadata public
zeevm Jul 20, 2025
e7aaeaa
add #[deprecated] to new_metadata and match_partitions
zeevm Jul 20, 2025
7c1f448
chore: itnroduce a regression case for invalid checkpoint reads
rtyler Jul 23, 2025
494d287
fix: ensure a consistent RecordBatch schema is used during checkpoint…
rtyler Jul 23, 2025
36ce68e
docs: fix daft links which were broken by Daft, oops
rtyler Jul 24, 2025
f6a440e
chore: bump Python for a fix release
rtyler Jul 24, 2025
7be5db9
fix: allow writing to DeltaTable objects across Python threads
rtyler Jul 24, 2025
6740f8a
fix: ensure openssl-sys doesn't creep into the dependency via the ker…
rtyler Jul 24, 2025
154c497
chore: Update to DataFusion 49.0.0
alamb Jul 18, 2025
9f52358
chore: Update to Rust version 1.85
alamb Jul 21, 2025
daf1abd
chore: finish datafusion 49 upgrade
rtyler Jul 27, 2025
201d4c6
chore(deps): update rstest requirement from 0.25.0 to 0.26.1
dependabot[bot] Jul 28, 2025
485b607
fix: coerce polars.Array into a suitable Arrow list type
rtyler Jul 28, 2025
2ea5c65
fix: make the docs link checking more useful/less faily
rtyler Jul 28, 2025
8010987
fix: avoid parsing generationExpressions as JSON
rtyler Jul 28, 2025
5f5286a
feat: build musl wheels upon release
rtyler Jul 28, 2025
6bdbdf5
chore: bump version for release
rtyler Jul 28, 2025
b56b961
Add keep_versions parameter to vacuum command for python
corwinjoy Jul 29, 2025
989c2f8
chore: remove deprecated use of kernel's Table
rtyler Jul 29, 2025
e38b38d
chore: use pytest-xdist for speeding up python tests
rtyler Jul 29, 2025
ab1d255
fix: aws special paths encoding
roeap Aug 5, 2025
ff5ceee
fix: handle checking partition filters in array/list when converting …
smeyerre Aug 5, 2025
d65691d
Fix formatting
smeyerre Aug 5, 2025
799ff5a
ci: run integration tests against next branches
roeap Aug 5, 2025
3d6557e
feat: support converting parquet with non-microsecond timestamps to d…
smeyerre Aug 4, 2025
6637e12
refactor schema field conversion functions
smeyerre Aug 6, 2025
1715a3e
chore: fmt
ion-elgreco Aug 7, 2025
55ea157
fix: use RFC3896 percent encoding
ion-elgreco Aug 7, 2025
ce7bfb9
chore: bump polars version
ion-elgreco Aug 7, 2025
fe964d5
test: increase ASCII table range
ion-elgreco Aug 7, 2025
6eab0d2
chore: remove print
ion-elgreco Aug 7, 2025
36f9151
chore: fmt
ion-elgreco Aug 7, 2025
d3d1873
chore: cleanup
ion-elgreco Aug 7, 2025
1dad7b1
chore: set protocol url docs
ion-elgreco Aug 7, 2025
57030dd
chore: bump python
ion-elgreco Aug 8, 2025
a31f7ef
feat!: kernel log replay
roeap Jul 18, 2025
f516c67
test: fix python tests after kernel log replay
roeap Jul 18, 2025
b127532
fix: refresh log store before log scans (#3659)
roeap Aug 5, 2025
ed621b7
fix: align stats schema with delta-spark implementation (#3663)
roeap Aug 7, 2025
5e8f564
refactor!: deprecate protocol, metadata, and schema methods on DeltaT…
roeap Jul 19, 2025
777dde3
refactor!: deprecate get_files_iter on DeltaTable
roeap Jul 19, 2025
97ae0aa
refactor: remove redundant NoMetadataError
roeap Aug 2, 2025
6da978c
refactor!: use kernel TableProperties
roeap Aug 2, 2025
c8b0c3f
chore: introduce a regression test to demonstrate reading of v2 check…
rtyler Aug 9, 2025
c54ab1f
chore: update hdfs object store to 0.15 (#3681)
Kimahriman Aug 18, 2025
a50ce6f
fix(pandas): implement-automatic-conversion-for-pandas-null-types
fvaleye Aug 22, 2025
e5a3416
feat(pandas): use field access using arro3 on type conversion
fvaleye Aug 24, 2025
861e96b
fix(format): fix formatting in Python for conversion file
fvaleye Aug 25, 2025
5d330fa
chore: remove unused dependencies
rtyler Aug 25, 2025
b830ae2
fix: enabling correctly pulling partition values out of column mapped…
rtyler Aug 25, 2025
0496d34
refactor: simplify file view iterators
roeap Aug 9, 2025
f0f8e28
feat: use kernel file skipping for getting files by partition
roeap Aug 10, 2025
5f1481a
refactor: move file (action) apis async/stream
roeap Aug 10, 2025
0a9522e
refactor: update method invocations to prepare migrations
roeap Aug 10, 2025
ab24be5
refactor: use replace EagerSnapshot with LogDataHandler in conflict c…
roeap Aug 11, 2025
595e448
chore: remove unnecessary parse_scalar function
rtyler Aug 16, 2025
4d783b4
docs: add some RustDocs
roeap Aug 16, 2025
4df2fdf
test: add some test for logical file view
roeap Aug 16, 2025
a2d94d8
Update crates/core/src/kernel/snapshot/mod.rs
rtyler Aug 25, 2025
f42193f
fix: reintroduce the 100 commit checkpoint interval
rtyler Aug 25, 2025
c961ac3
chore: gcp requires the cloud feature on the core dependency
rtyler Aug 27, 2025
2e0c4c6
chore: update the changelog for the 0.28 crate release
rtyler Aug 27, 2025
e7a574b
chore(cargo): add cargo-machete to detect and remove unused dependenc…
fvaleye Aug 28, 2025
9b49c40
chore: update kernel to 0.15.1
roeap Aug 28, 2025
fe0af4a
feat(storage): expand user with tilde in local path
fvaleye Aug 29, 2025
5e9b6ef
chore: bump to a minor version for a small core release with the new …
rtyler Aug 29, 2025
57c6e53
chore: remove unused json module
roeap Aug 16, 2025
28ba993
feat: domain metadata read support
roeap Aug 17, 2025
6b53efc
chore!: remove deprecated methods
roeap Aug 28, 2025
039b00d
refactor: move table provider to dedicated mod (#3726)
roeap Sep 1, 2025
941bc0a
feat(url): use Url in Rust for accessing to DeltaTable, use only stri…
fvaleye Aug 25, 2025
609cfa3
chore: minor conflict fixes to green up the build again
rtyler Sep 1, 2025
8762e09
chore(ci): cache rust dependencies in the CI to avoid unnecessary dow…
fvaleye Sep 2, 2025
a9aa3da
refactor: avoid explicit mutex in MergeBarrier (#3734)
roeap Sep 3, 2025
99e677c
fix: re-export the DecimalType for consumers
rtyler Sep 6, 2025
ea8c435
fix schema
FrankPortman Sep 9, 2025
56fde11
test and fmt
FrankPortman Sep 9, 2025
df376e9
add merge test
FrankPortman Sep 9, 2025
5e0d18d
feat: add per column Parquet Encoding support for Delta Table column …
niltecedu Sep 11, 2025
388ae34
fix: better error handles in unity client (#3752)
hntd187 Sep 12, 2025
edac020
feat(datafusion): add insert_into operation to TableProvider and expo…
fvaleye Sep 15, 2025
9e6add9
fix: ensure that invalid URLs are bubbled up as errors when parsed
rtyler Sep 17, 2025
48b09f6
chore: convert the history() API to return an impl Iterator
rtyler Sep 16, 2025
c62a2ab
chore: bump all the versions for a breaking semantic version change
rtyler Sep 16, 2025
355669f
chore: update docs (#3761)
ion-elgreco Sep 18, 2025
b6419b6
feat: update to DataFusion 50, pyo3 24, pyo3-arrow 0.11 (#3749)
alamb Sep 19, 2025
cab33d1
feat: OptimizeBuilder accepts SessionConfig for finer-grained control…
rtyler Sep 16, 2025
274eb80
feat(unity-catalog): support credentials via storage options (#3769)
fvaleye Sep 20, 2025
72250fb
fix: check if eligible to read (#3771)
ion-elgreco Sep 20, 2025
004574d
chore: upgrade the aws dependencies in deltalake-aws
rtyler Sep 20, 2025
2278f66
chore: pin cargo-machete action to the sha right before a regression
rtyler Sep 22, 2025
a0cbb04
chore: fix some typos in comment
juejinyuxitu Sep 23, 2025
3bb222f
chore: reduce the number of warnings spewed during builds
rtyler Sep 20, 2025
72a8758
chore: remote unnecessary maplit dependency from core
rtyler Sep 20, 2025
9749b26
chore: make humantime optional unless it's actually used
rtyler Sep 20, 2025
32badc4
chore: upgrade to delta-kernel-rs 0.16
rtyler Sep 21, 2025
5aeac96
chore: remove no longer valid pytest
rtyler Sep 22, 2025
35a796f
Signed-off-by: ohadmata <[email protected]>
Sep 3, 2025
b656725
Signed-off-by: ohadamta <[email protected]>
Sep 7, 2025
c7dfe7b
Signed-off-by: ohadmata <[email protected]>
Sep 8, 2025
2d3dbf7
chore: add a regression test for the DeltaTable.count() method
rtyler Sep 23, 2025
28ed5c0
fix: somehow the right test value didn't make it into the pr
rtyler Sep 25, 2025
502bb01
fix: introduce a failing test for RecordBatchWriter interior schema m…
rtyler Sep 23, 2025
5132d48
fix: only mutate the internal schema of RecordBatchWriter during sche…
rtyler Sep 23, 2025
976c945
fix: use a safe checkpoint when cleaning up metadata (#3748)
corwinjoy Sep 30, 2025
f137e69
chore(deps): update sqlparser requirement from 0.56.0 to 0.59.0
dependabot[bot] Sep 29, 2025
6089231
chore(ci): add automatic cache cleanup for closed main branch PRs (#3…
fvaleye Oct 1, 2025
07f81b2
chore(deps): update foyer requirement from 0.17.2 to 0.20.0
dependabot[bot] Sep 29, 2025
5cd2299
refactor: use EagerSnapshot in datafusion module (#3796)
roeap Oct 1, 2025
0727e4f
feat: allow passing a `SessionState` into a `OptimizeBuilder` (#3802)
abhi-airspace-intelligence Oct 2, 2025
4a49a89
refactor: remove table_url from Snapshot (#3803)
roeap Oct 3, 2025
80f7358
fix: maintaining load config from state (#3805)
ion-elgreco Oct 3, 2025
d246f47
refactor: consolidate extension planners (#3804)
roeap Oct 3, 2025
ce8d514
ci: split out integration tests (#3806)
roeap Oct 5, 2025
431413c
feat: access tombstones via TombstoneView (#3809)
roeap Oct 5, 2025
0d2440a
refactor: use EagerSnapshot in vacuum operation #3784 (#3812)
roeap Oct 5, 2025
f94ff90
fix: track log files as vec of batches
roeap Oct 2, 2025
be94ec7
feat: process log repaly on stream
roeap Oct 4, 2025
ec542c0
fix: track single representation of file stats #3371
roeap Oct 4, 2025
9ea568b
refactor: better log data processing isolation
roeap Oct 4, 2025
e88772c
refactor: avoid downcasting to SessionState (#3813)
roeap Oct 5, 2025
d3e6f03
add deletion_vector_descriptor method
zeevm Aug 31, 2025
56a1162
chore: add some rustdoc for a new/temporary public method
rtyler Sep 6, 2025
5f27c6a
refactors: move find_files into dedicated mod (#3815)
roeap Oct 5, 2025
d682c65
chore: remove unreferenced file (#3819)
roeap Oct 6, 2025
966d3ba
feat: shim kernel Scan and ScanBuilder (#3818)
roeap Oct 6, 2025
3fd349a
chore(deps): update datatest-stable requirement from 0.2 to 0.3 (#3817)
dependabot[bot] Oct 6, 2025
fab0582
feat: expose arrow schema on snapshots (#3822)
roeap Oct 7, 2025
ee0ae37
fix: update deprecation versions to next release (#3828)
roeap Oct 8, 2025
6f5d8eb
chore: unify inconsistent `SessionState` in datafusion operations (#3…
abhi-airspace-intelligence Oct 9, 2025
8ad33f9
fix: Protect recently written uncommitted files in vacuum Full mode
Oct 10, 2025
6efd277
feat: enable ability to do writes through Unity Catalog (#3834)
hntd187 Oct 11, 2025
5d3a071
chore(performance): optimize JSON parsing in get_actions and snapshot…
fvaleye Oct 9, 2025
2927e18
refactor(bench): remove baseline and keep json_parsing benchmark
fvaleye Oct 11, 2025
153c714
perf(path): only clone string for the path (#3841)
fvaleye Oct 12, 2025
51dc1df
fix: Protect recently written uncommitted files in vacuum Full mode
Oct 10, 2025
f0e569b
changed into streams and made python use them
zen-zap Oct 10, 2025
4778197
made changes to the python bindings and modified tests
zen-zap Oct 11, 2025
26049b3
changed BoxStream to have 'static lifetime and changes applied to pyt…
zen-zap Oct 11, 2025
bf4d350
Merge branch 'main' into main
zen-zap Oct 12, 2025
26f38ef
feat(tracing): add tracing spans to all I/O sections (#3795)
fvaleye Oct 13, 2025
0bc3030
feat(bench): add new benchmarking script, harness, and profiling guid…
abhiaagarwal Oct 13, 2025
3d1964d
chore: bump version from 1.1.4 to 1.2.0 (#3842)
ion-elgreco Oct 13, 2025
05a46a9
feat: datafusiuon based kernel engine
roeap Oct 9, 2025
956fb00
Merge branch 'delta-io:main' into main
zen-zap Oct 15, 2025
a09edaa
made streams version specific
zen-zap Oct 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
11 changes: 11 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[target.x86_64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]

[target.aarch64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
15 changes: 15 additions & 0 deletions .commitlintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
module.exports = {
extends: ['@commitlint/config-conventional'],
// Workaround for https://github.com/dependabot/dependabot-core/issues/5923
ignores: [(message) => /^Bumps \[.+]\(.+\) from .+ to .+\.$/m.test(message)],
rules: {
'body-max-length': [0, 'always'],
'body-max-line-length': [0, 'always'],
'footer-max-length': [0, 'always'],
'footer-max-line-length': [0, 'always'],
'header-max-length': [0, 'always'],
'scope-max-length': [0, 'always'],
'subject-max-length': [0, 'always'],
'type-max-length': [0, 'always'],
},
}
14 changes: 14 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
crates/core @roeap @rtyler @hntd187
crates/aws @rtyler
crates/lakefs @ion-elgreco
crates/catalog-glue @rtyler
crates/catalog-unity @roeap @hntd187

delta-inspect/ @rtyler

proofs/ @houqp
python/ @ion-elgreco

tlaplus/ @houqp

.github/ @rtyler
27 changes: 27 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
name: Bug report
about: Problems and issues with delta-rs
title: ''
labels: 'bug'
---
# Environment

**Delta-rs version**:

**Binding**:

**Environment**:
- **Cloud provider**:
- **OS**:
- **Other**:

***
# Bug

**What happened**:

**What you expected to happen**:

**How to reproduce it**:

**More details**:
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: Ask a question, ideas or get support
url: https://delta-users.slack.com/ssb/redirect
about: Ask a question or request support for using delta-rs
12 changes: 12 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
name: Feature request
about: Suggest an idea for this project or its docs
title: ''
labels: 'enhancement'
---

# Description

**Use Case**

**Related Issue(s)**
36 changes: 36 additions & 0 deletions .github/actions/setup-env/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: "Setup Python and Rust Environment"
description: "Set up Python, virtual environment, and Rust toolchain"

inputs:
python-version:
description: "The Python version to set up"
required: false
default: "3.10"

rust-toolchain:
description: "The Rust toolchain to set up"
required: false
default: "stable"

runs:
using: "composite"

steps:
- name: Install uv
uses: astral-sh/setup-uv@v3

- name: Set up Python ${{ inputs.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ inputs.python-version }}

- name: Setup sccache
uses: mozilla-actions/[email protected]

- name: Setup Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: ${{ inputs.rust-toolchain }}
cache: true
# if we ever get rid of all the clippy warnings, we can remove this line
rustflags: ""
18 changes: 18 additions & 0 deletions .github/codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
coverage:
status:
project:
default:
# allow some leniency on the deviation of pull requests
threshold: 1
if_ci_failed: error
informational: false
patch:
default:
if_ci_failed: error
informational: false

ignore:
- "delta-inspect/"
- "proofs/"
- "**/*.toml"
- "crates/benchmarks/"
14 changes: 14 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Please see the documentation for all configuration options:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates

version: 2
updates:
- package-ecosystem: "cargo"
directory: "/"
schedule:
interval: "weekly"
ignore:
# arrow and datafusion are bumped manually
- dependency-name: "arrow*"
- dependency-name: "datafusion*"
- dependency-name: "parquet"
112 changes: 112 additions & 0 deletions .github/mockoon_data_files/unitycatalog_databricks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
{
"uuid": "0696d6d5-62b6-4a0f-8524-917b0c3848d4",
"lastMigration": 33,
"name": "Unity Catalog (Databricks) Test API",
"endpointPrefix": "",
"latency": 0,
"port": 8080,
"hostname": "",
"folders": [],
"routes": [
{
"uuid": "16d88cdc-abcf-4b0d-a4d6-18a3205a9a6c",
"type": "http",
"documentation": "Temporary Table Credentials API",
"method": "post",
"endpoint": "api/2.1/unity-catalog/temporary-table-credentials",
"responses": [
{
"uuid": "50bba5c1-1e2c-4d5b-a611-ab478320f2b0",
"body": "{\n \"aws_temp_credentials\": {\n \"access_key_id\": \"string\",\n \"secret_access_key\": \"string\",\n \"session_token\": \"string\",\n \"access_point\": \"string\"\n },\n \"azure_user_delegation_sas\": {\n \"sas_token\": \"string\"\n },\n \"r2_temp_credentials\": {\n \"access_key_id\": \"string\",\n \"secret_access_key\": \"string\",\n \"session_token\": \"string\"\n },\n \"expiration_time\": 0,\n \"url\": \"string\"\n}",
"latency": 0,
"statusCode": 200,
"label": "Default response",
"headers": [],
"bodyType": "INLINE",
"filePath": "",
"databucketID": "",
"sendFileAsBody": false,
"rules": [],
"rulesOperator": "OR",
"disableTemplating": false,
"fallbackTo404": false,
"default": true,
"crudKey": "id",
"callbacks": []
}
],
"responseMode": null,
"streamingMode": null,
"streamingInterval": 0
},
{
"uuid": "66f51ad5-2ba3-4cb0-b633-c362a9f75836",
"type": "http",
"documentation": "Get Table Details API",
"method": "get",
"endpoint": "api/2.1/unity-catalog/tables/unity.default.testtable",
"responses": [
{
"uuid": "da280b2c-07b6-45ee-8da7-92c50e4e540e",
"body": "{\n \"name\": \"string\",\n \"catalog_name\": \"string\",\n \"schema_name\": \"string\",\n \"table_type\": \"MANAGED\",\n \"data_source_format\": \"DELTA\",\n \"columns\": [\n {\n \"name\": \"string\",\n \"type_text\": \"string\",\n \"type_name\": \"BOOLEAN\",\n \"position\": 0,\n \"type_precision\": 0,\n \"type_scale\": 0,\n \"type_interval_type\": \"string\",\n \"type_json\": \"string\",\n \"comment\": \"string\",\n \"nullable\": true,\n \"partition_index\": 0,\n \"mask\": {\n \"function_name\": \"string\",\n \"using_column_names\": [\n \"string\"\n ]\n }\n }\n ],\n \"storage_location\": \"../crates/test/tests/data/delta-0.8.0-partitioned\",\n \"view_definition\": \"string\",\n \"view_dependencies\": {\n \"dependencies\": [\n {\n \"table\": {\n \"table_full_name\": \"string\"\n },\n \"function\": {\n \"function_full_name\": \"string\"\n }\n }\n ]\n },\n \"sql_path\": \"string\",\n \"owner\": \"string\",\n \"comment\": \"string\",\n \"properties\": {\n \"property1\": \"string\",\n \"property2\": \"string\"\n },\n \"storage_credential_name\": \"string\",\n \"table_constraints\": [\n {\n \"primary_key_constraint\": {\n \"name\": \"string\",\n \"child_columns\": [\n \"string\"\n ]\n },\n \"foreign_key_constraint\": {\n \"name\": \"string\",\n \"child_columns\": [\n \"string\"\n ],\n \"parent_table\": \"string\",\n \"parent_columns\": [\n \"string\"\n ]\n },\n \"named_table_constraint\": {\n \"name\": \"string\"\n }\n }\n ],\n \"row_filter\": {\n \"function_name\": \"string\",\n \"input_column_names\": [\n \"string\"\n ]\n },\n \"enable_predictive_optimization\": \"DISABLE\",\n \"metastore_id\": \"string\",\n \"full_name\": \"string\",\n \"data_access_configuration_id\": \"string\",\n \"created_at\": 0,\n \"created_by\": \"string\",\n \"updated_at\": 0,\n \"updated_by\": \"string\",\n \"deleted_at\": 0,\n \"table_id\": \"string\",\n \"delta_runtime_properties_kvpairs\": {\n \"delta_runtime_properties\": {\n \"property1\": \"string\",\n \"property2\": \"string\"\n }\n },\n \"effective_predictive_optimization_flag\": {\n \"value\": \"DISABLE\",\n \"inherited_from_type\": \"CATALOG\",\n \"inherited_from_name\": \"string\"\n },\n \"access_point\": \"string\",\n \"pipeline_id\": \"string\",\n \"browse_only\": true\n}",
"latency": 0,
"statusCode": 200,
"label": "Default response",
"headers": [],
"bodyType": "INLINE",
"filePath": "",
"databucketID": "",
"sendFileAsBody": false,
"rules": [],
"rulesOperator": "OR",
"disableTemplating": false,
"fallbackTo404": false,
"default": true,
"crudKey": "id",
"callbacks": []
}
],
"responseMode": null,
"streamingMode": null,
"streamingInterval": 0
}
],
"rootChildren": [
{
"type": "route",
"uuid": "16d88cdc-abcf-4b0d-a4d6-18a3205a9a6c"
},
{
"type": "route",
"uuid": "66f51ad5-2ba3-4cb0-b633-c362a9f75836"
}
],
"proxyMode": false,
"proxyHost": "",
"proxyRemovePrefix": false,
"tlsOptions": {
"enabled": false,
"type": "CERT",
"pfxPath": "",
"certPath": "",
"keyPath": "",
"caPath": "",
"passphrase": ""
},
"cors": true,
"headers": [],
"proxyReqHeaders": [
{
"key": "",
"value": ""
}
],
"proxyResHeaders": [
{
"key": "",
"value": ""
}
],
"data": [],
"callbacks": []
}
112 changes: 112 additions & 0 deletions .github/mockoon_data_files/unitycatalog_oss.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
{
"uuid": "0696d6d5-62b6-4a0f-8524-917b0c3848d4",
"lastMigration": 33,
"name": "Unity Catalog (OSS) Test API",
"endpointPrefix": "",
"latency": 0,
"port": 8080,
"hostname": "",
"folders": [],
"routes": [
{
"uuid": "16d88cdc-abcf-4b0d-a4d6-18a3205a9a6c",
"type": "http",
"documentation": "Temporary Table Credentials API",
"method": "post",
"endpoint": "api/2.1/unity-catalog/temporary-table-credentials",
"responses": [
{
"uuid": "50bba5c1-1e2c-4d5b-a611-ab478320f2b0",
"body": "{\n \"aws_temp_credentials\": {\n \"access_key_id\": \"string\",\n \"secret_access_key\": \"string\",\n \"session_token\": \"string\"\n },\n \"azure_user_delegation_sas\": {\n \"sas_token\": \"string\"\n },\n \"gcp_oauth_token\": {\n \"oauth_token\": \"string\"\n },\n \"expiration_time\": 0\n}",
"latency": 0,
"statusCode": 200,
"label": "Default response",
"headers": [],
"bodyType": "INLINE",
"filePath": "",
"databucketID": "",
"sendFileAsBody": false,
"rules": [],
"rulesOperator": "OR",
"disableTemplating": false,
"fallbackTo404": false,
"default": true,
"crudKey": "id",
"callbacks": []
}
],
"responseMode": null,
"streamingMode": null,
"streamingInterval": 0
},
{
"uuid": "66f51ad5-2ba3-4cb0-b633-c362a9f75836",
"type": "http",
"documentation": "Get Table Details API",
"method": "get",
"endpoint": "api/2.1/unity-catalog/tables/unity.default.testtable",
"responses": [
{
"uuid": "da280b2c-07b6-45ee-8da7-92c50e4e540e",
"body": "{\n \"name\": \"string\",\n \"catalog_name\": \"string\",\n \"schema_name\": \"string\",\n \"table_type\": \"MANAGED\",\n \"data_source_format\": \"DELTA\",\n \"columns\": [\n {\n \"name\": \"string\",\n \"type_text\": \"string\",\n \"type_json\": \"string\",\n \"type_name\": \"BOOLEAN\",\n \"type_precision\": 0,\n \"type_scale\": 0,\n \"type_interval_type\": \"string\",\n \"position\": 0,\n \"comment\": \"string\",\n \"nullable\": true,\n \"partition_index\": 0\n }\n ],\n \"storage_location\": \"string\",\n \"comment\": \"string\",\n \"properties\": {\n \"additionalProp1\": \"string\",\n \"additionalProp2\": \"string\",\n \"additionalProp3\": \"string\"\n },\n \"owner\": \"string\",\n \"created_at\": 0,\n \"created_by\": \"string\",\n \"updated_at\": 0,\n \"updated_by\": \"string\",\n \"table_id\": \"string\"\n}",
"latency": 0,
"statusCode": 200,
"label": "Default response",
"headers": [],
"bodyType": "INLINE",
"filePath": "",
"databucketID": "",
"sendFileAsBody": false,
"rules": [],
"rulesOperator": "OR",
"disableTemplating": false,
"fallbackTo404": false,
"default": true,
"crudKey": "id",
"callbacks": []
}
],
"responseMode": null,
"streamingMode": null,
"streamingInterval": 0
}
],
"rootChildren": [
{
"type": "route",
"uuid": "16d88cdc-abcf-4b0d-a4d6-18a3205a9a6c"
},
{
"type": "route",
"uuid": "66f51ad5-2ba3-4cb0-b633-c362a9f75836"
}
],
"proxyMode": false,
"proxyHost": "",
"proxyRemovePrefix": false,
"tlsOptions": {
"enabled": false,
"type": "CERT",
"pfxPath": "",
"certPath": "",
"keyPath": "",
"caPath": "",
"passphrase": ""
},
"cors": true,
"headers": [],
"proxyReqHeaders": [
{
"key": "",
"value": ""
}
],
"proxyResHeaders": [
{
"key": "",
"value": ""
}
],
"data": [],
"callbacks": []
}
Loading