Skip to content

DataSet parition ID to URL user defined function #9527

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 33 commits into from
Closed

DataSet parition ID to URL user defined function #9527

wants to merge 33 commits into from

Conversation

timsaucer
Copy link
Contributor

@timsaucer timsaucer commented Apr 7, 2025 •
edited
Loading

Related

Demo 25-04

What

This adds a udf that can be used thusly:

import datafusion as dfn
url_from_id = dfn.udf(dataset.partition_url_udf,  pa.utf8(), pa.utf8(), 'stable')
df = df.with_column("partition_url", url_from_id(df.col("rerun_partition_id")))

Wumpf and others added 30 commits April 4, 2025 20:56
@Wumpf
don't warn for adding already added server
537d7f5
@jleibs
Merge schemas when querying across partitions
60d782e
@jleibs
lint
4d39f3e
@grtlr @jleibs
Initial gRPC proposal
dc4a253
@grtlr @jleibs
Feedback from @jprochazk
6defce8
@grtlr @jleibs
pixi run codegen-protos
cd19fa3
@grtlr @jleibs
savepoint
31127a9
@grtlr @jleibs
savepoint 2
8884a4a
@grtlr @jleibs
remaining grpc calls
87042bb
@grtlr @jleibs
actually use the receivers
98ebe1f
@grtlr @jleibs
basic client
29e27f4
@grtlr @jleibs
move TableId to common
49761f0
@grtlr @jleibs
implement basic Rust API
cae1884
@grtlr @jleibs
Improve API
7ac3ed6
@grtlr @jleibs
Provide Rust snippet
7ed14a6
@grtlr @jleibs
Implement basic Python SDK
82f3628
@jprochazk @jleibs
temp
71fcb06
@grtlr @jleibs
Create notebook example
c8444e3
@grtlr @jleibs
Try to get notebook to work
47993e8
@grtlr @jleibs
Fix Table selection
aff94dd
@grtlr @jleibs
Better logging output
0c943fa
@jleibs
Fix the schema
4a3d382
@Wumpf
move logic for automatically opening server to add_receiver
00f29db
@Wumpf
Add send_file utilities and expose them to the python notebook
cb3e6c6
@Wumpf
Add send_file utilities and expose them to the python notebook
759c230
@Wumpf
Merge branch 'andreas/send_file' into jleibs/demo_rebasing
c5912cf
@Wumpf
Merge branch 'andreas/improved-auto-server-load' into jleibs/demo_reb…
6256c2b 
…asing
@Wumpf
remove info log on table transfer
e00f863
@Wumpf
Don't show "no recordings loaded" when there's tables or a server con…
7947d3c 
…nected
@Wumpf
Don't force changing the display mode to local recordings
6e7da55
@timsaucer
Copy link
Contributor Author

Superseded by #9530

@timsaucer timsaucer closed this Apr 7, 2025
@timsaucer timsaucer deleted the tsaucer/dataset_url_udf branch April 7, 2025 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@timsaucer @Wumpf @jleibs @grtlr @jprochazk