How can I write objects to a branch using the API?

[prat0088]

Hello,

I'm trying to figure out the best way to add 1,000s to 1,000s of 1-50MB objects via API. At the moment I can't figure out how to make anything work. The documentation seems to focus on lakectl. I read through the swagger doc and tried posting an object. The command succeeds but I can't see the result in the UI. I suspect I might want to write to lakefs using the s3 api to write the object, but then how would I stage that object for a commit? I think I'm probably missing some link/stage step.

I don't need to import the data or make it accessible outside of lakeFS. I want to keep it simple and easy and let lakeFS manage my data.

Can anyone explain what endpoints in the API I need to use to write 1-50MB objects to a branch, commit, and merge?

Thanks.

[itaiad200]

Hey @prat0088 - Thank you for posting what seems to be our first Github Discussion ever!

There are 2 endpoints for uploading objects through lakeFS:

1. OpenAPI endpoint - documented in the swagger doc you referenced, under "POST repositories/{repository}/branches/{branch}/objects". It is also where you would find the endpoints for committing and merging.
2. lakeFS s3 gateway - we support the s3 api protocol, which you can understand more in our architecture page, and configure it with the required tool like aws cli, lakectl or any other tool you see fit.

lakeFS uncommitted data is always staged by default. We currently don't support unstaged/untracked objects. It means that every commit to a branch will always apply and commit all uncommitted objects in that branch (like git commit -a ...).

If you configured everything correctly, got a successful response and can't see the object in the UI, something isn't right. Please attach the logs from lakeFS (with verbosity set to TRACE) and logs from the tool you used from the client side.

[prat0088]

Yay first post!

Regarding the s3 gateway, I presume the order of operations would be:

1. Create a branch using the API
2. Write the data to the S3 gateway in the branch
3. Commit the data in the branch using the API

Is that right?

[itaiad200]

Yes, assuming you've created a repository first. A repo is created with the default branch (normally main), so you don't have to create a branch if you're ok with using that default branch.

[prat0088]

Ok, cool. I've got a repo.

The reason the request was failing was my POST wasn't multipart, but I'm still getting some errors:

response_body="{\"message\":\"http: no such file\"}\n" 

I probably need to drop back down to the shell (from C#'s HTTP client that I'm using) and craft a curl POST to debug this further and verify I'm even calling it correctly. I take it that path is just the filename, and can be "foo.txt" without a branch name or anything.

I wanted to try out the gateway, too, but I'm not sure which port it starts on. The docs mention it starts on a separate port, but not what the default port is. I couldn't find any reference in the config doc, the lakefs tool help text, or the logs.

Thanks for your help.

[itaiad200]

Default port for lakeFS openAPI and s3 gateway is 8000. Try the aws cli as I believe it is the quickest way to check gateway access.