[Python][FlightRPC] How to Extend PyArrow Flight Server with Custom RPCs?

[amitbar101]

Describe the usage question you have. Please include as many useful details as possible.

I'm using a Python Flight server which comes with the built-in RPCs which defined at the Flight.proto file.

However, I want to add a new RPC similar to DoGet, but with a different return type.
For example:

rpc DoGet(Ticket) returns (FlightData) {}

Unlike DoGet, this RPC does not return a stream but a single FlightData message.

I found a C++ example that registers a gRPC service alongside a Flight service, allowing both to be accessed on the same port. However, my goal is to integrate a new RPC directly into the Flight server rather than serve two separate services at different threads.

• It is possible to modify or extend the FlightServerBase to support a new RPC while keeping it part of the Flight protocol?
• If not, is the only way to achieve this by running it separately, like in the C++ example? What are the consequences of implementing it this way (serving each server on different threads)?

I’d appreciate any guidance on the best approach for achieving this in Python. Are there any examples or best practices to follow?

Thanks!

Component(s)

Python, FlightRPC

[lidavidm]

This is a pain point that I have wanted to solve for a long time, but couldn't convince other contributors that it's worthwhile. Basically you can't do it. (You could use DoAction but that also returns a stream, and there's a proposal for adding data to GetFlightInfo that went nowhere.) Your only way is to just define your own gRPC service and not use Flight.

(Note that this won't work even if it were possible because FlightData encodes a single IPC message and you generally need two: one for the schema, one for the record batch - unless you transferred the schema (and any dictionary batches) separately)

[lidavidm]

You may want to watch #41840.