Branch for YMQ integration to Scaler

[gxuu]

READY FOR REVIEW.

The test cases when using tcp_zmq as SCALER_NETWORK_BACKEND is passing, after fixing a bug in OSS.

There are still two test cases that are not passed when SCALER_NETWORK_BACKEND is ymq. They likely are caused by the same underlying bug that I couldn't find.

There are still two test cases that are not passed when SCALER_NETWORK_BACKEND is ymq. They likely are caused by the same underlying bug that I couldn't find.

There is a bug in the worker side. The worker, when shutdown requested through signal handle, is sending a DisconnectRequest. It should wait for DisconnectResponse afterwards. This was never done. The old version of the code exits immediately after sending DisconnectRequest.

This results to hang in the Scheduler side. The Scheduler, upon receiving DisconnectRequest from downstream, tries to send DisconnectResponse back. The Scheduler equipped with ymq connectors will stuck on statement:

    async def on_disconnect(self, worker_id: WorkerID, request: DisconnectRequest):
        await self.__disconnect_worker(request.worker)
        await self._binder.send(worker_id, DisconnectResponse.new_msg(request.worker)) # <-- HERE

and no other recv/send operation can be completed due to the interface's implementation. Apparently, we don't have this problem when zmq connectors equipped. This is because their implementation contains internal polling in the interface side. We don't have it.

But the culprit is the worker not waiting on DisconnectResponse. Hence, this really is a bug in scaler itself, but was hidden by zmq connectors.