V1.0.0/1 allocated job resource usage API

[JosephKang]

Organization Name: Advantech

Short summary about the issue/question: V1.0.0/1 Job resource usage API

Brief what process you are following:
In OpenPAI v1.0.0/1, the smallest resource(GPU/CPU/MEM) are defined in hivescheduler yaml and the resource allocated on demand might be the same or multiple of the defined value invisible to the API user.
Therefore, API user is unable to know the exact resource usage through API each time when the job is submitted since the allocated resource is probably not exact the same as the requested resource.
Please share the API to know the real resource allocation upon job submission.

OpenPAI Environment: OpenStack VM

• OpenPAI version: v1.0.0/1

[fanyangCS]

@abuccts, consider to add actual resource usage in terms of sku in a RESTful API

[JosephKang]

Is it possible to know the total resource and used resource for a given cluster through API?

[abuccts]

You can use GET /api/v2/virtual-clusters or GET /api/v2/virtual-clusters/{vc} to get used resources and total resources for all virtual clusters or a given vc.

The actually used or allocated resources are multiple of skuType defined in scheduler config, requested resources may not be the same as allocated resources.

[JosephKang]

Thanks @abuccts ,

Do we have API to get total resource of a Node within Cluster?

[abuccts]

You could use GET /api/v2/virtual-clusters?nodes or k8s node api.

[JosephKang]

The API seems work fine for GPU worker only for GPU resource.
,
May I know how to get CPU and MEM resource?

Moreover, the CPU worker don't have GPU resource so the API seems unable to work on CPU worker node.

Any suggestion?

[abuccts]

To get the real resources in the node, k8s node api is preferred, or you can use the proxy GET /api/v2/kubernetes/nodes in rest server.

For hived scheduler, it allocates sku actually, you could get the resources in each sku and sku number in each node in its config file, then calculated the total resources in each node. So that file you configured is the baseline actually. We haven't provided an api to get resources in each node like each vc yet.