Skip to content
This repository was archived by the owner on Mar 22, 2021. It is now read-only.

Commit 2144081

Browse files
whowellwhowell
authored
Merge pull request #2 from UFResearchComputing/1.0
Version 1.0
2 parents 5367714 + 219d777 commit 2144081

27 files changed

+1799
-1510
lines changed

‎LICENSE

+4-671
Large diffs are not rendered by default.

‎README.md

+206-21
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,215 @@
1-
# Warning
2-
* This branch is a POC to allow first pass initial testing.
3-
* A formal first release will be developed in a 1.0 branch.
4-
* As such **NOTHING IN THIS BRANCH SHOULD BE USED FOR PERMANENT REFERENCE OR DEVELOPMENT**
5-
* v1.0 will coordinated against pyslurm api v2 or worked around v1 in order to reference objects and fields in the syntax common to scontrol, srun, and sbatch
1+
SAPI - SLURM REST API
2+
=====================
3+
sapi is a RESTful wrapper around the PySlurm interface utilizing Python Connexion. The goal of slurm-api is to provide a centralized interface for web applications, portals, and CLI utilities to interact with a site's scheduler instead of requiring each such instance to maintain the site's SLURM configuration files and correctly versioned packages.
64

7-
# Overview
8-
Provides a python flask RESTful API for SLURM
95

10-
## Notes
6+
sapi features
7+
-------------
8+
- Swagger UI for API
9+
- Centralized interface for web applications and CLI tools to interface with SLURM
10+
- Limits the spread and maintenance of SLURM configuration files, munge keys, and where root must be enabled to interact with the scheduler
11+
- Built-in authenticaion methods or delegation via bearer auth
1112

12-
* node running the API must have the slurm libraries installed and be able to resolve user names (for example an LDAP client)
1313

14-
* pyslurm of appropriate version compatibility must be build referencing slurm paths if they are non-standard
14+
current slurm features
15+
----------------------
16+
Features are implemented to the extent required by our projects. If certain slurm interactively would help your site or product, please open a feature request or upvote if it already exists.
1517

16-
* job submission requires the API be run as root to allow su to users for submission, or the API must run as user in slurm who is allowed to submit jobs for other users
18+
- partitions
19+
- (GET): utilizes the pyslurm/slurm source fields
20+
- reservations
21+
- (GET): utilizes the pyslurm/slurm source fields
22+
- nodes
23+
- (GET): utilizes the pyslurm/slurm source fields
24+
- jobs
25+
- note: job interaction is limited to the user name encoded in the jwt token used to access sapi
26+
- (GET): utilizes the pyslurm/slurm source fields
27+
- (POST): external call out to sbatch - minimal fields set to date based on sbatch options
28+
- (DELETE): cancel a scheduled or running job
1729

18-
* configuration options reside in a slurm-virt.ini config file. By default this is located at /etc/slurm-api.ini, but can be changed via the environment variable SLURM_API_CONFIG
1930

31+
Prerequisites
32+
=============
33+
Python 3.6+ (other versions may work but have not been tested)
2034

21-
## UFRC Installation
22-
* ** only valid in Library lifecycle environment **
23-
* dependencies have been built and published for python36-*
24-
* slurm-api is not RPM packaged yet due to waiting on v1.0
25-
- clone repo on target node
26-
- yum install the python36 dependencies
27-
- generate the sqlite auth database using referenced schema in conf/
28-
- setup /etc/slurm-api.ini based on the example configuration and referencing the sqlite database
29-
- run it from src/
35+
Example Development/Testing Environment:
36+
37+
```bash
38+
$ pip list
39+
Package Version
40+
---------------------- ---------
41+
asn1crypto 0.24.0
42+
certifi 2019.6.16
43+
cffi 1.12.3
44+
chardet 3.0.4
45+
Click 7.0
46+
clickclick 1.2.2
47+
connexion 2.3.0
48+
cryptography 2.7
49+
Cython 0.29.11
50+
Flask 1.1.0
51+
Flask-Cors 3.0.8
52+
idna 2.8
53+
inflection 0.3.1
54+
itsdangerous 1.1.0
55+
Jinja2 2.10.1
56+
jsonschema 2.6.0
57+
MarkupSafe 1.1.1
58+
openapi-spec-validator 0.2.7
59+
pip 10.0.1
60+
pycparser 2.19
61+
pyOpenSSL 19.0.0
62+
pyslurm 18.8.4.1
63+
PyYAML 5.1.1
64+
requests 2.22.0
65+
setuptools 39.0.1
66+
six 1.12.0
67+
swagger-ui-bundle 0.0.5
68+
urllib3 1.25.3
69+
Werkzeug 0.15.4
70+
```
71+
72+
Quick Start
73+
===========
74+
* Identify service node to deploy service on. This node must have the SLURM configuration files in place for the underlying libraries to properly interact with the scheduler
75+
76+
* Clone slurm-api
77+
78+
```bash
79+
$ cd /opt
80+
$ git clone [email protected]:UFResearchComputing/slurm-api.git sapi
81+
```
82+
83+
* Install dependencies (use a python3 virtual env or system python3)
84+
85+
```bash
86+
$ cd /opt/sapi
87+
$ pip install dependencies.txt
88+
```
89+
90+
Running:
91+
92+
```bash
93+
$ /opt/sapi/bin/sapi
94+
```
95+
96+
Some configuration may be required as documented below.
97+
98+
* The sapi will be accepting connections on the port defined in **SAPI_CONFIG** (see also configuration options)
99+
100+
* The swagger UI for the api should be available at http://example.com:9090/v1.0/ui/ (see also configuration options)
101+
102+
103+
Configuration
104+
=============
105+
The Swagger UI URL is read from the swagger.yaml definition currently. As such, set url correctly in
106+
107+
```bash
108+
$ grep url /etc/sapi/swagger.yaml
109+
- url: http://example.com:9090/v1.0
110+
```
111+
112+
slurm-api reads a INI configuration file. By default this is expected to be located at
113+
114+
```bash
115+
/etc/sapi/sapi.conf
116+
```
117+
118+
However, you may set **SAPI_CONFIG** to modify the path to the configuration file which can be helpful in running development versus production instances.
119+
120+
Example:
121+
--------
122+
123+
* To run a development instance on port 9090 over HTTP
124+
125+
```bash
126+
$ cat /opt/sapi/conf/sapi.conf.example
127+
[api]
128+
port=9090
129+
host=0.0.0.0
130+
debug=true
131+
132+
[ssl]
133+
enable=false
134+
cert=/etc/pki/tls/certs/example.com.crt
135+
key=/etc/pki/tls/private/example.com.key
136+
137+
[logging]
138+
path=/var/log/sapi.log
139+
140+
[auth]
141+
jwt_issuer=sapi
142+
jwt_secret=CHANGEME
143+
jwt_lifetime=3600
144+
jwt_algorithm=HS256
145+
146+
147+
$ export SAPI_CONFIG=/opt/sapi/conf/sapi.conf.example
148+
$ /opt/sapi/bin/sapi
149+
```
150+
151+
* To run directly over HTTPS, set ssl enable=true
152+
153+
* Sites running in production should place behind apache, nginx, or similar
154+
155+
156+
Token Management
157+
================
158+
A minimal administrative CLI interface is provided, sapadm, which can be used to generate tokens and inspect tokens generated. It utilizes the same SAPI_CONFIG as the API.
159+
160+
```bash
161+
[admin@example sapi]# /opt/sapi/bin/sapiadm
162+
usage: sapiadm [-h] (-u USER | -t TOKEN) {create,view}
163+
```
164+
165+
To generate a token:
166+
167+
```bash
168+
[admin@example sapi]# /opt/sapi/bin/sapiadm create -u exampleUser
169+
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzYXBpLnJjLnVmbC5lZHUiLCJpYXQiOjE1ODIxMTY0MzgsImV4cCI6MTYxMzY3MzM5MCwic3ViIjoiZXhhbXBsZVVzZXIifQ.arcSdErOsz5-b9tucH0VZKMYIWTZyuEHv0GmjDhVSN4
170+
```
171+
172+
To inspect a token which has been generated by this program with the same signing authority and secret:
173+
174+
```bash
175+
[root@sapi2 sapi]# /opt/sapi/bin/sapiadm view -t eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzYXBpLnJjLnVmbC5lZHUiLCJpYXQiOjE1ODIxMTY0MzgsImV4cCI6MTYxMzY3MzM5MCwic3ViIjoiZXhhbXBsZVVzZXIifQ.arcSdErOsz5-b9tucH0VZKMYIWTZyuEHv0GmjDhVSN4
176+
User: exampleUser
177+
Issuer: example.com
178+
Created: 2020-01-19 07:47:18
179+
Expires: 2021-01-18 13:36:30
180+
181+
```
182+
183+
184+
Packaging
185+
=========
186+
187+
188+
FIXME
189+
190+
191+
Documentation
192+
=============
193+
[GitHub Wiki](https://github.com/UFResearchComputing/slurm-api/wiki)
194+
195+
Changes
196+
=======
197+
[GitHub Releases](https://github.com/UFResearchComputing/slurm-api/releases)
198+
199+
200+
Features and Bug Reporting
201+
==========================
202+
Bugs and feature requests can be filed at
203+
204+
[GitHub Issues](https://github.com/UFResearchComputing/slurm-api/issues)
205+
206+
Licence
207+
=======
208+
Copyright 2020 UFIT Research Computing
209+
210+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
211+
212+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
213+
214+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
30215

‎bin/sapi

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/usr/bin/env python3
2+
3+
#/* ************************************************************************* */
4+
#/* Description: A Python REST API for slurm scheduling */
5+
#/* Dependencies: */
6+
#/* - pyslurm (https://github.com/PySlurm) */
7+
#/* Authors: */
8+
#/* - [email protected] */
9+
#/* ************************************************************************* */
10+
11+
#/* ************************************************************************* */
12+
#/* INCLUDE */
13+
#/* ************************************************************************* */
14+
from OpenSSL import SSL
15+
from connexion.resolver import RestyResolver
16+
from flask_cors import CORS
17+
from sapi.configure import configure
18+
from logging.config import dictConfig
19+
import connexion
20+
import logging
21+
import os
22+
23+
#/* ************************************************************************* */
24+
#/* MAIN */
25+
#/* ************************************************************************* */
26+
if __name__ == '__main__':
27+
app = connexion.FlaskApp(__name__)
28+
system_conf = '/etc/sapi/swagger.yaml'
29+
repo_conf = os.getcwd() + '/../conf/swagger.yaml'
30+
if os.path.isfile(system_conf) and os.access(system_conf,os.R_OK):
31+
swagger_config = system_conf
32+
elif os.path.isfile(repo_conf) and os.access(repo_conf,os.R_OK):
33+
swagger_config = repo_conf
34+
else:
35+
print("error: swagger.yaml does not exist or is not readable. PATHS=[" +
36+
system_conf + "," +
37+
repo_conf + "]")
38+
exit(1)
39+
app.add_api(swagger_config,
40+
arguments={'title': 'Slurm API'},
41+
resolver=RestyResolver('sapi')
42+
)
43+
#/* General Configuration */
44+
try:
45+
settings = configure()
46+
except Exception as e:
47+
print("Configuration Error: " + str(e))
48+
exit(1)
49+
50+
#/* Configure logging before app */
51+
dictConfig({
52+
'version': 1,
53+
'formatters': {
54+
'default': {
55+
'format': '[%(asctime)s] %(levelname)s: %(message)s',
56+
}
57+
},
58+
'handlers': {
59+
'file': {
60+
'class': 'logging.FileHandler',
61+
'level': settings.log_level,
62+
'formatter': 'default',
63+
'filename': settings.log_file,
64+
}
65+
},
66+
'root': {
67+
'level': 'DEBUG',
68+
'handlers': ['file'],
69+
}
70+
})
71+
logfile = logging.getLogger('file')
72+
settings.logger = logfile
73+
logfile.info("Logging to " + str(settings.log_file) + " with " + str(settings.log_level))
74+
logfile.info("Using configuration " + str(settings.config_path))
75+
if settings.debug:
76+
logfile.debug(str(settings))
77+
78+
#/* API Startup */
79+
if settings.use_ssl:
80+
app.run( host=settings.host,
81+
port=settings.port,
82+
debug=settings.debug,
83+
ssl_context=(settings.ssl_cert,settings.ssl_key)
84+
)
85+
else:
86+
app.run( host=settings.host,
87+
port=settings.port,
88+
debug=settings.debug,
89+
)

0 commit comments

Comments
 (0)