Skip to content

Commit 9e16114

Browse files
authored
Change the Workflow for Lablink (#40)
* api for client service * add database class to allocator * add API request * Delete database.py * Revert "Delete database.py" This reverts commit dd2fa07. * fix import * not expose port 5432 * remove port 5432 * modify config for client * update allocator requirements * add omega config for allocator * debug closing database * debug non-overriding config file * update docs for client * update doc for allocator
1 parent 9b34c42 commit 9e16114

File tree

14 files changed

+394
-71
lines changed

14 files changed

+394
-71
lines changed

lablink-allocator/Dockerfile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,8 @@ COPY lablink-allocator/generate-init-sql.py /app
3838
COPY db_config.py /app
3939
RUN python generate-init-sql.py
4040

41-
# Expose ports for Flask and PostgreSQL
42-
EXPOSE 5000 5432
41+
# Expose ports for Flask
42+
EXPOSE 5000
4343

4444
# Copy and set permissions for startup script
4545
COPY lablink-allocator/start.sh /app/start.sh

lablink-allocator/README.md

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ Lablink has folowing componenets:
1313

1414

1515
- main.tf (in root directory) - For allocator-image EC2 server creation
16-
-- Creates a security group for EC2, exposing ports 5000(flask port), 5432(postgre port), 22(SSH port)
16+
-- Creates a security group for EC2, exposing ports 5000(flask port), 22(SSH port)
1717
-- Creates ec2 instance, with allocator test image
1818

1919
- generate-init-sql.py - This file contains python script to generate init.sql file, after importing config.
@@ -38,7 +38,17 @@ Lablink has folowing componenets:
3838
-- /admin - Renders admin page with two options, create instances or view instances
3939
-- /admin/instances - Renders view instances, containing a table of existing instances
4040
-- /launch - called from /create page. Using subprocess and terraform creates VM instances. Takes instance_count as input. this runs the 'terraform/main.tf' file.
41-
41+
-- /admin/set-aws-credentials - Sets the AWS credentials for terraform to use. It takes in the access key and secret key as input. It creates a file called 'aws_credentials' in the terraform directory, which is used by terraform to create instances.
42+
-- /admin/instances - Displays the existing vms in a table
43+
-- /admin/destroy - Destroys the existing instances. It runs the 'terraform destroy' command using subprocess. It also deletes the security group created by terraform.
44+
-- /vm_startup - This is a POST method that takes in hostname as input. It listens for the message of database changeto be received from the database.
45+
-> Configurations in the structured_config.py file:
46+
- `db.dbname`: The name of the database to connect to.
47+
- `db.user`: The username for the database connection.
48+
- `db.password`: The password for the database connection.
49+
- `db.host`: The host address of the database.
50+
- `db.port`: The port number for the database connection.
51+
- `db.table_name`: The name of the table to query for VM assignments.
4252
-> templates/index.html - Takes email and crd command as input and submits to /request_vm
4353

4454
-> templates/create-instances.html - Takes in no of instances count and submits to /launch
@@ -49,7 +59,7 @@ Lablink has folowing componenets:
4959
Current issues/workarounds:
5060
- terraform/main.tf - We have currently tested this locally, that is by giving the AWS credentials from access keys in AWS(in main.tf file).
5161
- When creating both allocator instance or VM instance, we can't modify the same instance the next time from terraform. We need to delete the EC2 instances first. Once they are sucessfully deleted, we need to delete the security group, associated with instances.
52-
- allocator EC2 creation - Once allocator EC2 is created, in order to allow client access postgre sql at 5432 port, Right now it is requiring us to manually restart the server every time. Tried multiple ways, but oculdn't debug and achieve it through code
62+
- allocator EC2 creation - Once allocator EC2 is created, in order to allow client access postgre sql, Right now it is requiring us to manually restart the server every time. Tried multiple ways, but oculdn't debug and achieve it through code
5363
Following are the steps to restart the postgre server:
5464
- ssh -i "sleap-lablink.pem" [email protected] (Replace with the EC2 public IP)
5565
- sudo docker ps
@@ -80,10 +90,10 @@ docker pull ghcr.io/talmolab/lablink-allocator-image:latest
8090
To run the LabLink Allocator service locally, use the following command:
8191

8292
```bash
83-
docker run -d -p 5000:5000 -p 5432:5432 ghcr.io/talmolab/lablink-allocator-image:latest
93+
docker run -d -p 5000:5000 ghcr.io/talmolab/lablink-allocator-image:latest
8494
```
8595

86-
This will expose the Flask application on port `5000` and the PostgreSQL database on port `5432`.
96+
This will expose the Flask application on port `5000`.
8797

8898
### Endpoints
8999
- **Home Page**: Accessible at `http://localhost:5000/`. Displays a form for submitting VM details.
@@ -98,15 +108,6 @@ curl -X POST http://localhost:5000/request_vm \
98108
-d "crd_command=example_command"
99109
```
100110

101-
## Connecting to the Database
102-
The PostgreSQL database is exposed on port `5432`. You can connect to it using any PostgreSQL client with the following credentials:
103-
104-
- **Host**: `localhost`
105-
- **Port**: `5432`
106-
- **Database**: `lablink_db`
107-
- **User**: `lablink`
108-
- **Password**: `lablink`
109-
110111
## Deployment with Terraform
111112
The LabLink Allocator can be deployed to AWS using the Terraform configuration provided in the `terraform` directory. Follow these steps:
112113

@@ -148,12 +149,12 @@ docker pull ghcr.io/talmolab/lablink-allocator-image:linux-amd64-test
148149
Then:
149150

150151
```bash
151-
docker run -d -p 5000:5000 -p 5432:5432 ghcr.io/talmolab/lablink-allocator-image:linux-amd64-test
152+
docker run -d -p 5000:5000 ghcr.io/talmolab/lablink-allocator-image:linux-amd64-test
152153
```
153154

154155
To build locally for testing, use the command:
155156

156157
```bash
157158
docker build --no-cache -t lablink-allocator -f .\lablink-allocator\Dockerfile .
158-
docker run -d -p 5000:5000 -p 5432:5432 --name lablink-allocator lablink-allocator
159+
docker run -d -p 5000:5000 --name lablink-allocator lablink-allocator
159160
```
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
db:
2+
dbname: "lablink_db"
3+
user: "lablink"
4+
password: "lablink"
5+
host: "localhost"
6+
port: 5432
7+
table_name: "vms"
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
"""This module defines the database configuration structure for the LabLink Allocator Service."""
2+
3+
from hydra.core.config_store import ConfigStore
4+
from dataclasses import dataclass, field
5+
6+
@dataclass
7+
class DatabaseConfig:
8+
dbname: str = field(default="lablink")
9+
user: str = field(default="lablink")
10+
password: str = field(default="lablink_password")
11+
host: str = field(default="localhost")
12+
port: int = field(default=5432)
13+
table_name: str = field(default="vm_table")
14+
15+
@dataclass
16+
class Config:
17+
db: DatabaseConfig = field(default_factory=DatabaseConfig)
18+
19+
cs = ConfigStore.instance()
20+
cs.store(name="config", node=Config)
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
import select
2+
import json
3+
import logging
4+
5+
# Set up logging
6+
logging.basicConfig(
7+
level=logging.DEBUG,
8+
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
9+
datefmt="%Y-%m-%d %H:%M:%S",
10+
)
11+
logger = logging.getLogger(__name__)
12+
logger.setLevel(logging.DEBUG)
13+
14+
try:
15+
import psycopg2
16+
except ImportError as e:
17+
logger.error(
18+
"psycopg2 is not installed in the development environment. "
19+
"Please install it using `pip install psycopg2`"
20+
)
21+
raise e
22+
23+
24+
class PostgresqlDatabase:
25+
"""Class to interact with a PostgreSQL database.
26+
This class provides methods to connect to the database, insert data,
27+
retrieve data, and listen for notifications.
28+
"""
29+
30+
def __init__(
31+
self,
32+
dbname: str,
33+
user: str,
34+
password: str,
35+
host: str,
36+
port: int,
37+
table_name: str,
38+
):
39+
"""Initialize the database connection.
40+
41+
Args:
42+
dbname (str): The name of the database.
43+
user (str): The username to connect to the database.
44+
password (str): The password for the user.
45+
host (str): The host where the database is located.
46+
port (int): The port number for the database connection.
47+
table_name (str): The name of the table to interact with.
48+
"""
49+
self.dbname = dbname
50+
self.user = user
51+
self.password = password
52+
self.host = host
53+
self.port = port
54+
self.table_name = table_name
55+
56+
# Connect to the PostgreSQL database
57+
self.conn = psycopg2.connect(
58+
dbname=dbname,
59+
user=user,
60+
password=password,
61+
host=host,
62+
port=port,
63+
)
64+
65+
# Set the isolation level to autocommit so that each SQL command is immediately executed
66+
self.conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
67+
self.cursor = self.conn.cursor()
68+
69+
def get_column_names(self, table_name=None):
70+
"""Get the column names of a table.
71+
72+
Args:
73+
table_name (str): The name of the table.
74+
75+
Returns:
76+
list: A list of column names.
77+
"""
78+
if table_name is None:
79+
table_name = self.table_name
80+
81+
# Query to get the column names from the information schema
82+
self.cursor.execute(
83+
f"SELECT column_name FROM information_schema.columns WHERE table_name = '{table_name}'"
84+
)
85+
return [row[0] for row in self.cursor.fetchall()]
86+
87+
def insert_vm(self, hostname):
88+
"""Insert a new row into the table.
89+
90+
Args:
91+
hostname (str): The hostname of the VM.
92+
"""
93+
column_names = self.get_column_names()
94+
95+
values = []
96+
97+
for col in column_names:
98+
# Find the column that corresponds to the hostname and set its value
99+
if col == "hostname":
100+
values.append(hostname)
101+
elif col == "inuse":
102+
values.append(False)
103+
else:
104+
values.append(None)
105+
106+
# Construct the SQL query
107+
columns = ", ".join(column_names)
108+
placeholders = ", ".join(["%s" for _ in column_names])
109+
110+
sql = f"INSERT INTO {self.table_name} ({columns}) VALUES ({placeholders});"
111+
self.cursor.execute(sql, values)
112+
self.conn.commit()
113+
logger.debug(f"Inserted data: {values}")
114+
115+
def listen_for_notifications(self, channel, target_hostname):
116+
"""Listen for notifications on a specific channel.
117+
118+
Args:
119+
channel (str): The name of the notification channel.
120+
target_hostname (str): The hostname of the VM to connect to.
121+
"""
122+
self.cursor.execute(f"LISTEN {channel};")
123+
logger.debug(f"Listening for notifications on '{channel}'...")
124+
125+
# Infinite loop to wait for notifications
126+
try:
127+
while True:
128+
# Wait for notifications
129+
if select.select([self.conn], [], [], 10) == ([], [], []):
130+
continue
131+
else:
132+
self.conn.poll() # Process any pending notifications
133+
while self.conn.notifies:
134+
notify = self.conn.notifies.pop(0)
135+
logger.debug(
136+
f"Received notification: {notify.payload} from channel {notify.channel}"
137+
)
138+
# Parse the JSON payload
139+
try:
140+
payload_data = json.loads(notify.payload)
141+
logger.debug(f"Payload data: {payload_data}")
142+
hostname = payload_data.get("HostName")
143+
pin = payload_data.get("Pin")
144+
command = payload_data.get("CrdCommand")
145+
146+
if hostname is None or pin is None or command is None:
147+
logger.error(
148+
"Invalid payload data. Missing required fields."
149+
)
150+
continue
151+
152+
# Check if the hostname matches the current hostname
153+
if hostname != target_hostname:
154+
logger.debug(
155+
f"Hostname '{hostname}' does not match the current hostname '{target_hostname}'."
156+
)
157+
continue
158+
159+
logger.debug(
160+
"Chrome Remote Desktop connected successfully. Exiting listener loop."
161+
)
162+
return {
163+
"status": "success",
164+
"pin": pin,
165+
"command": command,
166+
}
167+
168+
except json.JSONDecodeError as e:
169+
logger.error(f"Error decoding JSON payload: {e}")
170+
continue
171+
except Exception as e:
172+
logger.error(f"Error processing notification: {e}")
173+
continue
174+
except KeyboardInterrupt:
175+
logger.debug("Exiting...")
176+
177+
def get_crd_command(self, hostname):
178+
"""Get the command assigned to a VM.
179+
180+
Args:
181+
hostname (str): The hostname of the VM.
182+
183+
Returns:
184+
str: The command assigned to the VM.
185+
"""
186+
if not self.vm_exists(hostname):
187+
logger.error(f"VM with hostname '{hostname}' does not exist.")
188+
return None
189+
190+
query = f"SELECT crdcommand FROM {self.table_name} WHERE hostname = %s"
191+
self.cursor.execute(query, (hostname,))
192+
return self.cursor.fetchone()[0]
193+
194+
def get_unassigned_vms(self):
195+
"""Get the VMs that are not assigned to any command.
196+
197+
Returns:
198+
list: A list of VMs that are not assigned to any command.
199+
"""
200+
query = f"SELECT hostname FROM {self.table_name} WHERE crdcommand IS NULL"
201+
try:
202+
self.cursor.execute(query)
203+
return [row[0] for row in self.cursor.fetchall()]
204+
except Exception as e:
205+
logger.error(f"Error retrieving unassigned VMs: {e}")
206+
return []
207+
208+
def vm_exists(self, hostname):
209+
"""Check if a VM with the given hostname exists in the table.
210+
211+
Args:
212+
hostname (str): The hostname of the VM.
213+
214+
Returns:
215+
bool: True if the VM exists, False otherwise.
216+
"""
217+
query = f"SELECT EXISTS (SELECT 1 FROM {self.table_name} WHERE hostname = %s)"
218+
self.cursor.execute(query, (hostname,))
219+
return self.cursor.fetchone()[0]
220+
221+
def get_assigned_vms(self):
222+
"""Get the VMs that are assigned to a command.
223+
224+
Returns:
225+
list: A list of VMs that are assigned to a command.
226+
"""
227+
query = f"SELECT hostname FROM {self.table_name} WHERE crdcommand IS NOT NULL"
228+
try:
229+
self.cursor.execute(query)
230+
return [row[0] for row in self.cursor.fetchall()]
231+
except Exception as e:
232+
logger.error(f"Error retrieving assigned VMs: {e}")
233+
234+
@classmethod
235+
def load_database(cls, dbname, user, password, host, port, table_name):
236+
"""Loads an existing database from PostgreSQL.
237+
238+
Args:
239+
dbname (str): The name of the database.
240+
user (str): The username to connect to the database.
241+
password (str): The password for the user.
242+
host (str): The host where the database is located.
243+
port (int): The port number for the database connection.
244+
table_name (str): The name of the table to interact with.
245+
246+
Returns:
247+
PostgresqlDtabase: An instance of the PostgresqlDtabase class.
248+
"""
249+
return cls(dbname, user, password, host, port, table_name)
250+
251+
def __del__(self):
252+
"""Close the database connection when the object is deleted."""
253+
self.cursor.close()
254+
self.conn.close()
255+
logger.debug("Database connection closed.")
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
from hydra import initialize, compose
2+
from omegaconf import OmegaConf
3+
from conf.structured_config import Config
4+
5+
def get_config() -> Config:
6+
"""
7+
Load the configuration file using Hydra and return it as a dictionary.
8+
"""
9+
with initialize(config_path="conf"):
10+
cfg = compose(config_name="config")
11+
print(OmegaConf.to_yaml(cfg))
12+
return cfg

0 commit comments

Comments
 (0)