To-Do List

[Costaki33]

1. isConnected()

2. Need to implement complete list of indexes

3. Create cursors on the fly, pass in the connection, create cursor, close cursor in the respective function

   • Play it safe

4. Injection

   • Perform all the correct conversion and input into the table
   • Dates to be datetimes (UTC), timelimit to be minutes (int -> look into regex), nnodes (int), jobid(str)
   • Consider what type for nodelist in database
     • Search for how to represent list in mysql

5. Injection Improvements

   • Create a last read table
     • Insert the last read in file name
     • First time on creation, insert a date that is earlier than all the dates in the sets
     • Commit after read in last file
   • Ignore any files before that specified date

6. On the command line

   • Pass in all parameters in the command line without prompting
     • Security for the user log in information
       • Password file with proper permissions

[Costaki33]

Notes:

• connect() is set up, establishes the connection, and returns it. Each function passes the connection around, creating cursors on the fly and closing them in their respective functions.

• Indexes have been created for each of the columns. JobID is now read in as a string. I decided to leave State as a varchar, as a simple search for the state definition (COMPLETED, FAILED, etc.) would be easiest. I could not index the Nodelist column, as it is a TEXT type. MySQL can only index the first few N values of TEXT. It can't guarantee its uniqueness because the size keeps changing. I decide to leave it, with no indexing -> can easily search for a specific node.

• Properly converted the datetime columns from their local time (Still unsure - waiting on Joe to reply) to UTC time. Also created if statement cases for different SLURM time defined cases for the Timelimit column (Working) and converted all time in time-limit column to minutes (max-minutes)

• Data all properly inserted into table and can query properly in MySQL

UPDATE:

• Ran into -> pytz.exceptions.AmbiguousTimeError: 2021-11-07 01:25:50 -> Currently

[Costaki33]

Add Debug statements to see how long the data is running

[Costaki33]

Point to a directory and load into a database all the data from the database all the files it hasn't run yet. Admins -> We move the new files to a new directory and read those in. No archive. Grab the new ones, read in, and delete.

Christian's idea:
List of all the names of the HPCS and directory path, SSH client into each hpc through a loop, grab file or scp via list based on position in comparison to the position of the last read in file

Initial load:

• SCP directorys, read in and delete post read in, save the file name of the last read in file & position (via list) to the readIn table

For each update, above

Contact Virginia