New file creation issue fixes#8
Conversation
Prevents loop from quitting when an error in file is found due to hour/minutes being unmatched.
Print statments removed from read_functions.py, which had been used for development
| # Code now attempts to find file with matching hour start time | ||
| try: | ||
| os.path.isfile(l1c_dir+string_date+l1c_filename) | ||
| #l1c_filename = rf.construct_l1c(l1c_dir,string_date,g,gac_dir) |
There was a problem hiding this comment.
This commented line should be removed as it is copied above.
|
|
||
| # If hour-matched start time is missed, it now finds the closest in time and replaces | ||
| # l1c_filename with the a new version which contains the time of the nearest file | ||
| except: |
There was a problem hiding this comment.
I don't like this change because if problem is that the file is on the next day, rather than the current day due to a timing issue, your code will now read the previous file (closest in time) which doesn't necessarily overlap. It should run in the sense that no overlap should be found but it is an unnecessary processing overhead. You could also perform a check for a maximum time delta on an accepted filename which would improve things.
| return hirs_dir,fiduceo_dir,gac_dir,l1c_dir | ||
|
|
||
| def closest(num,arr): | ||
| curr = arr[0] |
There was a problem hiding this comment.
Please add a docstring to describe what the function does, inputs and outputs.
| # l1c_filename with the a new version which contains the time of the nearest file | ||
| except: | ||
| print "Mismatched file found in l2p and l1c for: ",g | ||
| old_g_time = int(g[8:14]) |
There was a problem hiding this comment.
I would be inclined to move these ~five lines of code to a separate function for clarity.
2 participants
This pull fixes two issues with the file name timing of l1c and l2p files.
create_fiduceo path is tried before failing using the hour of the start time on the file. This prevents later files not being created in the day due to the job exiting early at a mis-timed file. It will instead simply be skipped.
For filenames with times that match the hour, but not minute or seconds, a new function determines the closest matching time file, which should be the same file but offset by a few seconds., and uses this filename instead. This does not currently ensure the files are matched within a time (new fix needed for that, coming soon) but prevents failure of the file and early exit of the job when filenames don't match exactly. Instances of mismatched files and complete mismatch failures are recorded in the log file.
Further work is needed on restricting the closest file by time, and to extend for scenarios where differences in time lead to different hours or even days.