initial nabat models/api script #122

BryonLewis · 2025-02-04T19:41:17Z

resolves #121

NOTE: Use the ./scripts/USGS/sampleURL.txt for testing, just inject your apiToken.

Creates new /nabat under /models for nabat specific models
- AcousticBatch -> Recording
- NaBatSpectrogram -> Spectrogram
- NaBatCompressedSpectrogram -> CompressedSpectrogram
Added a sample script to read from the NaBat GraphQL API
Under ./tasks/nabat/ added a tasks.py and a nabat_data_retrieval.py
- The nabat_data_retrieval has a main acoustic_batch_initialize that does:
  - Using the ApiToken and BatchId it requests some basic information about the NaBat file for the batchId
  - Using a ProjectId and the filename it requests a presigned S3 URL for downloading the file
  - Downloads the wav file
  - Creates an AcousticBatch using the metadata that we have
  - Using the downloaded audio file it generates a spectrogram
  - From the spectrogram it generates a compressed spectrogram
  - It also creates an annotation if one exists for the file and uses the proper species
  - Then it runs the prediction on the compressed spectrogram and logs the results
Under the /views there is an /nabat sub directory that contains the new endpoints for the nabat integration
- NABat access relies on having API key access to the file for displaying
- The endpoints are all public and don't require a django username but require that the NABat API Token has access to the file
- The above is why in a lot of cases there is a check for getting the presigned S3 URL for the file before viewing or editing annotations.
Client Side
- Added a new /nabat route that avoids the standard login because the API endpoints are all unpotected
- There is a default /nabat/{recordingId}/?surveyEventId={surveyEventId}&apiToken={apiToken}
  - This endpoint will initially look for an existing NABatRecording withthe recording Id.
  - If that is not found it will check for an in progress processing task
  - If both of the above are false it starts a task for downloading and processing the spectrogram. This also creates a processingTask that can be used to check if a version is in progress
  - Finally when complete it will redirect to the /nabat/spectrogram/{id} route to display the spectrogram
- The Spectrogram Annotation viewer has some minor changes
  - Users can only make a single annotation (this is an NABat limitation)
  - Only File Annotaitons are enabled currently
Removed references to flower and the container from the docker compose settings
Added a generateSpectrograms.py script to test the generation on a folder of files and log any errors that happen.
Added a configuration option for NABat that allows importing the species list from NABat.

TODO:

20240404 TODO:

Update to utilize new Query for presigned URL
Update the endpoint to be outside of Use Credentials/Logging in
Extract from API Token the username and email for associatiating a user with the system
Update the Front-End to respect the new UI updates

20250408 TODO:

Front-end Annotation UI Implementation
- Handle the difference between model-suggested annotations and user annotations
- Adding existing annotations on import (we only have the user email)
- Testing of creation/deletion/updating file annotations
Integration of the Prediction settings to turn on/off automatic annotation calculation
Restrict to a single Annotation per file per user
Pushing annotations to NABat
Update the Species List from NaBat (maybe a configuration endpoint to run?)

20250411 TODO:

CONVERT THE PUSHING OF SPECIES ANNOTATIONS TO mutation instead of query

…on to tasks, update NABat spectro generating process

…pressed

…ecordings

* renames acoustic batch to recording based on conversations * add migrations

Add additional settings to the admin page

BryonLewis · 2025-04-04T13:06:24Z

Updated queries:
add a basic query:
SpeciesId is from the species we utilize.
Takes the email from the JWT token for pushing value.
It has an assumed manual species classifier list.

SoftwareId is a Kitware SoftwareId.

Query VetQuery {
updateAcousticFileVet(
surveyEventId: 4481017
acousticFileId: 174988990
softwareId: 81
speciesId: 29
) {
acousticFileBatchId
}
}

query PresignedUrlQuery {
presignedUrlFromAcousticFile(acousticFileId: "174988990")
{
s3PresignedUrl
}
}

BryonLewis · 2025-04-07T17:45:43Z

Add in new query to get existing annotations and match them up with any thing we currently have:

query GetFileBatchesBySurveyEventIdAndFileId {

  surveyEventById(id: "4481017") {

    acousticBatchesBySurveyEventId(filter: {softwareId: {equalTo:81}}) {

      nodes {

        id

        acousticFileBatchesByBatchId(filter: {fileId: {equalTo: "174988990"}}) {

          nodes {

            autoId

            manualId

            vetter

            speciesByManualId {

              speciesCode

            }

          }

        }

      }

    }

  }

}

naglepuff · 2025-04-22T19:54:37Z

bats_ai/core/admin/__init__.py

+    'NABatRecordingAnnotation',
+    'NABatCompressedSpectrogram',
+    'NABatSpectrogram',
+    'NABatRecording',


No action needed, just observing that these don't follow the convention of other classes here of ending with ...Admin. Not sure if you want to keep that convention for these or move forward as-is.

You're correct, I'll update it

scripts/USGS/naBatSpecies.py

naglepuff · 2025-04-22T20:21:55Z

client/src/api/api.ts

+export interface NATBatFiles {
+    id: number,
+    recording_time: string;
+    recording_location: string | null;
+    file_name: string | null;
+    s3_verified: boolean | null;
+    length_ms: number | null;
+    size_bytes: number | null;
+    survey_event: null;


Is there an argument to be made for moving this into NABatApi.ts?

Probably good idea.

naglepuff · 2025-04-22T20:51:43Z

client/src/api/NABatApi.ts

+    getSpectrogram,
+    getSpectrogramCompressed,


Do you think these functions should include NABat in their names? Is there a reason they share names with functions in the non-NABat api.ts?

naglepuff · 2025-04-22T20:58:17Z

client/src/components/RecordingAnnotationDetails.vue

+    apiToken: {
+      type: String,
+      default: () => undefined,
+    },


For my own learning, why have the default be a function that returns undefined instead of just using the empty string as the default?

I should probably change it to String as PropType<string | undefined>;

naglepuff · 2025-04-22T21:03:55Z

client/src/components/RecordingAnnotationEditor.vue

@@ -85,7 +105,8 @@ export default defineComponent({
        confidence,
        comments,
        updateAnnotation,
-        deleteAnno
+        deleteAnno,


Would it be a big change/touch a ton of files if we changed this to deleteAnnotation?

This is getting removed/disabled right now. But I'll change the name currently.

naglepuff · 2025-04-22T21:16:08Z

bats_ai/core/views/nabat/nabat_configuration.py

+    if not request.user.is_authenticated or not request.user.is_superuser:
+        return JsonResponse({'error': 'Permission denied'}, status=403)
+    existing_task = ProcessingTask.objects.filter(
+        metadata__type='Updating Species',


Any value in using an enum for metadata.type at this point? Maybe at least for this one we can use a variable for the string 'Updating Species'

Implment this as an enum instead of.

naglepuff · 2025-04-22T21:17:46Z

bats_ai/core/views/processing_tasks.py

Can the commented out code here be removed?

naglepuff · 2025-04-22T21:22:42Z

bats_ai/tasks/nabat/nabat_update_species.py

+
+@app.task(bind=True)
+def update_nabat_species(self):
+    processing_task = ProcessingTask.objects.filter(celery_id=self.request.id)


Is this guaranteed to exist at this point or is that up to the code that starts this task?

Make sure you check for the existence of the task or in the calling request doing something different.

Co-authored-by: Michael Nagler <[email protected]>

BryonLewis · 2025-04-24T20:11:53Z

bats_ai/core/views/processing_tasks.py

+# @router.get("/filtered", response=List[ProcessingTaskSerializer])
+# def filtered_tasks(request, status: Optional[str] = None):
+#     if status and status not in ProcessingTask.Status.values:
+#         return {"error": f"Invalid status value. Allowed values are {ProcessingTask.Status.values}."}, 400
+
+#     tasks = ProcessingTask.objects.filter(status=status) if status else ProcessingTask.objects.all()
+#     return ProcessingTaskSerializer(tasks, many=True).data


Remove these lines.

BryonLewis · 2025-04-24T20:12:08Z

bats_ai/core/views/processing_tasks.py

+# class ProcessingTaskSerializer(serializers.ModelSerializer):
+#     class Meta:
+#         model = ProcessingTask
+#         fields = '__all__'


Remove these lines.

…_API_URL environment value

BryonLewis and others added 28 commits February 4, 2025 14:32

initial nabat models/api script

a54aef9

model updates and reset endpoints

d8440c2

outlining tasks

760acc9

nabat retrieval logic

da7d35c

NABat ingestion working

0574aa4

rename acoustic_batch_initialize

e83ef4c

remove flower, add spectrogram viewer script, move spectrogram creati…

91be416

…on to tasks, update NABat spectro generating process

nabat integration

ab9977d

deduplicating functions

45bb4b3

Merge branch 'main' into nabat-integration

e495458

add species manual and auto ids to nabat retrieval

8849abb

non-compressed X-axis highlight, zooming legend fix, 2.5x default com…

456cad6

…pressed

cursor time compressed fix, axis length fix, edit column reorder in R…

9d05588

…ecordings

recording annotation details

c31154b

removing some commented out emits

5491064

remove footer from details for inference annotations

f42c986

Ignore egg-info directories

230e8e3

Add additional configuration settings

024147d

Add new settings to admin page

488a78c

Use configuration to set compressed ref

96caa74

Move save button closer to settings

370ac19

Check config setting before running inference

31c089c

Use config setting for initial compressed scale

2329da7

reference the outputs.tags instead of outputs.results

b3ca903

renames acoustic batch to recording (#128)

e139f7a

* renames acoustic batch to recording based on conversations * add migrations

Merge branch 'main' into nabat-integration

f49e501

Use slider for stretch setting

d0bc693

Merge pull request #138 from Kitware/issue-134-configuration-settings

5d3f361

Add additional settings to the admin page

recording based nabat spectrograms

e18b046

BryonLewis added 3 commits April 7, 2025 12:56

Merge branch 'main' into ui-improvements

ced6425

Merge branch 'ui-improvements' into nabat-integration

aa07187

migrations modifications

de0adb1

BryonLewis added 10 commits April 7, 2025 15:00

public apis, adding apiToken access control

d2b0b10

recording annotation endpoints

b0a28db

nabat local annotaitons

1a2ee94

precition config, requesting annotations, updating species

9e78dcf

swapping to user_email from user_id

d8be1bb

pushing annotations back

df75702

auth cleanup

d801530

Merge branch 'main' into nabat-integration

71190d8

Merge branch 'main' into nabat-integration

6ee235d

convert to mutation

c0c0aca

BryonLewis requested a review from naglepuff April 17, 2025 17:39

naglepuff requested changes Apr 24, 2025

View reviewed changes

Update scripts/USGS/naBatSpecies.py

4d00bdc

Co-authored-by: Michael Nagler <[email protected]>

BryonLewis commented Apr 24, 2025

View reviewed changes

BryonLewis added 3 commits April 24, 2025 18:28

addressing some of the comments

a7d5e1b

enum for processing task types, processing task get_or_create

798fd4d

client_id option, base environment file, updating instructions, NABAT…

454725a

…_API_URL environment value

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial nabat models/api script #122

initial nabat models/api script #122

BryonLewis commented Feb 4, 2025 •

edited

Loading

BryonLewis commented Apr 4, 2025

BryonLewis commented Apr 7, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

naglepuff Apr 22, 2025

naglepuff Apr 22, 2025

BryonLewis Apr 24, 2025

BryonLewis Apr 24, 2025

BryonLewis Apr 24, 2025

initial nabat models/api script #122

Are you sure you want to change the base?

initial nabat models/api script #122

Conversation

BryonLewis commented Feb 4, 2025 • edited Loading

BryonLewis commented Apr 4, 2025

BryonLewis commented Apr 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BryonLewis commented Feb 4, 2025 •

edited

Loading