Add pod-job model #43

Synicix · 2024-11-01T23:33:19Z

Added pod job, InputStoreMapping, Input, StorePointer, and OutputStorePointer.

Update issue #11 to explained design choices found in this pull request

Synicix · 2024-11-08T01:07:55Z

src/model.rs

+    pub hash: String,
+    /// Details about the pod from which the pod job was created from
+    pub pod: Pod,
+    input_volume_map: BTreeMap<PathBuf, PathBuf>,


File hash -> stream key. Bring this up for dicussion tomrrow.
Perhaps need to implement input / store first.

guzman-raphael

A summary from our discussion yesterday is that your implementation of the current design is great and accurate here, however, I have some doubts about our design.

Mainly, feel that:

Input model should be implemented first (or with this PR) so that PodJob can have a reference to it (like how PodJob refers to a Pod).
A stream's key doesn't seem to be referenced in PodJob. That way less things would break if there are updates to the associated Pod.

Oh, and please update your linked issues and the project board to reflect the current status of the issues you are working on.

Synicix · 2024-11-16T08:45:17Z

Summary of changes:

Add pod job model
Add save and load functions for store for the file store
Add compute_check_sum for file or folder to the store trait and implmented it in filestore with merkle_hash for efficient computation (it uses Blake3 which is apprently faster than sha256, but need to actually indepdent test it)
Add wipe functionailty for wiping entire store
Change filestore functions to fully support annotationless operations
Redesign filestore test to full test all model more parallelization

Synicix

Old review

Cargo.toml

src/error.rs

src/model.rs

tests/fixture/mod.rs

src/model.rs

Synicix · 2024-12-17T01:21:03Z

src/model.rs

+    pub fn get_store(&self) -> Result<impl FileStore> {
+        // Load the yaml into a Btreemap, pull out the class, then build the store
+
+        let storage_class_name = self.uri.split("::").collect::<Vec<&str>>()[0];


change it to // instead of ;:

Synicix

Notes from meeting

Synicix · 2024-12-17T01:49:58Z

src/store/mod.rs

+}
+
+/// Trait to be implemented by file stores
+pub trait FileStore: Sized {


Change name to data store

Synicix · 2024-12-17T01:50:20Z

src/store/mod.rs

 /// Standard behavior of any store backend supported.
-pub trait Store {
-    /// How a pod is stored.
+pub trait ModelStore: Sized {


Add filestore as a trait dependency

Synicix · 2024-12-17T01:52:23Z

tests/model.rs

+    Ok(())
+}
+
+#[test]


Add data store trait tests

Defer till orchestrator

Synicix · 2024-12-17T10:50:55Z

Changes made, ready for review @eywalker @guzman-raphael

…d order and update `hash` to be used in more contexts.

…user data in `{store_directory}/orcapod_data`, and update tests to reflect our use-case example from Stanford 11/2024 trip.

…, add pod deserializer based on default, add clone support for models to give greater flexibility in tests, add minimal pod job model, change output of `load_model` to allow caller to control how to set fields, allow new min ident, (for now) add `anyhow` as dev-dependency to get stack trace when debugging, fix bug in `list` call in `basic_test`, and add new `pod_job` tests.

…et backtrace to display by default in DevContainer/GHA, and defer default user data subdirectory creation to orchestrator (i.e. later).

…hecksums, enable `todo!` for some untested cases, add user data for tests that need it, and update tests.

…ve unneeded error handling, rename `find_annotation` -> `find_model_metadata` to account for `spec.yaml` search, simplify, and update tests.

Apply review suggestions (going to edit the tests)

Cargo.toml

src/lib.rs

src/util.rs

Synicix added 2 commits November 1, 2024 23:32

Reset git commit for pod-job

a09cb8b

Add docs to deal with clippy complaints

2673a90

Synicix requested review from eywalker and guzman-raphael and removed request for eywalker November 1, 2024 23:44

Synicix marked this pull request as ready for review November 1, 2024 23:44

Synicix commented Nov 8, 2024

View reviewed changes

guzman-raphael suggested changes Nov 8, 2024

View reviewed changes

guzman-raphael linked an issue Nov 8, 2024 that may be closed by this pull request

Add PodJob, Input, InputStoreMapping, and OutputStoreMapping Model #11

Closed

Synicix marked this pull request as draft November 14, 2024 08:13

Synicix added 7 commits November 14, 2024 08:14

Merge and added input for pod job

72c7391

Change input data to use store compute_checksum

73c61c0

Add pod job scaffolding

b2c987a

Fix a few bugs and added tests

60db1a3

Fix filepath bug

a47b2fb

remove unused code

0a50223

Remove random image generation to prevent hash from changing

beb7b81

Synicix marked this pull request as ready for review November 16, 2024 08:41

Synicix requested a review from guzman-raphael November 16, 2024 08:45

Synicix linked an issue Nov 16, 2024 that may be closed by this pull request

Add InputSource model #13

Closed

Synicix added 5 commits December 6, 2024 12:31

Save progress for now, still need to be cleaned up

4465013

Fix a few clippy problems, still need to add some test cases

47018de

Change design of StorePointer

82623e1

Added test for StorePointer

7f7dd55

Rename and added to_yaml for StorePointer

44884c9

Synicix commented Dec 12, 2024

View reviewed changes

Fix last few remianing change request

90078b5

Synicix commented Dec 17, 2024

View reviewed changes

Renaming

a031713

Synicix mentioned this pull request Dec 30, 2024

Store_map and some improvements #46

Merged

guzman-raphael and others added 12 commits January 15, 2025 01:18

Simplify to_yaml by using serde builtin features, update Pod fiel…

8046483

…d order and update `hash` to be used in more contexts.

Move model storage to {store_directory}/orcapod_model, add default …

ac0a4f8

…user data in `{store_directory}/orcapod_data`, and update tests to reflect our use-case example from Stanford 11/2024 trip.

Refactor error module to use thiserror, remove unneeded anyhow, s…

b40daa2

…et backtrace to display by default in DevContainer/GHA, and defer default user data subdirectory creation to orchestrator (i.e. later).

remove store pointer

2601c8a

Update pod job to use new loading key system for pod

678f724

Change to new seralizer with

e53ccb9

Add input/output config to PodJob, add blob interface for computing c…

326f3f4

…hecksums, enable `todo!` for some untested cases, add user data for tests that need it, and update tests.

Remove restriction on a model requiring at least one annotation, remo…

4fbccf0

…ve unneeded error handling, rename `find_annotation` -> `find_model_metadata` to account for `spec.yaml` search, simplify, and update tests.

Clean up.

b749f64

Merge branch 'pod_job' into Synicix/pod-job2

4b7ffad

Fix missing files.

c0fb2cf

This was referenced Jan 22, 2025

Apply review suggestions Synicix/orcapod#2

Merged

[NDT-561] Base orchestrator, PodRun, PodResult + LocalDockerOrchestrator implementation #44

Merged

Merge pull request #2 from guzman-raphael/Synicix/pod-job2

e9e3342

Apply review suggestions (going to edit the tests)

guzman-raphael approved these changes Jan 24, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

src/lib.rs Outdated Show resolved Hide resolved

src/util.rs Outdated Show resolved Hide resolved

src/util.rs Outdated Show resolved Hide resolved

guzman-raphael merged commit a44c844 into nauticalab:dev Jan 24, 2025
1 check passed

guzman-raphael linked an issue Jan 31, 2025 that may be closed by this pull request

Implement thiserror #40

Closed

Synicix deleted the pod-job2 branch March 17, 2025 23:09

+                  Ok(())
+              }
+              #[test]

Add pod-job model #43

Add pod-job model #43

Uh oh!

Conversation

Synicix commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guzman-raphael left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Synicix commented Nov 16, 2024

Uh oh!

Synicix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Synicix left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Synicix commented Dec 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Synicix commented Nov 1, 2024 •

edited

Loading

guzman-raphael left a comment •

edited

Loading