Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,9 @@ benchmarkingjob:
name: "benchmarkingjob"
# the url address of job workspace that will reserve the output of tests; string type;
workspace: "/ianvs/sam_bench/robot-workspace"
#workspace: "/home/hsj/ianvs/sam_bench/cloud-robot-workspace"

# the url address of test environment configuration file; string type;
# the file format supports yaml/yml;
testenv: "./examples/robot/lifelong_learning_bench/testenv/testenv-robot-small.yaml"
#testenv: "./examples/robot/lifelong_learning_bench/testenv/testenv-robot.yaml"

testenv: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testenv/testenv-robot-small.yaml"
# the configuration of test object
test_object:
# test type; string type;
Expand All @@ -18,19 +14,10 @@ benchmarkingjob:
# test algorithm configuration files; list type;
algorithms:
# algorithm name; string type;
#- name: "rfnet_lifelong_learning"
- name: "sam_rfnet_lifelong_learning"
#- name: "vit_lifelong_learning"
#- name: "sam_vit_lifelong_learning"
# the url address of test algorithm configuration file; string type;
# the file format supports yaml/yml
#url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/rfnet_algorithm.yaml"
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/sam_algorithm.yaml"
#url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/vit_algorithm.yaml"
#url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/sam_vit_algorithm.yaml"
# the url address of test algorithm configuration file; string type;
# the file format supports yaml/yml

url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/sam_algorithm.yaml"
# the configuration of ranking leaderboard
rank:
# rank leaderboard with metric of test case's evaluation and order ; list type;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ benchmarkingjob:

# the url address of test environment configuration file; string type;
# the file format supports yaml/yml;
testenv: "./examples/robot/lifelong_learning_bench/testenv/testenv-robot.yaml"
testenv: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testenv/testenv-robot.yaml"

# the configuration of test object
test_object:
Expand All @@ -19,7 +19,7 @@ benchmarkingjob:
- name: "rfnet_lifelong_learning"
# the url address of test algorithm configuration file; string type;
# the file format supports yaml/yml
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/rfnet_algorithm-simple.yaml"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/rfnet_algorithm-simple.yaml"

# the configuration of ranking leaderboard
rank:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,14 +162,14 @@ def confidence(self, input_output):
return sum_3

def sam_predict_ssa(self, image_name, pred):
with open('/home/hsj/ianvs/project/cache.pickle', 'rb') as file:
with open('/ianvs/project/cache.pickle', 'rb') as file:
cache = pickle.load(file)
Comment on lines +165 to 166
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The cache file path /ianvs/project/cache.pickle is hardcoded. This makes the code less flexible and harder to configure for different environments. It would be better to define this path as a constant or pass it as a parameter. Furthermore, the current implementation will raise a FileNotFoundError if the cache file doesn't exist (e.g., on a fresh run). It's better to handle this gracefully, for example with a try...except block:

cache_path = getattr(self.args, "cache_path", "/ianvs/project/cache.pickle")
try:
    with open(cache_path, 'rb') as file:
        cache = pickle.load(file)
except (FileNotFoundError, EOFError):
    cache = {}

img = mmcv.imread(image_name)
if image_name in cache.keys():
mask = cache[image_name]
print("load cache")
else:
sam = sam_model_registry["vit_h"](checkpoint="/home/hsj/ianvs/project/segment-anything/sam_vit_h_4b8939.pth").to('cuda:1')
sam = sam_model_registry["vit_h"](checkpoint="/ianvs/project/segment-anything/sam_vit_h_4b8939.pth").to('cuda:1')
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The SAM checkpoint path ("/ianvs/project/segment-anything/sam_vit_h_4b8939.pth") and the CUDA device ('cuda:1') are hardcoded. This reduces portability and makes configuration difficult. It's better to pass these as arguments, for instance via self.args. Using a hardcoded device ID is particularly brittle and assumes a specific multi-GPU setup. A better approach would be:

sam_checkpoint = getattr(self.args, "sam_checkpoint", "/ianvs/project/segment-anything/sam_vit_h_4b8939.pth")
device = f"cuda:{self.args.gpu_ids}" if self.args.cuda else "cpu"
sam = sam_model_registry["vit_h"](checkpoint=sam_checkpoint).to(device)

mask_branch_model = SamAutomaticMaskGenerator(
model=sam,
#points_per_side=64,
Expand All @@ -184,7 +184,7 @@ def sam_predict_ssa(self, image_name, pred):
print('[Model loaded] Mask branch (SAM) is loaded.')
mask = mask_branch_model.generate(img)
cache[image_name] = mask
with open('/home/hsj/ianvs/project/cache.pickle', 'wb') as file:
with open('/ianvs/project/cache.pickle', 'wb') as file:
pickle.dump(cache, file)
print("save cache")

Expand Down Expand Up @@ -234,14 +234,14 @@ def sam_predict_ssa(self, image_name, pred):
return semantc_mask, mask

def sam_predict(self, image_name, pred):
with open('/home/hsj/ianvs/project/cache.pickle', 'rb') as file:
with open('/ianvs/project/cache.pickle', 'rb') as file:
cache = pickle.load(file)
Comment on lines +237 to 238
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The cache file path /ianvs/project/cache.pickle is hardcoded here as well. This makes the code less flexible. Consider making it a configurable parameter and handling potential FileNotFoundError or EOFError as suggested for the sam_predict_ssa method.

cache_path = getattr(self.args, "cache_path", "/ianvs/project/cache.pickle")
try:
    with open(cache_path, 'rb') as file:
        cache = pickle.load(file)
except (FileNotFoundError, EOFError):
    cache = {}

img = mmcv.imread(image_name)
if image_name in cache.keys():
mask = cache[image_name]
print("load cache")
else:
sam = sam_model_registry["vit_h"](checkpoint="/home/hsj/ianvs/project/segment-anything/sam_vit_h_4b8939.pth").to('cuda:1')
sam = sam_model_registry["vit_h"](checkpoint="/ianvs/project/segment-anything/sam_vit_h_4b8939.pth").to('cuda:1')
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The SAM checkpoint path and CUDA device are hardcoded here as well, similar to the sam_predict_ssa method. These should be parameterized for better flexibility and portability.

sam_checkpoint = getattr(self.args, "sam_checkpoint", "/ianvs/project/segment-anything/sam_vit_h_4b8939.pth")
device = f"cuda:{self.args.gpu_ids}" if self.args.cuda else "cpu"
sam = sam_model_registry["vit_h"](checkpoint=sam_checkpoint).to(device)

mask_branch_model = SamAutomaticMaskGenerator(
model=sam,
#points_per_side=64,
Expand All @@ -256,7 +256,7 @@ def sam_predict(self, image_name, pred):
print('[Model loaded] Mask branch (SAM) is loaded.')
mask = mask_branch_model.generate(img)
cache[image_name] = mask
with open('/home/hsj/ianvs/project/cache.pickle', 'wb') as file:
with open('/ianvs/project/cache.pickle', 'wb') as file:
pickle.dump(cache, file)
print("save cache")

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def set_weights(self, weights):
train_loss = self.trainer.my_training(epoch)
#train_loss = self.trainer.training(epoch)
loss_all.append(train_loss)
with open('/home/shijing.hu/ianvs/project/ianvs/train_loss_2.txt', 'a+') as file:
with open('/ianvs/project/ianvs/train_loss_2.txt', 'a+') as file:
np.savetxt(file, loss_all)
file.close

Expand Down Expand Up @@ -83,7 +83,7 @@ def train(self, train_data, valid_data=None, **kwargs):
}, is_best)

self.trainer.writer.close()
with open('/home/shijing.hu/ianvs/project/ianvs/train_loss.txt', 'a+') as file:
with open('/ianvs/project/ianvs/train_loss.txt', 'a+') as file:
np.savetxt(file, loss_all)
file.close
return self.train_model_url
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ algorithm:
# the method of splitting dataset; string type; optional;
# currently the options of value are as follows:
# 1> "default": the dataset is evenly divided based train_ratio;
#splitting_method: "default"
splitting_method: "fwt_splitting"

# algorithm module configuration in the paradigm; list type;
Expand All @@ -25,7 +24,7 @@ algorithm:
# example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking;
name: "BaseModel"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/basemodel-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/basemodel-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -40,7 +39,7 @@ algorithm:
# name of python module; string type;
name: "TaskDefinitionByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_definition_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_definition_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -53,7 +52,7 @@ algorithm:
# name of python module; string type;
name: "TaskAllocationByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ algorithm:
# the method of splitting dataset; string type; optional;
# currently the options of value are as follows:
# 1> "default": the dataset is evenly divided based train_ratio;
#splitting_method: "default"
#splitting_method: "fwt_splitting"
splitting_method: "hard-example_splitting"

# algorithm module configuration in the paradigm; list type;
Expand All @@ -26,7 +24,7 @@ algorithm:
# example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking;
name: "BaseModel"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/basemodel-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/basemodel-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -41,7 +39,7 @@ algorithm:
# name of python module; string type;
name: "TaskDefinitionByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_definition_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_definition_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -54,7 +52,7 @@ algorithm:
# name of python module; string type;
name: "TaskAllocationByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -67,7 +65,7 @@ algorithm:
# name of python module; string type;
name: "HardSampleMining"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/hard_sample_mining.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/hard_sample_mining.py"
hyperparameters:
- threhold:
values:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ algorithm:
# the method of splitting dataset; string type; optional;
# currently the options of value are as follows:
# 1> "default": the dataset is evenly divided based train_ratio;
#splitting_method: "default"
#splitting_method: "fwt_splitting"
splitting_method: "hard-example_splitting"

# algorithm module configuration in the paradigm; list type;
Expand All @@ -26,7 +24,7 @@ algorithm:
# example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking;
name: "BaseModel"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/basemodel-sam.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/basemodel-sam.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -41,7 +39,7 @@ algorithm:
# name of python module; string type;
name: "TaskDefinitionByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_definition_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_definition_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -54,7 +52,7 @@ algorithm:
# name of python module; string type;
name: "TaskAllocationByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -67,7 +65,7 @@ algorithm:
# name of python module; string type;
name: "HardSampleMining"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/hard_sample_mining.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/hard_sample_mining.py"
hyperparameters:
- threhold:
values:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ algorithm:
# the method of splitting dataset; string type; optional;
# currently the options of value are as follows:
# 1> "default": the dataset is evenly divided based train_ratio;
#splitting_method: "default"
#splitting_method: "fwt_splitting"
splitting_method: "hard-example_splitting"

# algorithm module configuration in the paradigm; list type;
Expand All @@ -26,7 +24,7 @@ algorithm:
# example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking;
name: "BaseModel"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/basemodel-vit-sam.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/basemodel-vit-sam.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -41,7 +39,7 @@ algorithm:
# name of python module; string type;
name: "TaskDefinitionByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_definition_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_definition_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -54,7 +52,7 @@ algorithm:
# name of python module; string type;
name: "TaskAllocationByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -67,7 +65,7 @@ algorithm:
# name of python module; string type;
name: "HardSampleMining"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/hard_sample_mining.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/hard_sample_mining.py"
hyperparameters:
- threhold:
values:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ algorithm:
# the method of splitting dataset; string type; optional;
# currently the options of value are as follows:
# 1> "default": the dataset is evenly divided based train_ratio;
#splitting_method: "default"
#splitting_method: "fwt_splitting"
splitting_method: "hard-example_splitting"

# algorithm module configuration in the paradigm; list type;
Expand All @@ -26,7 +24,7 @@ algorithm:
# example: basemodel.py has BaseModel module that the alias is "FPN" for this benchmarking;
name: "BaseModel"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/basemodel-vit-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/basemodel-vit-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -41,7 +39,7 @@ algorithm:
# name of python module; string type;
name: "TaskDefinitionByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_definition_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_definition_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -54,7 +52,7 @@ algorithm:
# name of python module; string type;
name: "TaskAllocationByOrigin"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/task_allocation_by_origin-simple.py"
# hyperparameters configuration for the python module; list type;
hyperparameters:
# name of the hyperparameter; string type;
Expand All @@ -67,7 +65,7 @@ algorithm:
# name of python module; string type;
name: "HardSampleMining"
# the url address of python module; string type;
url: "./examples/robot/lifelong_learning_bench/testalgorithms/rfnet/hard_sample_mining.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testalgorithms/rfnet/hard_sample_mining.py"
hyperparameters:
- threhold:
values:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ testenv:
# dataset configuration
dataset:
# the url address of train dataset index; string type;
train_url: "/home/shijing.hu/ianvs/dataset/all-train-index.txt"
train_url: "/data/datasets/all-train-index.txt"
# the url address of test dataset index; string type;
test_url: "/home/shijing.hu/ianvs/dataset/all-test-index.txt"
test_url: "/data/datasets/all-test-index.txt"

# model eval configuration of incremental learning;
model_eval:
Expand All @@ -13,7 +13,7 @@ testenv:
# metric name; string type;
name: "accuracy"
# the url address of python file
url: "./examples/robot/lifelong_learning_bench/testenv/accuracy.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testenv/accuracy.py"
mode: "no-inference"

# condition of triggering inference model to update
Expand All @@ -28,7 +28,7 @@ testenv:
# metric name; string type;
- name: "accuracy"
# the url address of python file
url: "./examples/robot/lifelong_learning_bench/testenv/accuracy.py"
url: "./examples/robot/lifelong_learning_bench/semantic-segmentation/testenv/accuracy.py"
- name: "samples_transfer_ratio"

# incremental rounds setting; int type; default value is 2;
Expand Down
Loading