Upgrade SAM to SAM2 (released 30.07.2024)

[mashb1t]

https://github.com/facebookresearch/segment-anything-2
https://ai.meta.com/sam2/

Sadly it's only compatible with CUDA, which would leave AMD, MacOS (MPS) and CPU users behind.
Also see facebookresearch/sam2#34
Let me know if you find a way to use it on CPU only!

console logs

(fooocus)  15:25:21  /Volumes/data/private/development/ai/fooocus/libs   develop ±  git clone [email protected]:facebookresearch/segment-anything-2.git

Cloning into 'segment-anything-2'...
remote: Enumerating objects: 285, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 285 (delta 2), reused 1 (delta 1), pack-reused 279
Receiving objects: 100% (285/285), 42.91 MiB | 11.24 MiB/s, done.
Resolving deltas: 100% (10/10), done.

(fooocus)  15:25:33  /Volumes/data/private/development/ai/fooocus/libs   develop ±  cd segment-anything-2; pip install -e .
Obtaining file:///Volumes/data/private/development/ai/fooocus/libs/segment-anything-2
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build editable did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      /private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:258: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:84.)
        cpu = _conversion_method_template(device=torch.device("cpu"))
      Traceback (most recent call last):
        File "/opt/homebrew/anaconda3/envs/fooocus/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/homebrew/anaconda3/envs/fooocus/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/homebrew/anaconda3/envs/fooocus/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable
          return hook(config_settings)
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 458, in get_requires_for_build_editable
          return self.get_requires_for_build_wheel(config_settings)
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 70, in <module>
        File "<string>", line 51, in get_extensions
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1076, in CUDAExtension
          library_dirs += library_paths(cuda=True)
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1207, in library_paths
          if (not os.path.exists(_join_cuda_home(lib_dir)) and
        File "/private/var/folders/j6/l5hq5k2s7hq11nyh00n__k4m0000gq/T/pip-build-env-zzmad1sl/overlay/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2416, in _join_cuda_home
          raise OSError('CUDA_HOME environment variable is not set. '
      OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[bobsponge76]

But maybe you could implement as a option for users with nvidia? Please? S2

[p1agnut]

CUDA only - don't quite see the problem here 🙃

[mashb1t]

which would leave AMD, MacOS (MPS) and CPU users behind.

Roughly 1/3 of all users of Fooocus would not be able to use this.

[NBICtech]

2/3 will still be able to... Please consider the whole

[Wontfallo]

SAM2 CPU Fork
https://github.com/SauravMaheshkar/samv2