Merge pull request #9 from Unity-Technologies/release/AME.2025.02

Release 0.3.0

Author: jean-lou-laliberte

CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this package will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] - 2025-02-04
+
+### Added
+- Added the column description in the csv to allow edit of individual asset descriptions.
+- Add multiple user input validations during the interactive config mode.
+- Added `app_settings.json` file to configure applications amount of parallel workers and environment variables when needed.
+
+### Changed
+- Organizations and projects are now selected via a list in the interactive config mode and delete mode.
+
+### Fixed
+- Bug with special characters when reading CSV files.
+- Bug with boolean metadata always being read as false.
+- CSV generated headlessly now respect the same template as CSV generated in an interactive run.
+- Relative path used to indicate assets location are now supported.
+- Fix tags and collection not being added to config when saving it.
+
 ## [0.2.0] - 2024-11-19
 
 ### Added

bulk_upload_cli/README.md

@@ -19,6 +19,8 @@ Find and connect support services on the [Help & Support](https://cloud.unity.co
 	- [Creating a csv from a Unity Cloud project](#creating-a-csv-from-a-unity-cloud-project)
     - [Editing metadata in the csv file](#editing-metadata-in-the-csv-file)
     - [Use an existing configuration file](#use-an-existing-configuration-file)
+    - [Fine-tune the asset creation and upload](#fine-tune-the-asset-creation-and-upload)
+  - [Troubleshoot](#troubleshoot)
   - [See also](#see-also)
   - [Tell us what you think](#tell-us-what-you-think)
 
@@ -62,10 +64,10 @@ The bulk upload sample script is provided under the [Unity ToS license](../LICEN
 
 ### Select the input method
 
-If you have a CSV respecting the template, when prompted about it, you can select yes and provide the path to the CSV file. Otherwise, you will be prompted about your assets location.
-
 Select one of the three strategies as the input method for bulk asset creation:
 
+- **listed in a casv respecting the CLI tool template**: Select this option if you built a CSV listing your assets location and details using the provided template.
+  * Provide the path to the csv file.
 - **in a .unitypackage file**: Select this option if your assets are in a .unitypackage file. The tool extracts the assets from the .unitypackage file and uploads them to the cloud.
   * Provide the path to the .unitypackage file.
 - **in a local unity project**: Select this option if your assets are in a local Unity project.
@@ -125,6 +127,21 @@ To use an existing configuration file, follow these steps:
 3. On the next run with the `--create` flag, you can add the `--config` flag followed by the name of the configuration file you created. All the answers you gave during the first run will be loaded from the configuration file.
 4. Alternatively, you can use the `--config-select` flag to select a configuration file from the list of existing configuration files.
 
+### Fine-tune the asset creation and upload
+
+With the `app_settings.json` file, you can fine-tune the amount of assets created and uploaded in parallel. Depending on your network, the number of assets, and the size of the assets, you can adjust the following settings:
+- `parallelCreationEdit`: The number of assets created and updated in parallel. This settings can be kept high as it is not resource intensive.
+- `parallelAssetUpload`: The number of assets that will have their files uploaded in parallel. This setting should be adjusted depending on the size of the assets and the network speed. When dealing with large files (>100MB), it is recommended to keep this setting low (1-2) to avoid time out.
+- `parallelFileUploadPerAsset`: The number of files uploaded in parallel for each asset. This setting should be adjusted depending on the number of files and the network speed. It is recommended to adjust it according to `parallelAssetUpload`, as the total number of files uploaded in parallel will be `parallelAssetUpload * parallelFileUploadPerAsset`.
+
+In the `app_settings.json` file, you can also add environment variables that will be set at runtime. This is useful when running the CLI tool in a private network environment.
+
+## Troubleshoot
+
+Here's a list of common problems you might encounter while using the CLI Tool.
+- `error ModuleNotFoundError: No module named ...`: This can be caused by a uncompleted installation. Start by uninstalling `unity_cloud` with `pip(3) uninstall unity_cloud`, then re-run the CLI tool installation.
+- Timeout exception during the upload step: When uploading large files, it is recommended to lower the amount of parallel uploads allowed. To do so, refer to the [Fine-tune the asset creation and upload](#fine-tune-the-asset-creation-and-upload) section.
+
 ## See also
 For more information, see the [Unity Cloud Python SDK](https://docs.unity.com/cloud/en-us/asset-manager/python-sdk) documentation.
 

bulk_upload_cli/app_settings.json

@@ -0,0 +1,6 @@
+{
+    "parallelCreationEdit": 20,
+    "parallelAssetUpload": 2,
+    "parallelFileUploadPerAsset": 5,
+    "environmentVariables": {}
+}

bulk_upload_cli/bulk_upload/asset_deleter.py

@@ -39,8 +39,22 @@ def delete_assets_in_project():
     key_id, key = ask_for_login()
     login(key_id, key)
 
-    org_id = inquirer.text(message="Enter your organization ID:").execute()
-    project_id = inquirer.text(message="Enter your project ID:").execute()
+    organizations = uc.identity.get_organization_list()
+    if len(organizations) == 0:
+        print("No organizations found. Please create an organization first.")
+        exit(1)
+    org_selected = inquirer.select(message="Select an organization:",
+                                   choices=[org.name for org in organizations]).execute()
+    org_id = [org.id for org in organizations if org.name == org_selected][0]
+
+    projects = uc.identity.get_project_list(org_id)
+    if len(projects) == 0:
+        print("No projects found. Please create a project first.")
+        exit(1)
+
+    selected_project = inquirer.select(message="Select a project:",
+                                       choices=[project.name for project in projects]).execute()
+    project_id = [project.id for project in projects if project.name == selected_project][0]
 
     project_assets = uc.assets.get_asset_list(org_id, project_id)
     confirm = inquirer.confirm(message=f"Are you sure you want to delete {len(project_assets)} assets?").execute()

bulk_upload_cli/bulk_upload/asset_mappers.py

@@ -104,8 +104,9 @@ class FolderGroupingAssetMapper(AssetMapper):
 
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
 
-        hierarchical_level = int(config.hierarchical_level) + os.path.abspath(config.assets_path).count(os.sep)
-        folders = [x[0] for x in os.walk(config.assets_path) if x[0].count(os.sep) == hierarchical_level]
+        abs_path = os.path.abspath(config.assets_path)
+        hierarchical_level = int(config.hierarchical_level) + abs_path.count(os.sep)
+        folders = [x[0] for x in os.walk(abs_path) if x[0].count(os.sep) == hierarchical_level]
 
         if len(folders) == 0:
             print(f"No folders found in the assets path. Only the root folder will be considered as an asset")
@@ -116,7 +117,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
             asset_name = PurePath(folder).name
             assets[asset_name] = AssetInfo(asset_name)
 
-            files = [get_file_info(PurePath(f), config.assets_path) for x in os.walk(folder) for f in glob(os.path.join(x[0], '*'))]
+            files = [get_file_info(PurePath(f), abs_path) for x in os.walk(folder) for f in glob(os.path.join(x[0], '*'))]
             # remove files with excluded extensions
             files = [f for f in files if not any(f.path.suffix.endswith(ext) for ext in config.excluded_file_extensions)]
             assets[asset_name].files = files
@@ -129,7 +130,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
                     asset.files.remove(file)
 
                 if config.preview_detection and self.is_preview_file(file.path):
-                    asset.preview_files.append(get_file_info(file.path, config.assets_path))
+                    asset.preview_files.append(get_file_info(file.path, abs_path))
                     asset.files.remove(file)
 
         return list(assets.values())
@@ -219,6 +220,7 @@ def clean_up(self):
         pass
 
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
+        absolute_path = os.path.abspath(config.assets_path)
         # find all files in the assets folder and sub folders
         files = [y for x in os.walk(config.assets_path) for y in glob(os.path.join(x[0], '*'))]
         # remove files with excluded extensions
@@ -247,7 +249,7 @@ def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
 
             file_path = PurePath(file)
             asset = AssetInfo(os.path.basename(file))
-            asset.files.append(FileInfo(file_path, config.assets_path))
+            asset.files.append(FileInfo(file_path, PurePosixPath(file_path.relative_to(config.assets_path))))
 
             if file_path.stem.lower() in potential_previews:
                 preview_file = potential_previews[file_path.stem.lower()]
@@ -285,7 +287,7 @@ def clean_up(self):
 
     def map_assets(self, config: ProjectUploaderConfig) -> [AssetInfo]:
         assets = []
-        with open(config.assets_path, 'r') as file:
+        with open(config.assets_path, 'r', encoding='utf-8') as file:
             reader = csv.DictReader(file)
             input_row = next(reader)
 

bulk_upload_cli/bulk_upload/assets_customization_providers.py

@@ -21,8 +21,9 @@ def apply_asset_customization(self, assets: [AssetInfo], config: ProjectUploader
             asset_customization.tags.append(PurePath(config.assets_path).name.replace(".unitypackage", ""))
 
         asset_customization.collection = self.get_collection(config.org_id, config.project_id)
-        #asset_customization.metadata = self.get_metadata()
 
+        config.tags = asset_customization.tags
+        config.collection = asset_customization.collection
         for asset in assets:
             asset.customization.tags = asset_customization.tags
             asset.customization.collection = asset_customization.collection

bulk_upload_cli/bulk_upload/assets_uploaders.py

@@ -1,21 +1,20 @@
-import json
 import logging
+import time
 
-import unity_cloud as uc
 import unity_cloud.assets.asset_reference
 
 from bulk_upload.asset_mappers import *
+from bulk_upload.models import *
 from concurrent.futures import ThreadPoolExecutor, wait
 from unity_cloud.models import *
-from pathlib import PurePath, PurePosixPath
 
 
 logger = logging.getLogger(__name__)
 
 
 class AssetUploader(ABC):
     @abstractmethod
-    def upload_assets(self, asset_infos: [AssetInfo], config: ProjectUploaderConfig):
+    def upload_assets(self, asset_infos: [AssetInfo], config: ProjectUploaderConfig, app_settings: AppSettings):
         pass
 
 
@@ -25,7 +24,7 @@ def __init__(self):
         self.config = None
         self.futures = list()
 
-    def upload_assets(self, asset_infos: [AssetInfo], config: ProjectUploaderConfig):
+    def upload_assets(self, asset_infos: [AssetInfo], config: ProjectUploaderConfig, app_settings: AppSettings):
         self.config = config
 
         cloud_assets = [] if config.strategy == Strategy.CLOUD_ASSET else uc.assets.get_asset_list(self.config.org_id, self.config.project_id)
@@ -42,7 +41,7 @@ def upload_assets(self, asset_infos: [AssetInfo], config: ProjectUploaderConfig)
                     asset.is_frozen_in_cloud = project_asset.is_frozen
                     break
 
-        with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor:
+        with ThreadPoolExecutor(max_workers=app_settings.parallel_creation_edit) as executor:
             for asset in asset_infos:
                 if not asset.already_in_cloud:
                     self.futures.append(executor.submit(self.create_asset, asset))
@@ -53,25 +52,28 @@ def upload_assets(self, asset_infos: [AssetInfo], config: ProjectUploaderConfig)
         self.futures = list()
 
         print("Setting asset dependencies", flush=True)
-        with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor:
+        with ThreadPoolExecutor(max_workers=app_settings.parallel_creation_edit) as executor:
             for asset in asset_infos:
                 self.futures.append(executor.submit(self.set_asset_references, asset, asset_infos))
 
         wait(self.futures)
         self.futures = list()
 
+        #sleep for 5 seconds to allow the asset to be created with their dataset
+        time.sleep(5)
+
         if self.config.update_files and self.config.strategy == Strategy.CLOUD_ASSET:
             self.config.update_files = False
             print("File update not supported for cloud assets, skipping file upload", flush=True)
-        with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor:
+        with ThreadPoolExecutor(max_workers=app_settings.parallel_asset_upload) as executor:
             for asset in asset_infos:
                 if not asset.already_in_cloud or self.config.update_files:
-                    self.futures.append(executor.submit(self.upload_asset_files, asset))
+                    self.futures.append(executor.submit(self.upload_asset_files, asset, app_settings))
 
         self.futures = list()
 
         print("Setting tags and collections for assets", flush=True)
-        with ThreadPoolExecutor(max_workers=self.config.amount_of_parallel_uploads) as executor:
+        with ThreadPoolExecutor(max_workers=app_settings.parallel_creation_edit) as executor:
             for asset in asset_infos:
                 self.futures.append(executor.submit(self.set_asset_decorations, asset))
 
@@ -108,7 +110,7 @@ def create_new_version(self, asset: AssetInfo):
             print(f'Failed to create new version for asset: {asset.name}', flush=True)
             print(e, flush=True)
 
-    def upload_asset_files(self, asset: AssetInfo):
+    def upload_asset_files(self, asset: AssetInfo, app_settings: AppSettings):
         try:
 
             dataset_id = uc.assets.get_dataset_list(self.config.org_id, self.config.project_id, asset.am_id, asset.version)[0].id
@@ -118,7 +120,7 @@ def upload_asset_files(self, asset: AssetInfo):
 
             print(f"Uploading files for asset: {asset.name}", flush=True)
             files_upload_futures = []
-            with ThreadPoolExecutor(max_workers=200) as executor:
+            with ThreadPoolExecutor(max_workers=app_settings.parallel_file_upload_per_asset) as executor:
                 for file in asset.files:
                     files_upload_futures.append(executor.submit(self.upload_file, asset, dataset_id, file))
 
@@ -203,7 +205,8 @@ def set_asset_decorations(self, asset: AssetInfo):
             for metadata_field in asset.customization.metadata:
                 asset_update.metadata[metadata_field.field_definition] = metadata_field.field_value
 
-        if asset.customization.description is not None:
+        if asset.customization.description is not None and asset.customization.description != "":
+            print(asset.customization.description)
             asset_update.description = asset.customization.description
 
         try:

bulk_upload_cli/bulk_upload/bulk_upload_pipeline.py

@@ -1,7 +1,8 @@
 import unity_cloud as uc
+import os
 
 from bulk_upload.config_providers import InteractiveConfigProvider, FileConfigProvider, SelectConfigProvider
-from bulk_upload.models import ProjectUploaderConfig, Strategy, DependencyStrategy
+from bulk_upload.models import ProjectUploaderConfig, Strategy, DependencyStrategy, AppSettings
 from bulk_upload.asset_mappers import NameGroupingAssetMapper, FolderGroupingAssetMapper, UnityPackageAssetMapper, \
     UnityProjectAssetMapper, SingleFileAssetMapper, CsvAssetMapper, CloudAssetMapper
 from bulk_upload.assets_uploaders import AssetUploader, CloudAssetUploader
@@ -39,6 +40,10 @@ def login_with_user_account():
             uc.identity.user_login.login()
 
     def run(self, config_file=None, select_config=False):
+        app_settings = AppSettings()
+        app_settings.load_from_json()
+        self.set_environment_variables(app_settings)
+
         is_headless_run = config_file is not None or select_config
 
         # Step 1: Get base the configuration
@@ -69,11 +74,16 @@ def run(self, config_file=None, select_config=False):
 
         # Step 7: Upload
         asset_uploader = self.get_asset_uploader(config)
-        asset_uploader.upload_assets(assets, config)
+        asset_uploader.upload_assets(assets, config, app_settings)
 
         # Step 8: Post upload actions, Clean up
         asset_mapper.clean_up()
 
+    @staticmethod
+    def set_environment_variables(app_settings: AppSettings):
+        for key, value in app_settings.environment_variables.items():
+            os.environ[key] = value
+
     @staticmethod
     def get_config_provider(select_config=False, config_file=None):
         if select_config: