Clean up code

Author: forsyth2

docs/source/usage.rst

@@ -56,6 +56,8 @@ Additional optional arguments:
   than MAXSIZE except when individual input files exceed MAXSIZE (as 
   individual files are never split up between different tar files).
 * ``--non-blocking`` Zstash will submit a Globus transfer and immediately create a subsequent tarball. That is, Zstash will not wait until the transfer completes to start creating a subsequent tarball. On machines where it takes more time to create a tarball than transfer it, each Globus transfer will have one file. On machines where it takes less time to create a tarball than transfer it, the first transfer will have one file, but the number of tarballs in subsequent transfers will grow finding dynamically the most optimal number of tarballs per transfer. NOTE: zstash is currently always non-blocking.
+* ``--error-on-duplicate-tar`` Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash's behavior will depend on whether or not the --overwrite-duplicate-tar flag is set.
+* ``--overwrite-duplicate-tars`` If a duplicate tar is encountered, overwrite the existing tar file with the new one (i.e., it will assume the latest tar is the correct one). If this flag is not set, zstash will permit multiple entries for the same tar in its database.
 * ``-v`` increases output verbosity.
 
 Local tar files as well as the sqlite3 index database (index.db) will be stored
@@ -153,6 +155,7 @@ where
   an incomplete tar file, then the archive you're checking
   must have been created using ``zstash >= v1.1.0``.
 * ``--tars`` to specify specific tars to check. See below for example usage.
+* ``--error-on-duplicate-tar`` Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash will check if the sizes and md5sums match *at least one* of the tars.
 * ``-v`` increases output verbosity.
 * ``[files]`` is a list of files to check (standard wildcards supported).
 
@@ -240,6 +243,8 @@ where
   they have been extracted from the archive. Normally, they are deleted after
   successful transfer.
 * ``--non-blocking`` Zstash will submit a Globus transfer and immediately create a subsequent tarball. That is, Zstash will not wait until the transfer completes to start creating a subsequent tarball. On machines where it takes more time to create a tarball than transfer it, each Globus transfer will have one file. On machines where it takes less time to create a tarball than transfer it, the first transfer will have one file, but the number of tarballs in subsequent transfers will grow finding dynamically the most optimal number of tarballs per transfer. NOTE: zstash is currently always non-blocking.
+* ``--error-on-duplicate-tar`` Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash's behavior will depend on whether or not the --overwrite-duplicate-tar flag is set.
+* ``--overwrite-duplicate-tars`` If a duplicate tar is encountered, overwrite the existing tar file with the new one (i.e., it will assume the latest tar is the correct one). If this flag is not set, zstash will permit multiple entries for the same tar in its database.
 * ``-v`` increases output verbosity.
 
 Note: in the event that an update includes revisions to files previously archived, ``zstash update``
@@ -319,6 +324,7 @@ where
   an incomplete tar file, then the archive you're extracting from
   must have been created using ``zstash >= v1.1.0``.
 * ``--tars`` to	specify	specific tars to extract. See "Check" above for example usage.
+* ``--error-on-duplicate-tar`` Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash will check if the sizes and md5sums match *at least one* of the tars.
 * ``-v`` increases output verbosity.
 * ``[files]`` is a list of files to be extracted (standard wildcards supported).
 

tests/scripts/database_corruption.bash

@@ -1,8 +1,4 @@
-# TODO: Commit these changes
-# TODO: Factor out functions in this script
-# TODO: Run the unit tests too
-# TODO: Add any functionality added to create to update as well.
-# TODO: Add any new parameters to the usage docs
+# TODO: Run the unit tests on Perlmutter
 
 setup()
 {
@@ -58,6 +54,15 @@ run_test_cases()
     fail_count=0
     review_str=""
 
+    # Test case explanations ##################################################
+    # 1.`zstash create`, then run `zstash_check` from a different directory.
+    # 2. `zstash create`, then run `zstash_check` from a directory that already has `zstash/index.db`.
+    # 3. `zstash_create` with `--for-developers-force-database-corruption="simulate_row_existing" --error-on-duplicate-tar`. Errors out on create, so we don't even get to check.
+    # 4. `zstash create` with `--for-developers-force-database-corruption="simulate_row_existing_bad_size" --overwrite-duplicate-tars`. We see there's a duplicate tar and we overwrite it with the latest data. `zstash check` confirms the tar is correct.
+    # 5. `zstash create` with `--for-developers-force-database-corruption="simulate_row_existing"`. We simply add a duplicate tar, but `zstash check` with `--error-on-duplicate-tar` errors out because it finds two entries for the same tar.
+    # 6. `zstash create` with `--for-developers-force-database-corruption="simulate_no_correct_size"` to construct a very bad database: two entries for the same tar, both with incorrect sizes. `zstash check` confirms that no entries match the actual file size.
+    # 7. `zstash create` with `--for-developers-force-database-corruption="simulate_row_existing_bad_size"`. We add a duplicate tar, but with the wrong size. `zstash check` confirms that the other entry matches the actual file size, so it succeeds.
+
     # Standard cases ##########################################################
 
 
@@ -98,12 +103,12 @@ run_test_cases()
     else
         ((success_count++))
     fi
-    # Do NOT change directory
+    cd zstash_demo # Use a directory that already has a zstash/index.db!
     zstash check --hpss=${DST_DIR}/${case_name} 2>&1 | tee check.log
     grep "INFO: 000000.tar: Found a single database entry." check.log
     if [ $? != 0 ]; then
         ((fail_count++))
-        review_str+="${case_name}_check/check.log,"
+        review_str+="${case_name}_create/zstash_demo/check.log," # Notice this is a different path!
     else
         ((success_count++))
     fi

zstash/update.py

@@ -105,6 +105,16 @@ def setup_update() -> Tuple[argparse.Namespace, str]:
         action="store_true",
         help="do not wait for each Globus transfer until it completes.",
     )
+    optional.add_argument(
+        "--error-on-duplicate-tar",
+        action="store_true",
+        help="Raise an error if a tar file with the same name already exists in the database. If this flag is set, zstash will exit if it sees a duplicate tar. If it is not set, zstash's behavior will depend on whether or not the --overwrite-duplicate-tar flag is set.",
+    )
+    optional.add_argument(
+        "--overwrite-duplicate-tars",
+        action="store_true",
+        help="If a duplicate tar is encountered, overwrite the existing tar file with the new one (i.e., it will assume the latest tar is the correct one). If this flag is not set, zstash will permit multiple entries for the same tar in its database.",
+    )
     optional.add_argument(
         "-v", "--verbose", action="store_true", help="increase output verbosity"
     )
@@ -265,6 +275,8 @@ def update_database(  # noqa: C901
                 keep,
                 args.follow_symlinks,
                 non_blocking=args.non_blocking,
+                error_on_duplicate_tar=args.error_on_duplicate_tar,
+                overwrite_duplicate_tars=args.overwrite_duplicate_tars,
             )
         except FileNotFoundError:
             raise Exception("Archive update failed due to broken symlink.")
@@ -279,6 +291,8 @@ def update_database(  # noqa: C901
             keep,
             args.follow_symlinks,
             non_blocking=args.non_blocking,
+            error_on_duplicate_tar=args.error_on_duplicate_tar,
+            overwrite_duplicate_tars=args.overwrite_duplicate_tars,
         )
 
     # Close database