Version 624

Author: hydrusnetwork

docs/changelog.md

@@ -34,6 +34,51 @@
         <div class="content">
             <h1 id="changelog"><a href="#changelog">changelog</a></h1>
             <ul>
+                <li>
+                    <h2 id="version_624"><a href="#version_624">version 624</a></h2>
+                    <ul>
+                        <li><h3>locked pages</h3></li>
+                        <li>the search page state where it could hold files but have no active search is back with some bells and whistles. it is now much easier to manage a 'scratchpad' file page that won't suddenly get messed up if you hit F5 (issue #1602, and likely others in different ways)</li>
+                        <li>all normal file search pages now have a padlock icon beside the autocomplete text input. clicking this collapses the current search to a single system:hash of the current media in view and swaps the search panel for an 'unlock' button. this page will not refresh its query until unlocked, at which point you are given the system:hash back</li>
+                        <li>if you add files to or remove files from the page, the underlying system:hash will update to the new contents of the page</li>
+                        <li>lock status is saved in the session, so it persists through restart etc..</li>
+                        <li>any time you create a page with files, which typically means, 'open files in a new page' and subscription file-popups, they now start in the locked state. since the system:hash updates for new files, you can drag and drop new files onto the page, or add more subscription files from a later sync, and the underlying system:hash now stays updated. play around with it--you'll see how it works</li>
+                        <li>if you are big brain and wish in some cases _not_ to sync the underlying system:hash on add/remove file events, a cog icon beside the unlock button lets you set this</li>
+                        <li>if you have a bunch of existing subscription landing pages and other 'I'll process this later' pages, please go through them after update and lock them. it'll say 'hey, you already have a system:hash but it differs to what is currently in view, do you want to overwrite this old one with one for the current files?' and you do, so click yes</li>
+                        <li>next step here is to finally figure out some sort of 'sort files by system:hash/downloader' sort type so these sorts of pages can restore original sort</li>
+                        <li><h3>duplicates</h3></li>
+                        <li>the 'preview' panel of an auto-resolution rule has had a complete pipeline overhaul. after a much, much, briefer one-time initialisation, it now loads results in fast small batches and streams pass/fail pairs to the lists one by one. it shows search progress and presents ongoing count of pairs. it no longer gets the total count of the search every time--unless you let it run all the way and get everything. the refresh buttons are more efficient, the 'only sample this many' widget updates the dialog live as you edit it, and there are pause buttons for both search and testing. it also cancels work faster if the dialog is closed amidst work. this dialog is now ready for the relatively CPU-expensive 'A and B are visual duplicates' integration</li>
+                        <li>when the core duplicate file searches they rely on are CPU-expensive, the duplicates filter batch pair fetch, duplicates filter count fetch, duplicates 'show some random pairs', and duplicates auto-resolution rule preview now load up faster, particularly when you have few potential pairs remaining</li>
+                        <li>auto-resolution rules that are in semi-automatic mode now only queue up a max of 512 files in the 'pending' queue. it can take some CPU to run these tests, so it is good to keep the queue short and snappy in case you need to change anything (thus resetting the queue) in future. you can alter the 512 limit, including removing it completely, in the edit rule panel</li>
+                        <li>auto-resolution rules now categorise their comparators into a cross-reference of fast + order-possibly-important. the main testing routine and 'can this pair fit either way' test now cascades through some logic to get a dispositive true negative result faster</li>
+                        <li>the program now recognises and gracefully handles the situation where the duplicate filter is closed and program shut down initiated before a slow-loading visual duplicate pair can finish (it was outputting some mysterious error lines to the log)</li>
+                        <li><h3>a and b are visual duplicates</h3></li>
+                        <li>if the images are very simple, e.g. just one or two flat colours, they are now considered 'too simple to compare' and not visual duplicates. these edge cases are often deceptively tricky</li>
+                        <li>if the images have a perfectly matching interesting tile, the presence of any non-matching tile now results in 'probably not visual duplicates/(small difference?)'</li>
+                        <li>I experimented with many new techniques: pre-decimation histogram population, histogram absolute difference comparison, tile phashes, longer tile phashes, gaussian edge detection histograms, gaussian edge map average block 2d wasserstein scoring, and even gaussian edge map phashes, but unfortunately these all pretty much failed. the algorithm as-is is good, but there are still a couple of false positive situations it cannot catch. the delicate edge detection is the main thing--a difference with a subtle shade (e.g. a soft line, or a transparent water droplet) that is clear to the human eye can still be missed. please keep sending in any false positives, and I'll keep thinking about all this</li>
+                        <li>cleaned and polished a bunch of this code</li>
+                        <li><h3>misc</h3></li>
+                        <li>thanks to a user, the new resizable ratings have improved visuals! the little padding clipping bugs are fixed and everything lines up nice</li>
+                        <li>'less than or equal to' and 'greater than or equal to' are added to all the modernised places you can set a number comparison, such as `system:height` or duplicates auto-resolution comparison statements</li>
+                        <li>fixed the broken `Ctrl+C = copy selected files` default 'media' shortcut for newer users--if it is broken for you (it'll say 'Unknown Application Command' in the shortcut UI), it will fix itself on update</li>
+                        <li><h3>updated Linux and Windows builds</h3></li>
+                        <li>the 'future build' last week went without any reports of problems, so I am folding the changes into today's normal build. both Windows and Linux are updating several core components and will gain a host of small and large bug fixes and performance improvements. It does not seem that a clean install is needed this week, but if you use the zip or tar.zst extracts, you might like to take this opportunity to do one anyway: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs</li>
+                        <li>the WIndows installer basically does a clean install every week, so no special instructions there</li>
+                        <li>the changes are--</li>
+                        <li>Windows and Linux: moved from python 3.11 to python 3.12</li>
+                        <li>Windows and Linux: moved from PySide6 (Qt) 6.7.3 to 6.8.2.1</li>
+                        <li>Windows: moved from windows-2019 runner to windows-2022 (2019 runner is being retired next month)</li>
+                        <li>if you use a particularly old, un-updated version of Windows 10, it is possible the new Windows builds will not run for you. try running from source and choosing an older Qt version: https://hydrusnetwork.github.io/hydrus/running_from_source.html</li>
+                        <li>if you already run from source, you might like to rebuild your venv this week to get the new Qt. users on Python 3.13 no longer have to choose the (a)dvanced mode for the (t)est Qt version-- on a typical OS, all pythons from 3.10-13 should now work with (s) out of the box!</li>
+                        <li><h3>weird/boring stuff</h3></li>
+                        <li>I fixed some encoding in my png gamma/chromaticity ICC Profile generator. it still isn't there, but I know where to look. also wangled the `wtpt` and `chad` to handle non-D65 source whitepoints</li>
+                        <li>I may have fixed an issue where the thumbnail banners could be drawn with bold text</li>
+                        <li>my asynchronous updater object can now deal with several mid-job cancel events</li>
+                        <li>my thumbnail pair list can now handle streaming appends</li>
+                        <li>deleted the monolith 'get potential pairs that match this duplicate context' routine the preview panel was relying on. I'll migrate more duplicates stuff to incremental fetches like this, and I feel good we can make the duplicate filter and so on initialise much quicker</li>
+                        <li>updated the client api help's Hydrus Video Deduplicator link to https://github.com/hydrusvideodeduplicator/hydrus-video-deduplicator</li>
+                    </ul>
+                </li>
                 <li>
                     <h2 id="version_623"><a href="#version_623">version 623</a></h2>
                     <ul>
@@ -42,7 +87,7 @@ <h2 id="version_623"><a href="#version_623">version 623</a></h2>
                         <li>the client now tries to adjust for timezone when it parses a modified date from a server response. some downloaders will now get a more precise 'modified date' for their domains</li>
                         <li>fixed a typo bug that was causing existing 'set numerical or inc/dec rating' commands to fail to initialise in the shortcut panel on edit (staying as the default 'simple command, archive file')</li>
                         <li>a new `DEBUG: Activate Main GUI when closing the media viewer` checkbox under `options-&gt;media viewer`, default fa!se, lets you force an "activate" call on the Main GUI (bringing it to the front and setting keyboard focus) on any media viewer close</li>
-                        <li>the hydrus server now better tries to clear out ongoing file downloads when the connection is disconnected early. if you noticed in Hydrus Web or similar that scrubbing through a bunch of big videos one after another could get laggy, let me know if it is any better now</li>
+                        <li>the core hydrus server/client api response object now better tries to clear out ongoing file downloads when the connection is disconnected early. if you noticed in Hydrus Web or similar that scrubbing through a bunch of big videos one after another could get laggy, let me know if it is any better now</li>
                         <li><h3>duplicates</h3></li>
                         <li>all duplicate filters now have the 'A and B are visual duplicates' comparison statement. this is a custom algorthim that inspects the images and attempts to differentiate resizes/re-encodes from recolours/alternates. please rely on it as you do normal duplicate filtering of images from now on, and if it produces any false positives, I'd love to see the files!</li>
                         <li>I improved this 'A and B are visual duplicates' algorithm after our test. overall it went very well, with only a couple false positives found. thank you for your feedback, everyone. I improved the 'skewness' test to reduce the number of false negatives, and I added a new special test to catch the false positives submitted. further, I have softened the 'no' result texts and categorised the 'yes' results into three clear 'very probably'/'definitely'/'perfectly' lines so you can better recognise edge cases. when this test is integrated into the auto-resolution system, I will default to just definitely/perfectly results to start</li>

docs/old_changelog.html

@@ -640,6 +640,9 @@ def _Initialise( self ):
         self.highlight = QG.QPixmap( os.path.join( HC.STATIC_DIR, 'highlight.png' ) )
         self.clear_highlight = QG.QPixmap( os.path.join( HC.STATIC_DIR, 'clear_highlight.png' ) )
 
+        self.lock = QG.QPixmap( os.path.join( HC.STATIC_DIR, 'lock.png' ) )
+        self.lock_open = QG.QPixmap( os.path.join( HC.STATIC_DIR, 'lock_open.png' ) )
+        
         self.star = QG.QPixmap( os.path.join( HC.STATIC_DIR, 'star.png' ) )
 
         #self.listctrl = QG.QPixmap( os.path.join( HC.STATIC_DIR, 'listctrl.png' ) )

hydrus/client/ClientConstants.py

@@ -314,7 +314,7 @@ def GetDefaultShortcuts():
 
     media.SetCommand(
         ClientGUIShortcuts.Shortcut( ClientGUIShortcuts.SHORTCUT_TYPE_KEYBOARD_CHARACTER, ord( 'C' ), ClientGUIShortcuts.SHORTCUT_PRESS_TYPE_PRESS, [ ClientGUIShortcuts.SHORTCUT_MODIFIER_CTRL ] ),
-        CAC.ApplicationCommand.STATICCreateSimpleCommand( CAC.SIMPLE_COPY_FILES )
+        CAC.ApplicationCommand.STATICCreateSimpleCommand( CAC.SIMPLE_COPY_FILES, simple_data = CAC.FILE_COMMAND_TARGET_SELECTED_FILES )
     )
 
     shortcuts.append( media )

hydrus/client/ClientDefaults.py

@@ -3834,6 +3834,7 @@ def _InitCommandsToMethods( self ):
 
         self._read_commands_to_methods.update(
             {
+                'all_potential_duplicate_pairs_and_distances' : self.modules_files_duplicates.GetAllPotentialDuplicatePairsAndDistances,
                 'autocomplete_predicates' : self.modules_tag_search.GetAutocompletePredicates,
                 'client_files_subfolders' : self.modules_files_physical_storage.GetClientFilesSubfolders,
                 'deferred_delete_data' : self.modules_db_maintenance.GetDeferredDeleteTableData,
@@ -3861,7 +3862,7 @@ def _InitCommandsToMethods( self ):
                 'missing_archive_timestamps_legacy_test' : self.modules_files_inbox.WeHaveMissingLegacyArchiveTimestamps,
                 'missing_repository_update_hashes' : self.modules_repositories.GetRepositoryUpdateHashesIDoNotHave,
                 'num_deferred_file_deletes' : self.modules_files_storage.GetDeferredPhysicalDeleteCounts,
-                'potential_duplicate_pairs' : self.modules_files_duplicates_file_query.GetPotentialDuplicatePairs,
+                'potential_duplicate_pairs_fragmentary' : self.modules_files_duplicates_file_query.GetPotentialDuplicatePairsForAutoResolutionMediaResults,
                 'recent_tags' : self.modules_recent_tags.GetRecentTags,
                 'repository_progress' : self.modules_repositories.GetRepositoryProgress,
                 'repository_update_hashes_to_process' : self.modules_repositories.GetRepositoryUpdateHashesICanProcess,
@@ -9190,6 +9191,46 @@ def notify_deleting_auto_resolution_rules():
 
 
 
+        if version == 623:
+            
+            try:
+                
+                from hydrus.client.gui import ClientGUIShortcuts
+                from hydrus.client import ClientApplicationCommand as CAC
+                
+                media_shortcuts_set: ClientGUIShortcuts.ShortcutSet = self.modules_serialisable.GetJSONDumpNamed( HydrusSerialisable.SERIALISABLE_TYPE_SHORTCUT_SET, 'media' )
+                
+                jobs = [
+                    (
+                        ClientGUIShortcuts.Shortcut( ClientGUIShortcuts.SHORTCUT_TYPE_KEYBOARD_CHARACTER, ord( 'C' ), ClientGUIShortcuts.SHORTCUT_PRESS_TYPE_PRESS, [ ClientGUIShortcuts.SHORTCUT_MODIFIER_CTRL ] ),
+                        CAC.ApplicationCommand.STATICCreateSimpleCommand( CAC.SIMPLE_COPY_FILES ),
+                        CAC.ApplicationCommand.STATICCreateSimpleCommand( CAC.SIMPLE_COPY_FILES, simple_data = CAC.FILE_COMMAND_TARGET_SELECTED_FILES ) 
+                    )
+                ]
+                
+                for ( shortcut, bad_command, good_command ) in jobs:
+                    
+                    if media_shortcuts_set.HasCommand( shortcut ):
+                        
+                        if media_shortcuts_set.GetCommand( shortcut ) == bad_command:
+                            
+                            media_shortcuts_set.SetCommand( shortcut, good_command )
+                            
+                        
+                    
+                
+                self.modules_serialisable.SetJSONDump( media_shortcuts_set )
+                
+            except Exception as e:
+                
+                HydrusData.PrintException( e )
+                
+                message = 'Trying to update some shortcuts failed! Please let hydrus dev know!'
+                
+                self.pub_initial_message( message )
+                
+            
+        
         self._controller.frame_splash_status.SetTitleText( 'updated db to v{}'.format( HydrusNumbers.ToHumanInt( version + 1 ) ) )
 
         self._Execute( 'UPDATE version SET version = ?;', ( version + 1, ) )

hydrus/client/db/ClientDB.py

@@ -521,6 +521,11 @@ def FilterExistingPotentialDuplicatePairs( self, potential_pair_ids_table_name:
         return existing_pairs
 
 
+    def GetAllPotentialDuplicatePairsAndDistances( self ):
+        
+        return self._Execute( 'SELECT smaller_media_id, larger_media_id, distance FROM potential_duplicate_pairs;' ).fetchall()
+        
+    
     def GetAlternatesGroupId( self, media_id, do_not_create = False ):
 
         result = self._Execute( 'SELECT alternates_group_id FROM alternate_file_group_members WHERE media_id = ?;', ( media_id, ) ).fetchone()

hydrus/client/db/ClientDBFilesDuplicates.py

@@ -94,10 +94,32 @@ def get_row():
             return self.modules_files_duplicates_auto_resolution_storage.GetMatchingUntestedPair( rule )
 
 
+        current_num_pending_pairs = rule.GetCountsCacheDuplicate()[ ClientDuplicatesAutoResolution.DUPLICATE_STATUS_MATCHES_SEARCH_PASSED_TEST_READY_TO_ACTION ]
+        
+        def pending_pairs_full_up():
+            
+            if rule.GetOperationMode() == ClientDuplicatesAutoResolution.DUPLICATES_AUTO_RESOLUTION_RULE_OPERATION_MODE_WORK_BUT_NO_ACTION:
+                
+                max_pending_pairs = rule.GetMaxPendingPairs()
+                
+                if max_pending_pairs is not None and current_num_pending_pairs >= max_pending_pairs:
+                    
+                    return True
+                    
+                
+            
+            return False
+            
+        
         pair_to_work = get_row()
 
         while pair_to_work is not None:
 
+            if pending_pairs_full_up():
+                
+                return False
+                
+            
             ( smaller_media_id, larger_media_id ) = pair_to_work
 
             smaller_hash_id = self.modules_files_duplicates.GetBestKingId( smaller_media_id, db_location_context = db_location_context )
@@ -145,6 +167,8 @@ def get_row():
 
             if operation_mode == ClientDuplicatesAutoResolution.DUPLICATES_AUTO_RESOLUTION_RULE_OPERATION_MODE_WORK_BUT_NO_ACTION:
 
+                current_num_pending_pairs += 1
+                
                 self.modules_files_duplicates_auto_resolution_storage.SetPairToPendingAction( rule, smaller_media_id, larger_media_id, hash_id_a, hash_id_b )
 
             else: