improvement(decode_backtrace): improve logic #10646
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Current logic for decoding backtrace was locking db log processing for duration of getting path to scylla debug file. Also it was downloading debug file once - possibly resulting in wrong backtrace decode in upgrade tests. Moved logic for getting debug file info to decoding queue, so it's not locking db log processing loop. Also downloading debug file to monitor node (where decoding is done) to `/tmp/debug_<build_id>` and skipping download if it's already there. closes: scylladb/qa-tasks#57
soyacz
added
backport/2024.1
Contributor
Author
|
@dimakr ping |
This was referenced Apr 29, 2025
scylladbbot
added
backport/perf-v16-done
backport/2025.1-done
backport/2024.2-done
fruch
added a commit
to fruch/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string
2 tasks
soyacz
pushed a commit
that referenced
this pull request
May 12, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string
scylladbbot
pushed a commit
to scylladbbot/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
2 tasks
scylladbbot
pushed a commit
to scylladbbot/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
2 tasks
scylladbbot
pushed a commit
to scylladbbot/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
2 tasks
scylladbbot
pushed a commit
to scylladbbot/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
2 tasks
scylladbbot
pushed a commit
to scylladbbot/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
2 tasks
scylladbbot
pushed a commit
to scylladbbot/scylla-cluster-tests
that referenced
this pull request
May 12, 2025
scylladb#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
2 tasks
vponomaryov
pushed a commit
that referenced
this pull request
May 12, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
vponomaryov
pushed a commit
that referenced
this pull request
May 12, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
vponomaryov
pushed a commit
that referenced
this pull request
May 12, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
vponomaryov
pushed a commit
that referenced
this pull request
May 12, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
vponomaryov
pushed a commit
that referenced
this pull request
May 12, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
fruch
added a commit
that referenced
this pull request
May 13, 2025
#10646 introduced some unsafe code the when debug info doesn't it fail like the following: ``` 20:58:29 raise Exception("Couldn't find scylla debug information") 20:58:29 Exception: Couldn't find scylla debug information 20:58:29 20:58:29 During handling of the above exception, another exception occurred: 20:58:29 20:58:29 Traceback (most recent call last): 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner 20:58:29 self.run() 20:58:29 File "/usr/local/lib/python3.10/threading.py", line 953, in run 20:58:29 self._target(*self._args, **self._kwargs) 20:58:29 File "/home/ubuntu/scylla-cluster-tests/sdcm/cluster.py", line 1600, in decode_backtrace 20:58:29 if "is closed" in details: 20:58:29 TypeError: argument of type 'Exception' is not iterable ``` the mistake was to treat Exception as string, with turn it into a string (cherry picked from commit c2b8ce4)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Current logic for decoding backtrace was locking db log processing for duration of getting path to scylla debug file. Also it was downloading debug file once - possibly resulting in wrong backtrace decode in upgrade tests.
Moved logic for getting debug file info to decoding queue, so it's not locking db log processing loop. Also downloading debug file to monitor node (where decoding is done) to
/tmp/debug_<build_id>and skipping download if it's already there.closes: https://github.com/scylladb/qa-tasks/issues/57
Testing
PR pre-checks (self review)
backportlabelsReminders
sdcm/sct_config.py)unit-test/folder)