fix: some issues with soft_delete #3657

jaegeral · 2026-01-06T14:06:18Z

Closing OpenSearch Indices on Soft Delete:
- Updated SketchResource.delete in timesketch/api/v1/resources/sketch.py to close all associated OpenSearch indices when a sketch is soft-deleted. This saves cluster resources.
Allowing Force Deletion of Soft-Deleted Sketches:
- Modified get_with_acl in timesketch/models/__init__.py to accept an include_deleted parameter.
  This allows retrieving soft-deleted objects if the user is an admin and explicitly requests it.
- Updated SketchResource.delete in timesketch/api/v1/resources/sketch.py to move the force_delete
  detection logic earlier and pass it to get_with_acl(include_deleted=force_delete). This ensures
  that admins can find and permanently delete sketches that are already soft-deleted.

And adding e2e tests for both

This PR is dependend on #3661

jaegeral · 2026-01-06T14:18:09Z

/gemini review

gemini-code-assist

Code Review

This pull request effectively addresses issues with sketch deletion by introducing index closing on soft delete and allowing admins to force-delete already soft-deleted sketches. The changes are well-implemented across the API, client, and database models. The accompanying e2e tests are thorough and provide good coverage for the new functionality. The documentation updates are also clear and will be very helpful for users. I have a few minor suggestions to improve maintainability and robustness.

gemini-code-assist · 2026-01-06T14:19:35Z

end_to_end_tests/client_test.py

+        es = opensearchpy.OpenSearch(
+            [
+                {
+                    "host": interface.OPENSEARCH_HOST,
+                    "port": interface.OPENSEARCH_PORT,
+                }
+            ],
+            http_compress=True,
+        )


To improve maintainability and reduce code duplication, consider creating a helper method to instantiate the opensearchpy.OpenSearch client. This client setup is duplicated in test_force_delete_soft_deleted_sketch on lines 500-508.

timesketch/api/v1/resources/sketch.py

gemini-code-assist · 2026-01-06T14:19:36Z

timesketch/api/v1/resources/sketch.py

+                        sketch.id,
+                        e,
+                    )
+            db_session.commit()


This db_session.commit() appears to be redundant. The call to sketch.set_status(status="deleted") on line 624 likely already handles committing the session state. Removing this would prevent an unnecessary database transaction.

jaegeral · 2026-01-07T11:09:20Z

Multiple errors like:

  [2026-01-07 09:30:33,276] timesketch.tasks/ERROR PLASO_LOG_FOLDER is set to [/var/log/timesketch/psort/] but could not be created/accessed: [Errno 13] Permission denied: '/var/log/timesketch'. Logging falling back to /dev/null.
  [2026-01-07 09:30:33,276] timesketch.tasks/INFO Plaso cmd line: ['psort.py', '-o', 'opensearch_ts', '/tmp/cd43b1ea9cb0445caab15bd6a2af8f8f', '--server', 'opensearch', '--port', '9200', '--status_view', 'none', '--index_name', 'ffa7b5ee55e9436c9eee25d72b6e4f7d', '--logfile', '/dev/null', '--opensearch_mappings', '/etc/timesketch/plaso.mappings', '--timeline_identifier', '5', '--flush_interval', '5000', '--custom_formatter_definitions', '/etc/timesketch/plaso_formatters.yaml'] start
  [2026-01-07 09:30:36,467] timesketch.tasks/INFO Plaso cmd line: ['psort.py', '-o', 'opensearch_ts', '/tmp/cd43b1ea9cb0445caab15bd6a2af8f8f', '--server', 'opensearch', '--port', '9200', '--status_view', 'none', '--index_name', 'ffa7b5ee55e9436c9eee25d72b6e4f7d', '--logfile', '/dev/null', '--opensearch_mappings', '/etc/timesketch/plaso.mappings', '--timeline_identifier', '5', '--flush_interval', '5000', '--custom_formatter_definitions', '/etc/timesketch/plaso_formatters.yaml'] finish
  [2026-01-07 09:30:36,526] celery.app.trace/INFO Task timesketch.lib.tasks.run_plaso[fc5b3eeec3fb405696531c7e4efe6a92] succeeded in 3.3619505690000153s: 'ffa7b5ee55e9436c9eee25d72b6e4f7d'
  [2026-01-07 09:30:43,600] timesketch.aggregator_interface/ERROR Unable to generate chart [hbarchart] with title [Top 100 identified tags]. The error was: Values and/or Encoding missing from data. Values type: NoneType, empty: True. Encoding type: NoneType, empty: True.
  Traceback (most recent call last):
    File "/opt/venv/lib/python3.12/site-packages/timesketch/lib/aggregators/interface.py", line 124, in to_chart
      chart_object = chart_class(
                     ^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/timesketch/lib/charts/interface.py", line 78, in __init__
      raise RuntimeError(error_message)
  RuntimeError: Values and/or Encoding missing from data. Values type: NoneType, empty: True. Encoding type: NoneType, empty: True.
  [2026-01-07 09:30:43,747] celery.worker.strategy/INFO Task timesketch.lib.tasks.run_csv_jsonl[b1a17180dfa145229dfb3a26513db252] received
  [2026-01-07 09:30:45,792] timesketch.index_api/ERROR Unable to find index: 3a678acf760d40be95957ac892cde55a
  [2026-01-07 09:30:47,770] timesketch.tasks/ERROR Unable to refresh index: 3a678acf760d40be95957ac892cde55a, not found after 5 attempts.

jaegeral · 2026-01-07T11:12:28Z

#3659 filed for the plaso error

jaegeral · 2026-01-07T11:54:52Z

[2026-01-07 11:08:58,220] timesketch.aggregator_interface/ERROR Unable to generate chart [hbarchart] with title [Top 100 identified tags]. The error was: 'Series' object has no attribute 'iteritems'
  Traceback (most recent call last):
    File "/opt/venv/lib/python3.12/site-packages/timesketch/lib/aggregators/interface.py", line 165, in to_chart
      return chart.to_html()
             ^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/altair/vegalite/v4/api.py", line 410, in to_html
      self.to_dict(),
      ^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/altair/vegalite/v4/api.py", line 363, in to_dict
      copy.data = _prepare_data(original_data, context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/altair/vegalite/v4/api.py", line 84, in _prepare_data
      data = _pipe(data, data_transformers.get())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/toolz/functoolz.py", line 628, in pipe
      data = func(data)
             ^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/toolz/functoolz.py", line 304, in __call__
      return self._partial(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/altair/vegalite/data.py", line 19, in default_data_transformer
      return curried.pipe(data, limit_rows(max_rows=max_rows), to_values)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/toolz/functoolz.py", line 628, in pipe
      data = func(data)
             ^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/toolz/functoolz.py", line 304, in __call__
      return self._partial(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/altair/utils/data.py", line 149, in to_values
      data = sanitize_dataframe(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/altair/utils/core.py", line 283, in sanitize_dataframe
      for col_name, dtype in df.dtypes.iteritems():
                             ^^^^^^^^^^^^^^^^^^^
    File "/opt/venv/lib/python3.12/site-packages/pandas/core/generic.py", line 6299, in __getattr__
      return object.__getattribute__(self, name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'Series' object has no attribute 'iteritems'

fix some issues with soft_delete

8244b2a

jaegeral self-assigned this Jan 6, 2026

jaegeral added the Backend label Jan 6, 2026

jaegeral and others added 3 commits January 6, 2026 14:10

add tests

0732751

Merge branch 'master' into 2026-01-06_find_soft_deleted_sketches

b566bca

update docs and docstrings

4da50fe

gemini-code-assist bot reviewed Jan 6, 2026

View reviewed changes

jaegeral and others added 5 commits January 6, 2026 14:23

fix lint

1ee5762

docstring

6e99247

include delted in the sketch list.

b36bddd

Merge branch 'master' into 2026-01-06_find_soft_deleted_sketches

9a26f21

modify interface.py

1763ae7

more fixes

5ae5a54

jkppr mentioned this pull request Jan 8, 2026

Unable to delete or unarchive archived sketch with open indexes #3654

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: some issues with soft_delete #3657

fix: some issues with soft_delete #3657

Uh oh!

jaegeral commented Jan 6, 2026 •

edited

Loading

Uh oh!

jaegeral commented Jan 6, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jan 6, 2026

Uh oh!

Uh oh!

gemini-code-assist bot Jan 6, 2026

Uh oh!

jaegeral commented Jan 7, 2026

Uh oh!

jaegeral commented Jan 7, 2026

Uh oh!

jaegeral commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fix: some issues with soft_delete #3657

Are you sure you want to change the base?

fix: some issues with soft_delete #3657

Uh oh!

Conversation

jaegeral commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaegeral commented Jan 6, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist bot Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

jaegeral commented Jan 7, 2026

Uh oh!

jaegeral commented Jan 7, 2026

Uh oh!

jaegeral commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jaegeral commented Jan 6, 2026 •

edited

Loading