Unable to delete or unarchive archived sketch with open indexes

[cfiguereosupransc]

Describe the bug

The tsctl list-sketches --archived-with-open-indexes command lists a sketch that cannot be deleted or unarchived.

To Reproduce

First, run tsctl list-sketches --archived-with-open-indexes to list improperly archived sketches.

Pick a sketch, e.g., 1346:

...
Sketch ID: 1346, Name: 'openrelik-workflow-1679' (status: archived)
  - Timeline: 'REDACTED' (ID: 22099), SearchIndex DB: 'c59705f1748f40a686acfd72365d9883' (ID: 6770), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22100), SearchIndex DB: '312d2159ace047f3a816eaef45575ae1' (ID: 6771), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22101), SearchIndex DB: '23c2a800c40d42a9a283f891d3061e53' (ID: 6772), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22102), SearchIndex DB: '81fe31037f454f878214b1b7abf5eefa' (ID: 6773), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22103), SearchIndex DB: '48651579eee340abb427948ee812642a' (ID: 6774), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22104), SearchIndex DB: '5a2a7c76e03d4c7f9e8c4101402e3cfb' (ID: 6775), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22105), SearchIndex DB: '4c5caa64679d4850a25d48c8ba571756' (ID: 6776), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22106), SearchIndex DB: 'bdb0900d62214f16a91f522a298fec3e' (ID: 6777), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22107), SearchIndex DB: '3f65910cf1cf40aba459122a5314ac1a' (ID: 6778), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22108), SearchIndex DB: 'b1b2f5c2e2744a7c9c436d14ed811642' (ID: 6779), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22109), SearchIndex DB: '1663520a8cef45d78b15bd7104bc3713' (ID: 6780), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22110), SearchIndex DB: '325d388d18bc4f25841896a88ba9f50b' (ID: 6781), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22111), SearchIndex DB: '5ca9f2fb6fb7469cae4bb20b7a3b1bc4' (ID: 6782), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22112), SearchIndex DB: 'ada089d51dd64cdb82ac1205a1808d00' (ID: 6783), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22113), SearchIndex DB: 'b4466f6461de47b093a04a6fa85ae0aa' (ID: 6784), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22114), SearchIndex DB: '1f3578fabfe7466ea7604c723f8c5512' (ID: 6785), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22115), SearchIndex DB: 'b5a9640867bb409c9a688a4e0aa56220' (ID: 6786), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22116), SearchIndex DB: '3f6c99fffd2c4100a4b92ce82164331b' (ID: 6787), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22117), SearchIndex DB: 'f749eef9105b4c10b2803fbab1a97e2d' (ID: 6788), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22118), SearchIndex DB: '8b80ebe5d1044dbc9e7529b9147c7ba4' (ID: 6789), DB Status: 'fail' (Inconsistent)
  - Timeline: 'REDACTED' (ID: 22119), SearchIndex DB: 'f6d1368b8f65466f8393184d5f190503' (ID: 6790), DB Status: 'fail' (Inconsistent)
...

tsctl sketch-info 1346 output:

Sketch 1346 Name: (openrelik-workflow-1679)

Timelines:
ID    Name     Search Index ID Index Name                       Created At                 User ID Description Status
22099 REDACTED 6770            c59705f1748f40a686acfd72365d9883 2025-08-16 21:02:38.611088 27      REDACTED    fail
22100 REDACTED 6771            312d2159ace047f3a816eaef45575ae1 2025-08-16 21:02:39.405920 27      REDACTED    fail
22101 REDACTED 6772            23c2a800c40d42a9a283f891d3061e53 2025-08-16 21:02:40.021361 27      REDACTED    fail
22102 REDACTED 6773            81fe31037f454f878214b1b7abf5eefa 2025-08-16 21:02:40.849856 27      REDACTED    fail
22103 REDACTED 6774            48651579eee340abb427948ee812642a 2025-08-16 21:02:41.112181 27      REDACTED    fail
22104 REDACTED 6775            5a2a7c76e03d4c7f9e8c4101402e3cfb 2025-08-16 21:02:41.728080 27      REDACTED    fail
22105 REDACTED 6776            4c5caa64679d4850a25d48c8ba571756 2025-08-16 21:02:42.518436 27      REDACTED    fail
22106 REDACTED 6777            bdb0900d62214f16a91f522a298fec3e 2025-08-16 21:02:42.788205 27      REDACTED    fail
22107 REDACTED 6778            3f65910cf1cf40aba459122a5314ac1a 2025-08-16 21:02:43.060954 27      REDACTED    fail
22108 REDACTED 6779            b1b2f5c2e2744a7c9c436d14ed811642 2025-08-16 21:02:43.350720 27      REDACTED    fail
22109 REDACTED 6780            1663520a8cef45d78b15bd7104bc3713 2025-08-16 21:02:43.655723 27      REDACTED    fail
22110 REDACTED 6781            325d388d18bc4f25841896a88ba9f50b 2025-08-16 21:02:43.972187 27      REDACTED    fail
22111 REDACTED 6782            5ca9f2fb6fb7469cae4bb20b7a3b1bc4 2025-08-16 21:02:44.297946 27      REDACTED    fail
22112 REDACTED 6783            ada089d51dd64cdb82ac1205a1808d00 2025-08-16 21:02:44.618743 27      REDACTED    fail
22113 REDACTED 6784            b4466f6461de47b093a04a6fa85ae0aa 2025-08-16 21:02:44.956738 27      REDACTED    fail
22114 REDACTED 6785            1f3578fabfe7466ea7604c723f8c5512 2025-08-16 21:02:45.353359 27      REDACTED    fail
22115 REDACTED 6786            b5a9640867bb409c9a688a4e0aa56220 2025-08-16 21:02:49.646196 27      REDACTED    fail
22116 REDACTED 6787            3f6c99fffd2c4100a4b92ce82164331b 2025-08-16 21:02:50.909598 27      REDACTED    fail
22117 REDACTED 6788            f749eef9105b4c10b2803fbab1a97e2d 2025-08-16 21:02:51.300847 27      REDACTED    fail
22118 REDACTED 6789            8b80ebe5d1044dbc9e7529b9147c7ba4 2025-08-16 21:02:51.702260 27      REDACTED    fail
22119 REDACTED 6790            f6d1368b8f65466f8393184d5f190503 2025-08-16 21:02:52.135449 27      REDACTED    fail

Data Sources per Timeline:

Timeline: REDACTED (ID: 22099)
ID    File Path Status Error Message
27583           fail   Index 'c59705f1748f40a686acfd72365d9883' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22100)
ID    File Path Status Error Message
27584           fail   Index '312d2159ace047f3a816eaef45575ae1' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22101)
ID    File Path Status Error Message
27585           fail   Index '23c2a800c40d42a9a283f891d3061e53' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22102)
ID    File Path Status Error Message
27586           fail   Index '81fe31037f454f878214b1b7abf5eefa' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22103)
ID    File Path Status Error Message
27587           fail   Index '48651579eee340abb427948ee812642a' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22104)
ID    File Path Status Error Message
27588           fail   Index '5a2a7c76e03d4c7f9e8c4101402e3cfb' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22105)
ID    File Path Status Error Message
27589           fail   Index '4c5caa64679d4850a25d48c8ba571756' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22106)
ID    File Path Status Error Message
27590           fail   Index 'bdb0900d62214f16a91f522a298fec3e' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22107)
ID    File Path Status Error Message
27591           fail   Index '3f65910cf1cf40aba459122a5314ac1a' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22108)
ID    File Path Status Error Message
27592           fail   Index 'b1b2f5c2e2744a7c9c436d14ed811642' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22109)
ID    File Path Status Error Message
27593           fail   Index '1663520a8cef45d78b15bd7104bc3713' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22110)
ID    File Path Status Error Message
27594           fail   Index '325d388d18bc4f25841896a88ba9f50b' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22111)
ID    File Path Status Error Message
27595           fail   Index '5ca9f2fb6fb7469cae4bb20b7a3b1bc4' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22112)
ID    File Path Status Error Message
27596           fail   Index 'ada089d51dd64cdb82ac1205a1808d00' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22113)
ID    File Path Status Error Message
27597           fail   Index 'b4466f6461de47b093a04a6fa85ae0aa' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22114)
ID    File Path Status Error Message
27598           fail   Index '1f3578fabfe7466ea7604c723f8c5512' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22115)
ID    File Path Status Error Message
27599           fail   Index 'b5a9640867bb409c9a688a4e0aa56220' was created but did not become ready within the timeout period of 10s.
27600           fail   Index 'b5a9640867bb409c9a688a4e0aa56220' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22116)
ID    File Path Status Error Message
27601           fail   Index '3f6c99fffd2c4100a4b92ce82164331b' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22117)
ID    File Path Status Error Message
27602           fail   Index 'f749eef9105b4c10b2803fbab1a97e2d' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22118)
ID    File Path Status Error Message
27603           fail   Index '8b80ebe5d1044dbc9e7529b9147c7ba4' was created but did not become ready within the timeout period of 10s.

Timeline: REDACTED (ID: 22119)
ID    File Path Status Error Message
27604           fail   Index 'f6d1368b8f65466f8393184d5f190503' was created but did not become ready within the timeout period of 10s.

Created by: REDACTED
Shared with:
        Users: (user_id, username, access_level)
                5: REDACTED (read/write)
        Groups (1): (group_name, access_level)
                REDACTED (read/write)
Sketch Status: archived
Sketch is public: True
Sketch Labels: ([],)
Status:
id   status   created_at                 user_id
4219 archived 2025-11-10 23:54:48.029938 None

The bug here is that we're unable to use the Timesketch UI or API to get rid of the offending sketch.

1. If we click "Delete Sketch" in the UI, the page displays error "Unable to delete a sketch that is already archived."

2. If we click "BRING IT BACK" in the UI, the page displays error "Server side error. Please contact your server administrator for troubleshooting."

3. If we try to delete the sketch via the Python API, we get error:

...
  File "/Users/cfiguereosupran/Dev/doodle/main.py", line 70, in delete
    sketch.delete()
  File "/Users/cfiguereosupran/Dev/doodle/.venv/lib/python3.11/site-packages/timesketch_api_client/sketch.py", line 491, in delete
    raise RuntimeError(
RuntimeError: Unable to delete an archived sketch, first unarchive then delete.

4. If we try to unarchive the sketch via the Python API, we get error:

Failed response: [500] Failed to unarchive sketch. One or more OpenSearch indices could not be opened. Details: Unarchiving sketch 1346, but it contains timeline  (ID: 22099) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22100) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22101) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22102) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22103) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22104) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22105) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22106) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22107) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22108) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22109) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22110) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22111) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22112) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22113) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22114) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22115) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22116) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22117) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22118) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.; Unarchiving sketch 1346, but it contains timeline  (ID: 22119) in a 'fail' state. It is recommended to fix this timeline (e.g., by deleting it) because the sketch cannot be archived again in this state. You can use 'tsctl find-inconsistent-archives' to find such sketches.
Error: Failed to unarchive sketch ID 1346.

5. If we try to delete timelines from the sketch, we get:

...
  File "/Users/cfiguereosupran/Dev/doodle/main.py", line 95, in delete_timeline
    timeline = sketch.get_timeline(timeline_id=timeline_id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cfiguereosupran/Dev/doodle/.venv/lib/python3.12/site-packages/timesketch_api_client/sketch.py", line 852, in get_timeline
    raise RuntimeError("Unable to get timelines on an archived sketch.")
RuntimeError: Unable to get timelines on an archived sketch.

Expected behavior

The sketch is successfully unarchived, then successfully deleted. (Or, we're open to any other approach that would get rid of the sketch.)

Screenshots

n/a

Desktop (please complete the following information):

• OS: macOS
• Browser: Chrome
• Version: 143.0.7499.170

Additional context

@daschwanden asked us to file this issue.

[jaegeral]

Taking a look, thank you for reporting.

[jaegeral]

First glance, did you try to do a sketch.delete(force_delete=True)? Edit: okay that will likely also not work because we check if an sketch is archived first, if yes it will not even reach the check if force_delete was set.

[jkppr]

Here is some additional information to assist with debugging and fixing this:

• Clicking "Delete" in the UI only performs a soft-delete (marking the sketch as deleted in the DB). It does not delete the actual data nor close the index. Therefore, I do not recommend using the UI for this specific case.

  • This behavior is being addressed in a separate issue.

• When using the API to delete a sketch, you must use sketch.delete(force_delete=True) as mentioned by @jaegeral above. Please note that this requires the user to have the admin role on the instance.

  • Reference: tsctl guide
  • Alternatively, you can use the timesketch cli sketch delete tool if you prefer not to write a custom API script.

Assessment of the deadlock

The root cause of this deadlock is the presence of timelines with a "fail" status within an archived sketch.

When we introduced the rule preventing the archiving of sketches with failed timelines (since the original status cannot be determined upon unarchiving), we also added backwards compatibility logic. This was intended to handle legacy cases where an instance already contained archived sketches with failed timelines prior to the update.

It appears this backwards compatibility logic is failing in this instance.

Recommendation

We need to ensure the unarchive method correctly handles these legacy scenarios. It should allow the unarchiving of sketches where a timeline is in a fail (or otherwise broken) state by simply leaving those timelines as-is without attempting to restore them. Once the sketch is successfully unarchived, it should be possible to delete it.

@jaegeral will look into the fix. Thanks

[jkppr]

@cfiguereosupransc alright, with the changes made in #3656 you can escape the deadlock now. So once this change hits your instance you can un-archvie these sketches and then delete them properly.

With #3657 you can also use the CLI client to delete prior soft-deleted sketches now.

Please note that a permanent deletion (wipe) that includes removing the index on your OpenSearch cluster only works with the Timesketch CLI client and as admin user out of the box. But you can use the same API calls to write your own deletion script with the API client.

Please keep us posted if this fixed the deadlock, thanks.