Clarify Empty region directory recovery workflow for standalone deployments

[yimeng]

Summary

The current documentation for table metadata reconciliation describes the Empty region directory recovery path, but it is unclear whether the documented Recovery Mode + ADMIN reconcile_* workflow applies to standalone deployments.

In a standalone GreptimeDB v1.1.1 environment, I was unable to use the documented workflow to recover from Empty region directory metadata/data inconsistency.

Environment

• GreptimeDB: v1.1.1
• Deployment mode: standalone
• Storage: local file storage
• WAL provider: raft_engine

Symptom

Startup fails when a table/region still exists in metadata, but its physical region directory no longer exists:

Error: 0: Failed to start datanode
1: Unexpected, violated: Failed to open batch regions
1: Empty region directory, region_id: <region_id>(<table_id>, 0), region_dir: data/greptime/<database>/<table_id>/<table_id>_0000000000/

This matches the Empty region directory scenario described in the docs.

What I tried

1. Enable recovery_mode = true in the standalone config

I created a recovery config based on the production standalone config and added:

recovery_mode = true

However, startup still attempted to open all regions synchronously and failed with the same Empty region directory error.

I also checked:

greptime standalone start --help

but did not find any recovery / recovery_mode related option in the CLI help output.

2. Use SQL reconciliation functions

After starting the standalone instance with background region initialization enabled:

init_regions_in_background = true

I tried:

ADMIN reconcile_database('some_database');

But standalone returned:

Unsupported operation reconcile

So it was not clear how the documented reconciliation workflow should be applied in standalone mode.

3. DROP TABLE works only when table metadata is still visible

For a Mito table that was still visible in information_schema.tables, DROP TABLE could remove it.

But for metadata residue where:

SHOW TABLES FROM some_database;

returned empty, and:

SELECT * FROM information_schema.tables WHERE table_schema = 'some_database';

also returned zero rows, DROP DATABASE could still fail with:

Table info not found: greptime.some_database.some_table

In that state, normal SQL operations could not clean the remaining metadata residue.

Documentation questions

Could the docs clarify the following?

1. Is Recovery Mode supported in standalone mode, or only in distributed Metasrv/Datanode deployments?
2. If standalone supports Recovery Mode, what is the exact config key and placement? For example, should it be top-level recovery_mode = true, under a specific section, or enabled by a CLI flag?
3. Which GreptimeDB versions support this recovery workflow?
4. Are ADMIN reconcile_table, ADMIN reconcile_database, and ADMIN reconcile_catalog expected to work in standalone mode?
5. For standalone mode, what is the recommended recovery procedure when:
   • Empty region directory prevents normal startup, and
   • the problematic table is no longer visible in information_schema.tables, and
   • DROP DATABASE fails with Table info not found?
6. Is init_regions_in_background = true an expected temporary workaround to bring the instance up, or is it unsafe for this scenario?

Suggested documentation improvement

It would be helpful to add a standalone-specific section for Empty region directory, including:

• whether Recovery Mode is available in standalone;
• exact example config/command;
• limitations of ADMIN reconcile_* in standalone;
• what to do when metadata residue is not visible via SQL;
• a safe checklist for backup, temporary startup, cleanup, and verification.

[yimeng]

Additional update from further investigation and recovery attempt.

In standalone mode, I found that the effective recovery switch is the metadata runtime switch key:

__switches/recovery

Adding the following to the standalone config did not appear to enable the recovery behavior used by datanode startup:

recovery_mode = true

From the v1.1.1 source code, datanode startup checks recovery mode via RuntimeSwitchManager::recovery_mode(), which reads the metadata key above.

For a standalone deployment using raft-engine metadata, I was able to set the key offline after stopping the standalone process:

greptime cli meta put key '__switches/recovery' \
  --backend raft-engine-store \
  --store-addrs 'raftengine:///path/to/metadata' \
  --value-stdin \
  --no-validate < /dev/null

After that, starting standalone with the normal config succeeded. The instance opened the available regions and they became writable again:

Leader(Writable)  Leader  1  <all_regions>

This also explains a previous intermediate state I observed:

• init_regions_in_background = true allowed the process to become healthy from /health's point of view;
• but because open_all_regions() still failed in the background when recovery mode was not actually enabled, the code path that promotes opened regions to Leader(Writable) was not reached;
• as a result, all regions remained Follower / writable=0, and writes failed with errors like:

Region <region_id>(<table_id>, 0) is in Follower state, expect: Leader(Writable)

After setting __switches/recovery and restarting with the normal config, the regions became Leader(Writable) and writes recovered.

However, this still leaves an important operational question:

The metadata/data inconsistency is not fully cleaned up. Recovery mode only allows startup to ignore missing/nonexistent region directories. In my case there are still metadata residues for historical tables/regions, including some that are no longer visible via information_schema.tables, so normal SQL cleanup is not possible.

Could you clarify the recommended standalone procedure for the next step?

1. Is it expected to keep __switches/recovery enabled until all residue is cleaned?
2. How should standalone users safely identify and remove metadata residues for tables/regions that are not visible via SQL?
3. Is using greptime cli meta del key ... on keys such as __table_name/..., __table_info/..., __table_route/..., __dn_table/..., __topic_region/..., etc. considered safe if done from an offline full backup, or is there a higher-level supported repair command?
4. After cleanup, what is the recommended verification before deleting __switches/recovery and returning to normal startup?

This would be very useful to document, because for standalone deployments the currently documented ADMIN reconcile_* path is unavailable, while SQL cleanup may fail when table metadata is already partially missing.

[WenyXu]

Hi @yimeng Thanks for the detailed investigation.

To clarify the current behavior:

1. Recovery Mode is not supported in standalone mode. It is intended for distributed Metasrv/Datanode deployments.

2. There is no standalone config key or CLI flag that enables Recovery Mode. So recovery_mode = true in a standalone config is not expected to work.

3. ADMIN reconcile_table, ADMIN reconcile_database, and ADMIN reconcile_catalog are not supported in standalone mode.

4. For standalone deployments, if Empty region directory prevents normal startup and the problematic table metadata can no longer be cleaned up through SQL, the recommended path is to delete the corresponding table metadata manually.

   Please stop the standalone instance first and make a full backup of both the data directory and metadata directory.

   In standalone mode, the metadata KV backend is stored under:

   <storage.data_home>/metadata
   

   For example, if the standalone config contains:

[storage]
data_home = "./greptimedb_data"

then the metadata store path is:
./greptimedb_data/metadata
The CLI should point to this standalone metadata store with the raft-engine-store backend:

greptime cli meta del table \
  --backend raft-engine-store \
  --store-addrs 'raftengine:///absolute/path/to/greptimedb_data/metadata' \
  --table-id <table_id>

Or, if the table name metadata is still available:

greptime cli meta del table \
  --backend raft-engine-store \
  --store-addrs 'raftengine:///absolute/path/to/greptimedb_data/metadata' \
  --catalog-name greptime \
  --schema-name <database_name> \
  --table-name <table_name>

Note that the raftengine:// URL should use the absolute path to the standalone metadata directory.
The related documentation is here:
https://docs.greptime.cn/reference/command-lines/utilities/metadata-interaction

We should improve the documentation to make these standalone limitations explicit and add a standalone-specific recovery section.