Scope replication-failure read-only to repo/project, not the whole server (#1305)

Motivation:

Central Dogma is a multi-tenant store: a single replica hosts many
projects and repositories belonging to unrelated tenants. When a
follower fails to replay a single log (state divergence, lock
acquisition timeout, or transient I/O error), the existing failure path
puts the whole server into read-only mode. A problem in one repository
then propagates to every other tenant on that replica: writes stop
everywhere until an operator investigates and lifts the server-wide
read-only state.

The goal of this change is to shrink that blast radius. A replay failure
flips only the smallest enclosing scope (repository, project, or server)
to read-only, and the executor keeps replicating unrelated logs for
everyone else. The read-only state is persisted so it survives restarts
— in `dogma/dogma` for repo/project scope, and in a properties file for
server scope. The existing administrative API can lift it once the
operator has investigated.

Modifications:

- Add `UpdateProjectStatusCommand` and `UpdateRepositoryStatusCommand`,
routed through the same replication pipeline as ordinary commands so
scope changes converge across the cluster. Their effect lands in
`dogma/dogma/status/{project}/{repo}.json` — the same storage as
ordinary commands, intentionally: a replay failure of the status update
itself is then treated as a `dogma/dogma` failure and escalates to
server scope.
- Carry `commandType`, `projectName`, and `repoName` on every `LogMeta`.
- Add `ReplicationLogContext` attached to `ReplicationException` so
`ZooKeeperCommandExecutor.handleReplicationFailure()` can pick the
read-only scope:
  - null project name (root commands or `dogma/dogma`) → server
- non-null project name with null repo (or the dogma metadata repo) →
project
  - otherwise → repository
- Add `RepoStatusManager` to manage the read-only status of projects and
repositories; remove `MetadataService.updateRepositoryStatus()` and
migrate callers.
- On a replay error, skip past the failing log (so subsequent unrelated
logs keep replaying) and issue the scope-appropriate
`Update(Project|Repository)StatusCommand`. Fall back to a server-wide
read-only state only if recovery itself fails.
- Wire `StandaloneCommandExecutor` to consult `RepoStatusManager` on
every push, so a per-repo / per-project read-only state rejects writes
with `ReadOnlyException` before reaching the storage layer.
- Rename `DefaultCrud*` → `ReplicatingCrud*` and introduce
`StandaloneCrud*` to bypass replication when the standalone executor
applies state updates directly.
- Move `ServerStatusManager` to the internal package.
- Add `ReplicationStatus` to describe the status of projects and
repositories, and migrate `RepositoryStatus` in `RepositoryDto` to
`ReplicationStatus`.

Result:

- A replication failure no longer puts the whole server into read-only
mode; only the affected project or repository becomes read-only.

Author: ikhoon

.gitignore

@@ -141,5 +141,11 @@ typings/
 # macOS folder meta-data
 .DS_Store
 
+# Claude
+.claude/
+
 # Codex
 AGENTS.md
+
+# IntelliJ AI agent
+.junie/

common/src/main/java/com/linecorp/centraldogma/common/ReplicationStatus.java

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2026 LY Corporation
+ *
+ * LY Corporation licenses this file to you under the Apache License,
+ * version 2.0 (the "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at:
+ *
+ *   https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
+ * under the License.
+ */
+
+package com.linecorp.centraldogma.common;
+
+/**
+ * The replication status of a project or repository. It indicates whether the project or repository is active
+ * and writable or in read-only mode.
+ */
+public enum ReplicationStatus {
+
+    /**
+     * The project or repository is active and writable. It can accept write operations and is being replicated
+     * to other nodes.
+     */
+    WRITABLE,
+
+    /**
+     * The project or repository is in read-only mode. It cannot accept write operations and is not being
+     * replicated to other nodes.
+     */
+    READ_ONLY
+}

common/src/main/java/com/linecorp/centraldogma/common/RepositoryStatus.java

@@ -17,7 +17,11 @@
 
 /**
  * The status of a repository.
+ *
+ * @deprecated Use {@link ReplicationStatus} instead.
  */
+// TODO(ikhoon): Remove this enum in the future.
+@Deprecated
 public enum RepositoryStatus {
 
     /**

common/src/main/java/com/linecorp/centraldogma/common/jsonpatch/JsonPatchOperation.java

@@ -229,6 +229,9 @@ public static ReplaceOperation replace(String path, JsonNode value) {
     /**
      * Creates a new JSON Patch {@code safeReplace} operation.
      *
+     * <p>This operation is similar to {@link #replace(JsonPointer, JsonNode)}, but it throws an error if the
+     * path does not have the expected value.
+     *
      * @param path the JSON Pointer for this operation
      * @param oldValue the old value to replace
      * @param newValue the new value to replace the old value
@@ -240,6 +243,9 @@ public static SafeReplaceOperation safeReplace(JsonPointer path, JsonNode oldVal
     /**
      * Creates a new JSON Patch {@code safeReplace} operation.
      *
+     * <p>This operation is similar to {@link #replace(String, JsonNode)}, but it throws an error if the
+     * path does not have the expected value.
+     *
      * @param path the JSON Pointer for this operation
      * @param oldValue the old value to replace
      * @param newValue the new value to replace the old value

common/src/main/java/com/linecorp/centraldogma/internal/api/v1/RepositoryDto.java

@@ -32,7 +32,7 @@
 import com.google.common.base.MoreObjects;
 
 import com.linecorp.centraldogma.common.Author;
-import com.linecorp.centraldogma.common.RepositoryStatus;
+import com.linecorp.centraldogma.common.ReplicationStatus;
 import com.linecorp.centraldogma.common.Revision;
 
 @JsonInclude(Include.NON_NULL)
@@ -58,7 +58,7 @@ public static RepositoryDto removed(String name) {
     private final String createdAt;
 
     @Nullable
-    private final RepositoryStatus status;
+    private final ReplicationStatus status;
 
     RepositoryDto(String name) {
         this.name = requireNonNull(name, "name");
@@ -70,7 +70,7 @@ public static RepositoryDto removed(String name) {
     }
 
     public RepositoryDto(String projectName, String repoName, Author creator, Revision headRevision,
-                         long creationTimeMillis, RepositoryStatus status) {
+                         long creationTimeMillis, ReplicationStatus status) {
         this(requireNonNull(repoName, "repoName"), requireNonNull(creator, "creator"),
              requireNonNull(headRevision, "headRevision"),
              PROJECTS_PREFIX + '/' + requireNonNull(projectName, "projectName") + REPOS + '/' + repoName,
@@ -83,7 +83,7 @@ public RepositoryDto(@JsonProperty("name") String name,
                          @JsonProperty("headRevision") @Nullable Revision headRevision,
                          @JsonProperty("url") @Nullable String url,
                          @JsonProperty("createdAt") @Nullable String createdAt,
-                         @JsonProperty("status") @Nullable RepositoryStatus status) {
+                         @JsonProperty("status") @Nullable ReplicationStatus status) {
         this.name = requireNonNull(name, "name");
         this.creator = creator;
         this.headRevision = headRevision;
@@ -123,7 +123,7 @@ public String createdAt() {
 
     @Nullable
     @JsonProperty("status")
-    public RepositoryStatus status() {
+    public ReplicationStatus status() {
         return status;
     }
 

server/src/main/java/com/linecorp/centraldogma/server/CentralDogma.java

@@ -165,17 +165,18 @@
 import com.linecorp.centraldogma.server.internal.api.sysadmin.MirrorAccessControlService;
 import com.linecorp.centraldogma.server.internal.api.sysadmin.ServerStatusService;
 import com.linecorp.centraldogma.server.internal.api.variable.VariableServiceV1;
+import com.linecorp.centraldogma.server.internal.management.RepoStatusManager;
+import com.linecorp.centraldogma.server.internal.management.ServerStatusManager;
 import com.linecorp.centraldogma.server.internal.mirror.DefaultMirrorAccessController;
 import com.linecorp.centraldogma.server.internal.mirror.DefaultMirroringServicePlugin;
 import com.linecorp.centraldogma.server.internal.mirror.MirrorAccessControl;
 import com.linecorp.centraldogma.server.internal.mirror.MirrorRunner;
 import com.linecorp.centraldogma.server.internal.replication.ZooKeeperCommandExecutor;
 import com.linecorp.centraldogma.server.internal.storage.project.DefaultProjectManager;
 import com.linecorp.centraldogma.server.internal.storage.project.ProjectApiManager;
-import com.linecorp.centraldogma.server.internal.storage.repository.CrudRepository;
-import com.linecorp.centraldogma.server.internal.storage.repository.git.GitCrudRepository;
+import com.linecorp.centraldogma.server.internal.storage.repository.crud.CrudRepository;
+import com.linecorp.centraldogma.server.internal.storage.repository.crud.ReplicatingCrudRepository;
 import com.linecorp.centraldogma.server.management.ServerStatus;
-import com.linecorp.centraldogma.server.management.ServerStatusManager;
 import com.linecorp.centraldogma.server.metadata.MetadataService;
 import com.linecorp.centraldogma.server.mirror.MirrorProvider;
 import com.linecorp.centraldogma.server.mirror.MirroringServicePluginConfig;
@@ -304,6 +305,8 @@ public static CentralDogma forConfig(File configFile) throws IOException {
     @Nullable
     private ServerStatusManager statusManager;
     @Nullable
+    private RepoStatusManager repoStatusManager;
+    @Nullable
     private InternalProjectInitializer projectInitializer;
     @Nullable
     private volatile MirrorRunner mirrorRunner;
@@ -620,20 +623,22 @@ private CommandExecutor startCommandExecutor(
         }
 
         statusManager = new ServerStatusManager(cfg.dataDir());
+        repoStatusManager = new RepoStatusManager(statusManager, pm);
         logger.info("Startup mode: {}", statusManager.serverStatus());
         final CommandExecutor executor;
         final ReplicationMethod replicationMethod = cfg.replicationConfig().method();
         switch (replicationMethod) {
             case ZOOKEEPER:
-                executor = newZooKeeperCommandExecutor(pm, repositoryWorker, statusManager, meterRegistry,
+                executor = newZooKeeperCommandExecutor(pm, repositoryWorker, statusManager, repoStatusManager,
+                                                       meterRegistry,
                                                        sessionManager, encryptionStorageManager,
                                                        onTakeLeadership, onReleaseLeadership,
                                                        onTakeZoneLeadership, onReleaseZoneLeadership);
                 break;
             case NONE:
                 logger.info("No replication mechanism specified; entering standalone");
-                executor = new StandaloneCommandExecutor(pm, repositoryWorker, statusManager, sessionManager,
-                                                         encryptionStorageManager,
+                executor = new StandaloneCommandExecutor(pm, repositoryWorker, statusManager, repoStatusManager,
+                                                         sessionManager, encryptionStorageManager,
                                                          onTakeLeadership, onReleaseLeadership,
                                                          onTakeZoneLeadership, onReleaseZoneLeadership);
                 break;
@@ -647,6 +652,7 @@ private CommandExecutor startCommandExecutor(
         executor.setWritable(initialServerStatus.writable());
         if (!initialServerStatus.replicating()) {
             projectInitializer.initializeInReadOnlyMode();
+            repoStatusManager.initialize();
             setMirrorAccessControllerRepository(pm, executor);
             return executor;
         }
@@ -669,19 +675,21 @@ private CommandExecutor startCommandExecutor(
             // Trigger the exception if any.
             startFuture.get();
             projectInitializer.initialize();
+            repoStatusManager.initialize();
         } catch (Exception e) {
             logger.warn("Failed to start the command executor. Entering read-only.", e);
             projectInitializer.initializeInReadOnlyMode();
+            repoStatusManager.initialize();
         }
         setMirrorAccessControllerRepository(pm, executor);
         return executor;
     }
 
     private void setMirrorAccessControllerRepository(ProjectManager pm, CommandExecutor executor) {
         final CrudRepository<MirrorAccessControl> accessControlRepository =
-                new GitCrudRepository<>(MirrorAccessControl.class, executor, pm,
-                                        INTERNAL_PROJECT_DOGMA, Project.REPO_DOGMA,
-                                        MIRROR_ACCESS_CONTROL_PATH);
+                new ReplicatingCrudRepository<>(MirrorAccessControl.class, executor, pm,
+                                               INTERNAL_PROJECT_DOGMA, Project.REPO_DOGMA,
+                                               MIRROR_ACCESS_CONTROL_PATH);
         mirrorAccessController.setRepository(accessControlRepository);
     }
 
@@ -924,6 +932,7 @@ private AuthProvider createAuthProvider(
     private CommandExecutor newZooKeeperCommandExecutor(
             ProjectManager pm, Executor repositoryWorker,
             ServerStatusManager serverStatusManager,
+            RepoStatusManager repoStatusManager,
             MeterRegistry meterRegistry,
             @Nullable SessionManager sessionManager,
             EncryptionStorageManager encryptionStorageManager,
@@ -945,8 +954,8 @@ private CommandExecutor newZooKeeperCommandExecutor(
         //                so that we can recover from ZooKeeper maintenance automatically.
         return new ZooKeeperCommandExecutor(
                 zkCfg, dataDir,
-                new StandaloneCommandExecutor(pm, repositoryWorker, serverStatusManager, sessionManager,
-                                              encryptionStorageManager,
+                new StandaloneCommandExecutor(pm, repositoryWorker, serverStatusManager, repoStatusManager,
+                                              sessionManager, encryptionStorageManager,
                         /* onTakeLeadership */ null, /* onReleaseLeadership */ null,
                         /* onTakeZoneLeadership */ null, /* onReleaseZoneLeadership */ null),
                 meterRegistry, zone,
@@ -1008,12 +1017,13 @@ private void configureHttpApi(ServerBuilder sb,
             decoratorBuilder.build(decorator);
         }
 
-        assert statusManager != null;
+        assert statusManager != null && repoStatusManager != null;
         final ContextPathServicesBuilder apiV1ServiceBuilder = sb.contextPath(API_V1_PATH_PREFIX);
         apiV1ServiceBuilder
                 .annotatedService(new ServerStatusService(executor, statusManager))
                 .annotatedService(new ProjectServiceV1(projectApiManager, executor))
-                .annotatedService(new RepositoryServiceV1(executor, mds, encryptionStorageManager))
+                .annotatedService(new RepositoryServiceV1(executor, mds, encryptionStorageManager,
+                                                          repoStatusManager))
                 .annotatedService(new CredentialServiceV1(projectApiManager, executor))
                 .annotatedService(new VariableServiceV1(pm, executor));
         if (LOGBACK_ENABLED) {
@@ -1241,6 +1251,7 @@ private void doStop() {
         final ExecutorService purgeWorker = this.purgeWorker;
         final SessionManager sessionManager = this.sessionManager;
         final MirrorRunner mirrorRunner = this.mirrorRunner;
+        final ServerStatusManager statusManager = this.statusManager;
 
         this.server = null;
         this.executor = null;
@@ -1249,13 +1260,18 @@ private void doStop() {
         this.repositoryWorker = null;
         this.sessionManager = null;
         this.mirrorRunner = null;
+        this.statusManager = null;
         if (meterRegistryToBeClosed != null) {
             assert meterRegistry instanceof CompositeMeterRegistry;
             ((CompositeMeterRegistry) meterRegistry).remove(meterRegistryToBeClosed);
             meterRegistryToBeClosed.close();
             meterRegistryToBeClosed = null;
         }
 
+        if (statusManager != null) {
+            statusManager.close();
+        }
+
         logger.info("Stopping the Central Dogma ..");
         if (!doStop(server, executor, pm, repositoryWorker, purgeWorker, sessionManager, mirrorRunner,
                     encryptionStorageManager)) {

server/src/main/java/com/linecorp/centraldogma/server/command/Command.java

@@ -30,6 +30,7 @@
 import com.linecorp.centraldogma.common.Author;
 import com.linecorp.centraldogma.common.Change;
 import com.linecorp.centraldogma.common.Markup;
+import com.linecorp.centraldogma.common.ReplicationStatus;
 import com.linecorp.centraldogma.common.Revision;
 import com.linecorp.centraldogma.server.EncryptionConfig;
 import com.linecorp.centraldogma.server.auth.Session;
@@ -64,6 +65,8 @@
         @Type(value = RemoveSessionCommand.class, name = "REMOVE_SESSIONS"),
         @Type(value = CreateSessionMasterKeyCommand.class, name = "CREATE_SESSION_MASTER_KEY"),
         @Type(value = UpdateServerStatusCommand.class, name = "UPDATE_SERVER_STATUS"),
+        @Type(value = UpdateProjectStatusCommand.class, name = "UPDATE_PROJECT_STATUS"),
+        @Type(value = UpdateRepositoryStatusCommand.class, name = "UPDATE_REPOSITORY_STATUS"),
         @Type(value = ForcePushCommand.class, name = "FORCE_PUSH_COMMAND"),
 })
 public interface Command<T> {
@@ -331,8 +334,8 @@ static Command<Void> migrateToEncryptedRepository(@Nullable Long timestamp, Auth
      * @param changes the changes to be applied
      */
     static Command<Revision> push(Author author, String projectName, String repositoryName,
-                                      Revision baseRevision, String summary, String detail,
-                                      Markup markup, Change<?>... changes) {
+                                  Revision baseRevision, String summary, String detail,
+                                  Markup markup, Change<?>... changes) {
 
         return push(null, author, projectName, repositoryName, baseRevision, summary, detail, markup, changes);
     }
@@ -478,6 +481,34 @@ static Command<Void> updateServerStatus(ServerStatus serverStatus) {
         return new UpdateServerStatusCommand(null, null, serverStatus);
     }
 
+    /**
+     * Returns a new {@link Command} which is used to update the status of a project.
+     */
+    static Command<Void> updateProjectStatus(String projectName, ReplicationStatus projectStatus) {
+        requireNonNull(projectName, "projectName");
+        requireNonNull(projectStatus, "projectStatus");
+        return new UpdateProjectStatusCommand(null, null, projectName, projectStatus);
+    }
+
+    /**
+     * Returns a new {@link Command} which is used to update the status of a repository.
+     */
+    static Command<Void> updateRepositoryStatus(String projectName, String repositoryName,
+                                                ReplicationStatus replicationStatus) {
+        return updateRepositoryStatus(projectName, repositoryName, Author.SYSTEM, replicationStatus);
+    }
+
+    /**
+     * Returns a new {@link Command} which is used to update the status of a repository.
+     */
+    static Command<Void> updateRepositoryStatus(String projectName, String repositoryName, Author author,
+                                                ReplicationStatus replicationStatus) {
+        requireNonNull(projectName, "projectName");
+        requireNonNull(repositoryName, "repositoryName");
+        requireNonNull(replicationStatus, "replicationStatus");
+        return new UpdateRepositoryStatusCommand(null, author, projectName, repositoryName, replicationStatus);
+    }
+
     /**
      * Returns a new {@link Command} which is used to force-push {@link Command} even the server is in
      * read-only mode. This command is useful for migrating the repository content during maintenance mode.

server/src/main/java/com/linecorp/centraldogma/server/command/CommandExecutorStatusManager.java

@@ -21,8 +21,8 @@
 
 import com.google.common.base.MoreObjects;
 
+import com.linecorp.centraldogma.server.internal.management.ServerStatusManager;
 import com.linecorp.centraldogma.server.management.ServerStatus;
-import com.linecorp.centraldogma.server.management.ServerStatusManager;
 
 /**
  * Manages the status of a {@link CommandExecutor}.

server/src/main/java/com/linecorp/centraldogma/server/command/CommandType.java

@@ -47,6 +47,8 @@ public enum CommandType {
     PURGE_PROJECT(Void.class),
     PURGE_REPOSITORY(Void.class),
     UPDATE_SERVER_STATUS(Void.class),
+    UPDATE_PROJECT_STATUS(Void.class),
+    UPDATE_REPOSITORY_STATUS(Void.class),
     // The result type of FORCE_PUSH is Object because it can be any type.
     FORCE_PUSH(Object.class);
 

server/src/main/java/com/linecorp/centraldogma/server/command/ProjectCommand.java

@@ -30,7 +30,7 @@
  *
  * @param <T> the result type of a {@link Command}
  */
-abstract class ProjectCommand<T> extends AbstractCommand<T> {
+public abstract class ProjectCommand<T> extends AbstractCommand<T> {
 
     private final String projectName;
 
@@ -40,6 +40,9 @@ abstract class ProjectCommand<T> extends AbstractCommand<T> {
         this.projectName = requireNonNull(projectName, "projectName");
     }
 
+    /**
+     * Returns the project name.
+     */
     @JsonProperty
     public final String projectName() {
         return projectName;