Skip to content

Expose WAL-G runtime settings in BackupConfig #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3loymag wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Expose WAL-G runtime settings in BackupConfig #67

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,22 @@ This plugin adds backup and restore functionality to [CloudNativePG](https://clo

You should encounter new CNPG `Cluster` with encrypted WAL archivation and periodic auto-backups performed by plugin and `WAL-G`.

### WAL-G runtime settings

`BackupConfig` supports optional WAL-G runtime overrides under `spec.walg`.
Unset fields keep the plugin defaults, and existing top-level fields remain supported for backward compatibility.

```yaml
apiVersion: cnpg-extensions.yandex.cloud/v1beta1
kind: BackupConfig
spec:
walg:
compressionMethod: lz4
uploadConcurrency: 16
uploadDiskConcurrency: 8
networkRateLimitBytesPerSecond: 536870912
```

## Development

### Prerequisites
Expand Down
76 changes: 76 additions & 0 deletions api/v1beta1/backupconfig_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,8 +118,84 @@ type StorageConfig struct {
S3 *S3StorageConfig `json:"s3,omitempty"` // S3-specific parameters
}

// WalgConfig defines WAL-G runtime configuration overrides.
type WalgConfig struct {
// GOMAXPROCS value used by WAL-G. Default value is evaluated at runtime.
// +optional
GoMaxProcs *int `json:"goMaxProcs,omitempty"`

// Limit of files uploaded in background by WAL-G. Default value is evaluated at runtime.
// +optional
TotalBgUploadedLimit *int `json:"totalBgUploadedLimit,omitempty"`

// WAL-G alive check interval value, e.g. "30s". Default value is evaluated at runtime.
// +optional
AliveCheckInterval string `json:"aliveCheckInterval,omitempty"`

// Compression method used by WAL-G, e.g. "lz4", "brotli" or "zstd".
// Default value is evaluated at runtime.
// +optional
CompressionMethod string `json:"compressionMethod,omitempty"`

// Determines how many delta backups can be between full backups. Default value is evaluated at runtime.
// +optional
DeltaMaxSteps *int `json:"deltaMaxSteps,omitempty"`

// Disk read rate limit during backup creation in bytes per second. Default value is evaluated at runtime.
// +optional
DiskRateLimitBytesPerSecond *int `json:"diskRateLimitBytesPerSecond,omitempty"`

// How many goroutines to use during backup && wal downloading. Default value is evaluated at runtime.
// +optional
DownloadConcurrency *int `json:"downloadConcurrency,omitempty"`

// How many times failed file will be retried during backup / wal download. Default value is evaluated at runtime.
// +optional
DownloadFileRetries *int `json:"downloadFileRetries,omitempty"`

// WAL-G failover storages cache lifetime value, e.g. "0m". Default value is evaluated at runtime.
// +optional
FailoverStoragesCacheLifetime string `json:"failoverStoragesCacheLifetime,omitempty"`

// Whether WAL-G should check failover storages. Default value is evaluated at runtime.
// +optional
FailoverStoragesCheck *bool `json:"failoverStoragesCheck,omitempty"`

// WAL-G failover storages check size value, e.g. "1KB". Default value is evaluated at runtime.
// +optional
FailoverStoragesCheckSize string `json:"failoverStoragesCheckSize,omitempty"`

// Network upload rate limit during backup uploading in bytes per second. Default value is evaluated at runtime.
// +optional
NetworkRateLimitBytesPerSecond *int `json:"networkRateLimitBytesPerSecond,omitempty"`

// Disable calling fsync after writing files when extracting tar files. Default value is evaluated at runtime.
// +optional
TarDisableFsync *bool `json:"tarDisableFsync,omitempty"`

// Threshold in bytes is size of one backup bundle. Default value is evaluated at runtime.
// +optional
TarSizeThreshold *int64 `json:"tarSizeThreshold,omitempty"`

// How many concurrency streams to use during backup uploading. Default value is evaluated at runtime.
// +optional
UploadConcurrency *int `json:"uploadConcurrency,omitempty"`

// How many concurrency streams are reading disk during backup uploading. Default value is evaluated at runtime.
// +optional
UploadDiskConcurrency *int `json:"uploadDiskConcurrency,omitempty"`

// Whether WAL-G should prevent WAL overwrite. Default value is evaluated at runtime.
// +optional
PreventWalOverwrite *bool `json:"preventWalOverwrite,omitempty"`
}

// BackupConfigSpec defines the desired state of BackupConfig.
type BackupConfigSpec struct {
// WAL-G runtime configuration overrides
// +optional
Walg *WalgConfig `json:"walg,omitempty"`

// How many goroutines to use during backup && wal downloading. Default value is evaluated at runtime
DownloadConcurrency *int `json:"downloadConcurrency,omitempty"`

Expand Down
85 changes: 85 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

74 changes: 74 additions & 0 deletions chart/templates/crd/cnpg-extensions.yandex.cloud_backupconfigs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -504,6 +504,80 @@ spec:
description: Network upload rate limit during backup uploading in
bytes per second. Default value is evaluated at runtime
type: integer
walg:
description: WAL-G runtime configuration overrides
properties:
aliveCheckInterval:
description: WAL-G alive check interval value, e.g. "30s". Default
value is evaluated at runtime.
type: string
compressionMethod:
description: |-
Compression method used by WAL-G, e.g. "lz4", "brotli" or "zstd".
Default value is evaluated at runtime.
type: string
deltaMaxSteps:
description: Determines how many delta backups can be between
full backups. Default value is evaluated at runtime.
type: integer
diskRateLimitBytesPerSecond:
description: Disk read rate limit during backup creation in bytes
per second. Default value is evaluated at runtime.
type: integer
downloadConcurrency:
description: How many goroutines to use during backup && wal downloading.
Default value is evaluated at runtime.
type: integer
downloadFileRetries:
description: How many times failed file will be retried during
backup / wal download. Default value is evaluated at runtime.
type: integer
failoverStoragesCacheLifetime:
description: WAL-G failover storages cache lifetime value, e.g.
"0m". Default value is evaluated at runtime.
type: string
failoverStoragesCheck:
description: Whether WAL-G should check failover storages. Default
value is evaluated at runtime.
type: boolean
failoverStoragesCheckSize:
description: WAL-G failover storages check size value, e.g. "1KB".
Default value is evaluated at runtime.
type: string
goMaxProcs:
description: GOMAXPROCS value used by WAL-G. Default value is
evaluated at runtime.
type: integer
networkRateLimitBytesPerSecond:
description: Network upload rate limit during backup uploading
in bytes per second. Default value is evaluated at runtime.
type: integer
preventWalOverwrite:
description: Whether WAL-G should prevent WAL overwrite. Default
value is evaluated at runtime.
type: boolean
tarDisableFsync:
description: Disable calling fsync after writing files when extracting
tar files. Default value is evaluated at runtime.
type: boolean
tarSizeThreshold:
description: Threshold in bytes is size of one backup bundle.
Default value is evaluated at runtime.
format: int64
type: integer
totalBgUploadedLimit:
description: Limit of files uploaded in background by WAL-G. Default
value is evaluated at runtime.
type: integer
uploadConcurrency:
description: How many concurrency streams to use during backup
uploading. Default value is evaluated at runtime.
type: integer
uploadDiskConcurrency:
description: How many concurrency streams are reading disk during
backup uploading. Default value is evaluated at runtime.
type: integer
type: object
required:
- storage
type: object
Expand Down
74 changes: 74 additions & 0 deletions config/crd/bases/cnpg-extensions.yandex.cloud_backupconfigs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -504,6 +504,80 @@ spec:
description: Network upload rate limit during backup uploading in
bytes per second. Default value is evaluated at runtime
type: integer
walg:
description: WAL-G runtime configuration overrides
properties:
aliveCheckInterval:
description: WAL-G alive check interval value, e.g. "30s". Default
value is evaluated at runtime.
type: string
compressionMethod:
description: |-
Compression method used by WAL-G, e.g. "lz4", "brotli" or "zstd".
Default value is evaluated at runtime.
type: string
deltaMaxSteps:
description: Determines how many delta backups can be between
full backups. Default value is evaluated at runtime.
type: integer
diskRateLimitBytesPerSecond:
description: Disk read rate limit during backup creation in bytes
per second. Default value is evaluated at runtime.
type: integer
downloadConcurrency:
description: How many goroutines to use during backup && wal downloading.
Default value is evaluated at runtime.
type: integer
downloadFileRetries:
description: How many times failed file will be retried during
backup / wal download. Default value is evaluated at runtime.
type: integer
failoverStoragesCacheLifetime:
description: WAL-G failover storages cache lifetime value, e.g.
"0m". Default value is evaluated at runtime.
type: string
failoverStoragesCheck:
description: Whether WAL-G should check failover storages. Default
value is evaluated at runtime.
type: boolean
failoverStoragesCheckSize:
description: WAL-G failover storages check size value, e.g. "1KB".
Default value is evaluated at runtime.
type: string
goMaxProcs:
description: GOMAXPROCS value used by WAL-G. Default value is
evaluated at runtime.
type: integer
networkRateLimitBytesPerSecond:
description: Network upload rate limit during backup uploading
in bytes per second. Default value is evaluated at runtime.
type: integer
preventWalOverwrite:
description: Whether WAL-G should prevent WAL overwrite. Default
value is evaluated at runtime.
type: boolean
tarDisableFsync:
description: Disable calling fsync after writing files when extracting
tar files. Default value is evaluated at runtime.
type: boolean
tarSizeThreshold:
description: Threshold in bytes is size of one backup bundle.
Default value is evaluated at runtime.
format: int64
type: integer
totalBgUploadedLimit:
description: Limit of files uploaded in background by WAL-G. Default
value is evaluated at runtime.
type: integer
uploadConcurrency:
description: How many concurrency streams to use during backup
uploading. Default value is evaluated at runtime.
type: integer
uploadDiskConcurrency:
description: How many concurrency streams are reading disk during
backup uploading. Default value is evaluated at runtime.
type: integer
type: object
required:
- storage
type: object
Expand Down
Loading