WAL-G

• wal-g/wal-gGitHubwal-g/wal-gwal-g/wal-g 是什么？ · Apache 2.0, lzo 部分 GPL 3.0+ · PostgreSQL 备份、恢复工具 · WAL-E 后继 - Golang 重写 Python - 性能 x4 · 支持 PostgreSQL, MySQL/MariaDB, Mongo, FoundationDB, Redis, Greenplum笔记：WAL-G 是什么？
  • Apache 2.0, lzo 部分 GPL 3.0+
  • PostgreSQL 备份、恢复工具
  • WAL-E 后继 - Golang 重写 Python - 性能 x4
  • 支持 PostgreSQL, MySQL/MariaDB, Mongo, FoundationDB, Redis, Greenplum
  • 支持 S3, GCS, Azure, Swift, 本地, SSH 存储
  • PostgreSQLGitHub Filewal-g/wal-g/PostgreSQL.md
• WAL-E 不支持关闭 S3 SSE - SelfHost 常见,不便于使用
• 参考
  • camptocamp/wal-g-prometheus-exporterGitHubcamptocamp/wal-g-prometheus-exporterwal-g/wal-g 是什么？ · Apache 2.0, lzo 部分 GPL 3.0+ · PostgreSQL 备份、恢复工具 · WAL-E 后继 - Golang 重写 Python - 性能 x4 · 支持 PostgreSQL, MySQL/MariaDB, Mongo, FoundationDB, Redis, Greenplum笔记：WAL-G
    • 监控指标

env	desc
WALG_S3_PREFIX	s3://bucket/path/to/folder
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY	AWS_SESSION_TOKEN,~/.aws/credentials,AWS_PROFILE
AWS_REGION	us-west-2
AWS_ENDPOINT	s3-like-service
AWS_S3_FORCE_PATH_STYLE	false http://BUCKET.s3.amazonaws.com/KEY -> http://s3.amazonaws.com/BUCKET/KEY
WALG_FILE_PREFIX	/tmp/wal-g-test-data
WALG_SSH_PREFIX	ssh://localhost/walg-folder
SSH_PORT
SSH_USERNAME
SSH_PASSWORD
TOTAL_BG_UPLOADED_LIMIT	32
WALG_S3_STORAGE_CLASS	STANDARD
WALG_S3_SSE	false
WALG_S3_SSE_KMS_ID
WALG_COMPRESSION_METHOD	lz4,lzma,brotli
S3_USE_LIST_OBJECTS_V1	false

# Minio
AWS_ACCESS_KEY_ID: '<minio-key>'
AWS_SECRET_ACCESS_KEY: '<minio-secret>'
WALG_S3_PREFIX: 's3://my-minio-bucket/sub-dir'
AWS_ENDPOINT: 'http://minio:9000'
AWS_S3_FORCE_PATH_STYLE: 'true'
AWS_REGION: us-east-1
WALG_S3_CA_CERT_FILE: '/path/to/custom/ca/file'

docker pull bitnami/wal-g

# --pretty --json --detail
wal-g backup-list

# --confirm
# retain, before, everything, target
wal-g delete

目录结构

• /
  • basebackup_005/
    • base_000000020000000000000005_backup_stop_sentinel.json
    • base_000000020000000000000005/
      • metadata.json
      • tar_partitions/
        • pg_control.tar.lz4
        • part_001.tar.lz4
  • wal_005/
    • 00000002.history.lz4
    • 000000020000000000000005.lz4
    • 000000020000000000000007.partial.lz4
    • 000000020000000000000005.00000028.backup.lz4

PostgreSQL

• PG 配置 PGHOST, PGPORT, PGUSER, PGPASSWORD, PGPASSFILE, ~/.pgpass
• Gitlab 文档 PostgreSQL Backups: WAL-G

| env | | ---------------------------- | --- | | WALG_DISK_RATE_LIMIT | | WALG_NETWORK_RATE_LIMIT | | WALG_DOWNLOAD_CONCURRENCY | | WALG_PREFETCH_DIR | | WALG_UPLOAD_CONCURRENCY | 16 | | WALG_UPLOAD_DISK_CONCURRENCY | | TOTAL_BG_UPLOADED_LIMIT | | WALG_SENTINEL_USER_DATA | | WALG_PREVENT_WAL_OVERWRITE | | WALG_DELTA_MAX_STEPS | | WALG_DELTA_ORIGIN | | WALG_TAR_SIZE_THRESHOLD | | WALG_TAR_DISABLE_FSYNC | | WALG_PG_WAL_SIZE | | WALG_UPLOAD_WAL_METADATA | | WALG_ALIVE_CHECK_INTERVAL | | WALG_STOP_BACKUP_TIMEOUT |

Redis

• 只能是全量
• backup-push, backup-list, backup-fetch, delete

WALG_STREAM_CREATE_COMMAND='redis_cli --rdb /dev/stdout'
WALG_STREAM_RESTORE_COMMAND='cat > /var/lib/redis/dump.rdb'

• redis-cli >= 6.2 使用 redis_cli.shGitHub Filewal-g/wal-g/redis_cli.sh
  • redis-cli --rdb /dev/stdout > /dev/null

操作

• wal-e Primary CommandsGitHubwal-e/wal-e

# Push base backup
wal-g backup-push /data
# Push WAL 片段
wal-g wal-push /data

postgresql.conf

wal_level = archive # hot_standby and logical in 9.x is also acceptable
archive_mode = on
archive_command = 'envdir /etc/wal-e.d/env wal-e wal-push %p'
archive_timeout = 60

wal-g backup-fetch /data LATEST
wal-g backup-fetch /data base_LONGWALNUMBER_POSITION_NUMBER

recovery.conf

# https://www.postgresql.org/docs/11/recovery-config.html
restore_command = 'envdir /etc/wal-e.d/env wal-e wal-fetch %f %p'
standby_mode = on

# 提升 standby
# https://www.postgresql.org/docs/current/app-pg-ctl.html
pg_ctl promote