Skip to main content

WAL-G

  • wal-g/wal-g 是什么?
    • Apache 2.0, lzo 部分 GPL 3.0+
    • PostgreSQL 备份、恢复工具
    • WAL-E 后继 - Golang 重写 Python - 性能 x4
    • 支持 PostgreSQL, MySQL/MariaDB, Mongo, FoundationDB, Redis, Greenplum
    • 支持 S3, GCS, Azure, Swift, 本地, SSH 存储
    • PostgreSQL
  • WAL-E 不支持关闭 S3 SSE - SelfHost 常见,不便于使用
  • 参考
envdesc
WALG_S3_PREFIXs3://bucket/path/to/folder
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEYAWS_SESSION_TOKEN,~/.aws/credentials,AWS_PROFILE
AWS_REGIONus-west-2
AWS_ENDPOINThttp://s3-like-service:9000
AWS_S3_FORCE_PATH_STYLEfalse
http://BUCKET.s3.amazonaws.com/KEY -> http://s3.amazonaws.com/BUCKET/KEY
WALG_FILE_PREFIX/tmp/wal-g-test-data
WALG_SSH_PREFIXssh://localhost/walg-folder
SSH_PORT
SSH_USERNAME
SSH_PASSWORD
TOTAL_BG_UPLOADED_LIMIT32
WALG_S3_STORAGE_CLASSSTANDARD
WALG_S3_SSEfalse
WALG_S3_SSE_KMS_ID
WALG_COMPRESSION_METHODlz4,lzma,brotli
S3_USE_LIST_OBJECTS_V1false
# Minio
AWS_ACCESS_KEY_ID: '<minio-key>'
AWS_SECRET_ACCESS_KEY: '<minio-secret>'
WALG_S3_PREFIX: 's3://my-minio-bucket/sub-dir'
AWS_ENDPOINT: 'http://minio:9000'
AWS_S3_FORCE_PATH_STYLE: 'true'
AWS_REGION: us-east-1
WALG_S3_CA_CERT_FILE: '/path/to/custom/ca/file'
docker pull bitnami/wal-g
# --pretty --json --detail
wal-g backup-list
namelast_modifiedwal_segment_backup_startstart_timefinish_timehostnamedata_dirpg_versionstart_lsnfinish_lsnis_permanent
base_0000000200000000000000052021-03-09T08:07:49Z000000020000000000000005Tuesday, 09-Mar-21 08:07:43 UTCTuesday, 09-Mar-21 08:07:49 UTCacid-demo/home/postgres/pgdata/pgroot/data1300028388612083927800false
base_0000000300000000000000082021-03-09T08:14:48Z000000030000000000000008Tuesday, 09-Mar-21 08:14:42 UTCTuesday, 09-Mar-21 08:14:48 UTCacid-demo/home/postgres/pgdata/pgroot/data130002134217768134250592false
# --confirm
# retain, before, everything, target
wal-g delete

目录结构

  • /
    • basebackup_005/
      • base_000000020000000000000005_backup_stop_sentinel.json
      • base_000000020000000000000005/
        • metadata.json
        • tar_partitions/
          • pg_control.tar.lz4
          • part_001.tar.lz4
    • wal_005/
      • 00000002.history.lz4
      • 000000020000000000000005.lz4
      • 000000020000000000000007.partial.lz4
      • 000000020000000000000005.00000028.backup.lz4

PostgreSQL

| env | | ---------------------------- | --- | | WALG_DISK_RATE_LIMIT | | WALG_NETWORK_RATE_LIMIT | | WALG_DOWNLOAD_CONCURRENCY | | WALG_PREFETCH_DIR | | WALG_UPLOAD_CONCURRENCY | 16 | | WALG_UPLOAD_DISK_CONCURRENCY | | TOTAL_BG_UPLOADED_LIMIT | | WALG_SENTINEL_USER_DATA | | WALG_PREVENT_WAL_OVERWRITE | | WALG_DELTA_MAX_STEPS | | WALG_DELTA_ORIGIN | | WALG_TAR_SIZE_THRESHOLD | | WALG_TAR_DISABLE_FSYNC | | WALG_PG_WAL_SIZE | | WALG_UPLOAD_WAL_METADATA | | WALG_ALIVE_CHECK_INTERVAL | | WALG_STOP_BACKUP_TIMEOUT |

Redis

  • 只能是全量
  • backup-push, backup-list, backup-fetch, delete
env
WALG_STREAM_CREATE_COMMAND
WALG_STREAM_RESTORE_COMMAND
WALG_REDIS_PASSWORD
WALG_STREAM_CREATE_COMMAND='redis_cli --rdb /dev/stdout'
WALG_STREAM_RESTORE_COMMAND='cat > /var/lib/redis/dump.rdb'
  • redis-cli >= 6.2 使用 redis_cli.sh
    • redis-cli --rdb /dev/stdout > /dev/null

操作

# Push base backup
wal-g backup-push /data
# Push WAL 片段
wal-g wal-push /data

postgresql.conf

wal_level = archive # hot_standby and logical in 9.x is also acceptable
archive_mode = on
archive_command = 'envdir /etc/wal-e.d/env wal-e wal-push %p'
archive_timeout = 60
wal-g backup-fetch /data LATEST
wal-g backup-fetch /data base_LONGWALNUMBER_POSITION_NUMBER

recovery.conf

# https://www.postgresql.org/docs/11/recovery-config.html
restore_command = 'envdir /etc/wal-e.d/env wal-e wal-fetch %f %p'
standby_mode = on
# 提升 standby
# https://www.postgresql.org/docs/current/app-pg-ctl.html
pg_ctl promote