Skip to main content

Storage

Tips

  • restic/restic
    • restic
    • Fast, secure, efficient backup program
  • Rclone
    • ncw/rclone
    • Rclone is a command line program to sync files and directories to and from
  • Mountain Duck
    • mount server and cloud storage as a local disk
    • comparison
  • Cyberduck
    • iterate-ch/cyberduck
    • Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, OpenStack Swift, Backblaze B2, Microsoft Azure & OneDrive, Google Drive and Dropbox browser for Mac and Windows.
  • Cryptomator
    • cryptomator/cryptomator
    • Free client-side encryption for your cloud files. Open source software: No backdoors, no registration.
    • Transparent encryption
    • Files get encrypted individually
    • cryptomator/cryptofs
      • Java Filesystem Provider with integrated encryption
  • Java
  • Golang
  • S3
    • s3fs-fuse/s3fs-fuse
      • 限制
        • 随机写或 append 会重写整个文件
        • list 整个目录的性能较差, 取决于网络延迟
        • 最终一致性可能会导致读取过期数据
        • 重命名不是原子性的
        • 多个端挂载不会进行协调
        • 不支持硬连接
    • kahing/goofys
      • Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
  • B2 Cloud Storage
    • Pricing
    • 非常便宜的云存储
    • 存储 0.005$/GB/Month
    • 下载 0.02$/GB
  • FS
    • kahing/catfs
      • Catfs is a caching filesystem written in Rust
  • Optimize Storage Efficiency
  • 不能使用的文件名
    • Windows \/:*?"<>|
    • Linux, macOS null/
brew install s3fs

brew install goofys

export AWS_ACCESS_KEY_ID=${ID}
export AWS_SECRET_ACCESS_KEY=${KEY}
# goofys <bucket> <mountpoint>
# goofys <bucket:prefix> <mountpoint>
find dir/ -name "offender1" -mtime -1 -print0 | du --files0-from=- -hc | tail -n1

Transfer

# 两个服务器之间传输大量小文件时非常快
# 觉得 gzip 慢可以使用 pigz
tar c some/dir/ | gzip - | ssh host2 tar xz -C /other/dir/
# 全量同步后再进行增量同步
rsync -azv some/dir/ user@host:/other/dir/

Disk

#
lsblk --output NAME,SIZE,VENDOR,FSTYPE,LABEL,UUID,MODE

# 克隆单个磁盘
dd if=/dev/sda1 of=/dev/sdb1 bs=64K conv=noerror,sync status=progress
# 克隆整个磁盘
dd if=/dev/sdX of=/dev/sdY bs=64K conv=noerror,sync status=progress

# 备份
# 备份整个系统
rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup/folder
# 使用 --info=progress2 显示总体的进度, 而不是每个文件
rsync -aAX --info=progress2 --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup/folder

# 备份启动分区
# https://wiki.archlinux.org/index.php/disk_cloning
dd if=/dev/sda1 of=/mnt/dst/bios-boot bs=64K conv=noerror,sync status=progress

# 备份分区
sfdisk -d /dev/sda > part_table
# 恢复分区
sfdisk /dev/sda < part_table
# wipefs 也可以