Storage

Tips

restic/restic
- restic
- Fast, secure, efficient backup program
Rclone
- ncw/rclone
- Rclone is a command line program to sync files and directories to and from
Mountain Duck
- mount server and cloud storage as a local disk
- comparison
Cyberduck
- iterate-ch/cyberduck
- Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, OpenStack Swift, Backblaze B2, Microsoft Azure & OneDrive, Google Drive and Dropbox browser for Mac and Windows.
Cryptomator
- cryptomator/cryptomator
- Free client-side encryption for your cloud files. Open source software: No backdoors, no registration.
- Transparent encryption
- Files get encrypted individually
- cryptomator/cryptofs
  - Java Filesystem Provider with integrated encryption
Java
- lookfirst/sardine
  - an easy to use webdav client for java
- dCache/nfs4j
  - Pure Java NFSv3 and NFSv4.1 implementation
- ssh, scp and sftp for java
Golang
- spf13/afero
S3
- s3fs-fuse/s3fs-fuse
  - 限制
    - 随机写或 append 会重写整个文件
    - list 整个目录的性能较差, 取决于网络延迟
    - 最终一致性可能会导致读取过期数据
    - 重命名不是原子性的
    - 多个端挂载不会进行协调
    - 不支持硬连接
- kahing/goofys
  - Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
B2 Cloud Storage
- Pricing
- 非常便宜的云存储
- 存储 0.005$/GB/Month
- 下载 0.02$/GB
FS
- kahing/catfs
  - Catfs is a caching filesystem written in Rust
Optimize Storage Efficiency
不能使用的文件名
- Windows \/:*?"<>|
- Linux, macOS null 或 /

brew install s3fs

brew install goofys

export AWS_ACCESS_KEY_ID=${ID}
export AWS_SECRET_ACCESS_KEY=${KEY}
# goofys <bucket> <mountpoint>
# goofys <bucket:prefix> <mountpoint>

find dir/ -name "offender1" -mtime -1 -print0 | du --files0-from=- -hc | tail -n1

Transfer

# 两个服务器之间传输大量小文件时非常快
# 觉得 gzip 慢可以使用 pigz
tar c some/dir/ | gzip - | ssh host2 tar xz -C /other/dir/
# 全量同步后再进行增量同步
rsync -azv some/dir/ user@host:/other/dir/

Disk

Disk cloning

#
lsblk --output NAME,SIZE,VENDOR,FSTYPE,LABEL,UUID,MODE

# 克隆单个磁盘
dd if=/dev/sda1 of=/dev/sdb1 bs=64K conv=noerror,sync status=progress
# 克隆整个磁盘
dd if=/dev/sdX of=/dev/sdY bs=64K conv=noerror,sync status=progress

# 备份
# 备份整个系统
rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup/folder
# 使用 --info=progress2 显示总体的进度, 而不是每个文件
rsync -aAX --info=progress2 --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup/folder

# 备份启动分区
# https://wiki.archlinux.org/index.php/disk_cloning
dd if=/dev/sda1 of=/mnt/dst/bios-boot bs=64K conv=noerror,sync status=progress

# 备份分区
sfdisk -d /dev/sda > part_table
# 恢复分区
sfdisk /dev/sda < part_table
# wipefs 也可以

Storage

Tips​

Transfer​

Disk​

Tips

Transfer

Disk