S3

• garage
• MinIO
• oxyno-zeta/s3-proxy
  • Apache-2.0, Go
  • Reverse Proxy with GET, PUT and DELETE methods and authentication
• gaul/s3proxy
  • Apache-2.0, Java
• S3 Proxy with overlay
  • Switch-TV/s3proxy/overlay-middleware
• dandanthedev/littletinystorage
  • GPLv3, TS
  • S3 兼容接口
  • 可以作为实现的参考
• glacier - 冷存储, 90 天 删除
• pottava/aws-s3-proxy
  • MIT, Go

Limits

bucket name

(?!(^xn--|.+-s3alias$))^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$
(?!(^((2(5[0-5]|[0-4][0-9])|[01]?[0-9]{1,2})\.){3}(2(5[0-5]|[0-4][0-9])|[01]?[0-9]{1,2})$|^xn--|.+-s3alias$))^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$

• key
  • UTF8
  • 安全字符 [0-9a-zA-z!_.*'()-]
• docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html
• docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
  • BucketName 规则
• S3 Limits
• minio Thresholds and Limits

headers

• x-amz-meta-
• x-amz-acl
• x-amz-server-side-encryption-
• x-amz-server-side-encryption
• x-amz-storage-class

content-type
cache-control
content-encoding
content-disposition
content-language
x-amz-website-redirect-location

Content-MD5: ${BASE64}

# SigV4 - Signature Version 4
# UNSIGNED-PAYLOAD 不计算 payload hash
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-content-sha256: ${hex(sha256)}

# 自动处理
x-amz-sdk-checksum: SHA256

• docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

API

### List Objects V2
GET /?list-type=2&continuation-token=ContinuationToken&delimiter=Delimiter&encoding-type=EncodingType&fetch-owner=FetchOwner&max-keys=MaxKeys&prefix=Prefix&start-after=StartAfter HTTP/1.1
Host: Bucket.s3.amazonaws.com
x-amz-request-payer: RequestPayer
x-amz-expected-bucket-owner: ExpectedBucketOwner
x-amz-optional-object-attributes: OptionalObjectAttributes

• ObjectIdentifier
  • Key
    • XML 需要做 escape
  • ETag
  • LastModifiedTime
  • Size
  • VersionId
• write once, read many (WORM) model
  • Lock
• S3 API compatibility
  • docs.openio.io/latest/source/arch-design/s3_compliancy.html
  • docs.riak.com/riak/cs/2.1.1/references/apis/storage/s3/index.html
  • docs.ceph.com/en/latest/radosgw/s3
  • Garage garagehq.deuxfleurs.fr/documentation/reference-manual/s3-compatibility
  • Cloudflare R2 developers.cloudflare.com/r2/api/s3/api
  • MinIO min.io/docs/minio/linux/reference/s3-api-compatibility.html
  • 阿里云 help.aliyun.com/zh/oss/developer-reference/compatibility-with-amazon-s3
• 参考
  • docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html
  • min.io/docs/minio/linux/developers/javascript/API.html

Profile

• AWS_PROFILE
• ~/.aws/credentials
• %USERPROFILE%\.aws\credentials
• ~/.aws/config
• docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

[your_profile_name]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

aws --profile project01 --endpoint-url http://localhost:9000 s3 ls s3://my-bucket01/

FAQ

• Aliyun
  • s3.oss-cn-hongkong.aliyuncs.com
  • 内网 s3.oss-cn-hongkong-internal.aliyuncs.com
  • help.aliyun.com/zh/oss/developer-reference/compatibility-with-amazon-s3-1
• Testing
  • buckets.grayhatwarfare.com
  • registry.opendata.aws
  • aws.amazon.com/opendata

API

• listObjects
  • 如果传入了 delimiter, 会返回 CommonPrefixes, 类似于目录列表
• docs.aws.amazon.com/s3
• docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRequestHeaders.html