Object Storage

Object storage is one of the most fundamental storage services in cloud computing. It stores data in a flat key-value structure, offering high availability, high scalability, and low cost. The AWS S3 protocol has become the de facto standard for object storage, and nearly all cloud providers (Alibaba Cloud OSS, Tencent Cloud COS, Huawei Cloud OBS) offer S3-compatible interfaces.

Object Storage Core Concepts

Concept	Description
Bucket	A container for objects with a globally unique name
Object	The basic unit of storage, containing data and metadata
Key	The unique identifier of an object within a bucket, similar to a file path
Region	The geographic region where the bucket is located
ACL	Access Control List
Presigned URL	A signed temporary access link

Using the AWS CLI for Object Storage

Installing the AWS CLI

# Method 1: Install via pip
sudo dnf install -y python3 python3-pip
pip3 install --user awscli

# Method 2: Install via the official package
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo dnf install -y unzip
unzip awscliv2.zip
sudo ./aws/install

# Verify installation
aws --version

Configuring Credentials

# Interactive configuration
aws configure
# AWS Access Key ID: YOUR_ACCESS_KEY
# AWS Secret Access Key: YOUR_SECRET_KEY
# Default region name: ap-northeast-1
# Default output format: json

# Configuration files are located at
cat ~/.aws/credentials
cat ~/.aws/config

# Use Named Profiles to manage multiple accounts
aws configure --profile aliyun-oss

For S3-compatible services (such as Alibaba Cloud OSS), you need to specify the endpoint:

# Alibaba Cloud OSS
aws configure set default.s3.endpoint_url https://oss-cn-hangzhou.aliyuncs.com

# Or specify it with each command
aws s3 ls --endpoint-url https://oss-cn-hangzhou.aliyuncs.com

Bucket Operations

# Create a bucket
aws s3 mb s3://my-backup-bucket

# List all buckets
aws s3 ls

# List bucket contents
aws s3 ls s3://my-backup-bucket/
aws s3 ls s3://my-backup-bucket/logs/ --recursive

# View bucket size statistics
aws s3 ls s3://my-backup-bucket --recursive --summarize --human-readable

# Delete an empty bucket
aws s3 rb s3://my-backup-bucket

# Delete a bucket and all its contents
aws s3 rb s3://my-backup-bucket --force

Uploading and Downloading Files

# Upload a single file
aws s3 cp /var/log/messages s3://my-backup-bucket/logs/messages

# Download a single file
aws s3 cp s3://my-backup-bucket/logs/messages /tmp/messages

# Upload an entire directory
aws s3 cp /var/www/html/ s3://my-backup-bucket/website/ --recursive

# Download an entire directory
aws s3 cp s3://my-backup-bucket/website/ /var/www/html/ --recursive

# Use exclude and include filters
aws s3 cp /var/log/ s3://my-backup-bucket/logs/ \
  --recursive \
  --exclude "*" \
  --include "*.log"

File Synchronization

s3 sync only transfers changed files, similar to rsync:

# Sync a local directory to S3
aws s3 sync /var/www/html/ s3://my-backup-bucket/website/

# Sync from S3 to local
aws s3 sync s3://my-backup-bucket/website/ /var/www/html/

# Sync and delete extra files at the destination
aws s3 sync /var/www/html/ s3://my-backup-bucket/website/ --delete

# Exclude specific files
aws s3 sync /data/ s3://my-backup-bucket/data/ \
  --exclude "*.tmp" \
  --exclude ".git/*"

# Dry run (no actual transfers)
aws s3 sync /data/ s3://my-backup-bucket/data/ --dryrun

Other Operations

# Move a file
aws s3 mv s3://my-bucket/old-path/file.txt s3://my-bucket/new-path/file.txt

# Delete a file
aws s3 rm s3://my-backup-bucket/logs/old.log

# Delete recursively
aws s3 rm s3://my-backup-bucket/logs/ --recursive

# Generate a presigned URL (valid for 3600 seconds)
aws s3 presign s3://my-backup-bucket/files/report.pdf --expires-in 3600

# Set object ACL
aws s3api put-object-acl \
  --bucket my-backup-bucket \
  --key public/image.png \
  --acl public-read

MinIO Self-Hosted Object Storage

MinIO is a high-performance S3-compatible object storage server, ideal for private cloud and on-premises environments.

Installing the MinIO Server

# Download the MinIO binary
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

# Create the data directory
sudo mkdir -p /data/minio

# Create a MinIO system user
sudo useradd -r -s /sbin/nologin minio-user
sudo chown minio-user:minio-user /data/minio

Configuring MinIO Environment Variables

sudo cat > /etc/default/minio << 'EOF'
# MinIO data directory
MINIO_VOLUMES="/data/minio"

# Admin credentials
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=ChangeThisStrongPassword123

# Listen address
MINIO_OPTS="--console-address :9001"
EOF

Creating a Systemd Service

sudo cat > /etc/systemd/system/minio.service << 'EOF'
[Unit]
Description=MinIO Object Storage
Documentation=https://min.io/minio/linux/index.html
After=network-online.target
Wants=network-online.target

[Service]
User=minio-user
Group=minio-user
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server $MINIO_VOLUMES $MINIO_OPTS
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# Start MinIO
sudo systemctl daemon-reload
sudo systemctl enable --now minio

# Check status
sudo systemctl status minio

Configuring the Firewall

# Open the MinIO API port (9000) and console port (9001)
sudo firewall-cmd --permanent --add-port=9000/tcp
sudo firewall-cmd --permanent --add-port=9001/tcp
sudo firewall-cmd --reload

MinIO console access URL: http://your-server-ip:9001

Installing the MinIO Client

# Download the mc client
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin/

# Configure connection to local MinIO
mc alias set local http://localhost:9000 minioadmin ChangeThisStrongPassword123

# You can also configure connections to other S3-compatible services
mc alias set myaws https://s3.amazonaws.com YOUR_ACCESS_KEY YOUR_SECRET_KEY
mc alias set aliyun https://oss-cn-hangzhou.aliyuncs.com YOUR_ACCESS_KEY YOUR_SECRET_KEY

# Verify connection
mc admin info local

Using the MinIO Client

# Create buckets
mc mb local/my-bucket
mc mb local/backups

# List buckets
mc ls local

# Upload files
mc cp /etc/hosts local/my-bucket/hosts
mc cp --recursive /var/log/ local/backups/logs/

# Download files
mc cp local/my-bucket/hosts /tmp/hosts-backup

# Sync directories
mc mirror /var/www/html/ local/my-bucket/website/

# Continuous watch and sync
mc mirror --watch /var/www/html/ local/my-bucket/website/

# View object information
mc stat local/my-bucket/hosts

# Set bucket policy
mc anonymous set download local/my-bucket/public/

# Manage users
mc admin user add local newuser newpassword123
mc admin policy attach local readwrite --user newuser

# View service information
mc admin info local

MinIO Multi-Disk Deployment (Erasure Coding)

# In production, use multiple disks for erasure code protection
# At least 4 disks are required
sudo mkdir -p /data/minio-disk{1..4}
sudo chown minio-user:minio-user /data/minio-disk{1..4}

# Modify environment variables
sudo sed -i 's|MINIO_VOLUMES=.*|MINIO_VOLUMES="/data/minio-disk{1...4}"|' /etc/default/minio

# Restart the service
sudo systemctl restart minio

rclone Multi-Cloud Storage Management

rclone is called “rsync for cloud storage” and supports over 70 storage backends.

Installing rclone

# Method 1: Official installation script
curl https://rclone.org/install.sh | sudo bash

# Method 2: Manual installation
RCLONE_VERSION="1.67.0"
wget https://downloads.rclone.org/v${RCLONE_VERSION}/rclone-v${RCLONE_VERSION}-linux-amd64.rpm
sudo dnf install -y ./rclone-v${RCLONE_VERSION}-linux-amd64.rpm

# Verify
rclone version

Configuring Storage Backends

# Interactive configuration
rclone config

# Configuration file location
cat ~/.config/rclone/rclone.conf

You can also edit the configuration file directly:

cat > ~/.config/rclone/rclone.conf << 'EOF'
[aws-s3]
type = s3
provider = AWS
access_key_id = YOUR_AWS_ACCESS_KEY
secret_access_key = YOUR_AWS_SECRET_KEY
region = ap-northeast-1

[aliyun-oss]
type = s3
provider = Alibaba
access_key_id = YOUR_ALICLOUD_ACCESS_KEY
secret_access_key = YOUR_ALICLOUD_SECRET_KEY
endpoint = oss-cn-hangzhou.aliyuncs.com

[tencent-cos]
type = s3
provider = TencentCOS
access_key_id = YOUR_TENCENT_SECRET_ID
secret_access_key = YOUR_TENCENT_SECRET_KEY
endpoint = cos.ap-guangzhou.myqcloud.com

[local-minio]
type = s3
provider = Minio
access_key_id = minioadmin
secret_access_key = ChangeThisStrongPassword123
endpoint = http://localhost:9000
EOF

rclone Basic Operations

# List all configured remotes
rclone listremotes

# List buckets
rclone lsd aws-s3:

# List files in a bucket
rclone ls aws-s3:my-bucket/
rclone lsl aws-s3:my-bucket/  # Including time and size

# Copy a file
rclone copy /var/log/messages aws-s3:my-bucket/logs/

# Copy a directory
rclone copy /var/www/html/ aws-s3:my-bucket/website/ --progress

# Sync (mirror, deletes extra files at destination)
rclone sync /var/www/html/ aws-s3:my-bucket/website/ --progress

# Move files
rclone move /tmp/uploads/ aws-s3:my-bucket/uploads/

# Delete
rclone delete aws-s3:my-bucket/old-logs/
rclone purge aws-s3:my-bucket/temp-bucket/  # Delete path and all contents

Cross-Cloud Transfers

# Copy from AWS S3 to Alibaba Cloud OSS
rclone copy aws-s3:source-bucket/data/ aliyun-oss:dest-bucket/data/ --progress

# Sync from Alibaba Cloud OSS to local MinIO
rclone sync aliyun-oss:my-bucket/ local-minio:backup-bucket/ --progress

# Compare differences between two remotes
rclone check aws-s3:my-bucket/ aliyun-oss:my-bucket/

rclone Advanced Features

# Bandwidth limiting
rclone copy /data/ aws-s3:my-bucket/ --bwlimit 10M

# Encrypted storage (create an encrypted remote)
rclone config
# Select the "crypt" type and specify the underlying remote

# Mount as a local filesystem
sudo dnf install -y fuse3
rclone mount aws-s3:my-bucket /mnt/s3 --daemon --allow-other --vfs-cache-mode full

# Run the mount as a service
cat > ~/.config/systemd/user/rclone-mount.service << 'UNIT'
[Unit]
Description=rclone S3 mount
After=network-online.target

[Service]
ExecStart=/usr/bin/rclone mount aws-s3:my-bucket /mnt/s3 --allow-other --vfs-cache-mode full
ExecStop=/bin/fusermount -uz /mnt/s3
Restart=on-failure

[Install]
WantedBy=default.target
UNIT

systemctl --user enable --now rclone-mount

# View remote space usage
rclone about aws-s3:

Backing Up to Object Storage

Database Backup Script

/usr/local/bin/backup-db.sh

#!/bin/bash
# MySQL/MariaDB backup to object storage

set -euo pipefail

# Configuration
BACKUP_DIR="/tmp/db-backup"
BUCKET="s3://my-backup-bucket/database"
DB_NAME="myapp"
DATE=$(date +%Y%m%d-%H%M%S)
RETENTION_DAYS=30

# Create temporary directory
mkdir -p "${BACKUP_DIR}"

# Perform backup
echo "[$(date)] Starting backup of database ${DB_NAME}..."
mysqldump --single-transaction \
  --routines \
  --triggers \
  --databases "${DB_NAME}" | \
  gzip > "${BACKUP_DIR}/${DB_NAME}-${DATE}.sql.gz"

# Upload to object storage
echo "[$(date)] Uploading backup to object storage..."
aws s3 cp "${BACKUP_DIR}/${DB_NAME}-${DATE}.sql.gz" \
  "${BUCKET}/${DB_NAME}-${DATE}.sql.gz"

# Clean up local temporary files
rm -f "${BACKUP_DIR}/${DB_NAME}-${DATE}.sql.gz"

# Clean up expired remote backups
echo "[$(date)] Cleaning up backups older than ${RETENTION_DAYS} days..."
CUTOFF_DATE=$(date -d "-${RETENTION_DAYS} days" +%Y%m%d)
aws s3 ls "${BUCKET}/" | while read -r line; do
  FILE_DATE=$(echo "$line" | grep -oP '\d{8}' | head -1)
  FILE_NAME=$(echo "$line" | awk '{print $NF}')
  if [[ -n "${FILE_DATE}" && "${FILE_DATE}" < "${CUTOFF_DATE}" ]]; then
    echo "  Deleting expired backup: ${FILE_NAME}"
    aws s3 rm "${BUCKET}/${FILE_NAME}"
  fi
done

echo "[$(date)] Backup complete"

Incremental System File Backup

/usr/local/bin/backup-files.sh

#!/bin/bash
# Incremental backup to object storage using rclone

set -euo pipefail

REMOTE="aliyun-oss"
BUCKET="server-backups"
HOSTNAME=$(hostname)
LOG_FILE="/var/log/backup-files.log"

echo "[$(date)] Starting file backup..." >> "${LOG_FILE}"

# Sync configuration files
rclone sync /etc/ "${REMOTE}:${BUCKET}/${HOSTNAME}/etc/" \
  --exclude "shadow" \
  --exclude "gshadow" \
  --log-file="${LOG_FILE}" \
  --log-level INFO

# Sync website data
rclone sync /var/www/ "${REMOTE}:${BUCKET}/${HOSTNAME}/www/" \
  --log-file="${LOG_FILE}" \
  --log-level INFO

# Sync user data
rclone sync /home/ "${REMOTE}:${BUCKET}/${HOSTNAME}/home/" \
  --exclude ".cache/**" \
  --exclude ".local/share/Trash/**" \
  --log-file="${LOG_FILE}" \
  --log-level INFO

echo "[$(date)] Backup complete" >> "${LOG_FILE}"

Setting Up Scheduled Backups

# Set up daily database backup
sudo chmod +x /usr/local/bin/backup-db.sh
sudo cat > /etc/cron.d/db-backup << 'EOF'
# Run database backup daily at 2 AM
0 2 * * * root /usr/local/bin/backup-db.sh >> /var/log/backup-db.log 2>&1
EOF

# Set up daily incremental file backup
sudo chmod +x /usr/local/bin/backup-files.sh
sudo cat > /etc/cron.d/file-backup << 'EOF'
# Run file backup daily at 3 AM
0 3 * * * root /usr/local/bin/backup-files.sh
EOF

Using Lifecycle Policies for Automatic Cleanup

For AWS S3 or compatible services, you can configure lifecycle policies to automatically clean up old backups:

# Create a lifecycle rule configuration
cat > /tmp/lifecycle.json << 'EOF'
{
  "Rules": [
    {
      "ID": "DeleteOldBackups",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "database/"
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "ArchiveOldFiles",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "files/"
      },
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}
EOF

# Apply the lifecycle policy
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-backup-bucket \
  --lifecycle-configuration file:///tmp/lifecycle.json

Object storage is the foundational storage solution for the cloud era. Once you master the AWS CLI, MinIO, and rclone, you can flexibly manage data across public cloud, private cloud, and hybrid cloud environments, and build reliable backup strategies.