Using droplet S3 as a backingstore for backups.

The droplet S3 storage backend writes chunks of data in an S3 bucket.

For this you need to install the the bareos-storage-droplet packages which contains
the libbareossd-chunked*.so  and  libbareossd-droplet*.so shared objects and the droplet storage backend which implements a dynamic loaded storage backend.

In the following example all the backup data is placed in the "bareos-backup" bucket on the defined S3 storage.
A volume is a sub-directory in the defined bucket, and every chunk is placed in the Volume directory with the filename 0000-9999 and a size that is defined in the chunksize.

The droplet S3 can only be used with virtual-hosted-style buckets like http://<bucket>.<s3_server>/object
Path-style buckets are not supported when using the droplet S3.

On the Storage Daemon the following configuration is needed.
Example bareos-sd.d/device/S3_ObjectStorage.conf file:

Device {
  Name = S3_ObjectStorage
  Media Type = S3_Object1
  Archive Device = S3 Object Storage

  #
  # Device Options:
  #    profile= - Droplet profile path, e.g. /etc/bareos/bareos-sd.d/droplet/droplet.profile
  #    location= - AWS location (e.g. us-east etc.). Optional.
  #    acl= - Canned ACL
  #    storageclass= - Storage Class to use.
  #    bucket= - Bucket to store objects in.
  #    chunksize= - Size of Volume Chunks (minimum = default = 10 Mb)
  #    iothreads= - Number of IO-threads to use for upload (use blocking uploads if not defined)
  #    ioslots= - Number of IO-slots per IO-thread (0-255, default 10)
  #    retries= - Number of retires if a write fails (0-255, default = 0, which means unlimited retries)
  #    mmap - Use mmap to allocate Chunk memory instead of malloc().
  #

  # testing:
  Device Options = "profile=/etc/bareos/bareos-sd.d/droplet/droplet.profile,bucket=bareos-bucket,chunksize=100M,iothreads=0,retries=1"

  # performance:
  #Device Options = "profile=/etc/bareos/bareos-sd.d/droplet/droplet.profile,bucket=bareos-bucket,chunksize=100M"

  Device Type = droplet
  LabelMedia = yes                    # lets Bareos label unlabeled media
  Random Access = yes
  AutomaticMount = yes                # when device opened, read it
  RemovableMedia = no
  AlwaysOpen = no
  Description = "S3 device"
  Maximum File Size = 500M            # 500 MB (allows for seeking to small portions of the Volume)
  Maximum Concurrent Jobs = 1
  Maximum Spool Size = 15000M
}

The droplet.profile file holds the credentials for S3 storage
Example /etc/bareos/bareos-sd.d/droplet/droplet.profile file:

Make sure the file is only readable for bareos, credentials for S3 are listed here.

Config options profile:

use_https = True
host = <FQDN for S3>
access_key = <S3 access key>
secret_key = <S3 secret key>
pricing_dir = ""
backend = s3
aws_auth_sign_version = 2

If the pricing_dir is not empty,
it will create an <profile_directory>/droplet.csv file which will record all S3 operations.
See the code at https://github.com/bareos/Droplet/blob/bareos-master/libdroplet/src/pricing.c for an explanation.

The parameter "aws_auth_sign_version = 2" is for the connection to a CEPH S3 gateway.
For use with AWS S3 the aws_auth_sign_version, must be set to "4".

On the Director you connect to the Storage Device with the following configuration
Example bareos-dir.d/storage/S3_1-00.conf file:

Storage {
  Name = S3_Object
  Address  = "Replace this by the Bareos Storage Daemon FQDN or IP address"
  Password = "Replace this by the Bareos Storage Daemon director password"
  Device = S3_ObjectStorage
  Media Type = S3_Object1
}


Troubleshooting
===============

S3 Backend Unreachable
----------------------

The droplet device can run in two modes:
  * direct writing (iothreads  = 0)
  * cached writing (iothreads >= 1)

If iothreads >= 1, retries = 0 (unlimited retries) and the droplet backend (e.g. S3 storage) is not available, a job will continue running until the backend problem is fixed.
If this is the case and the job is canceled, it will only be canceled on the Director. It continues running on the Storage Daemon, until the S3 backend is available again or the Storage Daemon itself is restarted.

If iothreads >= 1, retries != 0 and the droplet backend (e.g. S3 storage) is not available, write operation will be silently discarded after the specified number of retries.
*Don't use this combination of options*.

Caching when S3 backend is not available:
This behaviour have not changed, but I fear problems can arise, if the backend is not available and all write operations are stored in memory.

The status of the cache can be determined with the "status storage=..." command.


Pending IO chunks (and inflight chunks):
```
...
Device "S3_ObjectStorage" (S3) is mounted with:
    Volume:      Full-0085
    Pool:        Full
    Media type:  S3_Object1
Backend connection is working.
Inflight chunks: 2
Pending IO flush requests:
   /Full-0085/0002 - 10485760 (try=0)
   /Full-0085/0003 - 10485760 (try=0)
   /Full-0085/0004 - 10485760 (try=0)
...
Attached Jobs: 175
...
```

If try > 0, problems did already occur. The system will continue retrying.


Status without pending IO chunks:
```
Device "S3_ObjectStorage" (S3) is mounted with:
    Volume:      Full-0084
    Pool:        Full
    Media type:  S3_Object1
Backend connection is working.
No Pending IO flush requests.
Configured device capabilities:
  EOF BSR BSF FSR FSF EOM !REM RACCESS AUTOMOUNT LABEL !ANONVOLS !ALWAYSOPEN
Device state:
  OPENED !TAPE LABEL !MALLOC APPEND !READ EOT !WEOT !EOF !NEXTVOL !SHORT MOUNTED
  num_writers=0 reserves=0 block=8
Attached Jobs:
```
