Using the s3fs module for media file hosting

Traditionally, many websites store their media assets on the same server that hosts the application. In the case of Drupal, this often leads to storing files in subdirectories under /docroot/sites/default/files/. While this approach may suffice for simple and small websites, it can present challenges when dealing with a large volume of files.

If you have multiple servers for performance you have to make sure that user-generated files are persistent and can be read by all your application servers. Most hosting providers will create a custom mount for the user-generated files to mitigate this problem. If you have a managed hosting service you often will notice that they charge quite a sum for storage space which seems unusually expensive since the prices per GB should be getting lower every year.

Utilizing S3FS offers several advantages:

1. Cloud storage allows us to gradually and seamlessly scale the file storage while not upgrading an entire hosting plan. Some hosting providers offer upgrading storage space while leaving the hosting plan untouched but this is not often the case.
2. Usually cloud storage providers integrate their own CDN to help reducing load time and latency globally. 
3. Since cloud storage providers specialize in providing storage they are often more reliable and resiliant. Replicating the storage buckets across multiple data center and backup / versioning systems in place ensure that the data can recover in a failure.
4. Cloud storage is usually more cost-efficient

Selecting a Storage Provider

There are a lot of S3-compatible storage providers available. There is of course the AWS S3 storage service but also other cloud provider offer compatible space, i.e. Digital Ocean spaces. When choosing a provider make sure to scan their pricing strategy to make sure that the cost not only matches your budget now but also later in your project life-span. There are also provider who bill not only storage space but also ingress and exgress traffic so make sure to take note of this as well.

For this demonstration, I chose Backblaze.com due to its cost-effectiveness, offering a free trial with 10GB of storage. Additionally, it can be seamlessly integrated with Cloudflare.

Configuring your Drupal site

To get started with storing images and other files into your S3-bucket you will need to download and install the S3FS module: https://www.drupal.org/project/s3fs

Head to the configuration page of the module at /admin/config/media/s3fs and start configuring. You will need:

• The name of your bucket where the files will be stored
• The domain of your S3-storage
• The application access key
• The application secret key

Authenticate to your bucket

I recommend setting most of the configuration details in your environment-specific settings.php file. The keys should be only available in your environment variables and then set in the settings.php file. This allows you to have different buckets for different environments, i.e. your staging bucket does not access your production bucket.

// Assign the auth data.
$settings['s3fs.access_key'] = $_ENV["S3FS_ACCESS_KEY"];
$settings['s3fs.secret_key'] = $_ENV["S3FS_SECRET_KEY"];

// Assign the environment-specific buckets.
$config['s3fs.settings']['bucket'] = "staging";

After providing the configuration needed for the connection you can validate your connection in the "Actions" tab next to the settings tab in the configuration page.

Note: There are specific settings you can set depending on your storage provider like Bucket Region, encryption and other stuff. I will not explain every setting but only a few more which I think should be set in most cases.

Configuring the file system

I highly recommend that you make sure that you set the storage for the compiled twig files to your local storage for security reasons. You can do this also in your settings file:

$settings['php_storage']['twig']['directory'] = '../tmp/php';

You can also decide if you want to upload all public files to S3 or decide on a field base. After installing the module there is a new option available for all file fields to set the file storage to s3. Since you probably do not want to upload your local dev files to an S3 storage I recommend to leave the settings you had before and override the storage per environment as well.

See this example for field_media_image:

$config['field.storage.media.field_media_image']['settings']['uri_scheme'] = "s3";

After you set this up all new files should be uploaded to the S3 storage.

You can also set the following setting which will upload all public files to S3 storage:

$settings['s3fs.use_s3_for_public'] = TRUE;

Configuring a Custom Domain

You will notice that depending on your provider media files no longer will be served from your domain but from a domain provided by the hoster. You can set a custom domain in the S3FS settings instead but you will have to make sure that your CDN and provider can serve the media assets. 

For backblaze there is a good tutorial on their blog page: https://www.backblaze.com/blog/free-image-hosting-with-cloudflare-transform-rules-and-backblaze-b2/