Adding Direct S3 Uploads (original) (raw)

This walkthrough shows how to add asynchronous uploads to a Rails app. The flow will go like this:

1. User selects file(s)
2. For each file a presign request is made to fetch AWS S3 upload parameters
3. Files are uploaded asynchronously to AWS S3
4. Uploaded file JSON data is written to a hidden field
5. Form is submitted instantaneously as it only has to submit the JSON data
6. JSON data is assigned to the Shrine attachment attribute

AWS S3 setup

You'll need to create an AWS S3 bucket, which is where the uploads will be stored. See this walkthrough on how to do that.

Next you'll need to configure CORS for that bucket, so that it accepts uploads directly from the client. In the AWS S3 Console go to your bucket, click on the "Permissions" tab and then on "CORS configuration". There paste in the following:

Replace https://my-app.com with the URL to your app (in development you can set this to *). Once you've hit "Save", it may take some time for the new CORS settings to be applied.

1. Installation

Add Shrine and aws-sdk-s3 to the Gemfile:

Gemfile

gem "shrine", "> 3.0" gem "aws-sdk-s3", "> 1.14"

and run bundle install.

2. Initializer

Add your S3 credentials to your application:

s3: bucket: "" region: "" access_key_id: "" secret_access_key: ""

...

Then create an initializer which configures your S3 storage with those credentials and loads default plugins:

config/initializers/shrine.rb

require "shrine" require "shrine/storage/s3"

s3_options = Rails.application.credentials.s3

Shrine.storages = { cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options), store: Shrine::Storage::S3.new(**s3_options), }

Shrine.plugin :activerecord # load Active Record integration Shrine.plugin :cached_attachment_data # for retaining cached file on form redisplays Shrine.plugin :restore_cached_data # refresh metadata for cached files

3. Migration

Add the <attachment>_data text or JSON column to the table to which you want to add the attachment:

$ rails generate migration add_image_data_to_photos image_data:text # or :jsonb

This should generate the following migration:

class AddImageDataToPhotos < ActiveRecord::Migration def change add_column :photos, :image_data, :text # or :jsonb end end

Run rails db:migrate to apply the migration.

4. Attachment

Create an uploader for the types of files you'll be uploading:

app/uploaders/image_uploader.rb

class ImageUploader < Shrine end

and add an attachment attribute to your model:

app/models/photo.rb

class Photo < ApplicationRecord include ImageUploader::Attachment(:image)

validates_presence_of :image end

5. Form

In your form you can now add form fields for the attachment attribute, and an image tag for the preview:

<%= form_for @photo do |f| %>

In your controller make sure to allow the attachment param:

app/controllers/photos_controller.rb

class PhotosController < ApplicationController

...

def create @photo = Photo.new(photo_params)

if @photo.save
  redirect_to @photo
else
  render :new
end

end

...

private

def photo_params params.require(:photo).permit(..., :image) # permit attachment param end end

Now you should have working synchronous file uploads.

6. Direct upload

We can now add asynchronous direct uploads to the mix. We'll be using Uppyand its AwsS3 plugin, which will upload selected files directly to S3 using params fetched from Shrine's presign endpoint.

6a. Presign endpoint

We'll first configure the presign_endpoint plugin and mount it in our routes:

config/initializers/shrine.rb

...

Shrine.plugin :presign_endpoint, presign_options: -> (request) {

Uppy will send the "filename" and "type" query parameters

filename = request.params["filename"] type = request.params["type"]

{ content_disposition: ContentDisposition.inline(filename), # set download filename content_type: type, # set content type (defaults to "application/octet-stream") content_length_range: 0..(1010241024), # limit upload size to 10 MB } }

config/routes.rb

Rails.application.routes.draw do

...

mount Shrine.presign_endpoint(:cache) => "/s3/params" end

6b. Uppy

Now we can setup Uppy to do the direct uploads. First we'll add the package to our bundle (we're assuming you're using webpacker):

Now we can setup direct uploads to S3, where upload params request will be fetched from Shrine's presign endpoint, and upload result will be written to the hidden attachment field:

// app/javascript/fileUpload.js

import 'uppy/dist/uppy.min.css'

import { Core, FileInput, Informer, ProgressBar, ThumbnailGenerator, AwsS3, } from 'uppy'

function fileUpload(fileInput) { const hiddenInput = document.querySelector('.upload-data'), imagePreview = document.querySelector('.upload-preview img'), formGroup = fileInput.parentNode

// remove our file input in favour of Uppy's formGroup.removeChild(fileInput)

const uppy = Core({ autoProceed: true, }) .use(FileInput, { target: formGroup, }) .use(Informer, { target: formGroup, }) .use(ProgressBar, { target: imagePreview.parentNode, }) .use(ThumbnailGenerator, { thumbnailWidth: 600, }) .use(AwsS3, { companionUrl: '/', // will call the presign endpoint on /s3/params })

uppy.on('thumbnail:generated', (file, preview) => { // show preview of the image using URL from thumbnail generator imagePreview.src = preview })

uppy.on('upload-success', (file, response) => { // construct uploaded file data in the format that Shrine expects const uploadedFileData = { id: file.meta['key'].match(/^cache/(.+)/)[1], // object key without prefix storage: 'cache', metadata: { size: file.size, filename: file.name, mime_type: file.type, } }

// set hidden field value to the uploaded file data so that it's submitted
// with the form as the attachment
hiddenInput.value = JSON.stringify(uploadedFileData)

}) }

export default fileUpload

You may wish to initialize on turbolinks (or turbo) loading: or alternatively, using a stimulus controller:

(A) Initialize on Turbo load

// app/javascript/packs/application.js // ... import fileUpload from 'fileUpload'

// if you are using turbo, listen on turbo:load instead // listen on 'turbolinks:load' instead of 'DOMContentLoaded' if using Turbolinks document.addEventListener('DOMContentLoaded', () => { document.querySelectorAll('.upload-file').forEach(fileInput => { fileUpload(fileInput) }) })

(B) Initialize using a Stimulus Controller

• Multiple Upload Controller and see the ruby code needed in the view.
• Single Upload Controller and see the ruby code in the view.

And add the following CSS:

/* app/assets/stylesheets/application.css / / ... */ .upload-preview img { display: block; max-width: 100%; }

.upload-preview { margin-bottom: 10px; display: inline-block; height: 300px; }

img[src=""] { visibility: hidden; }

And that's it, now when a file is selected it will be asynchronously uploaded to your app, accompanied by an image preview and a nice progress bar.

Restricting Access to Presign Endpoint

You may want to restrict the access to your presign endpoint to protect your s3 bucket.

One way to do this (if you are using Rails) is to use constraints on your rails route (e.g., example using headers, or via plain ol' ruby):

Rails.application.routes.draw do # need a true statement with the constraint - in this case that a header exists using a lambda
mount Shrine.presign_endpoint(:cache) => "/s3/params", constraints: -> (req) { req.headers["HEADER_THAT_SHOULD_EXIST"] }

  # or you can use a ruby class
  mount Shrine.presign_endpoint(:cache), :constraints => AdminConstraint.new # only admins should get access

  # or you can route requests to a separate controller where you can return whatever rack response you design

end

./app/lib/admin_constraint.rb

class AdminConstraint def matches?(request) return false unless request.session[:user_id] user = User.find request.session[:user_id] user && user.site_admin? # only logged in admin users should be able to upload end end

See Also

• Adding Direct App Uploads
• Adding Resumable Uploads
• uppy-s3_multipart
• Demo