Skip to content

Move from Azure to AWS S3 for blob storage - #5023

Merged
cielf merged 4 commits into
mainfrom
move-to-s3
Feb 28, 2025
Merged

Move from Azure to AWS S3 for blob storage#5023
cielf merged 4 commits into
mainfrom
move-to-s3

Conversation

@dorner

@dorner dorner commented Feb 17, 2025

Copy link
Copy Markdown
Collaborator

This PR does the following:

  • Adds an amazon storage provider. I've set up blob storage buckets for both production and staging.
  • Changes the storage provider to amazon for staging only for now.
  • Adds a migration for staging only which downloads / uploads the contents of the blobs to the S3 bucket.
  • Updates the database backup and fetch script to use S3. There is a third bucket set up which both staging and prod can write to where the DBs will go. This bucket is set up to have a TTL of 14 days (so all files added to it will be deleted after 14 days).

My suggestion for testing:

  • Add some blobs on staging by attaching some files to various records.
  • Deploy the branch so the migration runs.
  • Validate that you can still download the existing blobs.
  • Add some more blobs and validate that they upload correctly and you can download them.

Once testing is complete, this branch is safe to merge, and we can put in a further PR that will apply these changes to prod.

@dorner
dorner requested a review from awwaiid February 17, 2025 00:18

logger.info("Copying the database...")
backup_filename = "#{current_time}.rds.dump"
backup_filename = "#{Rails.env}-#{current_time}.rds.dump"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

storage_account_name: account_name,
storage_access_key: account_key
)
client = Aws::S3::Client.new(region: 'us-east-2')

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the same as

client = ActiveStorage::Blob.services.fetch(:amazon)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically. In this case I'm going straight to the client rather than doing it in a roundabout way through ActiveStorage.

@danielabar

Copy link
Copy Markdown
Collaborator

Is CORS enabled on S3? Will be needed for direct uploads, introduced here: #4937

@dorner

dorner commented Feb 26, 2025

Copy link
Copy Markdown
Collaborator Author

Yep, it should be!

class MoveAzureBlobsToS3 < ActiveRecord::Migration[7.2]
# https://stackoverflow.com/questions/71699789/activestorage-transfer-all-assets-from-one-bucket-to-another-bucket
def up
return unless Rails.env.staging?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, so only run this migration in staging. I see.

@awwaiid awwaiid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good!

@awwaiid

awwaiid commented Feb 26, 2025

Copy link
Copy Markdown
Collaborator

We'll let @cielf merge as part of a controlled staging validation (upload files, merge this, see that they're still there, upload more files)

@cielf

cielf commented Feb 26, 2025

Copy link
Copy Markdown
Collaborator

I was hoping to get to this today but life interfered. Hopefully Friday.

@cielf

cielf commented Feb 28, 2025

Copy link
Copy Markdown
Collaborator

Merging for staging testing now.

@cielf
cielf merged commit e8b521f into main Feb 28, 2025
@cielf
cielf deleted the move-to-s3 branch February 28, 2025 17:33
@cielf

cielf commented Feb 28, 2025

Copy link
Copy Markdown
Collaborator

@dorner I think we have to revert this -- and retry --
Two things
1/ Attachments on the customized email in Edit My Organization don't seem to be persisting
2/ pdfs, on view, are showing gobbledygook in their tab

@github-actions

github-actions Bot commented Mar 2, 2025

Copy link
Copy Markdown
Contributor

@dorner: Your PR Move from Azure to AWS S3 for blob storage is part of today's Human Essentials production release: 2025.03.02.
Thank you very much for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants