Storing static files inside the computing system involves some work when you decide to scale horizontally, but having all static files stored in a cloud object storage service (as an external “microservice”) could ease the deployment process, at the same time reducing the complexity and improving the performance of the system. You can check some application architectures to guide you to the proper solution for your use case.
- For example:
- store CMS media files directly on the remote filesystem without touching the server disk.
- all the raw files could easily be served directly from there or through a built-in or third-party CDN.
- the products media files could also be stored in a cloud object storage, but in this case, the resized images should be served through a reverse proxy server with a fallback on the resizing endpoint.
- even better, you can extract the image resize feature as a microservice and reduce your project complexity. Read more about the Magento 2 headless implementation.
There are plenty of cloud object storage services (for static files) offered with various features, all having common features like:
- allow secure upload remotely of files
- deliver public or private static files
- built-in Content Delivery Network (CDN)
With this idea in mind, you can identify some advantages of using it in e-commerce websites for storing images and videos for products, categories, or CMS pages. Storing private content like downloadable products could be also covered.
- This project is under active development phase, and there are plans for a bunch of new features or Magento 2 Community platform refactoring. You can check there plans on Project Roadmap.
- Feel free to create new issues for feature requests, new ideas, and improvements or bugs related to these extensions: On GitHub
Extracting static files storage component as a microservice¶
There are few questions that could be raised related to the actual need of replacing the built-in media storage based on local disk with a microservice specialized in web asset management. You can find the answers by reading the architecture section of this documentation.
Having microservice responsible for storage, processing, and delivery of all static content is a happy case that allows developers to focus on scaling compute and storage systems independently.
Also, having a specialized microservice allows improving this area independently. It is clear now, that this may be a big plus because website performance affects customer experience, conversions, and advertisement costs. (lately, advertisers are more and more careful about website performance)
On-premises deployment with cloud file storage and CDN in cloud¶
Even if this is not the lucky case this module could be used for this use-case, but some latency problems may occur on the admin panel.
This scenario implies storing media files using remote cloud services and process images on the application server.
In this case, optimization could be achieved if the resize script could be relocated in the same data-center with media storage service.
Migration to cloud from on-premises infrastructure¶
In the migration stage, this module could be used as a way to migrate media files to cloud storage service and continue using them from there.
Existing deployment in cloud¶
All notes above apply to this case, and the advantage of having the storage system and computing ones in the same data-center will assure best performance and cost-effective setup.
Product file storage¶
Images in the gallery, product description images, or other product-related photos will be stored at upload time in the cloud and delivered from there each time is needed in frontend.
Downloadable product attachments are supported by installing bb/filesystem-downloadable.
The database and data structure is not changed after the installation of these modules.
Photo upload and management in admin¶
The extension implements cloud storage folders navigation for admin users in order to allow users to locate files directly on storage service and insert them into HTML without the need of being on application servers.
WYSIWYG images storage¶
In this area, images are stored in the cloud, and serve directly from there without processing or resizing.
OOB The image below proves changes in the media storage structure are minimal: the URL is the same.
Sync command between filesystems¶
For migration projects, there is a command to synchronize media files from one filesystem to another.
bin/magento bb:storage:sync Description: Sync media files between two filesystems on a particular directory. Usage: bb:storage:sync [options] [--] <source> <destination> [<directory>] Arguments: source Origin filesystem code destination Destination filesystem code directory Directory to be synchronized. Options: -o, --overwrite Overwrite files in destination
Report differences between filesystems¶
Also, for later use, there is a developer command to report the differences between the two filesystems in order to re-evaluate the migration process.
bin/magento bb:storage:diff files s3 media Description: Create a report with differences between different two filesystems on a particular directory. Usage: bb:storage:diff <filesystem_1> <filesystem_2> <directory> Arguments: filesystem_1 First filesystem code filesystem_2 Second filesystem code directory Directory to be compared
Multiple cloud locations mapping for subdirectories¶
Having multiple cloud objects buckets mapped to different media level directories allow websites to expose files with different levels of permission for frontend. For example for downloadable products, files should always be served through the application server in order to check permissions, not directly from storage service.
Overwriting the Magento media location (or another core directory) may cause some errors in modules not consistently using \Magento\Framework\Filesystem for directory and file operations (including Magento Core modules).
In some cases, operations on files or directories are performed directly with PHP functions, or \Magento\Framework\Filesystem\DriverInterface objects are obtained directly from ObjectManager. Obtaining a driver object from the filesystem object will avoid this type of problem.
Most of the inconsistencies from Magento core modules are fixed in bb/filesystem-cms, bb/filesystem-catalog, bb/filesystem-downloadable modules, and is part of the plan to include these improvements in the Magento Community project.
However overwriting the Magento core directories is not mandatory, because you can configure new directories for custom implementation with the base module bb/filesystem and the module implementing the driver of your choice eg: bb/filesystem-s3.
This module is not yet compatible with New Magento Media Gallery.
This extension is currently an active development phase, check Project Roadmap to see some ideas or ask for solutions on a particular use case.
- Feel free to create new issues for feature requests, questions, new ideas, and improvements or bugs related to these extensions: On GitHub
- 1.0.0 - proof of concept
- bb/filesystem could be installed independently and will allow custom/new directories for new modules
- To overwrite the Magento storage following modules should be installed: bb/filesystemOverwrite, bb/filesystem-cms, bb/filesystem-catalog, bb/filesystem-downloadable
- For now, when you create a mapping for a subdirectory, you need to create a configuration for all others subdirectories
- Available driver bb/filesystem-s3 for Amazon S3 like integration. More drivers will be published soon depending on demands. Please request new drivers on GitHub.