Skip to Content
Engineering11 Documentation 🔥
BackendServer API PackagesStorage

Storage API

Multi-cloud storage abstraction for seamless GCP and AWS S3 operations.

What It Does

The Storage API provides a unified interface for cloud storage operations across Google Cloud Platform (GCP) and Amazon AWS S3. It enables applications to perform file operations, manage metadata, generate signed URLs, and stream large files—all while maintaining the flexibility to switch between cloud providers without changing application code.

Key Capabilities

CapabilityDescription
Multi-Cloud SupportUnified interface for both GCP Cloud Storage and AWS S3
File OperationsCreate, read, copy, delete files with metadata management
StreamingEfficient handling of large files with pagination and gzip support
Signed URLsGenerate secure, time-limited URLs for file sharing
Access ControlMake files public or manage permissions
JSON StreamingParse JSON files in real-time with async iteration
Batch OperationsDelete multiple files efficiently (AWS: up to 1000 at once)
Metadata ManagementGet, set, and update file metadata and custom properties
Path BuildingFluent API for constructing file paths with proper formatting
Dependency InjectionAutomatic cloud provider selection via DI container

Configuration

Provider Selection

  • Environment Variable: CLOUD_STORAGE_CHOICE='aws'|'gcp'
  • Default: GCP

AWS S3 Configuration

  • Region: Default us-east-1
  • Credentials: From parameters or secrets manager
  • Batch Delete: Up to 1000 objects at once

GCP Configuration

  • Credentials: PATH_TO_STORAGE_CREDENTIALS or default
  • Signed URL Format: v4
  • Firebase Tokens: Supported

Exports

Services

  • CloudStorageService - Unified interface
  • GCPStorageService - GCP implementation
  • S3StorageService - AWS S3 implementation
  • StorageService - GCP wrapper (legacy)

Utilities

  • PathBuilder - Fluent path construction
  • S3Client - AWS client factory
  • StorageWriteStream - Safe write operations

Streaming Classes

  • AsyncJSONStream - JSON file streaming
  • PaginatedBucketStream - Large file streaming
  • PaginatedBucketJSONChunkStream - Batched JSON streaming
  • PaginatedBucketJSONStream - Legacy JSON streaming

Models

  • ICloudStorageFileMetadata - File metadata
  • ICopyObject - Copy parameters
  • StorageState - Storage tier enum
  • IListObjectResponse - List response

Container

  • storageContainer - DI container for automatic provider selection

Common Use Cases

  • Multi-cloud applications: Switch between GCP and AWS without code changes
  • File ingestion pipelines: Stream and process large files efficiently
  • Asset management: Store and retrieve images, documents, media
  • Data migrations: Copy files between buckets and providers
  • Secure file sharing: Generate time-limited signed URLs
  • ETL pipelines: Process JSON data from cloud storage
  • Backup systems: Copy and archive files with metadata preservation
  • Document processing: Stream and parse large documents

What Customers Don’t Have to Build

  • Multi-cloud storage abstraction
  • Provider-specific API integration
  • File streaming infrastructure
  • JSON parsing for large files
  • Signed URL generation
  • Metadata management
  • Path building utilities
  • Recursive file operations
  • Batch deletion optimization
  • Storage tier detection
  • Write stream wrappers
  • Firebase token management
  • S3 client configuration
  • Dependency injection for storage
  • Provider selection logic
  • Credentials management
  • Gzip decompression
  • Pagination for streaming
  • Backpressure handling
  • Error handling for batch operations
Last updated on
REST DatastoresRedis