ETL API

Extract-Transform-Load framework for payload-driven data updates.

What It Does

The ETL API provides a type-safe framework for implementing the ETL pattern for payload-driven updates. It enables efficient extraction of items from payloads, transformation with business logic, and loading into datastores with type safety and polymorphism.

Key Capabilities

• Polymorphic Payload Handling: Multiple updaters for different payload types
• Type-Safe Transformations: Generic types with TypeScript safety
• Async Extraction: Async support for complex extraction logic
• Streaming Updates: Process and load items efficiently
• Type Guards: Runtime payload type checking

Main Components

ItemUpdater Interface

interface ItemUpdater<T, R> {
  itemIsOfType(payload: unknown): payload is T
  extractUpdateItems(payload: T): R[] | Promise<R[]>
  extractLatestPayload(payload: T): T | Promise<T>
  transform(payload: T, updateItem: R): Partial<Indexable<R>>
}

PayloadUpdateStreamer

Abstract class orchestrating the ETL pipeline:

abstract class PayloadUpdateStreamer<R> {
  constructor(updaters: ItemUpdater<unknown, R>[])
  protected abstract load(updatedItems: Partial<Indexable<R>>[]): Promise<void>
  async updateItemFromPayload(payload: unknown): Promise<void>
}

Common Use Cases

• Event-driven data updates
• Webhook payload processing
• Microservice data synchronization
• Batch processing pipelines

What Customers Don’t Have to Build

• ETL orchestration framework
• Polymorphic payload routing
• Type-safe transformations
• Streaming update patterns