Multi-Format Export - Rich Domain

Overview

@woltz/rich-domain-export provides powerful multi-format export capabilities for your rich-domain entities. Export data to CSV, JSON, JSON Lines (JSONL), and custom formats with full type safety, custom formatting, and efficient streaming for large datasets.

Multiple Formats

CSV, JSON, JSONL, and extensible for custom formats

Type-Safe Exports

Full TypeScript support with discriminated unions

Streaming Support

Memory-efficient export for large datasets

Progress Tracking

Real-time progress callbacks for long-running exports

Installation

npm install @woltz/rich-domain-export

This is a backend-only package (Node.js). For frontend exports, use API endpoints.

Quick Start

Approach 1: Repository Extension

Extend your repository with export capabilities:

import { ExportableRepository } from "@woltz/rich-domain-export";
import { Criteria } from "@woltz/rich-domain";

class UserRepository extends ExportableRepository<User> {
  // Your repository implementation
}

const userRepository = new UserRepository();

// Export as CSV
const { data, stats } = await userRepository.export(
  Criteria.create<User>().where("status", "equals", "active"),
  {
    format: "csv",
    columns: ["name", "email", "createdAt"],
    headers: {
      name: "Full Name",
      email: "Email Address",
      createdAt: "Registration Date"
    }
  }
);

console.log(`Exported ${stats.totalRecords} records in ${stats.durationMs}ms`);

Approach 2: Composition with ExportService

Use the standalone service to keep export logic separate:

import { ExportService } from "@woltz/rich-domain-export";

const exportService = new ExportService();

// Export from any repository
const { data, stats } = await exportService.export(
  userRepository,
  criteria,
  { format: "csv", columns: ["name", "email"] }
);

Supported Formats

CSV Format

Export entities to comma-separated values format:

const { data, stats } = await repository.export(criteria, {
  format: "csv",
  columns: ["name", "email", "age"],
  headers: {
    name: "Full Name",
    email: "Email Address",
    age: "Age"
  },
  delimiter: ",",
  includeHeaders: true,
  formatters: {
    age: (value) => `${value} years old`
  }
});

// Output:
// Full Name,Email Address,Age
// John Doe,john@example.com,25 years old
// Jane Smith,jane@example.com,30 years old

CSV Options:

Option	Type	Default	Description
`format`	`"csv"`	-	Format discriminator (required)
`columns?`	`string[]`	all fields	Fields to include
`headers?`	`Record<string, string>`	-	Custom header labels
`delimiter?`	`string`	`","`	Delimiter character
`includeHeaders?`	`boolean`	`true`	Include header row
`formatters?`	`Record<string, Function>`	-	Custom formatters (returns string)
`batchSize?`	`number`	`1000`	Batch size for streaming

JSON Format

Export entities to standard JSON array:

const { data, stats } = await repository.export(criteria, {
  format: "json",
  pretty: true,
  indent: 2,
  fields: ["name", "email"],
  rootKey: "users",
  transformers: {
    email: (email) => email.toLowerCase()
  }
});

// Output:
// {
//   "users": [
//     { "name": "John Doe", "email": "john@example.com" },
//     { "name": "Jane Smith", "email": "jane@example.com" }
//   ]
// }

JSON Options:

Option	Type	Default	Description
`format`	`"json"`	-	Format discriminator (required)
`pretty?`	`boolean`	`false`	Pretty print with indentation
`indent?`	`number`	`2`	Number of spaces for indentation
`jsonLines?`	`boolean`	`false`	Use JSON Lines format
`fields?`	`string[]`	all fields	Fields to include
`transformers?`	`Record<string, Function>`	-	Custom transformers (returns any type)
`rootKey?`	`string`	-	Wrap output in root key
`batchSize?`	`number`	`1000`	Batch size for streaming

JSON Lines (JSONL) Format

Export entities to newline-delimited JSON (streaming-friendly):

const { data, stats } = await repository.export(criteria, {
  format: "json",
  jsonLines: true,
  fields: ["name", "email"]
});

// Output (each line is a valid JSON object):
// {"name":"John Doe","email":"john@example.com"}
// {"name":"Jane Smith","email":"jane@example.com"}

JSON Lines is ideal for streaming large datasets and is supported by many data processing tools. Learn more at jsonlines.org.

Common Use Cases

Export with Filters

Use Criteria to filter data before export:

import { Criteria } from "@woltz/rich-domain";

const criteria = Criteria.create<User>()
  .where("status", "equals", "active")
  .where("age", "greaterThan", 18)
  .orderBy("name", "asc");

const { data } = await repository.export(criteria, {
  format: "csv",
  columns: ["name", "email", "age"]
});

Custom Formatters (CSV)

Transform field values before CSV serialization:

const { data } = await repository.export(criteria, {
  format: "csv",
  columns: ["name", "createdAt", "balance", "status"],
  formatters: {
    createdAt: (date) => new Date(date).toLocaleDateString("en-US"),
    balance: (value) => `$${value.toFixed(2)}`,
    status: (value) => value.toUpperCase()
  }
});

Custom Transformers (JSON)

Transform field values with any type (not just strings):

const { data } = await repository.export(criteria, {
  format: "json",
  fields: ["name", "tags", "metadata", "amount"],
  transformers: {
    tags: (tags) => tags.join(", "),
    metadata: (meta) => ({ ...meta, exported: true }),
    amount: (num) => Number(num.toFixed(2))
  }
});

Common Formatters

The library provides pre-built formatters for CSV:

import { commonFormatters } from "@woltz/rich-domain-export";

const { data } = await repository.export(criteria, {
  format: "csv",
  columns: ["name", "amount", "createdAt", "active", "tags"],
  formatters: {
    amount: commonFormatters.currencyUSD,
    createdAt: commonFormatters.isoDate,
    active: commonFormatters.yesNo,
    tags: commonFormatters.array
  }
});

Available formatters:

Date Formatters

isoDate - ISO 8601 date string
localeDate - Locale date string (e.g., “1/1/2024”)
localeDateTime - Locale datetime string

Number Formatters

decimal2 - Number with 2 decimal places
currencyUSD - USD currency format ($X.XX)

Boolean Formatters

yesNo - Boolean as “Yes” or “No”
trueFalse - Boolean as “True” or “False”

Collection Formatters

array - Array as comma-separated string
json - Object as JSON string

Text Formatters

uppercase - Convert to uppercase
lowercase - Convert to lowercase
trim - Trim whitespace

Progress Tracking

Monitor export progress for large datasets:

const { data, stats } = await repository.export(
  criteria,
  { format: "csv", columns: ["name", "email"] },
  (processed, total) => {
    const percentage = (processed / total) * 100;
    console.log(`Export progress: ${percentage.toFixed(1)}%`);
  }
);

Streaming for Large Datasets

For large datasets, use streaming to avoid loading everything into memory:

CSV Stream

import * as fs from "fs";

const stream = await repository.exportStream(criteria, {
  format: "csv",
  columns: ["name", "email"],
  batchSize: 1000 // Process 1000 records at a time
});

stream.pipe(fs.createWriteStream("users.csv"));

JSON Lines Stream (Recommended for Large JSON Exports)

const stream = await repository.exportStream(criteria, {
  format: "json",
  jsonLines: true,
  batchSize: 500
});

stream.pipe(fs.createWriteStream("users.jsonl"));

HTTP Streaming (Fastify Example)

// CSV download
const stream = await repository.exportStream(criteria, {
  format: "csv",
  columns: ["name", "email"]
});

reply
  .header("Content-Type", "text/csv")
  .header("Content-Disposition", 'attachment; filename="users.csv"')
  .send(stream);

// JSON Lines download
const stream = await repository.exportStream(criteria, {
  format: "json",
  jsonLines: true
});

reply
  .header("Content-Type", "application/x-ndjson")
  .header("Content-Disposition", 'attachment; filename="users.jsonl"')
  .send(stream);

HTTP Streaming (Express Example)

const stream = await repository.exportStream(criteria, {
  format: "json",
  pretty: true
});

res.setHeader("Content-Type", "application/json");
res.setHeader("Content-Disposition", 'attachment; filename="users.json"');
stream.pipe(res);

Performance Considerations

Dataset Size	Recommended Method	Memory Usage
< 1,000 records	`export()`	~1-5 MB
1,000 - 10,000	`export()`	~5-50 MB
10,000 - 100,000	`exportStream()`	~10-20 MB (constant)
> 100,000	`exportStream()`	~10-20 MB (constant)

Tips:

Use exportStream() for datasets > 10,000 records
Use JSON Lines (jsonLines: true) for streaming large JSON exports
Adjust batchSize option to control memory usage (default: 1000)

Custom Formats

Extend the library with custom formats using the Strategy Pattern:

import {
  ExportFormatStrategy,
  FormatRegistry,
  ExportResult,
  ValidationResult
} from "@woltz/rich-domain-export";
import type { Readable } from "stream";

// Define custom options
interface ExcelExportOptions extends BaseExportOptions {
  format: "excel";
  sheetName?: string;
  columns?: string[];
}

// Implement the strategy
class ExcelFormatStrategy implements ExportFormatStrategy<any, ExcelExportOptions> {
  async export(records: any[], options: ExcelExportOptions): Promise<ExportResult> {
    // Your Excel export logic
    const data = this.generateExcel(records, options);

    return {
      data,
      stats: {
        totalRecords: records.length,
        sizeInBytes: data.length,
        durationMs: 0,
        // ... other stats
      }
    };
  }

  async exportStream(
    recordsIterator: AsyncIterable<any[]>,
    options: ExcelExportOptions
  ): Promise<Readable> {
    // Your streaming logic
    const { Readable } = await import("stream");
    return Readable.from(this.streamExcel(recordsIterator, options));
  }

  validateOptions(options: ExcelExportOptions): ValidationResult {
    const errors: string[] = [];

    if (options.sheetName && options.sheetName.length > 31) {
      errors.push("Sheet name must be 31 characters or less");
    }

    return {
      isValid: errors.length === 0,
      errors
    };
  }

  getMimeType(): string {
    return "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
  }

  getFileExtension(): string {
    return "xlsx";
  }

  getFormatName(): string {
    return "excel";
  }

  private generateExcel(records: any[], options: ExcelExportOptions): string {
    // Implementation here
    return "";
  }

  private async *streamExcel(
    recordsIterator: AsyncIterable<any[]>,
    options: ExcelExportOptions
  ): AsyncGenerator<string> {
    // Implementation here
  }
}

// Register custom format
FormatRegistry.register("excel", new ExcelFormatStrategy());

// Use it
const { data } = await repository.export(criteria, {
  format: "excel",
  sheetName: "Users",
  columns: ["name", "email"]
});

Export Statistics

Get detailed statistics about the export operation:

const { data, stats } = await repository.export(criteria, {
  format: "csv",
  columns: ["name", "email"]
});

console.log({
  totalRecords: stats.totalRecords,     // Number of records exported
  sizeInBytes: stats.sizeInBytes,       // Data size in bytes
  durationMs: stats.durationMs,         // Export duration
});

// CSV-specific stats
if (stats.totalColumns) {
  console.log(`Exported ${stats.totalColumns} columns`);
}

Type Safety

The library provides full type safety with discriminated unions:

// TypeScript enforces valid options for each format
const csvExport = await repository.export(criteria, {
  format: "csv",
  columns: ["name"],  // ✓ Valid for CSV
  delimiter: ","      // ✓ Valid for CSV
  // pretty: true     // ✗ Error: 'pretty' doesn't exist on CSV options
});

const jsonExport = await repository.export(criteria, {
  format: "json",
  pretty: true,       // ✓ Valid for JSON
  fields: ["name"]    // ✓ Valid for JSON
  // delimiter: ","   // ✗ Error: 'delimiter' doesn't exist on JSON options
});

Error Handling

The library provides specific error types:

import {
  ValidationError,
  FormatterError,
  ExportOperationError
} from "@woltz/rich-domain-export";

try {
  const { data } = await repository.export(criteria, {
    format: "csv",
    columns: ["name", "email"]
  });
} catch (error) {
  if (error instanceof ValidationError) {
    console.error("Invalid options:", error.validationErrors);
  } else if (error instanceof FormatterError) {
    console.error(`Formatter failed for field: ${error.field}`);
  } else if (error instanceof ExportOperationError) {
    console.error(`Export failed at phase: ${error.phase}`);
  }
}

Complete Example

import {
  ExportableRepository,
  commonFormatters
} from "@woltz/rich-domain-export";
import { Criteria } from "@woltz/rich-domain";

class UserRepository extends ExportableRepository<User> {
  // Your repository implementation
}

async function exportActiveUsers() {
  const userRepository = new UserRepository();

  // Create criteria for active users over 18
  const criteria = Criteria.create<User>()
    .where("status", "equals", "active")
    .where("age", "greaterThanOrEqual", 18)
    .orderBy("name", "asc");

  // Export as CSV with custom formatting
  const { data: csv, stats: csvStats } = await userRepository.export(
    criteria,
    {
      format: "csv",
      columns: ["name", "email", "age", "createdAt", "balance"],
      headers: {
        name: "Full Name",
        email: "Email Address",
        age: "Age",
        createdAt: "Registered",
        balance: "Account Balance"
      },
      formatters: {
        createdAt: commonFormatters.localeDate,
        balance: commonFormatters.currencyUSD
      }
    },
    (processed, total) => {
      const percentage = Math.round((processed / total) * 100);
      console.log(`CSV Export: ${percentage}%`);
    }
  );

  console.log(`✅ CSV: ${csvStats.totalRecords} users in ${csvStats.durationMs}ms`);

  // Export as JSON
  const { data: json, stats: jsonStats } = await userRepository.export(
    criteria,
    {
      format: "json",
      pretty: true,
      fields: ["name", "email", "age"],
      rootKey: "users",
      transformers: {
        email: (email) => email.toLowerCase()
      }
    }
  );

  console.log(`✅ JSON: ${jsonStats.totalRecords} users in ${jsonStats.durationMs}ms`);

  // Export as JSON Lines for streaming
  const stream = await userRepository.exportStream(criteria, {
    format: "json",
    jsonLines: true,
    batchSize: 500
  });

  stream.pipe(fs.createWriteStream("users.jsonl"));
}

API Reference

ExportableRepository

abstract class ExportableRepository<TDomain> extends Repository<TDomain> {
  export(
    criteria?: Criteria<TDomain>,
    options: ExportOptions<TDomain>,
    onProgress?: ExportProgressCallback
  ): Promise<ExportResult>;

  exportStream(
    criteria?: Criteria<TDomain>,
    options: ExportOptions<TDomain>
  ): Promise<Readable>;
}

ExportService

class ExportService {
  export<T>(
    repository: Repository<T>,
    criteria: Criteria<T> | undefined,
    options: ExportOptions<T>,
    onProgress?: ExportProgressCallback
  ): Promise<ExportResult>;

  exportStream<T>(
    repository: Repository<T>,
    criteria: Criteria<T> | undefined,
    options: ExportOptions<T>
  ): Promise<Readable>;

  getMimeType(format: string): string;
  getFileExtension(format: string): string;
}

FormatRegistry

class FormatRegistry {
  static register(
    format: string,
    strategyInstance: ExportFormatStrategy
  ): void;

  static getStrategy(format: string): ExportFormatStrategy;
  static hasFormat(format: string): boolean;
  static getRegisteredFormats(): string[];
}

ExportOptions

type ExportOptions<T> = CsvExportOptions<T> | JsonExportOptions<T>;

interface CsvExportOptions<T> extends BaseExportOptions {
  format: "csv";
  columns?: PropsOf<T>[];
  headers?: Partial<Record<PropsOf<T>, string>>;
  delimiter?: string;
  includeHeaders?: boolean;
  formatters?: Partial<Record<PropsOf<T>, (value: any) => string>>;
}

interface JsonExportOptions<T> extends BaseExportOptions {
  format: "json";
  pretty?: boolean;
  indent?: number;
  jsonLines?: boolean;
  fields?: PropsOf<T>[];
  transformers?: Partial<Record<PropsOf<T>, (value: any) => any>>;
  rootKey?: string;
}

interface BaseExportOptions {
  batchSize?: number; // Default: 1000
}

ExportResult

interface ExportResult {
  data: string;
  stats: BaseExportStats | CsvExportStats | JsonExportStats;
}

interface BaseExportStats {
  totalRecords: number;
  sizeInBytes: number;
  durationMs: number;
}

interface CsvExportStats extends BaseExportStats {
  totalColumns: number;
}

interface JsonExportStats extends BaseExportStats {
  // JSON-specific stats can be added here
}

Getting Started

Core Concepts

Validation

Criteria

Repository

Integrations

​Overview

Multiple Formats

Type-Safe Exports

Streaming Support

Progress Tracking

​Installation

​Quick Start

​Approach 1: Repository Extension

​Approach 2: Composition with ExportService

​Supported Formats

​CSV Format

​JSON Format

​JSON Lines (JSONL) Format

​Common Use Cases

​Export with Filters

​Custom Formatters (CSV)

​Custom Transformers (JSON)

​Common Formatters

​Progress Tracking

​Streaming for Large Datasets

​CSV Stream

​JSON Lines Stream (Recommended for Large JSON Exports)

​HTTP Streaming (Fastify Example)

​HTTP Streaming (Express Example)

​Performance Considerations

​Custom Formats

​Export Statistics

​Type Safety

​Error Handling

​Complete Example

​API Reference

​ExportableRepository

​ExportService

​FormatRegistry

​ExportOptions

​ExportResult

Overview

Installation

Quick Start

Approach 1: Repository Extension

Approach 2: Composition with ExportService

Supported Formats

CSV Format

JSON Format

JSON Lines (JSONL) Format

Common Use Cases

Export with Filters

Custom Formatters (CSV)

Custom Transformers (JSON)

Common Formatters

Progress Tracking

Streaming for Large Datasets

CSV Stream

JSON Lines Stream (Recommended for Large JSON Exports)

HTTP Streaming (Fastify Example)

HTTP Streaming (Express Example)

Performance Considerations

Custom Formats

Export Statistics

Type Safety

Error Handling

Complete Example

API Reference

ExportableRepository

ExportService

FormatRegistry

ExportOptions

ExportResult