Pattern 21: External Data Integration

Part III: Integration Patterns

Opening Scenario: The Form That Couldn't Remember Customers

Jessica was processing a new order for a returning customer at an online electronics retailer. She pulled up the order form:

New Order Form

Customer Name: [                    ]
Email: [                    ]
Phone: [                    ]
Shipping Address: [                    ]
                  [                    ]
                  [                    ]
Payment Method: [                    ]

Empty. Every field.

Jessica knew this customer - Sarah Martinez, customer ID 38472, had ordered from them 15 times in the past two years. All her information was in the CRM system. But the order form didn't know that.

Jessica spent 5 minutes: 1. Opening the CRM in another tab 2. Looking up Sarah Martinez (ID 38472) 3. Copy-pasting name 4. Copy-pasting email 5. Copy-pasting phone 6. Copy-pasting shipping address (4 lines) 7. Copy-pasting payment method

5 minutes of mindless data entry for information the company already had.

Jessica processed 40 orders per day. That's 200 minutes (3.3 hours) of redundant typing. For the whole order team of 8 people, that's 26.4 hours per day wasted on re-entering existing data.

The e-commerce director, Marcus, watched Jessica work. "Why doesn't the form just pull that data automatically?"

The developer, Tom, explained: "The order form doesn't connect to the CRM. They're separate systems."

"Can you connect them?"

Tom looked at the code:

// Bad: Form knows nothing about CRM
class OrderForm {
  render() {
    return `
      <input name="customerName" value="">
      <input name="email" value="">
      <input name="phone" value="">
      <!-- All fields empty, no integration -->
    `;
  }
}

// CRM data exists but is unreachable
// Customer 38472 has complete profile
// But form can't access it

Marcus asked: "What other data are we re-entering that already exists somewhere?"

Tom investigated:

Product information: - Product names, prices, descriptions all in inventory system - Order form makes user type everything manually - Frequent errors: wrong price, wrong description

Shipping rates: - Shipping API provides real-time rates - Form uses static, outdated rate table - Customers overcharged or undercharged

Address validation: - USPS API can validate any US address - Form accepts invalid addresses - Packages returned, reshipment costs

Tax calculation: - Tax service knows rates for every ZIP code - Form uses 6-month-old tax table - Wrong tax charged on 12% of orders

Inventory levels: - Real-time inventory in warehouse system - Form doesn't check stock - Orders placed for out-of-stock items

"We're maintaining duplicate data in five different systems," Tom said. "And none of them talk to each other."

Marcus approved the integration project. Tom rebuilt the order form:

class IntegratedOrderForm {
  async lookupCustomer(customerId) {
    // Connect to CRM
    const customer = await this.crmAPI.getCustomer(customerId);

    // Populate form automatically
    this.form.setValues({
      customerName: customer.name,
      email: customer.email,
      phone: customer.phone,
      shippingAddress: customer.defaultShippingAddress,
      paymentMethod: customer.defaultPaymentMethod,
      customerSince: customer.createdDate,
      lifetimeValue: customer.totalPurchases,
      loyaltyTier: customer.loyaltyTier
    });

    return customer;
  }

  async lookupProduct(sku) {
    // Connect to inventory system
    const product = await this.inventoryAPI.getProduct(sku);

    this.form.setValues({
      productName: product.name,
      description: product.description,
      price: product.currentPrice,
      inStock: product.quantityAvailable,
      estimatedShipping: product.shippingDays
    });

    return product;
  }

  async calculateShipping(address, weight) {
    // Connect to shipping API
    const rates = await this.shippingAPI.getRates({
      origin: this.warehouse.address,
      destination: address,
      weight: weight,
      dimensions: this.package.dimensions
    });

    this.form.setShippingOptions(rates);

    return rates;
  }

  async validateAddress(address) {
    // Connect to USPS API
    const validated = await this.uspsAPI.validateAddress(address);

    if (!validated.valid) {
      this.form.showAddressError(validated.suggestions);
    }

    return validated;
  }

  async calculateTax(amount, zipCode) {
    // Connect to tax service
    const tax = await this.taxAPI.calculateSalesTax({
      amount: amount,
      zipCode: zipCode,
      productCategory: this.product.category
    });

    this.form.setTax(tax.amount);

    return tax;
  }
}

Now when Jessica started an order:

New Order Form

Customer ID: [38472____________] [Search]

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AUTO-POPULATED FROM CRM:

Customer: Sarah Martinez
Email: sarah.martinez@email.com
Phone: (555) 123-4567
Shipping: 123 Main St, Apt 4B
          Springfield, IL 62701
Payment: Visa ****1234
Loyalty: Gold Member (15 previous orders)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Product SKU: [HD-4K-TV-55] [Lookup]

AUTO-POPULATED FROM INVENTORY:

Product: Samsung 55" 4K Smart TV
Price: $649.99
In Stock: 12 units at Chicago warehouse
Ships: Same-day if ordered by 2 PM

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

SHIPPING OPTIONS (from FedEx API):
○ Ground (3-5 days): $12.99
● Two-Day: $24.99  ← Customer's usual choice
○ Overnight: $49.99

TAX (calculated for IL 62701): $54.60

TOTAL: $729.58

[Place Order]

Jessica's time per order: 30 seconds (from 5 minutes).

Data entry: Zero (from 100% manual).

Errors: Nearly zero (from frequent mistakes).

The order team processed 45% more orders with the same staff. Customer satisfaction increased because: - Faster checkout - Accurate pricing - Real inventory status - Correct shipping estimates - Proper tax calculation

Connections made things flow as they should.

Context

External Data Integration applies when:

Data exists elsewhere: Customer data in CRM, products in inventory, addresses in USPS

Real-time accuracy needed: Tax rates change, inventory moves, prices fluctuate

Manual entry wasteful: Re-entering data that exists elsewhere

Consistency required: Single source of truth across systems

User experience matters: Pre-filling forms saves time and reduces errors

Validation necessary: External services can validate data (addresses, credit cards, tax IDs)

Scale demands efficiency: High-volume operations can't afford redundant entry

Integration possible: APIs or databases accessible for data retrieval

Problem Statement

Most forms operate in isolation, disconnected from valuable external data sources:

No CRM integration:

// Bad: Form doesn't know about existing customers
function createCustomerForm() {
  return {
    name: '',
    email: '',
    phone: '',
    address: ''
  };
}

// Customer exists in Salesforce with complete profile
// But form treats everyone as new
// Re-enter everything every time

No inventory connection:

// Bad: Static product data
const products = [
  { sku: 'ABC123', price: 99.99, inStock: true }
];

// Reality:
// - Price changed yesterday
// - Stock depleted this morning
// - New description added last week
// Form shows outdated information

No address validation:

// Bad: Accepts any text as address
function validateAddress(address) {
  return address.street && address.city && address.zip;
}

// User enters: "123 Main St, Anytown, 12345"
// Could be invalid, misspelled, non-deliverable
// No verification against postal service

No real-time calculation:

// Bad: Hardcoded tax rates
const TAX_RATE = 0.07;  // 7% - from when?

function calculateTax(amount) {
  return amount * TAX_RATE;
}

// Tax rates change by location, date, product type
// This is almost certainly wrong

No third-party services:

// Bad: Manual credit card entry with no validation
function processCreditCard(number, cvv, expiry) {
  // No validation service
  // No fraud detection
  // No BIN lookup
  // Just hope it works
}

No data enrichment:

// Bad: Only knows what user types
{
  companyName: "Acme Corp"
}

// Could know (from Clearbit, ZoomInfo, etc.):
// - Full company details
// - Industry
// - Employee count
// - Revenue range
// - Technologies used
// - Social profiles

We need forms that connect to external systems, fetch relevant data, validate against authoritative sources, and enrich user input with valuable context.

Forces

Performance vs Freshness

Real-time lookups are slow
But stale data is wrong
Balance speed with accuracy

Dependency vs Reliability

External APIs can fail
But isolated forms lack data
Balance integration with fallbacks

Security vs Connectivity

APIs expose attack surface
But isolation limits capability
Balance access with protection

Simplicity vs Integration

Standalone forms are simple
But integrated forms are powerful
Balance complexity with value

Cost vs Value

API calls cost money
But manual entry wastes time
Balance expenses with efficiency

Solution

Implement comprehensive external data integration that connects forms to CRM systems, inventory databases, validation services, calculation engines, and third-party APIs - providing real-time data lookup, automatic population, validation, and enrichment while maintaining performance, security, and reliability.

The pattern has seven key strategies:

1. API Integration Layer

Create abstraction for external API calls:

class ExternalDataAPI {
  constructor(config) {
    this.config = config;
    this.cache = new Map();
    this.rateLimiter = new RateLimiter(config.rateLimit);
  }

  async get(endpoint, params, options = {}) {
    // Build cache key
    const cacheKey = this.buildCacheKey(endpoint, params);

    // Check cache first
    if (options.cache !== false) {
      const cached = this.cache.get(cacheKey);
      if (cached && !this.isCacheExpired(cached)) {
        return cached.data;
      }
    }

    // Rate limiting
    await this.rateLimiter.checkLimit();

    // Build request
    const url = this.buildURL(endpoint, params);
    const headers = this.buildHeaders(options);

    try {
      const response = await fetch(url, {
        method: 'GET',
        headers: headers,
        timeout: options.timeout || 5000
      });

      if (!response.ok) {
        throw new Error(`API error: ${response.status}`);
      }

      const data = await response.json();

      // Cache successful response
      if (options.cache !== false) {
        this.cache.set(cacheKey, {
          data: data,
          timestamp: Date.now(),
          ttl: options.cacheTTL || 300000 // 5 minutes
        });
      }

      return data;

    } catch (error) {
      // Log error
      console.error(`API call failed: ${endpoint}`, error);

      // Return cached data if available (stale is better than nothing)
      const cached = this.cache.get(cacheKey);
      if (cached && options.fallbackToCache) {
        return cached.data;
      }

      throw error;
    }
  }

  async post(endpoint, data, options = {}) {
    await this.rateLimiter.checkLimit();

    const url = this.buildURL(endpoint);
    const headers = this.buildHeaders(options);

    const response = await fetch(url, {
      method: 'POST',
      headers: headers,
      body: JSON.stringify(data),
      timeout: options.timeout || 10000
    });

    if (!response.ok) {
      throw new Error(`API error: ${response.status}`);
    }

    return await response.json();
  }

  buildURL(endpoint, params) {
    let url = `${this.config.baseURL}${endpoint}`;

    if (params) {
      const queryString = new URLSearchParams(params).toString();
      url += `?${queryString}`;
    }

    return url;
  }

  buildHeaders(options) {
    return {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${this.config.apiKey}`,
      ...options.headers
    };
  }

  buildCacheKey(endpoint, params) {
    return `${endpoint}:${JSON.stringify(params || {})}`;
  }

  isCacheExpired(cached) {
    return (Date.now() - cached.timestamp) > cached.ttl;
  }
}

2. CRM Integration

Connect to customer relationship management systems:

class CRMIntegration {
  constructor(apiConfig) {
    this.api = new ExternalDataAPI(apiConfig);
  }

  async lookupCustomer(identifier) {
    // Lookup by email, phone, or customer ID
    let customer;

    if (identifier.includes('@')) {
      customer = await this.lookupByEmail(identifier);
    } else if (identifier.match(/^\d+$/)) {
      customer = await this.lookupByCustomerId(identifier);
    } else {
      customer = await this.lookupByPhone(identifier);
    }

    if (customer) {
      return this.enrichCustomerData(customer);
    }

    return null;
  }

  async lookupByEmail(email) {
    return await this.api.get('/customers/by-email', { email });
  }

  async lookupByCustomerId(id) {
    return await this.api.get(`/customers/${id}`);
  }

  async lookupByPhone(phone) {
    return await this.api.get('/customers/by-phone', { phone });
  }

  async enrichCustomerData(customer) {
    // Get additional data
    const [orders, preferences, loyaltyStatus] = await Promise.all([
      this.getCustomerOrders(customer.id),
      this.getCustomerPreferences(customer.id),
      this.getLoyaltyStatus(customer.id)
    ]);

    return {
      ...customer,
      orderHistory: orders,
      preferences: preferences,
      loyalty: loyaltyStatus,

      // Computed fields
      totalOrders: orders.length,
      lifetimeValue: orders.reduce((sum, o) => sum + o.total, 0),
      averageOrderValue: orders.length > 0 
        ? orders.reduce((sum, o) => sum + o.total, 0) / orders.length
        : 0,
      lastOrderDate: orders.length > 0 
        ? orders[0].date 
        : null,
      daysSinceLastOrder: orders.length > 0
        ? this.daysBetween(new Date(orders[0].date), new Date())
        : null
    };
  }

  async getCustomerOrders(customerId) {
    return await this.api.get(`/customers/${customerId}/orders`, {
      limit: 10,
      sort: 'date_desc'
    });
  }

  async getCustomerPreferences(customerId) {
    return await this.api.get(`/customers/${customerId}/preferences`);
  }

  async getLoyaltyStatus(customerId) {
    return await this.api.get(`/customers/${customerId}/loyalty`);
  }

  async updateCustomer(customerId, updates) {
    return await this.api.post(`/customers/${customerId}`, updates);
  }

  // Salesforce-specific implementation
  static createSalesforceIntegration(config) {
    return new CRMIntegration({
      baseURL: `https://${config.instance}.salesforce.com/services/data/v52.0`,
      apiKey: config.accessToken,
      rateLimit: {
        requests: 100,
        period: 60000 // per minute
      }
    });
  }

  // HubSpot-specific implementation
  static createHubSpotIntegration(config) {
    return new CRMIntegration({
      baseURL: 'https://api.hubapi.com',
      apiKey: config.apiKey,
      rateLimit: {
        requests: 100,
        period: 10000 // per 10 seconds
      }
    });
  }
}

3. Inventory & Product Integration

Connect to inventory management systems:

class InventoryIntegration {
  constructor(apiConfig) {
    this.api = new ExternalDataAPI(apiConfig);
  }

  async lookupProduct(sku) {
    const product = await this.api.get(`/products/${sku}`, null, {
      cache: true,
      cacheTTL: 60000 // 1 minute (inventory changes frequently)
    });

    if (!product) {
      return null;
    }

    return {
      sku: product.sku,
      name: product.name,
      description: product.description,
      price: product.currentPrice,
      msrp: product.msrp,
      discount: product.discount,

      // Inventory
      inStock: product.quantityAvailable > 0,
      quantity: product.quantityAvailable,
      warehouse: product.warehouse,

      // Shipping
      weight: product.weight,
      dimensions: product.dimensions,
      shippingDays: product.estimatedShippingDays,

      // Images
      images: product.images,
      thumbnail: product.thumbnail,

      // Categorization
      category: product.category,
      tags: product.tags,

      // Related
      relatedProducts: product.relatedSkus,
      alternatives: product.alternatives
    };
  }

  async checkAvailability(sku, quantity) {
    const availability = await this.api.get(`/products/${sku}/availability`, {
      quantity: quantity
    });

    return {
      available: availability.quantityAvailable >= quantity,
      quantityAvailable: availability.quantityAvailable,
      nextRestock: availability.nextRestockDate,
      alternativeLocations: availability.alternativeWarehouses
    };
  }

  async getPrice(sku, quantity, customerTier) {
    const pricing = await this.api.get(`/products/${sku}/price`, {
      quantity: quantity,
      customerTier: customerTier
    });

    return {
      unitPrice: pricing.unitPrice,
      totalPrice: pricing.totalPrice,
      discount: pricing.discount,
      discountReason: pricing.discountReason,
      volumeDiscounts: pricing.volumeDiscounts
    };
  }

  async reserveInventory(sku, quantity, reservationId) {
    // Reserve inventory for order
    return await this.api.post(`/products/${sku}/reserve`, {
      quantity: quantity,
      reservationId: reservationId,
      duration: 15 * 60 * 1000 // 15 minutes
    });
  }

  async searchProducts(query, filters = {}) {
    return await this.api.get('/products/search', {
      q: query,
      ...filters
    });
  }
}

4. Address Validation Services

Validate addresses against authoritative sources:

class AddressValidationService {
  constructor() {
    this.uspsAPI = new ExternalDataAPI({
      baseURL: 'https://secure.shippingapis.com/ShippingAPI.dll',
      apiKey: process.env.USPS_API_KEY
    });
  }

  async validateUSAddress(address) {
    try {
      const validated = await this.uspsAPI.get('/Verify', {
        API: 'Verify',
        XML: this.buildUSPSXML(address)
      });

      const result = this.parseUSPSResponse(validated);

      return {
        valid: result.valid,
        standardized: result.standardizedAddress,
        suggestions: result.suggestions,
        deliverable: result.deliverable,
        addressType: result.addressType, // Residential, Commercial, PO Box
        warnings: result.warnings
      };

    } catch (error) {
      return {
        valid: false,
        error: error.message,
        suggestions: []
      };
    }
  }

  buildUSPSXML(address) {
    return `
      <AddressValidateRequest USERID="${this.uspsAPI.config.apiKey}">
        <Address>
          <Address1>${address.line2 || ''}</Address1>
          <Address2>${address.line1}</Address2>
          <City>${address.city}</City>
          <State>${address.state}</State>
          <Zip5>${address.zip}</Zip5>
          <Zip4></Zip4>
        </Address>
      </AddressValidateRequest>
    `;
  }

  async validateInternationalAddress(address, country) {
    // Use Google Maps API, SmartyStreets, or similar
    const googleMaps = new ExternalDataAPI({
      baseURL: 'https://maps.googleapis.com/maps/api',
      apiKey: process.env.GOOGLE_MAPS_API_KEY
    });

    const result = await googleMaps.get('/geocode/json', {
      address: this.formatAddress(address),
      components: `country:${country}`
    });

    if (result.status === 'OK' && result.results.length > 0) {
      const validated = result.results[0];

      return {
        valid: true,
        standardized: this.parseGoogleAddress(validated),
        coordinates: validated.geometry.location,
        placeId: validated.place_id,
        formattedAddress: validated.formatted_address
      };
    }

    return {
      valid: false,
      suggestions: result.results.map(r => r.formatted_address)
    };
  }

  formatAddress(address) {
    return `${address.line1}, ${address.city}, ${address.state} ${address.zip}`;
  }
}

5. Tax Calculation Services

Calculate accurate sales tax:

class TaxCalculationService {
  constructor() {
    this.taxJarAPI = new ExternalDataAPI({
      baseURL: 'https://api.taxjar.com/v2',
      apiKey: process.env.TAXJAR_API_KEY
    });
  }

  async calculateSalesTax(order) {
    try {
      const taxData = await this.taxJarAPI.post('/taxes', {
        from_country: order.origin.country,
        from_zip: order.origin.zip,
        from_state: order.origin.state,
        from_city: order.origin.city,
        from_street: order.origin.street,

        to_country: order.destination.country,
        to_zip: order.destination.zip,
        to_state: order.destination.state,
        to_city: order.destination.city,
        to_street: order.destination.street,

        amount: order.amount,
        shipping: order.shippingCost,

        nexus_addresses: order.nexusAddresses,
        line_items: order.items.map(item => ({
          id: item.sku,
          quantity: item.quantity,
          product_tax_code: item.taxCode,
          unit_price: item.price,
          discount: item.discount
        }))
      });

      return {
        taxAmount: taxData.tax.amount_to_collect,
        taxRate: taxData.tax.rate,
        breakdown: taxData.tax.breakdown,
        jurisdiction: taxData.tax.jurisdictions,
        hasNexus: taxData.tax.has_nexus,
        freight_taxable: taxData.tax.freight_taxable
      };

    } catch (error) {
      console.error('Tax calculation failed:', error);

      // Fallback to estimated rate
      return {
        taxAmount: this.estimateTax(order),
        estimated: true,
        error: error.message
      };
    }
  }

  async getTaxRates(zipCode) {
    const rates = await this.taxJarAPI.get(`/rates/${zipCode}`);

    return {
      zip: rates.rate.zip,
      state: rates.rate.state,
      stateRate: rates.rate.state_rate,
      countyRate: rates.rate.county_rate,
      cityRate: rates.rate.city_rate,
      combinedRate: rates.rate.combined_rate,
      freightTaxable: rates.rate.freight_taxable
    };
  }

  estimateTax(order) {
    // Fallback estimation if API fails
    const estimatedRate = 0.07; // 7% conservative estimate
    return order.amount * estimatedRate;
  }
}

6. Shipping Rate APIs

Get real-time shipping rates:

class ShippingRateService {
  constructor() {
    this.fedexAPI = new ExternalDataAPI({
      baseURL: 'https://apis.fedex.com',
      apiKey: process.env.FEDEX_API_KEY
    });

    this.upsAPI = new ExternalDataAPI({
      baseURL: 'https://onlinetools.ups.com',
      apiKey: process.env.UPS_API_KEY
    });
  }

  async getRates(shipment) {
    // Get rates from multiple carriers in parallel
    const [fedexRates, upsRates] = await Promise.all([
      this.getFedExRates(shipment),
      this.getUPSRates(shipment)
    ]);

    // Combine and sort by price
    const allRates = [...fedexRates, ...upsRates].sort((a, b) => 
      a.price - b.price
    );

    return allRates;
  }

  async getFedExRates(shipment) {
    const response = await this.fedexAPI.post('/rate/v1/rates/quotes', {
      accountNumber: {
        value: process.env.FEDEX_ACCOUNT
      },
      requestedShipment: {
        shipper: {
          address: this.formatAddress(shipment.origin)
        },
        recipient: {
          address: this.formatAddress(shipment.destination)
        },
        pickupType: 'DROPOFF_AT_FEDEX_LOCATION',
        serviceType: 'FEDEX_GROUND',
        packagingType: 'YOUR_PACKAGING',
        rateRequestType: ['LIST', 'ACCOUNT'],
        requestedPackageLineItems: [{
          weight: {
            units: 'LB',
            value: shipment.weight
          },
          dimensions: {
            length: shipment.dimensions.length,
            width: shipment.dimensions.width,
            height: shipment.dimensions.height,
            units: 'IN'
          }
        }]
      }
    });

    return response.output.rateReplyDetails.map(rate => ({
      carrier: 'FedEx',
      service: rate.serviceName,
      price: rate.ratedShipmentDetails[0].totalNetCharge,
      deliveryDate: rate.commit.dateDetail.dayFormat,
      transitDays: rate.commit.transitDays
    }));
  }

  async getUPSRates(shipment) {
    const response = await this.upsAPI.post('/ship/v1/rating/Rate', {
      RateRequest: {
        Request: {
          TransactionReference: {
            CustomerContext: 'Rating Request'
          }
        },
        Shipment: {
          Shipper: {
            Address: this.formatAddress(shipment.origin)
          },
          ShipTo: {
            Address: this.formatAddress(shipment.destination)
          },
          Package: {
            PackagingType: {
              Code: '02' // Customer packaging
            },
            Dimensions: {
              UnitOfMeasurement: {
                Code: 'IN'
              },
              Length: shipment.dimensions.length.toString(),
              Width: shipment.dimensions.width.toString(),
              Height: shipment.dimensions.height.toString()
            },
            PackageWeight: {
              UnitOfMeasurement: {
                Code: 'LBS'
              },
              Weight: shipment.weight.toString()
            }
          }
        }
      }
    });

    return response.RateResponse.RatedShipment.map(rate => ({
      carrier: 'UPS',
      service: rate.Service.Code,
      price: parseFloat(rate.TotalCharges.MonetaryValue),
      deliveryDate: rate.TimeInTransit?.ServiceSummary?.EstimatedArrival,
      transitDays: rate.TimeInTransit?.ServiceSummary?.BusinessDaysInTransit
    }));
  }
}

7. Data Enrichment Services

Enrich user input with additional context:

class DataEnrichmentService {
  constructor() {
    this.clearbitAPI = new ExternalDataAPI({
      baseURL: 'https://company.clearbit.com/v2',
      apiKey: process.env.CLEARBIT_API_KEY
    });
  }

  async enrichCompanyData(domain) {
    try {
      const company = await this.clearbitAPI.get('/companies/find', {
        domain: domain
      });

      return {
        name: company.name,
        legalName: company.legalName,
        domain: company.domain,

        // Business info
        description: company.description,
        industry: company.category.industry,
        sector: company.category.sector,
        tags: company.tags,

        // Size
        employees: company.metrics.employees,
        employeesRange: company.metrics.employeesRange,
        estimatedRevenue: company.metrics.estimatedAnnualRevenue,

        // Technology
        tech: company.tech,

        // Location
        location: company.location,
        headquarters: company.geo,

        // Social
        twitter: company.twitter?.handle,
        linkedin: company.linkedin?.handle,
        facebook: company.facebook?.handle,

        // Contact
        phone: company.phone,
        emailDomain: company.emailProvider === false ? company.domain : null
      };

    } catch (error) {
      return null;
    }
  }

  async enrichPersonData(email) {
    try {
      const person = await this.clearbitAPI.get('/people/find', {
        email: email
      });

      return {
        name: person.name?.fullName,
        firstName: person.name?.givenName,
        lastName: person.name?.familyName,

        // Employment
        title: person.employment?.title,
        role: person.employment?.role,
        seniority: person.employment?.seniority,
        company: person.employment?.name,

        // Location
        city: person.geo?.city,
        state: person.geo?.state,
        country: person.geo?.country,

        // Social
        twitter: person.twitter?.handle,
        linkedin: person.linkedin?.handle,
        github: person.github?.handle,

        // Avatar
        avatar: person.avatar
      };

    } catch (error) {
      return null;
    }
  }

  async validateEmail(email) {
    // Use email validation service
    const kickboxAPI = new ExternalDataAPI({
      baseURL: 'https://api.kickbox.com/v2',
      apiKey: process.env.KICKBOX_API_KEY
    });

    const result = await kickboxAPI.get('/verify', { email });

    return {
      valid: result.result === 'deliverable',
      disposable: result.disposable,
      role: result.role,
      free: result.free,
      acceptAll: result.accept_all,
      suggestion: result.did_you_mean
    };
  }
}

Implementation Details

Complete External Integration System

class ExternalIntegrationSystem {
  constructor(config) {
    // Initialize all integrations
    this.crm = CRMIntegration.createSalesforceIntegration(config.salesforce);
    this.inventory = new InventoryIntegration(config.inventory);
    this.address = new AddressValidationService();
    this.tax = new TaxCalculationService();
    this.shipping = new ShippingRateService();
    this.enrichment = new DataEnrichmentService();
  }

  // Auto-populate form from CRM
  async populateCustomerData(form, customerId) {
    const customer = await this.crm.lookupCustomer(customerId);

    if (customer) {
      form.setValues({
        customerName: customer.name,
        email: customer.email,
        phone: customer.phone,
        shippingAddress: customer.defaultShippingAddress,
        billingAddress: customer.defaultBillingAddress,
        paymentMethod: customer.defaultPaymentMethod
      });

      // Show customer insights
      form.showInsights({
        orderHistory: customer.orderHistory,
        lifetimeValue: customer.lifetimeValue,
        loyaltyTier: customer.loyalty.tier
      });
    }

    return customer;
  }

  // Auto-populate product data from inventory
  async populateProductData(form, sku) {
    const product = await this.inventory.lookupProduct(sku);

    if (product) {
      form.setValues({
        productName: product.name,
        description: product.description,
        price: product.price,
        inStock: product.inStock,
        quantity: product.quantity
      });

      // Show product images
      form.showImages(product.images);

      // Show related products
      form.showRelated(product.relatedProducts);
    }

    return product;
  }

  // Validate and standardize address
  async validateAddress(form, address) {
    const validated = await this.address.validateUSAddress(address);

    if (!validated.valid) {
      form.showAddressError({
        message: 'Address could not be validated',
        suggestions: validated.suggestions
      });
      return false;
    }

    if (validated.standardized) {
      form.setValues({
        shippingAddress: validated.standardized
      });
    }

    return true;
  }

  // Calculate shipping rates
  async calculateShipping(form, shipment) {
    const rates = await this.shipping.getRates(shipment);

    form.setShippingOptions(rates.map(rate => ({
      label: `${rate.carrier} ${rate.service} (${rate.transitDays} days)`,
      price: rate.price,
      deliveryDate: rate.deliveryDate
    })));

    return rates;
  }

  // Calculate tax
  async calculateTax(form, order) {
    const tax = await this.tax.calculateSalesTax(order);

    form.setValues({
      taxAmount: tax.taxAmount,
      taxRate: tax.taxRate
    });

    if (tax.estimated) {
      form.showWarning('Tax amount is estimated');
    }

    return tax;
  }

  // Enrich company data
  async enrichCompany(form, domain) {
    const company = await this.enrichment.enrichCompanyData(domain);

    if (company) {
      form.setValues({
        companyName: company.name,
        industry: company.industry,
        employees: company.employeesRange,
        revenue: company.estimatedRevenue
      });

      form.showCompanyInsights(company);
    }

    return company;
  }
}

// Usage
const integration = new ExternalIntegrationSystem({
  salesforce: {
    instance: 'mycompany',
    accessToken: process.env.SALESFORCE_TOKEN
  },
  inventory: {
    baseURL: 'https://api.inventory.com',
    apiKey: process.env.INVENTORY_API_KEY
  }
});

// When user enters customer ID
form.on('customerIdEntered', async (customerId) => {
  await integration.populateCustomerData(form, customerId);
});

// When user enters product SKU
form.on('skuEntered', async (sku) => {
  await integration.populateProductData(form, sku);
});

// When user completes address
form.on('addressComplete', async (address) => {
  await integration.validateAddress(form, address);
});

// When calculating order total
form.on('calculateTotal', async (order) => {
  const [shipping, tax] = await Promise.all([
    integration.calculateShipping(form, order.shipment),
    integration.calculateTax(form, order)
  ]);

  form.setTotal(order.subtotal + shipping[0].price + tax.taxAmount);
});

Consequences

Benefits

Eliminates Redundant Entry: - Data entered once, used everywhere - Automatic population from sources - Massive time savings

Improves Accuracy: - Single source of truth - Real-time validation - Fewer human errors

Better User Experience: - Faster form completion - Less typing required - Immediate feedback

Real-Time Information: - Current prices - Live inventory - Accurate shipping

Data Enrichment: - Additional context automatically - Enhanced decision-making - Better insights

Liabilities

External Dependencies: - APIs can fail - Network issues affect forms - Downtime propagates

Performance Impact: - API calls add latency - Multiple lookups slow forms - Caching needed

Cost: - API calls cost money - Per-request pricing - Volume can be expensive

Complexity: - More integration points - More things to maintain - Debugging harder

Security Concerns: - API keys to manage - Data transmission security - Privacy compliance

Domain Examples

E-commerce: Order Forms

// Customer lookup from Shopify
await integration.crm.lookupCustomer(email);

// Product data from inventory system
await integration.inventory.lookupProduct(sku);

// Address validation via USPS
await integration.address.validateUSAddress(address);

// Shipping rates from FedEx/UPS
await integration.shipping.getRates(shipment);

// Tax calculation via TaxJar
await integration.tax.calculateSalesTax(order);

B2B Sales: Lead Forms

// Company enrichment from Clearbit
const company = await integration.enrichment.enrichCompanyData(domain);

// Person enrichment
const contact = await integration.enrichment.enrichPersonData(email);

// CRM integration to check existing lead
const existing = await integration.crm.lookupCustomer(email);

Healthcare: Patient Intake

// Patient lookup from EHR
const patient = await ehrAPI.getPatient(patientId);

// Insurance verification
const coverage = await insuranceAPI.verifyCoverage(
  patient.insuranceId
);

// Medication lookup
const medications = await rxAPI.getMedications(patientId);

Real Estate: Property Listing

// Property data from MLS
const property = await mlsAPI.getProperty(mlsNumber);

// Tax assessment from county
const assessment = await countyAPI.getTaxAssessment(parcelId);

// School district info
const schools = await schoolAPI.getDistrict(address);

Prerequisites: - Volume 3, Pattern 1: Progressive Disclosure (show data as retrieved) - Volume 3, Pattern 6: Domain-Aware Validation (validate against external rules)

Synergies: - Volume 3, Pattern 10: Semantic Suggestions (enrich with external data) - Volume 3, Pattern 22: Real-Time Lookup (next pattern - live validation) - All patterns (integration enhances all functionality)

Conflicts: - Offline-first applications - Air-gapped systems - Privacy-focused forms

Alternatives: - Manual data entry (no integration) - File imports (batch integration) - Data synchronization (periodic updates)

Known Uses

Salesforce: CRM data auto-population

Shopify: Product and customer integration

Stripe: Payment method validation

Google Maps: Address autocomplete and validation

Amazon: One-click ordering (all data integrated)

UPS/FedEx: Real-time shipping calculators

TurboTax: IRS data integration for returns

Zillow: Property data from multiple sources

Pattern 21: External Data Integration

Opening Scenario: The Form That Couldn't Remember Customers

Context

Problem Statement

Forces

Performance vs Freshness

Dependency vs Reliability

Security vs Connectivity

Simplicity vs Integration

Cost vs Value

Solution

1. API Integration Layer

2. CRM Integration

3. Inventory & Product Integration

4. Address Validation Services

5. Tax Calculation Services

6. Shipping Rate APIs

7. Data Enrichment Services

Implementation Details

Complete External Integration System

Consequences

Benefits

Liabilities

Domain Examples

E-commerce: Order Forms

B2B Sales: Lead Forms

Healthcare: Patient Intake

Real Estate: Property Listing

Known Uses

Further Reading

Academic Foundations

Practical Implementation

Standards & Specifications

Tools & Services

Implementation Examples

Pattern 21: External Data Integration

Opening Scenario: The Form That Couldn't Remember Customers

Context

Problem Statement

Forces

Performance vs Freshness

Dependency vs Reliability

Security vs Connectivity

Simplicity vs Integration

Cost vs Value

Solution

1. API Integration Layer

2. CRM Integration

3. Inventory & Product Integration

4. Address Validation Services

5. Tax Calculation Services

6. Shipping Rate APIs

7. Data Enrichment Services

Implementation Details

Complete External Integration System

Consequences

Benefits

Liabilities

Domain Examples

E-commerce: Order Forms

B2B Sales: Lead Forms

Healthcare: Patient Intake

Real Estate: Property Listing

Related Patterns

Known Uses

Further Reading

Academic Foundations

Practical Implementation

Standards & Specifications

Related Trilogy Patterns

Tools & Services

Implementation Examples