API Development Roadmap & Considerations

Current State

✅ Implemented

• JWT Authentication: Working with 7-day tokens

• Registration Codes: Email-based user signup

• Core Data APIs: Data file management, CSV upload

• Email Campaigns: Complete campaign workflow with Microsoft Graph

• Email Templates: HTML template management

• Live Connectors: Google Sheets, Excel, SQL Server

• Word Templates: Document generation

• Email Tracking: Opens, clicks, engagement

• Image Library: Image storage and serving

• Health Checks: API and database monitoring

• Rate Limiting: IP-based protection

• CORS: Configured for Office Add-in

⚠️ Not Yet Implemented

1. API Keys for Third-Party Developers

Current Issue: JWT tokens are designed for the Word add-in client, not for third-party API consumers.

Needed:

• API key generation system

• Key management (create, revoke, rotate)

• Per-key rate limits

• Per-key usage tracking

• Documentation portal with key management

• Scoped permissions (read-only, full access)

Implementation Priority: HIGH

2. Webhooks

Use Cases:

• Campaign completion notifications

• Email delivery events

• Data sync completion

• Error notifications

Needed:

• Webhook URL registration

• Event subscription system

• Retry logic for failed deliveries

• Webhook signature verification

• Event history/logs

Implementation Priority: MEDIUM

3. GraphQL API

Benefits:

• Flexible queries

• Reduced over-fetching

• Better for mobile apps

• Self-documenting

Implementation Priority: LOW (REST is sufficient for now)

4. Advanced Analytics

Needed:

• Campaign performance dashboards

• Email deliverability metrics

• Data source usage stats

• API usage analytics per user

• Cost tracking

Implementation Priority: MEDIUM

5. Batch Operations

Examples:

• Bulk file upload

• Batch email sending with priority

• Mass document generation

• Bulk data import

Implementation Priority: MEDIUM

---

API Key Implementation Plan

Phase 1: Basic API Keys

Database Schema:

CREATE TABLE ApiKeys (
  Id UNIQUEIDENTIFIER PRIMARY KEY DEFAULT NEWID(),

  UserId UNIQUEIDENTIFIER NOT NULL,

  KeyName NVARCHAR(255) NOT NULL,

  KeyHash NVARCHAR(255) NOT NULL, -- bcrypt hash

  KeyPrefix NVARCHAR(20) NOT NULL, -- First 8 chars for identification

  Status NVARCHAR(50) DEFAULT 'active', -- active, revoked, expired

  Scopes NVARCHAR(MAX), -- JSON array of permissions

  RateLimitTier NVARCHAR(50) DEFAULT 'standard',

  LastUsedAt DATETIME2,

  CreatedAt DATETIME2 DEFAULT GETDATE(),

  ExpiresAt DATETIME2,

  FOREIGN KEY (UserId) REFERENCES Users(Id)

);


CREATE TABLE ApiKeyUsage (

  Id UNIQUEIDENTIFIER PRIMARY KEY DEFAULT NEWID(),

  ApiKeyId UNIQUEIDENTIFIER NOT NULL,

  Endpoint NVARCHAR(255),

  RequestCount INT DEFAULT 1,

  Date DATE DEFAULT CAST(GETDATE() AS DATE),

  FOREIGN KEY (ApiKeyId) REFERENCES ApiKeys(Id)

);

API Key Format:

dp_live_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

• Prefix: dp_ (Data Publisher)

• Environment: live_ or test_

• Key: 32-character random string

Endpoints to Add:

POST   /api/keys               - Create new API key
GET    /api/keys               - List user's API keys

GET    /api/keys/:id           - Get key details (not the key itself)

PUT    /api/keys/:id           - Update key (name, scopes)

DELETE /api/keys/:id           - Revoke key

POST   /api/keys/:id/rotate    - Rotate key (invalidate & create new)

GET    /api/keys/:id/usage     - Get usage statistics

Authentication Middleware Update:

// Accept JWT OR API Key
export const authenticate = async (req, res, next) => {

  const authHeader = req.headers.authorization;


  if (authHeader.startsWith('Bearer dp_')) {

    // API Key authentication

    const apiKey = authHeader.substring(7);

    const user = await validateApiKey(apiKey);

    if (!user) return res.status(401).json({ error: 'Invalid API key' });

    req.user = user;

    req.authMethod = 'apikey';

  } else {

    // JWT authentication (existing)

    passport.authenticate('jwt', { session: false })(req, res, next);

    req.authMethod = 'jwt';

  }


  next();

};

Phase 2: Scoped Permissions

Permission Scopes:

{
  "scopes": [

    "data:read",

    "data:write",

    "email:send",

    "campaigns:read",

    "campaigns:write",

    "templates:read",

    "templates:write",

    "analytics:read",

    "users:read",

    "users:write"

  ]

}

Example Usage:

// Middleware to check scopes
export const requireScope = (scope: string) => {

  return (req, res, next) => {

    if (req.authMethod === 'jwt') {

      // JWT has full access

      return next();

    }


    if (!req.user.scopes.includes(scope)) {

      return res.status(403).json({

        error: 'Insufficient permissions',

        required: scope

      });

    }


    next();

  };

};


// Usage in routes

router.post('/email-campaigns', 

  authenticate, 

  requireScope('campaigns:write'), 

  createCampaign

);

Phase 3: Rate Limit Tiers

Tiers:

const RATE_LIMITS = {
  free: {

    requests: 1000,      // per day

    campaigns: 5,        // per month

    dataFiles: 10        // total

  },

  standard: {

    requests: 10000,

    campaigns: 50,

    dataFiles: 100

  },

  premium: {

    requests: 100000,

    campaigns: 500,

    dataFiles: 1000

  },

  enterprise: {

    requests: 'unlimited',

    campaigns: 'unlimited',

    dataFiles: 'unlimited'

  }

};

Phase 4: Developer Portal

Features Needed:

• Dashboard: API usage graphs, request logs

• Key Management: Create, revoke, view keys

• Documentation: Interactive API docs (Swagger/OpenAPI)

• Sandbox: Test environment with sample data

• Webhooks: Register and test webhooks

• Billing: Usage-based billing integration

Tech Stack:

• Frontend: React or Next.js

• API Docs: Swagger UI or Redoc

• Analytics: Custom charts with Chart.js

• Billing: Stripe integration

---

Webhook Implementation Plan

Webhook Events

enum WebhookEvent {
  // Campaign Events

  CAMPAIGN_STARTED = 'campaign.started',

  CAMPAIGN_COMPLETED = 'campaign.completed',

  CAMPAIGN_FAILED = 'campaign.failed',


  // Email Events

  EMAIL_SENT = 'email.sent',

  EMAIL_DELIVERED = 'email.delivered',

  EMAIL_OPENED = 'email.opened',

  EMAIL_CLICKED = 'email.clicked',

  EMAIL_BOUNCED = 'email.bounced',

  EMAIL_UNSUBSCRIBED = 'email.unsubscribed',


  // Data sync Events

  SYNC_COMPLETED = 'sync.completed',

  SYNC_FAILED = 'sync.failed',


  // Document Events

  DOCUMENT_GENERATED = 'document.generated',

  DOCUMENT_FAILED = 'document.failed'

}

Webhook Payload

{
  "event": "campaign.completed",

  "timestamp": "2026-02-12T10:30:00Z",

  "id": "event-uuid",

  "data": {

    "campaignId": "campaign-uuid",

    "campaignName": "Welcome Campaign",

    "sentCount": 150,

    "failedCount": 2,

    "completedAt": "2026-02-12T10:30:00Z"

  },

  "signature": "sha256-hmac-signature"

}

Webhook Endpoints (API)

POST   /api/webhooks              - Create webhook
GET    /api/webhooks              - List webhooks

GET    /api/webhooks/:id          - Get webhook

PUT    /api/webhooks/:id          - Update webhook

DELETE /api/webhooks/:id          - Delete webhook

POST   /api/webhooks/:id/test     - Test webhook

GET    /api/webhooks/:id/deliveries - Delivery history

POST   /api/webhooks/:id/retry    - Retry failed delivery

Webhook Security

HMAC Signature:

// Server generates signature
const signature = crypto

  .createHmac('sha256', webhookSecret)

  .update(JSON.stringify(payload))

  .digest('hex');


// Client verifies

const receivedSignature = req.headers['x-webhook-signature'];

const isValid = crypto.timingSafeEqual(

  Buffer.from(signature),

  Buffer.from(receivedSignature)

);

---

GraphQL Considerations

Pros

• Flexible queries - clients request exactly what they need

• Single endpoint for all operations

• Strong typing with schema

• Real-time subscriptions

• Self-documenting

Cons

• More complex to implement

• Requires client-side GraphQL library

• Caching more complex than REST

• Learning curve for developers unfamiliar with GraphQL

Decision

Stick with REST for now, add GraphQL later if needed. REST is:

• Well understood

• Easier to cache

• Simpler for basic use cases

• Better supported by tools

---

API Versioning Strategy

Current: URL Versioning (when needed)

/api/v2/email-campaigns

Alternatives Considered

Header Versioning:

Accept: application/vnd.datapublisher.v2+json

Query Parameter:

/api/email-campaigns?version=2

Recommendation

Continue with URL versioning. Only version endpoints that change incompatibly. Most changes should be additive (new optional fields, new endpoints).

---

Breaking Changes Policy

Never Break

• Remove required fields

• Change field types

• Remove endpoints

OK to Change

• Add optional fields

• Add new endpoints

• Deprecate (with warning)

• Add enums (preserve existing)

Deprecation Process

Announce 90 days in advance

Add deprecation header: X-API-Deprecation: true

Update documentation

Provide migration guide

Remove after 90 days

---

Performance Optimization

Current Bottlenecks

No pagination on some endpoints

Full file content in list responses (fixed for data files)

No caching layer

Synchronous email sending (being optimized)

Optimization Plan

1. Add Redis Caching

// Cache frequently accessed data
app.get('/api/email-templates', async (req, res) => {

  const cached = await redis.get(templates:${userId});

  if (cached) return res.json(JSON.parse(cached));


  const templates = await getTemplates(userId);

  await redis.setex(templates:${userId}, 3600, JSON.stringify(templates));


  res.json(templates);

});

2. Database Indexing

-- Add indexes for common queries
CREATE INDEX IX_EmailCampaigns_UserId ON EmailCampaigns(UserId);

CREATE INDEX IX_EmailCampaigns_Status ON EmailCampaigns(Status);

CREATE INDEX IX_EmailTracking_CampaignId ON EmailTracking(CampaignId);

CREATE INDEX IX_DataFiles_UserId_CreatedAt ON DataFiles(UserId, CreatedAt DESC);

3. Async Job Queue

Use Bull Queue for long-running operations:

• Email campaign sending

• Document generation

• Data exports

• File processing

import Queue from 'bull';

const emailQueue = new Queue('email-sending', redisConfig);


emailQueue.process(async (job) => {

  const { campaignId } = job.data;

  await sendCampaign(campaignId);

});


// In API

router.post('/campaigns/:id/send', async (req, res) => {

  const job = await emailQueue.add({ campaignId: req.params.id });

  res.json({ jobId: job.id });

});

---

Security Enhancements

Needed

API Key Encryption: Store hashed, not plaintext

Request Signing: HMAC for webhooks

IP Whitelisting: For API keys

Audit Log: Track all API key usage

Anomaly Detection: Unusual usage patterns

CAPTCHA: For registration code requests

---

Monitoring & Observability

Metrics to Track

• Request count per endpoint

• Response times (p50, p95, p99)

• Error rates

• Rate limit hits

• API key usage

• Campaign success rates

• Email deliverability

Tools to Integrate

• Application Performance Monitoring: New Relic, Datadog

• Logging: Winston + ELK Stack

• Alerting: PagerDuty for critical issues

• Status Page: Statuspage.io integration

---

Next Steps

Immediate (Sprint 1)

✅ Document existing API

⬜ Implement API key system (Phase 1)

⬜ Add usage tracking

⬜ Create developer portal homepage

Short-term (Sprint 2-3)

⬜ Scoped permissions

⬜ Interactive API documentation (Swagger)

⬜ Webhook system (basic)

⬜ Rate limit tiers

Medium-term (Q2)

⬜ Advanced analytics dashboard

⬜ Batch operations

⬜ SDK libraries (JS, Python)

⬜ Enhanced webhook features

Long-term (Q3+)

⬜ GraphQL endpoint (if needed)

⬜ API marketplace/integrations

⬜ Enterprise features (SLAs, dedicated support)

---

Questions to Resolve

Pricing Model: How to charge for API usage?

- Per request?

- Per email sent?

- Monthly subscription tiers?

Free Tier: What should be included?

- Generous enough for trials

- Limited enough to encourage upgrades

Enterprise Needs: What do large customers need?

- Dedicated IP ranges?

- Custom rate limits?

- On-premise deployment?

Partner Program: API partners/integrations?

- Revenue sharing?

- Co-marketing?

---

Contact

For API development questions:

• Technical Lead: api-dev@datapublisher.io

• Product Manager: product@datapublisher.io