docs.do.dev
Internal DocumentationArchived DocumentationDo dev legacyArchiveClerk migration

Clerk-Convex Data Synchronization Design

Overview

This document outlines the design for synchronizing user data between Clerk (authentication) and Convex (data storage) in the do.dev platform.

Architecture

graph TB
    subgraph "Authentication Layer"
        CLERK[Clerk User Management]
        WEBHOOK[Clerk Webhooks]
    end
    
    subgraph "Synchronization Layer"
        SYNC[Sync Service]
        QUEUE[Event Queue]
        MAPPER[Data Mapper]
    end
    
    subgraph "Data Layer"
        CONVEX_USERS[Convex Users Table]
        CONVEX_PROFILES[Convex Profiles Table]
        CONVEX_FILES[Convex Files Table]
    end
    
    CLERK --> WEBHOOK
    WEBHOOK --> QUEUE
    QUEUE --> SYNC
    SYNC --> MAPPER
    MAPPER --> CONVEX_USERS
    MAPPER --> CONVEX_PROFILES
    CONVEX_USERS --> CONVEX_FILES

Data Models

Clerk User Model

interface ClerkUser {
  id: string;
  emailAddresses: EmailAddress[];
  firstName: string | null;
  lastName: string | null;
  fullName: string | null;
  username: string | null;
  imageUrl: string;
  publicMetadata: {
    appId?: string;
    role?: string;
    organizationId?: string;
  };
  privateMetadata: {
    convexUserId?: string;
    syncedAt?: number;
  };
  createdAt: number;
  updatedAt: number;
}

Convex User Model

interface ConvexUser {
  _id: Id<"users">;
  clerkId: string;
  email: string;
  name: string | null;
  username: string | null;
  imageUrl: string;
  role: "user" | "admin" | "super_admin";
  organizationId: string | null;
  appId: string;
  createdAt: number;
  updatedAt: number;
  lastSyncAt: number;
}

Synchronization Events

Clerk Webhook Events

  1. user.created - New user registration
  2. user.updated - Profile changes (name, email, image)
  3. user.deleted - Account deletion
  4. organizationMembership.created - User joins organization
  5. organizationMembership.updated - Role changes
  6. organizationMembership.deleted - User leaves organization

Event Processing Flow

// apps/dodev/app/api/webhooks/clerk/route.ts
export async function POST(request: Request) {
  const payload = await request.text();
  const signature = request.headers.get('svix-signature');
  
  // Verify webhook signature
  const webhook = new Webhook(process.env.CLERK_WEBHOOK_SECRET!);
  const event = webhook.verify(payload, signature);
  
  // Process event based on type
  switch (event.type) {
    case 'user.created':
      await handleUserCreated(event.data);
      break;
    case 'user.updated':
      await handleUserUpdated(event.data);
      break;
    case 'user.deleted':
      await handleUserDeleted(event.data);
      break;
    default:
      console.log(`Unhandled event type: ${event.type}`);
  }
  
  return new Response('OK', { status: 200 });
}

Implementation Strategy

Phase 1: Basic Sync (Current)

  • ✅ Manual sync on first login
  • ✅ User creation in Convex from Clerk data
  • ✅ Basic role assignment based on metadata

Phase 2: Webhook Integration

  • Set up Clerk webhook endpoint
  • Implement event handlers for user lifecycle
  • Add retry logic for failed synchronizations
  • Implement sync status tracking

Phase 3: Advanced Features

  • Bidirectional sync for profile updates
  • Organization management sync
  • Bulk sync operations for existing users
  • Real-time sync status monitoring

Sync Service Implementation

Core Sync Functions

// tools/convex/convex/clerk-sync.ts
import { mutation, action } from "./_generated/server";
import { v } from "convex/values";

export const syncUserFromClerk = mutation({
  args: {
    clerkId: v.string(),
    email: v.string(),
    name: v.optional(v.string()),
    imageUrl: v.string(),
    role: v.string(),
    organizationId: v.optional(v.string()),
  },
  handler: async (ctx, args) => {
    // Check if user already exists
    const existingUser = await ctx.db
      .query("users")
      .withIndex("by_clerk_id", (q) => q.eq("clerkId", args.clerkId))
      .first();

    if (existingUser) {
      // Update existing user
      await ctx.db.patch(existingUser._id, {
        email: args.email,
        name: args.name,
        imageUrl: args.imageUrl,
        role: args.role as any,
        organizationId: args.organizationId,
        lastSyncAt: Date.now(),
        updatedAt: Date.now(),
      });
      return existingUser._id;
    } else {
      // Create new user
      return await ctx.db.insert("users", {
        clerkId: args.clerkId,
        email: args.email,
        name: args.name,
        imageUrl: args.imageUrl,
        role: args.role as any,
        organizationId: args.organizationId,
        appId: "dodev",
        createdAt: Date.now(),
        updatedAt: Date.now(),
        lastSyncAt: Date.now(),
      });
    }
  },
});

export const handleClerkWebhook = action({
  args: {
    eventType: v.string(),
    data: v.any(),
  },
  handler: async (ctx, args) => {
    switch (args.eventType) {
      case "user.created":
        await ctx.runMutation(internal.clerkSync.syncUserFromClerk, {
          clerkId: args.data.id,
          email: args.data.email_addresses[0]?.email_address || "",
          name: args.data.full_name,
          imageUrl: args.data.image_url,
          role: args.data.public_metadata?.role || "user",
          organizationId: args.data.public_metadata?.organizationId,
        });
        break;
      
      case "user.updated":
        await ctx.runMutation(internal.clerkSync.syncUserFromClerk, {
          clerkId: args.data.id,
          email: args.data.email_addresses[0]?.email_address || "",
          name: args.data.full_name,
          imageUrl: args.data.image_url,
          role: args.data.public_metadata?.role || "user",
          organizationId: args.data.public_metadata?.organizationId,
        });
        break;
      
      case "user.deleted":
        await ctx.runMutation(internal.clerkSync.deleteUser, {
          clerkId: args.data.id,
        });
        break;
    }
  },
});

Error Handling and Retry Logic

// Webhook handler with retry logic
export const processWebhookWithRetry = action({
  args: {
    eventType: v.string(),
    data: v.any(),
    retryCount: v.optional(v.number()),
  },
  handler: async (ctx, args) => {
    const maxRetries = 3;
    const retryCount = args.retryCount || 0;
    
    try {
      await ctx.runAction(internal.clerkSync.handleClerkWebhook, {
        eventType: args.eventType,
        data: args.data,
      });
    } catch (error) {
      if (retryCount < maxRetries) {
        // Schedule retry with exponential backoff
        await ctx.scheduler.runAfter(
          Math.pow(2, retryCount) * 1000, // 1s, 2s, 4s delays
          internal.clerkSync.processWebhookWithRetry,
          {
            eventType: args.eventType,
            data: args.data,
            retryCount: retryCount + 1,
          }
        );
      } else {
        // Log failure after max retries
        console.error(`Failed to process webhook after ${maxRetries} retries:`, error);
      }
    }
  },
});

Security Considerations

Webhook Security

  1. Signature Verification: Always verify Clerk webhook signatures
  2. HTTPS Only: Webhook endpoints must use HTTPS in production
  3. Rate Limiting: Implement rate limiting on webhook endpoints
  4. Idempotency: Handle duplicate webhook deliveries gracefully

Data Protection

  1. Minimal Data: Only sync necessary user data
  2. Encryption: Sensitive data encrypted at rest
  3. Access Control: Limit access to sync operations
  4. Audit Logging: Log all sync operations for compliance

Monitoring and Observability

Metrics to Track

  1. Sync Success Rate: Percentage of successful synchronizations
  2. Sync Latency: Time from Clerk event to Convex update
  3. Error Rate: Failed sync attempts and reasons
  4. Data Consistency: Periodic audits of Clerk vs Convex data

Alerting

  1. High Error Rate: Alert when sync failures exceed threshold
  2. Sync Delays: Alert when sync latency is too high
  3. Data Inconsistencies: Alert when audit finds mismatches

Migration Strategy

Existing Users

  1. Audit Current State: Identify users needing sync
  2. Bulk Sync Operation: One-time sync of existing Clerk users
  3. Validation: Verify sync completeness and accuracy
  4. Monitoring: Monitor sync health post-migration

Rollback Plan

  1. Backup Strategy: Regular backups before major sync operations
  2. Feature Flags: Ability to disable sync and fall back to manual mode
  3. Data Recovery: Process to restore from backups if needed

Testing Strategy

Unit Tests

  • Test individual sync functions
  • Mock Clerk webhook payloads
  • Verify error handling and retry logic

Integration Tests

  • End-to-end webhook processing
  • Database state verification
  • Performance testing under load

Manual Testing

  • Create/update/delete users in Clerk
  • Verify changes propagate to Convex
  • Test edge cases and error scenarios

Configuration

Environment Variables

# Clerk Configuration
CLERK_WEBHOOK_SECRET=whsec_xxx...
CLERK_SECRET_KEY=sk_live_xxx...

# Convex Configuration
CONVEX_URL=https://your-deployment.convex.cloud

# Sync Configuration
SYNC_ENABLED=true
SYNC_RETRY_MAX_ATTEMPTS=3
SYNC_RETRY_DELAY_MS=1000

Webhook URL

  • Development: https://your-dev-domain.dev/api/webhooks/clerk
  • Production: https://do.dev/api/webhooks/clerk

Last updated: January 2025

CLERK AUTHENTICATION ARCHITECTURE

Previous Page

CLERK ENV VARIABLES

Next Page

On this page

Clerk-Convex Data Synchronization DesignOverviewArchitectureData ModelsClerk User ModelConvex User ModelSynchronization EventsClerk Webhook EventsEvent Processing FlowImplementation StrategyPhase 1: Basic Sync (Current)Phase 2: Webhook IntegrationPhase 3: Advanced FeaturesSync Service ImplementationCore Sync FunctionsError Handling and Retry LogicSecurity ConsiderationsWebhook SecurityData ProtectionMonitoring and ObservabilityMetrics to TrackAlertingMigration StrategyExisting UsersRollback PlanTesting StrategyUnit TestsIntegration TestsManual TestingConfigurationEnvironment VariablesWebhook URL