---
title: "Discourse to Contentful Migration: A Technical Guide"
slug: discourse-to-contentful-migration-a-technical-guide
date: 2026-08-28
author: Abdul Wahab
categories: [Discourse, Contentful, Migration Guide]
excerpt: "A technical guide to migrating Discourse forum content into Contentful: API extraction, content modeling, Markdown-to-Rich-Text conversion, asset handling, and rate limits."
tldr: "Discourse to Contentful is a content remodel, not a lift-and-shift: extract topics and posts, design the Contentful schema first, convert content to Rich Text, and import with idempotent scripts."
canonical: https://clonepartner.com/blog/discourse-to-contentful-migration-a-technical-guide
---

# Discourse to Contentful Migration: A Technical Guide


# Discourse to Contentful Migration: A Technical Guide

Migrating content from Discourse to Contentful means reshaping threaded, user-generated forum data into structured, API-first content entries. There is no off-the-shelf connector for this. Much like a [Wix to Webflow migration](https://clonepartner.com/blog/blog/wix-to-webflow-migration-a-complete-technical-guide), you need a custom pipeline that extracts from Discourse's PostgreSQL database or REST API, transforms post content into Contentful's Rich Text document format, and loads entries through the Content Management API (CMA) — while respecting rate limits and dependency ordering on both sides.

The core challenge is a [data model mismatch](https://clonepartner.com/blog/blog/bloomreach-to-wordpress-migration-a-technical-guide). **Discourse is discussion-first:** topics contain posts, and post payloads expose both `raw` Markdown source and `cooked` rendered HTML. **Contentful is structured-content-first:** everything lives as typed entries and assets inside environments, and Rich Text is a JSON document model — not HTML, not Markdown. That mismatch is the real migration problem.

This guide covers the full technical workflow: scoping, extraction, content modeling, format conversion, asset handling, rate limit management, rollback strategy, and the edge cases that trip up most teams.

## Decide What to Migrate

Before you touch an API, define the target state:

- **Knowledge base extraction:** The first post of each topic becomes a canonical article. Replies are dropped, summarized, or linked as supporting discussion.
- **Discussion archive:** Each topic and its replies are preserved as content objects.
- **Editorial rewrite:** Selected topics become structured articles, FAQs, or release notes. The thread itself stays in Discourse as the historical record.

Most teams do not want a 1:1 forum clone inside Contentful. They want selected categories, first posts, attachments, authorship context, and enough thread history for traceability. If you skip this decision, you pay to import noise that editors will never maintain.

> [!NOTE]
> **Canonical content** means the version your editors and frontends treat as source of truth after cutover. Decide whether that is the first post, the full thread, or a rewritten article derived from the thread — before you write any extraction code.

## Understand the Discourse Data Model

Discourse uses a **PostgreSQL** database with a relational schema. The core entities:

- **Categories** — top-level content groupings
- **Topics** — individual discussion threads, each belonging to a category
- **Posts** — individual messages within a topic (the first post is the topic's body content)
- **Users** — authors with profiles, trust levels, and badges
- **Tags** — optional metadata attached to topics
- **Uploads** — images, files, and attachments referenced in posts

Key relationships: a category contains many topics. A topic contains many posts. Each post belongs to one user. Tags attach at the topic level.

Post content is stored in two forms: the **raw Markdown** source (what the author typed) and the **cooked HTML** (rendered output). Both are accessible via the API and useful during migration — which one you transform depends on your conversion approach, covered below.

## Extract Data from Discourse

You have two extraction paths: the **Discourse REST API** or **direct database access**. The right choice depends on your hosting situation and data volume.

### Option 1: Discourse REST API

For hosted Discourse or when you want application-level extraction, the REST API is your path. Key endpoints:

- `GET /categories.json` — list all categories
- `GET /c/{category_id}.json` — topics in a category
- `GET /t/{topic_id}.json` — a single topic with its first batch of posts
- `GET /t/{topic_id}/posts.json?post_ids []=...` — fetch specific posts by ID
- `GET /posts/{post_id}.json` — a single post with both `raw` and `cooked`

> [!WARNING]
> The topic endpoint returns only 20 posts per request by default. For topics with many replies, use the `post_stream.stream` array from the initial response to fetch remaining posts in batches via `/t/{topic_id}/posts.json?post_ids []=...`.

Rate limits on Discourse's API are configurable per instance. Self-hosted instances set limits in `app.yml` via `DISCOURSE_MAX_REQS_PER_IP_PER_MINUTE`. Hosted Discourse forums typically allow 60 requests per minute for admin API keys. Build a delay of at least 1 second between requests.

Create a scoped API key instead of reaching for a broad admin key by default. Scopes only reduce what the chosen user can already do.

```python
import requests
import time

DISCOURSE_URL = "https://your-forum.example.com"
API_KEY = "your-admin-api-key"
API_USERNAME = "system"

def get_topic(topic_id):
    resp = requests.get(
        f"{DISCOURSE_URL}/t/{topic_id}.json",
        headers={"Api-Key": API_KEY, "Api-Username": API_USERNAME}
    )
    resp.raise_for_status()
    return resp.json()

def get_all_posts_for_topic(topic_id):
    topic = get_topic(topic_id)
    posts = topic["post_stream"]["posts"]
    all_post_ids = topic["post_stream"]["stream"]
    fetched_ids = {p["id"] for p in posts}
    remaining = [pid for pid in all_post_ids if pid not in fetched_ids]

    # Fetch in chunks of 20
    for i in range(0, len(remaining), 20):
        chunk = remaining[i:i+20]
        params = "&".join(f"post_ids[]={pid}" for pid in chunk)
        resp = requests.get(
            f"{DISCOURSE_URL}/t/{topic_id}/posts.json?{params}",
            headers={"Api-Key": API_KEY, "Api-Username": API_USERNAME}
        )
        resp.raise_for_status()
        posts.extend(resp.json()["post_stream"]["posts"])
        time.sleep(1)  # Respect rate limits

    return posts
```

### Option 2: Direct PostgreSQL Access

If you have server access (self-hosted Discourse), querying PostgreSQL directly is faster and avoids API rate limits entirely. The key tables:

```sql
-- Core content
SELECT id, title, category_id, created_at, views, like_count
FROM topics WHERE deleted_at IS NULL AND visible = true;

SELECT id, topic_id, user_id, raw, cooked, created_at, post_number
FROM posts WHERE deleted_at IS NULL;

-- Categories
SELECT id, name, slug, parent_category_id, description
FROM categories;

-- Tags
SELECT t.name, tt.topic_id
FROM tags t JOIN topic_tags tt ON t.id = tt.tag_id;

-- Users (for author attribution)
SELECT id, username, name, email FROM users;
```

Self-hosted instances also have command-line tools like `export_category` and `export_topics`, which produce bounded source sets in Discourse-shaped JSON.

> [!TIP]
> Always extract to a local intermediate format (JSON or SQLite) first. Never transform and load in the same pass as extraction. This gives you a replayable pipeline and makes debugging far easier.

## Design the Contentful Content Model

This is where most migrations go wrong. Teams try to replicate Discourse's structure inside Contentful. **Don't do that.** Contentful is not a forum — it's a structured content platform. Design your content model around how the content will be *consumed*, not how it was *created*.

Each Contentful content type supports up to **50 fields**. Short text tops out at 256 characters, long text at 50,000, and Rich Text at 200,000 characters.

A practical content model for migrated forum content:

| Discourse object | Contentful content type | Key fields |
|---|---|---|
| Topic | `article` | title, slug, body (Rich Text), sourceRaw (Text), sourceTopicId (Integer), originalUrl, publishedAt, viewCount |
| First post | Body field on `article` | Converted from first post's Markdown or HTML |
| Reply | `reply` entries or omit | Only import if discussion history matters |
| Category | `category` | name, slug, description, parent (self-reference for subcategories) |
| Tag | Taxonomy entries | Use Contentful tags only for small tag sets |
| Upload | Asset | Rewrite references after asset IDs exist |
| User | `author` | username, displayName, avatar (Asset link) |

Every imported object should keep its legacy identifiers (Discourse topic ID, post ID, original URL) so you can rebuild links, deduplicate reruns, and trace where a record came from months later.

> [!WARNING]
> **Use Contentful tags only for low-cardinality metadata.** Contentful allows 1,000 tags per environment and 100 per entry. Large Discourse forums usually need taxonomy entries instead.

> [!NOTE]
> **Do not migrate forum replies into your main content model** unless you specifically need [thread preservation](https://clonepartner.com/blog/blog/desk365-to-jira-service-management-migration-guide). Headless CMS platforms are built for structured content, not high-volume user-generated comments. If replies matter, add a dedicated `reply` content type — but consider whether that complexity is worth maintaining.

### Define Content Models in Code

Use Contentful's migration tooling to create content types programmatically. This makes your schema repeatable and versioned before you import anything.

```js
module.exports = function (migration) {
  const article = migration.createContentType('article', {
    name: 'Article'
  })

  article.createField('title').name('Title').type('Symbol').required(true)
  article.createField('slug').name('Slug').type('Symbol').required(true)
  article.createField('body').name('Body').type('RichText')
  article.createField('sourceTopicId').name('Source Topic ID').type('Integer')
  article.createField('sourceRaw').name('Source Raw').type('Text')
  article.createField('originalUrl').name('Original URL').type('Symbol')
  article.createField('category').name('Category').type('Link').linkType('Entry')
  article.createField('author').name('Author').type('Link').linkType('Entry')
  article.createField('tags').name('Tags').type('Array').items({ type: 'Symbol' })
  article.createField('publishedAt').name('Published At').type('Date')
}
```

Build in a **sandbox environment** first. Contentful environments work like Git branches for content — create a dedicated `migration-test` environment to run imports against. Validate everything there before promoting to production.

If you have Premium or Enterprise, environment aliases give you the cleanest cutover. The alias is a static ID that can point to a different target environment. Without aliases, you need an application config switch or a short freeze window.

## Convert Discourse Content to Contentful Rich Text

This is the hardest part of the migration. Contentful's Rich Text is a structured JSON document format with a strict node type vocabulary. The complete set of supported node types is:

**Block nodes:** `document`, `paragraph`, `heading-1` through `heading-6`, `unordered-list`, `ordered-list`, `list-item`, `blockquote`, `hr`, `embedded-asset-block`, `embedded-entry-block`, `table`, `table-row`, `table-cell`, `table-header-cell`

**Inline nodes:** `hyperlink`, `asset-hyperlink`, `entry-hyperlink`, `embedded-entry-inline`

**Text marks:** `bold`, `italic`, `underline`, `code`, `superscript`, `subscript`

Every node must have a `content` array (even if empty), a `data` object, and the correct `nodeType` string. A missing `content` array will cause the entire field to render blank in the web app even though the API accepted the payload without error. Contentful does not support custom node types — any Discourse structure that does not map to this vocabulary must be converted, flattened, or dropped.

You have two viable conversion approaches.

### Approach 1: Raw Markdown → Rich Text

Use Contentful's official **`@contentful/rich-text-from-markdown`** library. It handles standard Markdown nodes: headings, paragraphs, bold, italic, links, lists, blockquotes, horizontal rules, and inline code.

What it does **not** handle:

- **Images** — you must provide a callback to upload as Contentful Assets and return `embedded-asset-block` nodes
- **Tables** — Contentful Rich Text added table support in newer schema versions; verify your `@contentful/rich-text-types` version includes `table`, `table-row`, `table-cell`, and `table-header-cell` before assuming table conversion will work
- **Fenced code blocks** — treated as unsupported nodes, need custom handling
- **Discourse-specific syntax** — `[quote]` blocks, `@mentions`, emoji shortcodes (`:smile:`), and onebox embeds need preprocessing

Preprocess Discourse extensions before conversion:

```javascript
function preprocessDiscourseMarkdown(raw) {
  let cleaned = raw;

  // Convert Discourse quote blocks to standard blockquotes
  cleaned = cleaned.replace(
    /\[quote="([^"]+)"\]([\s\S]*?)\[\/quote\]/g,
    '> **$1:**\n> $2'
  );

  // Convert @mentions to bold text
  cleaned = cleaned.replace(/@(\w+)/g, '**@$1**');

  // Strip emoji shortcodes or convert to unicode
  cleaned = cleaned.replace(/:([a-z_]+):/g, (match, code) => {
    return emojiMap[code] || match;
  });

  // Remove onebox / embed placeholders
  cleaned = cleaned.replace(/<aside class="onebox">[\s\S]*?<\/aside>/g, '');

  return cleaned;
}
```

Then convert with a callback for unsupported nodes:

```javascript
const { richTextFromMarkdown } = require('@contentful/rich-text-from-markdown');

const document = await richTextFromMarkdown(preprocessedMarkdown, async (node) => {
  if (node.type === 'image') {
    const asset = await uploadAsset(node.url, node.alt);
    return {
      nodeType: 'embedded-asset-block',
      content: [],
      data: { target: { sys: { type: 'Link', linkType: 'Asset', id: asset.sys.id } } }
    };
  }
  return null; // Skip other unsupported nodes
});
```

### Approach 2: Cooked HTML → Rich Text

If you want to skip preprocessing Discourse's custom Markdown extensions, work from the `cooked` HTML instead. Discourse has already rendered `[quote]` blocks, oneboxes, and mentions into standard HTML elements, giving you a DOM to work with.

Use an HTML-to-Rich-Text parser with custom node mapping rules. Because these parsers rely on DOM APIs, you need `jsdom` or similar in Node.js:

```javascript
const { JSDOM } = require('jsdom');
const dom = new JSDOM('');
global.document = dom.window.document;

// Parse cooked HTML and map Discourse-specific elements
// (aside.quote, img tags, etc.) to Contentful Rich Text nodes
// with custom handlers for each element type
```

**Discourse HTML → Contentful Rich Text node mapping:**

| Discourse HTML element | Contentful Rich Text node |
|---|---|
| `<p>` | `paragraph` |
| `<h1>`–`<h6>` | `heading-1` through `heading-6` |
| `<ul>` / `<ol>` | `unordered-list` / `ordered-list` |
| `<li>` | `list-item` |
| `<blockquote>`, `<aside class="quote">` | `blockquote` |
| `<strong>`, `<b>` | `bold` mark on text |
| `<em>`, `<i>` | `italic` mark on text |
| `<code>` (inline) | `code` mark on text |
| `<pre><code>` | `paragraph` with `code` mark (no native code block node) |
| `<img>` | `embedded-asset-block` (after uploading as Asset) |
| `<a>` | `hyperlink` |
| `<hr>` | `hr` |
| `<table>` | `table` (if supported in your schema version) |

The trade-off: `cooked` HTML avoids Discourse-specific Markdown parsing, but you take on HTML-to-AST conversion instead. Either path requires custom handling — pick the one that matches your team's comfort.

> [!WARNING]
> Some Discourse posts contain raw HTML mixed into the Markdown (common with older imported content or admin posts). The `rich-text-from-markdown` library will choke on this. If you go the Markdown route, run an HTML-to-Markdown preprocessor (like `turndown`) on the raw content first.

## Upload Assets Before Creating Entries

Contentful enforces strict referential integrity. If an article's Rich Text contains an embedded image, that image must exist as a **published Asset** before the article entry is created. You cannot pass raw image URLs into Rich Text fields.

Discourse stores uploads either locally on the server filesystem or in an S3-compatible object store. Each upload is referenced in posts by a short URL like `/uploads/default/original/1X/abc123.png`.

The asset migration sequence:

1. **Parse** Discourse content for image URLs and attachment references
2. **Download** from Discourse (resolve short URLs to full paths)
3. **Upload** to Contentful via the CMA
4. **Process** the asset (`asset.processForAllLocales()` — this is asynchronous)
5. **Poll** until processing completes (wait for `fields.file ['en-US'].url` to be populated)
6. **Publish** the asset
7. **Store** the mapping between the old Discourse URL and the new Contentful Asset ID

```javascript
const { createClient } = require('contentful-management');

const client = createClient({
  accessToken: process.env.CONTENTFUL_MANAGEMENT_TOKEN
});

async function uploadAssetToContentful(environment, imageUrl, fileName) {
  let asset = await environment.createAsset({
    fields: {
      title: { 'en-US': fileName },
      file: {
        'en-US': {
          contentType: 'image/jpeg',
          fileName: fileName,
          upload: imageUrl // Contentful fetches this URL
        }
      }
    }
  });

  asset = await asset.processForAllLocales();
  // Poll or wait — publishing an unprocessed asset will fail
  asset = await asset.publish();

  return asset.sys.id;
}
```

> [!WARNING]
> Uploads to the Contentful Upload API can be up to 1,000 MB but are not resumable. If an uploaded file is not associated and processed within 24 hours, Contentful deletes it. Note that `text/html` and `text/javascript` files cannot be uploaded as assets — PDFs and binaries are fine.

## Import Entries and Manage Rate Limits

The CMA enforces a default rate limit of **7 requests per second** (some plans allow 10). When you hit it, you get a `429 Too Many Requests` response with an `X-Contentful-RateLimit-Reset` header specifying how many seconds to wait.

**Deriving safe concurrency:** At 7 req/s, you have a budget of ~143ms per request slot. If each CMA call takes approximately 300–500ms end-to-end (network + processing), the safe concurrent request count is: `floor(7 × avg_latency_seconds)` = `floor(7 × 0.4)` ≈ 2–3. Setting `pLimit(3)` works when average latency is above ~430ms. If your network is fast and calls complete in under 200ms, `pLimit(3)` can burst above 7 req/s. Monitor your actual `429` rate during a test run and tune accordingly — or add an explicit minimum inter-request delay of `1000 / rate_limit_per_second` ms.

For a migration of 5,000 topics — each requiring entry creation plus publish (2 requests) — that's 10,000 requests minimum. At 7 req/s with no other overhead, that's approximately 24 minutes just for entries, not counting asset uploads (which involve create + processForAllLocales polling + publish = 3+ requests per asset).

Relying on SDK retry logic for bulk migrations is an anti-pattern. It leads to memory bloat, socket hangups, and unpredictable execution times. Control concurrency at the application level:

```javascript
const pLimit = require('p-limit');

// Tune this value based on measured average request latency
const limit = pLimit(3);

async function migrateTopics(topics, environment) {
  const tasks = topics.map(topic => limit(async () => {
    try {
      const entryId = `discourse-topic-${topic.id}`;
      const entry = await environment.createEntryWithId('article', entryId, {
        fields: {
          title: { 'en-US': topic.title },
          slug: { 'en-US': topic.slug },
          body: { 'en-US': topic.richTextAST },
          sourceTopicId: { 'en-US': topic.id },
          originalUrl: { 'en-US': topic.originalUrl }
        }
      });
      await entry.publish();
    } catch (error) {
      if (error.status === 429) {
        const wait = (error.headers?.['x-contentful-ratelimit-reset'] || 1) * 1000;
        await new Promise(r => setTimeout(r, wait));
      } else {
        console.error(`Failed to migrate topic ${topic.id}:`, error.message);
      }
    }
  }));

  await Promise.all(tasks);
}
```

### Entry Creation Order

Contentful uses references (links) between entries. Create entries in dependency order:

1. **Categories** first
2. **Authors** second
3. **Assets** (images) third — upload, process, publish
4. **Articles** last — these reference categories, authors, and embedded assets

If you create an article that references a category entry that doesn't exist yet, the publish call fails with `422 Unprocessable Entity` citing unresolvable links.

### Deterministic Entry IDs

The CMA lets you provide your own entry IDs (1–64 characters, alphanumerics, dots, hyphens, or underscores). A pattern like `discourse-topic-12345` prevents duplicates when you run your migration script multiple times — and you will run it multiple times.

For large batches, consider Contentful's bulk entry operations: up to 10,000 entries per job, one in-flight job per space, with asynchronous polling.

## Edge Cases That Break Naive Migrations

### CMA Error Reference

Understanding which error code maps to which migration cause prevents wasted debugging time:

| HTTP status | Common migration cause | Resolution |
|---|---|---|
| `400 Bad Request` | Malformed Rich Text AST (missing `content` array, invalid `nodeType`, wrong `data` structure) | Validate AST structure before sending; log the full payload |
| `409 Conflict` | Entry with that ID already exists and you used `createEntryWithId` | Use `getEntry` + update, or skip if already published |
| `422 Unprocessable Entity` | Missing required field, wrong locale key, unresolvable link reference, or entry references unpublished asset | Check that referenced entries/assets are published; verify locale strings match space configuration |
| `429 Too Many Requests` | CMA rate limit exceeded | Read `X-Contentful-RateLimit-Reset` header; wait that many seconds before retrying |
| `500 Internal Server Error` | Transient Contentful service error | Retry with exponential backoff; log payload for inspection |
| `413 Payload Too Large` | Rich Text field exceeds 200,000 character limit | Split content or truncate; store overflow in a separate `Text` field |

### Internal Link Resolution

Discourse posts frequently link to other topics using relative URLs (`/t/slug/1234`). These links break in Contentful. During transformation, either:

- Convert them to absolute URLs pointing to the original Discourse instance
- Map them to new Contentful entry IDs or frontend URLs (requires a two-pass migration)
- Strip them and add a "Related Articles" reference field instead

### Hidden, Unlisted, and Deleted Content

Discourse exposes flags like `visible`, `hidden`, `user_deleted`, and `deleted_at`. Moderators can also "unlist" topics or create "whispers" (internal staff notes). Decide upfront whether these become archived records, are filtered out, or exist only in an audit export. Do not let the importer make that policy accidentally.

### Optimistic Locking Conflicts

Contentful uses **version-based optimistic locking**. Every entry has a version number. To update or publish, you must send the current version via `X-Contentful-Version`. If another process modified the entry between your read and write, you get a `409 Conflict`. The fix: always fetch the latest entry version immediately before publishing.

```javascript
// Correct pattern: fetch fresh version immediately before publish
const freshEntry = await environment.getEntry(entryId);
await freshEntry.publish(); // uses the current version number
```

### Missing Locales

Contentful requires explicit locale definitions for every field (e.g., `{'en-US': 'My Title'}`). Discourse does not enforce locales. If your Contentful space has a required default locale, your migration script must map content to that exact locale string or the CMA rejects the payload with `422 Unprocessable Entity`.

**Locale fallback behavior in multilingual spaces:** Contentful evaluates locale fallback chains at delivery time, not at write time. If you import content only into `en-US` but your space has `de-DE` with fallback to `en-US`, the CDA returns `en-US` content for German requests — but the editor UI will show those fields as empty for `de-DE`, which confuses editors. Explicitly populate only the locales you plan to maintain, and document which locales fall back to which source.

For multilingual communities, set locale strategy before import. Only create locales you plan to maintain.

### Author History Loss

Contentful's CLI import recreates entries under the token owner, which loses author history. The safest pattern is to store legacy author, created-at, and source URLs as explicit fields on each entry if provenance matters.

### Post-Migration Retrieval Complexity

One trap surfaces after migration, not during it. Contentful's GraphQL API limits query complexity to 11,000 entities, and Rich Text links add cost. If you model every reply as a linked block inside one giant article, retrieval can become harder than import. Linked reply entries or paginated views are the safer design.

## Rollback Strategy

If you discover a systematic conversion error after importing 3,000 entries, you need a cleanup path. Contentful has no bulk delete by entry ID prefix — but deterministic entry IDs make scripted cleanup viable.

```javascript
// Cleanup script: delete all entries matching migration prefix
async function rollbackMigration(environment, prefix) {
  let skip = 0;
  const limit = 100;

  while (true) {
    const entries = await environment.getEntries({
      'sys.id[match]': prefix,
      limit,
      skip
    });

    if (entries.items.length === 0) break;

    for (const entry of entries.items) {
      try {
        if (entry.isPublished()) await entry.unpublish();
        await entry.delete();
      } catch (err) {
        console.error(`Failed to delete ${entry.sys.id}:`, err.message);
      }
    }

    skip += entries.items.length;
  }
}

// Usage: rollbackMigration(environment, 'discourse-topic-')
```

The same pattern applies to assets: query by a filename prefix or tag applied during upload, then unpublish and delete. Apply a Contentful tag (e.g., `migration-batch-2024-03`) to every entry and asset created during a run — this gives you a targeted handle for both rollback and audit without relying on ID pattern matching.

For large rollbacks, use the Contentful bulk operations API to unpublish up to 10,000 entries per job before deleting.

## Validation and Cutover

Do not assume a successful API response means the content looks correct. Contentful's Rich Text AST is finicky — a missing `content` array inside a paragraph node will render the entire field blank in the web app, even if the API accepted the payload.

Run a validation pass:

- [ ] Count topics, posts, uploads, and tags in scope before and after
- [ ] Verify every Contentful entry stores the legacy topic or post ID
- [ ] Programmatically check that `body` fields are not null and contain expected node types
- [ ] Verify all embedded asset links resolve to published assets
- [ ] Diff a sample of rendered bodies against Discourse pages
- [ ] Crawl internal links and attachment links after rewrite
- [ ] Check for locale gaps and tag overflow
- [ ] Re-run the importer on a sample set to prove idempotency
- [ ] Build a URL redirect map from old Discourse URLs to new content paths
- [ ] Confirm `409 Conflict` handling: re-running against existing entries should update, not fail

Validate through delivery-style reads, not just the editor UI. Use the Content Delivery API (CDA) to verify what your frontend will actually see.

### Phased Cutover

For low-downtime cutovers, Discourse webhooks can POST content to an external endpoint when topics or posts change, supporting SHA-256 secret signatures. Verify the signature before processing any payload:

```javascript
const crypto = require('crypto');

function verifyDiscourseWebhook(payload, signature, secret) {
  const expected = 'sha256=' + crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  );
}
```

On the Contentful side, webhooks can be filtered by environment and content type. This lets you keep late Discourse edits flowing while you rebuild caches and downstream consumers off Contentful.

> [!CAUTION]
> Zero downtime is realistic only if readers can switch to the new source instantly and late edits are replayed. Otherwise, take a short freeze window and make it explicit.

## Complexity Scaling Matrix

Migration effort scales non-linearly across three dimensions: content volume, media density, and locale count. Use this matrix to scope the pipeline architecture before committing to an approach:

| Forum size | Media volume | Locale count | Recommended approach |
|---|---|---|---|
| < 1,000 topics | Low (< 500 images) | 1 | Single-threaded script, sequential asset upload, ~4–8 hours total |
| 1,000–10,000 topics | Medium (500–5,000 images) | 1–2 | Concurrent workers (`pLimit(3)`), separate asset pipeline, ~1–3 days |
| 10,000–50,000 topics | High (5,000+ images) | 1–3 | Parallel workers with Redis job queue, S3 staging for assets, coordinated backoff, ~1–2 weeks |
| 50,000+ topics | High | 3+ | Distributed pipeline (e.g., worker queues + dead-letter handling), per-locale transformation passes, staged environment promotion, phased cutover with delta sync |

Additional complexity multipliers: custom Discourse plugins producing non-standard content require custom parsers; deeply nested category hierarchies require recursive reference handling; thread preservation with reply chains requires a recursive content model with GraphQL complexity budgeting at retrieval time.

For any migration above the first tier, instrument your pipeline with per-record success/failure logging from the start. Attempting to reconstruct which records failed after a partial run without logs adds days to recovery.

> Need help migrating Discourse content to Contentful? Our team builds custom migration pipelines that handle the content conversion, asset management, and rate limit orchestration — so you don't have to.
>
> [Talk to us](https://cal.com/clonepartner/meet?duration=30)

## Frequently asked questions

### Can you migrate Discourse posts to Contentful automatically?

No. There is no off-the-shelf connector. You need a custom pipeline that extracts Discourse data via the REST API or PostgreSQL, transforms Markdown (or cooked HTML) into Contentful Rich Text, and loads entries through the Content Management API.

### Should I use Discourse raw Markdown or cooked HTML for the Contentful migration?

Both work. Use raw Markdown with @contentful/rich-text-from-markdown if you're comfortable preprocessing Discourse-specific syntax ([quote], @mentions, oneboxes). Use cooked HTML with an HTML-to-Rich-Text parser if you'd rather work from standard DOM elements. Either path requires custom handling.

### What are the Contentful CMA rate limits for bulk imports?

The Content Management API enforces a default limit of 7 requests per second (some plans allow 10). When exceeded, you receive a 429 response with an X-Contentful-RateLimit-Reset header. Use a concurrency limiter like p-limit rather than relying on SDK retries for bulk migrations.

### Does Contentful Rich Text support Markdown tables and code blocks?

No. Rich Text has no native support for tables or fenced code blocks. Handle these via the callback in @contentful/rich-text-from-markdown — converting tables to lists or embedded entries, and code blocks to embedded entry types or separate fields.

### How do I extract all posts from a Discourse topic via the API?

The /t/{id}.json endpoint returns only 20 posts by default. Use the post_stream.stream array from the response to get all post IDs, then fetch remaining posts in batches of 20 via /t/{id}/posts.json?post_ids[]=...