ReadMe to Glean Migration: A Technical Guide

Migrating from ReadMe to Glean is not a content migration in the traditional sense — it is a content-to-search-index translation. ReadMe is a developer documentation platform that stores structured guides, API references, changelogs, and custom pages. Glean is an enterprise AI search and knowledge platform that indexes content from across your tool stack and makes it searchable via a knowledge graph. There is no native Glean connector for ReadMe, so you need to extract content, transform it into Glean's document schema, and push it through Glean's Indexing API as a custom datasource.

Teams typically make this move when they want developer documentation discoverable inside Glean alongside Confluence, Slack, Google Drive, and other internal knowledge sources. The objective is internal consolidation: taking siloed developer docs and integrating them into the company's central search index so engineers, support teams, and product managers can query them instantly. If you are looking to replace ReadMe entirely with a different docs publishing tool, this guide is not for that. Glean is the search and AI layer; ReadMe is the publishing system. Design the migration around that fact and the implementation gets simpler.

For background on knowledge base migration patterns, see our knowledge base migration checklist.

Why ReadMe Content Does Not Plug Into Glean

The core issue is architectural. ReadMe organizes content around a project-centric documentation hub: projects contain branches (versions), branches contain categories, and categories contain pages — guides, API reference pages, changelogs, custom pages, recipes. Each page type has its own schema. Content is stored as MDX — ReadMe's extended Markdown format with custom components like reusable content blocks, accordions, callouts, and embedded API explorers. API Reference pages do not have standard text bodies; they act as pointers to an OpenAPI (Swagger) file, and ReadMe's frontend dynamically renders the JSON/YAML into an interactive explorer.

Glean operates as an enterprise knowledge graph. It does not render documentation — it indexes it. Every piece of content becomes a document with a title, body, URL, permissions, and metadata. Glean's search engine builds relationships between content, people, and activity. It does not natively execute or render OpenAPI files.

This means you need to solve several translation problems:

• MDX → plain text or HTML. ReadMe's custom components (callouts, tabs, reusable content blocks) have no equivalent in Glean's index. They must be stripped or converted.
• Hierarchical categories → flat document list. ReadMe's nested category/page/subpage structure must be flattened into individual Glean documents with metadata that preserves the hierarchy.
• Interactive API reference → static text. ReadMe auto-generates interactive API explorers from OpenAPI specs. Glean indexes text, not interactive widgets. For Glean's AI to accurately answer a prompt like "What are the required parameters for the POST /v1/users endpoint?", the OpenAPI schema must be flattened into structured text before indexing.
• ReadMe-hosted images → external references. ReadMe's ZIP and PDF exports do not include images, only paths to them on ReadMe's CDN (files.readme.io). If your ReadMe account is closed, those images break.
• Versioned content → version-aware documents. ReadMe supports multiple doc versions via branches. You need to decide which versions to index and how to label them.

Three Ingestion Paths

There are three ways to get ReadMe content into Glean, each with different trade-offs.

Option 1: Crawl the Published Site with Glean's Website Connector

The lowest-engineering path when your docs are public. Glean's Website connector can crawl seed URLs or a sitemap, supports basic or bearer auth, custom headers, cookies, and respects robots.txt. (docs.glean.com)

Use this when:

• Your ReadMe docs are public
• You are fine with Glean indexing rendered HTML
• You do not need per-page ACL inheritance
• A slower refresh cycle is acceptable

The hard limits matter. The Website connector has no incremental crawl mode, and its default update rate is 28 days (configurable via Glean support). It does not propagate fine-grained source permissions. If pages require client-side rendering after login, dynamic indexing is not supported for password-protected or VPN-only pages. For private ReadMe deployments, a crawler usually stops being the right tool.

Option 2: Custom Datasource via ReadMe API + Glean Indexing API

The best path when you need predictable freshness, cleaner text, or permission control. You extract from ReadMe's API, transform the content, and push it through Glean's Indexing API. This is the focus of the rest of this guide.

This route is usually better for private ReadMe projects. ReadMe access is project-wide — if an end user can view Guides, they can also view API Reference. ReadMe cannot limit end-user access to specific sections within a project. (docs.readme.com) A custom Glean datasource lets you apply more granular document ACLs when your business rules need them.

Option 3: GitHub Route

If your ReadMe project uses bi-directional GitHub sync, each ReadMe version maps 1:1 to a Git branch. You can extract from GitHub instead of ReadMe directly. Glean's GitHub connector supports repository contents, issues, and metadata.

Use this when your docs team already reviews changes in pull requests and GitHub permissions should remain the access boundary. Do not use it if you need search results to point at polished, published ReadMe pages rather than repo files, or if your reference content depends on how ReadMe renders OpenAPI into reader-facing pages.

Ingestion Path Comparison

Step 0: Authenticate with Glean's Indexing API

Before writing any extraction code, obtain a Glean Indexing API token. This token is distinct from the Glean search token — the two use separate credential systems and cannot be substituted for one another.

To obtain an Indexing API token:

1. In Glean's admin console, navigate to Workspace Settings → Custom Apps.
2. Create a new custom app with the datasource:write scope.
3. Copy the bearer token — it is shown only once.

The Indexing API base URL follows the pattern https://<customer-subdomain>-be.glean.com/api/index/v1/. Your customer subdomain is the same prefix used for your Glean search instance.

Indexing API rate limits: Glean does not publish a hard requests-per-minute ceiling in its public documentation, but the Indexing API enforces per-datasource concurrency limits: only one active bulk upload is permitted at a time (concurrent /bulkindexdocuments calls for the same datasource are rejected). For /indexdocument (incremental), practical throughput in production deployments is typically 5–10 requests/second before receiving 429 responses. Implement exponential backoff starting at 1 second, doubling up to a 32-second cap.

Post-indexing processing lag: After a successful push to /bulkindexdocuments or /indexdocument, documents typically appear in Glean search within 15–60 minutes for text content. Permission propagation can take an additional 30–60 minutes. Plan your cutover window accordingly — do not schedule the final ReadMe shutdown within 2 hours of completing the last bulk push.

Step 1: Inventory Your ReadMe Project

Before writing extraction code, audit what you are migrating:

• How many versions/branches? Decide which to index. Most teams only need stable (the current published version). Indexing deprecated versions often degrades search accuracy by feeding Glean outdated context.
• Content types: Guides, API Reference, Changelog, Custom Pages, Recipes. Each has a different schema.
• Reusable content blocks: Shared snippets used across pages. When you export a page, the reusable content is typically included inline, but verify this.
• Total page count: This affects whether you use /indexdocument (incremental) or /bulkindexdocuments (full replacement).
• Custom pages and changelog: These use their own API families. ReadMe documents changelog as versionless, so teams that only pull guides often miss launch notes or standalone pages. (docs.readme.com)

Step 2: Extract Content via the ReadMe API

ReadMe's API requires a sequential extraction approach. You traverse the hierarchy from versions down to individual documents.

Fetching Versions, Categories, and Documents (v1 API)

import requests
import time
 
README_API_KEY = "your-api-key"
BASE_URL = "https://dash.readme.com/api/v1"
HEADERS = {"Authorization": f"Basic {README_API_KEY}"}
 
def get_all_pages(version: str) -> list:
    pages = []
    HEADERS["x-readme-version"] = version
    categories = requests.get(
        f"{BASE_URL}/categories?perPage=100", headers=HEADERS
    ).json()
 
    for cat in categories:
        cat_docs = requests.get(
            f"{BASE_URL}/categories/{cat['slug']}/docs", headers=HEADERS
        ).json()
        for doc in cat_docs:
            page = requests.get(
                f"{BASE_URL}/docs/{doc['slug']}", headers=HEADERS
            ).json()
            page["_category"] = cat["title"]
            pages.append(page)
            # Handle child pages
            for child in doc.get("children", []):
                child_page = requests.get(
                    f"{BASE_URL}/docs/{child['slug']}", headers=HEADERS
                ).json()
                child_page["_category"] = cat["title"]
                child_page["_parent"] = doc["title"]
                pages.append(child_page)
            time.sleep(0.25)  # Rate-limit: 200-300ms between calls
    return pages

ReadMe documents have a type field. If type == 'basic', the body field contains Markdown. If type == 'endpoint', the document is tied to an OpenAPI specification and the body will likely be empty or contain only a brief description.

You must also download your raw OpenAPI specification files via GET /api/v1/api-specification to process endpoint documentation separately.

Fetching Content via the v2 API (ReadMe Refactored Projects)

For v2 API projects, the extraction sequence uses branches instead of version headers:

import requests
import time
 
README_API_KEY = "your-api-key"
BASE_URL = "https://dash.readme.com/api/v2"
HEADERS = {
    "Authorization": f"Basic {README_API_KEY}",
    "Accept": "application/json"
}
 
def get_all_pages_v2(branch: str = "stable") -> list:
    pages = []
 
    # Fetch guide categories
    cats_resp = requests.get(
        f"{BASE_URL}/branches/{branch}/categories/guides",
        headers=HEADERS
    )
    cats_resp.raise_for_status()
    categories = cats_resp.json().get("data", [])
 
    for cat in categories:
        cat_title = cat.get("title", "Unknown")
        # Fetch pages within each category
        pages_resp = requests.get(
            f"{BASE_URL}/branches/{branch}/categories/guides/{cat['title']}/pages",
            headers=HEADERS
        )
        pages_resp.raise_for_status()
        cat_pages = pages_resp.json().get("data", [])
 
        for p in cat_pages:
            slug = p.get("slug", "")
            page_resp = requests.get(
                f"{BASE_URL}/branches/{branch}/guides/{slug}",
                headers=HEADERS
            )
            if page_resp.status_code == 200:
                page_data = page_resp.json()
                page_data["_category"] = cat_title
                pages.append(page_data)
            time.sleep(0.25)
 
    # Fetch custom pages
    custom_resp = requests.get(
        f"{BASE_URL}/branches/{branch}/custom_pages",
        headers=HEADERS
    )
    if custom_resp.status_code == 200:
        for cp in custom_resp.json().get("data", []):
            cp_detail = requests.get(
                f"{BASE_URL}/branches/{branch}/custom_pages/{cp['slug']}",
                headers=HEADERS
            )
            if cp_detail.status_code == 200:
                detail_data = cp_detail.json()
                detail_data["_category"] = "Custom Pages"
                pages.append(detail_data)
            time.sleep(0.25)
 
    return pages

ReadMe paginates collection endpoints to 10 items by default and allows per_page up to 100. Individual page endpoints support If-None-Match with ETags and can return 304 Not Modified, which is useful for incremental sync jobs. (docs.readme.com)

Step 3: Transform Content for Glean

This is where most migrations break. You have two distinct content types to handle: standard pages (MDX) and API reference content (OpenAPI).

Transforming MDX to Plain Text

ReadMe stores content as MDX, which includes standard Markdown, custom components (<Callout>, <Tabs>, <Tab>, reusable content embeds), inline HTML, and image references to ReadMe's CDN.

For Glean indexing, you need the text content:

1. Strip MDX components — Remove or unwrap custom component tags, keeping their text children.
2. Convert ReadMe block syntax — ReadMe uses proprietary [block:callout] and [block:code] JSON blocks in older content. Parse the JSON, extract the text, and convert to standard formatting.
3. Preserve code blocks — Glean can index code, so keep fenced code blocks as plain text.
4. Handle images — Replace image tags with their alt text. Download images from files.readme.io and re-host them if you need the images available at Glean's viewURL targets.

import re
import json
 
def transform_rdmd(content: str) -> str:
    # Convert ReadMe callout blocks to blockquotes
    def replace_callout(match):
        try:
            data = json.loads(match.group(1))
            title = data.get("title", "")
            body = data.get("body", "")
            ctype = data.get("type", "info").capitalize()
            return f"> **{title} ({ctype})**\n> {body}"
        except json.JSONDecodeError:
            return ""
 
    content = re.sub(
        r'\[block:callout\]\s*(\{.*?\})\s*\[/block\]',
        replace_callout, content, flags=re.DOTALL
    )
 
    # Convert ReadMe code blocks to fenced code
    def replace_code(match):
        try:
            data = json.loads(match.group(1))
            blocks = []
            for code in data.get("codes", []):
                lang = code.get("language", "")
                snippet = code.get("code", "")
                blocks.append(f"```{lang}\n{snippet}\n```")
            return "\n\n".join(blocks)
        except json.JSONDecodeError:
            return ""
 
    content = re.sub(
        r'\[block:code\]\s*(\{.*?\})\s*\[/block\]',
        replace_code, content, flags=re.DOTALL
    )
 
    # Handle newer :::callout syntax
    content = re.sub(
        r':::callout\{[^}]*\}\s*(.*?):::', 
        r'\1', content, flags=re.DOTALL
    )
 
    # Strip remaining MDX/HTML component tags, keep content
    content = re.sub(r'<[^>]+>', '', content)
    # Remove image markdown, keep alt text
    content = re.sub(r'!\[([^\]]*)\]\([^)]*\)', r'\1', content)
    # Collapse excessive whitespace
    content = re.sub(r'\n{3,}', '\n\n', content)
    return content.strip()

This is a simplified version. Production transforms need to handle ReadMe's specific MDX flavor — particularly the newer :::callout syntax, <HTMLBlock> components, embedded OpenAPI Try It widgets, and multi-language code tabs. Test your transform against your most code-heavy pages before running the full pipeline — pages with multi-language code tabs and embedded JSON are the most likely to produce encoding artifacts.

Flattening OpenAPI Specifications

This is the most critical transformation. Glean's search engine cannot effectively parse a 10,000-line OpenAPI JSON file to answer natural language queries. You have two options:

Option A: Index the entire spec as a single document. Preserves endpoint descriptions and schema definitions but gives poor search granularity — a user searching for "create user endpoint" has to wade through the entire spec.

Option B: Index each endpoint as a separate document. Almost always better for search quality. Parse the OpenAPI spec and create one Glean document per endpoint with structured text.

For Option B, generate a standalone text document for each endpoint:

import json
 
def openapi_to_documents(spec_path: str) -> list:
    with open(spec_path) as f:
        spec = json.load(f)
 
    docs = []
    for path, methods in spec.get("paths", {}).items():
        for method, details in methods.items():
            if method not in ("get", "post", "put", "patch", "delete"):
                continue
 
            summary = details.get("summary", f"{method.upper()} {path}")
            description = details.get("description", "")
 
            body_parts = [
                f"{method.upper()} {path}",
                f"Summary: {summary}",
                description,
            ]
 
            # Add parameter descriptions
            for param in details.get("parameters", []):
                required = "Required" if param.get("required") else "Optional"
                body_parts.append(
                    f"Parameter: {param['name']} "
                    f"({param.get('in', ''), param.get('schema', {}).get('type', param.get('type', 'string')), required}) "
                    f"- {param.get('description', '')}"
                )
 
            # Add request body schema if present (OpenAPI 3.x)
            request_body = details.get("requestBody", {})
            for content_type, media in request_body.get("content", {}).items():
                schema = media.get("schema", {})
                for prop_name, prop_details in schema.get("properties", {}).items():
                    required_props = schema.get("required", [])
                    req_flag = "Required" if prop_name in required_props else "Optional"
                    body_parts.append(
                        f"Body field: {prop_name} "
                        f"({prop_details.get('type', 'object'), req_flag}) "
                        f"- {prop_details.get('description', '')}"
                    )
 
            docs.append({
                "endpoint": f"{method.upper()} {path}",
                "title": summary,
                "body": "\n".join(body_parts),
                "operation_id": details.get("operationId", ""),
            })
    return docs

By flattening the API reference into explicit parameter lists and descriptions, Glean's indexer can tokenize the content properly, and the AI can surface accurate results for specific endpoint queries.

Step 4: Push to Glean via the Indexing API

Since there is no native connector, you use Glean's Indexing API (Push API) to create a custom datasource and push documents.

Create a Custom Datasource

Register a new datasource using the /adddatasource endpoint:

curl -X POST https://customer-be.glean.com/api/index/v1/adddatasource \
  -H 'Authorization: Bearer <your_indexing_token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "readme",
    "displayName": "ReadMe Docs",
    "urlRegex": "https://docs.yourcompany.com/.*",
    "datasourceCategory": "PUBLISHED_CONTENT"
  }'

datasourceCategory values and their implications:

For ReadMe documentation, PUBLISHED_CONTENT is the correct choice. Using TICKETS or COMMUNICATION will suppress these documents in results where reference documentation is expected.

After creating the datasource, configure its Results Display in Glean's admin console under Search → Custom Apps → [your datasource] → Object Definitions. If you skip this step, search results for your datasource show blank cards — the documents are indexed but display no title, snippet, or metadata. This is one of the most common post-migration confusion points.

Build the Document Payload

Map each transformed page to Glean's document schema:

import re
import requests
import time
 
def push_page_to_glean(
    page: dict, 
    base_url: str, 
    glean_token: str, 
    glean_url: str,
    max_retries: int = 4
):
    doc_id = re.sub(r'[^a-zA-Z0-9]', '', 
                    f"{page.get('_category', 'uncategorized')}{page['slug']}")
 
    payload = {
        "datasource": "readme",
        "document": {
            "id": doc_id,
            "objectType": "Guide",
            "title": page["title"],
            "body": {
                "mimeType": "text/plain",
                "textContent": transform_rdmd(page.get("body", ""))
            },
            "viewURL": f"{base_url}/docs/{page['slug']}",
            "permissions": {
                "allowAnonymousAccess": True
            },
            "customProperties": [
                {"name": "Category", "value": page.get("_category", "Uncategorized")},
                {"name": "ContentType", "value": page.get("type", "basic")}
            ]
        }
    }
 
    backoff = 1
    for attempt in range(max_retries):
        resp = requests.post(
            f"{glean_url}/api/index/v1/indexdocument",
            headers={"Authorization": f"Bearer {glean_token}",
                     "Content-Type": "application/json"},
            json=payload
        )
        if resp.status_code == 429 or resp.status_code >= 500:
            if attempt < max_retries - 1:
                time.sleep(backoff)
                backoff = min(backoff * 2, 32)
                continue
        resp.raise_for_status()
        return

Bulk Upload vs. Incremental

For initial loads, use /bulkindexdocuments. For ongoing sync, use /indexdocument.

def bulk_push_to_glean(documents: list, glean_token: str, glean_url: str):
    batch_size = 100
    upload_id = "readme-migration-001"
 
    for i in range(0, len(documents), batch_size):
        batch = documents[i:i + batch_size]
        is_first = (i == 0)
        is_last = (i + batch_size >= len(documents))
 
        payload = {
            "uploadId": upload_id,
            "isFirstPage": is_first,
            "isLastPage": is_last,
            "forceRestartUpload": is_first,
            "datasource": "readme",
            "documents": batch
        }
 
        resp = requests.post(
            f"{glean_url}/api/index/v1/bulkindexdocuments",
            headers={"Authorization": f"Bearer {glean_token}"},
            json=payload
        )
        resp.raise_for_status()

Concurrent uploads are not allowed — you cannot start a new upload before the previous one finishes. A new /bulkindexdocuments call with forceRestartUpload: true on an in-progress upload will abort the previous run and start fresh.

Handling Document Deletion in Incremental Sync

When a page is deleted from ReadMe, /indexdocument will not automatically remove it from Glean. You must explicitly tombstone the document using the /deletedocument endpoint:

def delete_glean_document(doc_id: str, glean_token: str, glean_url: str):
    resp = requests.post(
        f"{glean_url}/api/index/v1/deletedocument",
        headers={"Authorization": f"Bearer {glean_token}",
                 "Content-Type": "application/json"},
        json={"datasource": "readme", "id": doc_id}
    )
    resp.raise_for_status()

For incremental sync, maintain a local checkpoint store (a simple database table or JSON file) mapping ReadMe slugs to Glean document IDs. On each sync run: compare the current ReadMe page list against your checkpoint, call /deletedocument for any IDs present in the checkpoint but absent from ReadMe, and update the checkpoint.

Permissions Mapping

ReadMe's permission model is project-level: docs are either public or gated behind ReadMe's login. There is no per-page permission granularity in standard ReadMe projects.

Glean's permission model is document-level:

If you are migrating restricted docs (e.g., admin API references), map ReadMe user groups to your identity provider (Okta, Google Workspace) and pass those user or group emails in the Glean permissions object. Permissions are processed asynchronously — expect a lag of 30–60 minutes before documents are visible to the correct users after indexing.

Handling Versioned Documentation

ReadMe supports multiple documentation versions via branches. If you have versions v1, v2, and v3:

1. Index only the current version. Simplest approach. Only sync the stable branch. This is where most teams start.
2. Index all versions with metadata. Add a Version custom property to each document so Glean can disambiguate. Useful if users frequently reference older API versions.
3. Index all versions with prefixed IDs. Use IDs like v2gettingstarted to avoid collisions across versions.

Indexing deprecated versions can degrade search quality by feeding Glean outdated context. Be deliberate about which versions go in.

Asset Re-hosting and Link Resolution

Asset Migration Workflow

1. Scan all extracted Markdown bodies for https://files.readme.io/... URLs.
2. Download each asset programmatically.
3. Upload to an internal storage bucket (S3, GCS) that is accessible from wherever users will view the content.
4. Rewrite image tags in the Markdown to point to the new URLs.

Internal Link Resolution

ReadMe documents frequently link to one another using relative paths ([Authentication](/docs/authentication)). Because Glean assigns its own URLs, you need to either maintain a mapping of ReadMe Slug → Glean Document ID and rewrite links during the final transformation pass, or ensure document titles match exactly so Glean's entity resolution can connect them.

Setting Up Ongoing Sync

A one-time migration gets your content into Glean, but ReadMe docs change. You need ongoing sync.

Two approaches:

1. Scheduled full sync (daily cron). Re-run the full extraction pipeline and use /bulkindexdocuments to replace the entire corpus. Simple but destructive — every run replaces everything, and a bad inventory deletes good documents.
2. Incremental sync on change. Detect which pages changed and use /indexdocument for updates, /deletedocument for removals. ReadMe's page endpoints support If-None-Match with ETags and return 304 Not Modified, which makes them a good fit for incremental jobs. (docs.readme.com) Persist a checkpoint timestamp between runs. Pair with explicit deletion handling (see above) to avoid stale documents accumulating.

For teams using ReadMe's GitHub bi-directional sync, a GitHub Actions workflow triggered on push to the docs branch is the most reliable approach. It minimizes lag between doc changes and Glean index updates.

Validation and Cutover

Treat validation as a search-quality exercise, not just a row-count exercise. Row counts tell you completeness. They do not tell you whether Glean can retrieve the right page when an engineer searches for a phrase buried in a code example.

Validation Checklist

• Page count matches. Compare Glean's datasource document count against your ReadMe page count per branch, section, and category.
• Text content is intact. Spot-check 20–50 pages, especially pages with tables, code blocks, callouts, and long API reference content.
• URLs resolve. Every viewURL in Glean should open the corresponding ReadMe page.
• Permissions work. Log in as users with different access levels and verify they see the expected documents.
• API reference endpoints are granular. If you indexed per-endpoint, verify that searching for a specific endpoint name returns the right document.
• RAG accuracy. Ask Glean questions like "What are the required parameters for the POST /v1/users endpoint?" and verify the AI cites the correct endpoint document. If the AI hallucinates parameters, your OpenAPI flattening needs adjustment.
• Deletions are clean. After a second sync run, verify that documents no longer in ReadMe were tombstoned and removed from search results.
• Results display is configured. Search for a known page title and confirm the result card shows title, snippet, and metadata — not a blank card. If blank, the object definition in Glean's admin console needs configuration.
• No orphaned IDs. After a second sync run, verify that documents no longer in ReadMe were properly cleaned up.

Use Glean's search syntax for validation: exact phrase matching with quotes, app:readme filters, and updated:past_week to confirm freshness. But use source-side counts for completeness verification — Glean returns a bounded ranked result set, not exhaustive enumeration. (docs.glean.com)

Keep ReadMe live until search quality and permissions are both signed off. Post-indexing processing lag is 15–60 minutes for text content and up to 60 minutes for permissions. Crawler-based Glean ingestion can take days on larger deployments. Plan overlap.

For a broader cutover runbook, see our data migration playbook.

Common Failure Modes

MDX components rendered as garbage text. If you do not strip ReadMe's custom MDX components, Glean indexes raw component tags. Users searching for normal terms get results polluted with <HTMLBlock> and <Callout type="info"> markup.

Slug-to-ID collisions. ReadMe allows slugs like getting-started in different categories. After stripping non-alphanumeric characters, both become gettingstarted. Prefix the category slug to avoid collisions.

Bulk upload deletes pages. If your extraction script errors halfway and you push a partial batch as the final page, Glean deletes every document not in that batch. Validate extracted page counts before triggering bulk uploads.

Stale documents from deleted pages. In incremental sync mode, pages deleted from ReadMe persist in Glean indefinitely unless you call /deletedocument explicitly. Maintain a checkpoint store and tombstone removed slugs on each sync run.

Stale OpenAPI specs. If you export the OAS file once and index it, it becomes stale as your API evolves. Automate OAS extraction from ReadMe or pull it from your CI/CD pipeline where the spec is versioned.

Encoding issues with code blocks. ReadMe pages with complex code examples (multi-language tabs, embedded JSON) can produce malformed text after MDX stripping. Test your transform against your most code-heavy pages.

Missing display configuration. The Results Display area for a custom datasource stays blank until you create object definitions in Glean's admin console under Search → Custom Apps → [datasource] → Object Definitions. Teams sometimes think search is broken when the real issue is that the datasource was indexed without the display model being configured.

Permissions lag causing false negatives. Users may report not seeing documents in the first 30–60 minutes after a bulk push. This is expected behavior — Glean processes permissions asynchronously. Do not diagnose as a pipeline failure until the lag window has passed.

When This Migration Does Not Make Sense

Not every team needs ReadMe content in Glean:

• If your docs are public and well-indexed by Google, Glean's web connector may pick them up automatically. Check if a custom datasource adds value over Glean's built-in web indexing.
• If you have fewer than 50 pages, the engineering effort of building a custom connector may not justify the benefit. Consider Glean's browser extension approach, which makes page titles searchable from users' browsing history.
• If your ReadMe docs are external-facing only and your Glean instance is internal-only, there may be no audience overlap.

Effort Estimates

This migration is technically a custom connector build. The table below reflects typical engineering time for teams building this pipeline for the first time.

Smaller projects (under 100 pages, single version, no OpenAPI flattening) typically complete the extraction, transform, and initial push in 8–16 hours. Ongoing maintenance after initial build is primarily the incremental sync job and keeping the OpenAPI extraction current with API changes.

Do not assume Glean replaces ReadMe feature-for-feature. It does not. Glean is the search and AI layer. If you still need a public developer portal, keep or replace the docs publishing platform separately.