Connector-driven RAG: keeping your knowledge base fresh

A RAG system is only as good as its most recent sync. An AI assistant backed by a knowledge base that’s 48 hours stale will confidently serve outdated answers — and users will blame the AI, not the sync schedule. Freshness is a retrieval quality problem as much as it is an operational one, and it requires deliberate design.

The freshness problem in practice

Consider a common support bot use case: the bot answers employee questions by retrieving from a Confluence knowledge base. The knowledge base contains HR policies, IT procedures, and product documentation. Some of that content changes daily (IT incident updates), some changes quarterly (HR policy revisions), and some changes rarely (foundational architecture documents).

A flat 24-hour sync schedule treats all content the same. The IT incident runbooks that were updated at 9am are stale by 9am the next day. The architecture document that hasn’t changed in six months is re-indexed every 24 hours at unnecessary embedding cost.

The solution is a differentiated sync strategy: different content types get different sync frequencies, and high-churn sources use event-driven triggers instead of polling.

Sync strategies in ManyLayers Workspace

ManyLayers Workspace supports three sync trigger modes per connector:

Scheduled polling — the connector fetches all content on a cron interval. Simple to configure, appropriate for sources without webhook support. Use a short interval (15–30 minutes) for high-churn sources and a long interval (4–24 hours) for stable content.

Webhook-triggered sync — the source system sends a push notification to ManyLayers Workspace when content changes, and the connector incrementally fetches and re-indexes only the changed documents. This is the freshest option and the most efficient — only changed documents consume embedding quota. Supported by GitHub, Jira, Confluence (with an app plugin), and Zendesk.

Manual trigger — a REST API call to the Workspace management API triggers an on-demand sync for a specific connector. Useful for content published through internal workflows where you control the publication step.

Configure the trigger mode per connector in Workspace → Connectors → [Connector] → Sync Settings.

Designing a differentiated sync schedule

Map your knowledge sources to tiers based on content volatility:

Tier 1 — Real-time (webhook-triggered): incident runbooks, status pages, active issue queues. These sources change in response to live events and must reflect current state immediately. Configure webhook triggers and test the end-to-end latency from content change to updated retrieval (typically under 2 minutes in a well-configured setup).

Tier 2 — Near-real-time (15–30 minute polls): internal wikis for operational teams, support ticket databases, product documentation with frequent updates. Daily accuracy is not sufficient; hourly is acceptable; 15-minute is the practical minimum for scheduled polling before you start generating unnecessary load.

Tier 3 — Periodic (4–24 hour polls): policy documents, architecture documentation, onboarding materials, and historical content. These change rarely and tolerate a longer sync window.

Tier 4 — On-demand: regulatory filings, versioned specifications, and content that changes on a known release cycle. Trigger these manually as part of the release workflow rather than polling.

Incremental indexing vs full re-index

For large sources (tens of thousands of documents), a full re-index on every sync is expensive and slow. ManyLayers Workspace supports incremental indexing: the connector computes a content fingerprint for each document and only re-embeds documents whose fingerprint has changed since the last sync.

Enable incremental indexing in Workspace → Connectors → [Connector] → Advanced → Indexing Mode. Incremental indexing reduces embedding cost by 70–95% for stable sources and proportionally for sources with partial daily changes.

The tradeoff: incremental indexing uses the source system’s modification timestamp or content hash to detect changes. Sources that don’t provide reliable modification metadata (some REST APIs, some legacy document systems) require full re-indexing to guarantee freshness.

Handling deletions

Incremental indexing naturally handles updates but requires explicit handling of deletions. If a document is deleted from your Confluence space, its chunks remain in the knowledge base index until the next reconciliation pass.

ManyLayers Workspace runs a daily reconciliation that compares the current index contents against a manifest from the source connector and removes chunks belonging to documents that no longer exist. For most use cases, this daily reconciliation is sufficient — stale deleted content is rarely the cause of retrieval failures.

For sources where deletions are compliance-sensitive (e.g., a knowledge base that contains content subject to GDPR right-to-erasure requests), enable the accelerated reconciliation option, which runs reconciliation after every sync rather than daily.

Monitoring sync health and retrieval freshness

Sync health is straightforward: monitor the sync success rate and the last-successful-sync timestamp per connector in Workspace → Connectors → Health. Configure an alert for connectors that haven’t synced successfully in more than 2x their configured sync interval.

Retrieval freshness is harder to monitor directly, but you can approximate it. For sources where you know when content was last updated (e.g., a Confluence page with a visible last-modified date), sample recent retrievals and compare the document modification date in the returned metadata against the expected current state. A divergence of more than 2x the sync interval is a signal that the sync is lagging or missing updates.

Build this check into your eval pipeline: include a set of “current state” questions (questions whose answers change over time) in your retrieval eval dataset, run it weekly, and track whether the retrieved content matches the current ground truth. This closes the loop between sync configuration and retrieval quality — which is ultimately the only metric that matters for your users.