Connecting your knowledge sources

RAG-powered AI is only as useful as the knowledge it can retrieve. A well-tuned model backed by stale or incomplete knowledge returns confident but wrong answers. Getting your knowledge sources into ManyLayers Workspace correctly — and keeping them current — is the foundation everything else builds on.

How connectors work in ManyLayers Workspace

ManyLayers Workspace’s connector framework handles the extraction, chunking, embedding, and indexing of documents from external sources. Once a connector is configured, it syncs on a schedule (or via webhook) and maintains a live index that your AI workflows query at retrieval time.

The connector framework abstracts the plumbing: you don’t write ETL pipelines or manage vector databases directly. You configure a connector, map it to a knowledge base, and the Workspace handles the rest. The retrieval layer applies to all Workspace features: team chat, AI agents, structured workflows, and the API-accessible retrieval endpoint that your custom applications can query.

The connector catalog

ManyLayers Workspace ships with 20+ pre-built connectors. The most commonly configured fall into four categories.

Document storage

Confluence is the most common first connector for engineering and product teams. Configure it with an Atlassian API token and a space key list. The connector indexes page content, inline comments, and child page hierarchies. Exclude spaces by prefix pattern to avoid indexing legacy content that would dilute retrieval quality.

Google Drive and SharePoint/OneDrive cover organizations on Google Workspace or Microsoft 365. Both support folder-level scoping — index only the AI Platform shared drive, not every personal Drive in your organization.

Notion is common for product and operations teams. The connector indexes public and shared databases, pages, and linked databases. Note that Notion’s API doesn’t expose real-time webhooks; configure a 1-hour sync interval for most use cases.

Code and documentation

GitHub and GitLab connectors index repository contents, README files, and — critically — code comments and docstrings. For engineering support bots, indexing your internal libraries and their documentation dramatically improves answer quality for internal tooling questions.

The code connector applies language-aware chunking: a Python function is chunked as a unit rather than split mid-function at a token boundary. This preserves semantic coherence for code retrieval.

Ticketing and support

Jira and Linear connectors index issue descriptions, comments, and resolution notes. These are high-value sources for support bots and incident response assistants. Configure a label filter to index only closed issues (which have resolution context) rather than all open work.

Zendesk and Intercom index resolved support tickets. The connector can apply a recency filter — index only tickets closed in the last 24 months — to avoid surfacing outdated resolutions for deprecated products.

Databases and APIs

The SQL connector (Postgres, MySQL, BigQuery, Snowflake) indexes structured data as formatted text chunks. This works well for product catalogs, pricing tables, and configuration reference data where the answer is a specific row value. Configure which tables and columns to expose — don’t index tables that contain PII you haven’t cleared to flow into the knowledge base.

The REST API connector periodically polls a configured endpoint and indexes the JSON response. Use this for internal data APIs, status pages, or any service that surfaces structured data over HTTP.

Configuring chunking strategy

The default chunking strategy (512-token chunks, 64-token overlap) works well for prose documents. Adjust it based on your content type:

• Dense technical documentation: reduce to 256-token chunks. Smaller chunks improve precision for narrow factual queries.
• Long-form narratives (policy documents, legal agreements): increase to 1,024 tokens. Larger chunks preserve the context needed to answer questions about conditional clauses.
• Structured data (FAQs, tables): use the structured chunking mode, which treats each row or Q&A pair as an atomic chunk.

Set chunking strategy per knowledge base in Workspace → Knowledge Bases → [KB Name] → Settings → Chunking.

Managing sync schedules and freshness

Every connector has a sync schedule. Common patterns:

• Confluence/Notion: 1-hour sync. Content changes are usually not time-critical.
• Jira/Linear: 30-minute sync. Issue status changes matter for support assistants.
• GitHub: webhook-triggered sync. Code changes should reflect immediately in the index.
• Databases: 15-minute sync or on-demand via the refresh API. Pricing and configuration data may change frequently.

Monitor sync health in Workspace → Connectors → [Connector] → Sync History. A failed sync doesn’t immediately degrade retrieval quality, but a 24-hour sync gap on a high-churn source will produce stale results. Configure a sync failure alert to catch this proactively.

Access control alignment

Knowledge base access in Workspace respects team membership. Configure which teams can query each knowledge base in Workspace → Knowledge Bases → [KB Name] → Access. Teams only see results from knowledge bases they have access to.

For sources with internal access controls (Confluence spaces, Drive folders with restricted sharing), the connector indexes based on the service account’s access level. If the service account can read a document, the connector will index it. Audit the service account’s permissions before connecting — over-indexing confidential content is a harder problem to fix after the fact than setting conservative permissions up front.

Testing retrieval quality

After indexing, run a manual retrieval test before rolling out to users. In Workspace → Knowledge Bases → [KB Name] → Test Retrieval, enter 10–15 representative queries and review the top-3 returned chunks for each. Good retrieval returns specific, directly relevant chunks. Poor retrieval returns tangentially related content or overly generic sections.

If retrieval quality is poor, the most common causes are: chunks that are too large (diluting specificity), duplicate content (competing chunks with nearly identical text), or missing content (a source that wasn’t connected or was excluded by a filter). Address these before deploying to users — first impressions of an AI assistant are hard to reverse.