The knowledge worth putting in a RAG system almost never lives in a tidy directory of .txt files. It lives in Notion pages, Confluence spaces, and Google Drive documents — rich, interlinked, permission-controlled, and constantly changing. Ingesting these sources is where a lot of knowledge-base projects quietly get hard, because a workspace connector has to solve four problems at once: authentication, structure, permissions, and freshness. Here are the patterns that hold up in production.
Problem 1: authentication (OAuth, done right)
Workspace tools gate access behind OAuth, so a connector starts with an OAuth flow: the user authorizes your app, you receive tokens, and you use them to read their content. The patterns that matter:
- Store tokens encrypted, refresh them proactively. Access tokens expire; refresh tokens let you renew without re-prompting the user. Encrypt them at rest (a leaked workspace token is a serious breach) and handle refresh transparently. Kognita, for instance, keeps integration tokens under a dedicated encryption key precisely because these credentials are so sensitive.
- Handle revocation gracefully. Users disconnect apps. When a token stops working, fail cleanly and surface a re-auth prompt rather than silently breaking sync.
- Respect scopes. Request the minimum access you need. Over-broad scopes are a trust and security liability.
Problem 2: rich structure
Workspace content is not plain text. Notion has nested blocks, databases, and relations. Confluence has macros, panels, and hierarchies. Drive has Docs, Sheets, Slides, and PDFs — each a different format. Ingesting well means preserving the structure that carries meaning:
- Normalize to a common representation. Convert each source's native format into a consistent structured form (often Markdown-like) so downstream chunking and embedding don't need per-source special-casing. The connector absorbs the messiness; the pipeline stays uniform.
- Preserve hierarchy as metadata. A Notion page's location in its workspace, a Confluence page's space and parent — this context improves retrieval and lets you cite where an answer came from.
- Handle the structured types deliberately. A Notion database or a Google Sheet is tabular; treat it with the same care as any table (structure-preserving extraction), not by flattening it to prose.
- Follow the format-specific chunking rules. Once normalized, chunk along the document's real boundaries — headings, sections — rather than blind token counts.
Problem 3: permissions
This is the one teams underestimate, and it's the one that becomes an incident. Workspace content has per-item permissions — not everyone can see every page. If your knowledge base ingests everything and then answers everyone's queries from all of it, you can leak content across permission boundaries: someone asks a question and gets an answer sourced from a page they were never allowed to see.
- Decide your permission model up front. Either ingest only content appropriate for the audience, or carry permission metadata through and filter retrieval by what the querying user is allowed to see. The second is more powerful and much harder.
- Treat permission leakage as a security bug, not a nice-to-have. An answer that quotes a restricted document is a confidentiality breach even if the retrieval was technically correct.
- Re-check permissions on change. Access changes over time; a page shared today may be restricted tomorrow. Permission state is part of what sync has to keep current.
Problem 4: freshness via change events
Workspace content changes constantly, and a snapshot goes stale immediately. The durable pattern is event-driven sync:
- Use webhooks where the source offers them. Notion, for example, can notify you when content changes. An edit fires an event, you re-ingest just that page, and the knowledge base reflects it within seconds — far better than periodic full re-crawls.
- Fall back to incremental polling where webhooks aren't available, using timestamps or change feeds to fetch only what changed since last sync.
- Reconcile periodically to catch missed events and — critically — handle deletes, so a page removed at the source doesn't linger in your index forever.
- Make re-ingestion idempotent. The same page will sync many times; deterministic IDs and upserts mean re-processing overwrites in place instead of creating duplicates.
Putting it together
A robust workspace connector is a small system: an OAuth layer (encrypted tokens, refresh, revocation), a normalization layer (native format → common structured representation, structure preserved as metadata), a permission layer (model decided, metadata carried, retrieval filtered), and an event-driven sync layer (webhooks + incremental polling + reconciliation, idempotent writes). Skip any one and you get a predictable failure: broken auth, mangled structure, leaked confidential content, or stale answers.
The reason this is worth the effort is that these tools are where your organization's actual knowledge already lives. A knowledge base that can only ingest clean uploaded files is asking humans to duplicate their content — which they won't, and which would go stale if they did. A knowledge base that connects directly to Notion, Confluence, and Drive, and stays in sync with them, turns the tools people already use into a live, queryable knowledge layer without anyone changing how they work. That's the whole payoff — and it's why the connector, unglamorous as it is, is often the most important part of the system to get right.