Enterprise Data Integration

Enterprise RAG and agents are only as good as the data pipeline behind them: connectors that pull content from SharePoint, Drive, Confluence, ticketing systems and databases; synchronization that keeps the index fresh and propagates deletes; permissions that follow every chunk so users only retrieve what they may see; and governance for sensitive data, lineage and erasure. This note covers that pipeline - the part that most demos skip and most production incidents come from.

The Ingestion Pipeline

flowchart LR
    subgraph SRC["🏢 Sources"]
        SP["SharePoint / OneDrive"]
        GD["Google Drive"]
        CF["Confluence / wikis"]
        TK["Tickets / CRM"]
        DB["Databases"]
    end
    SRC --> CON["🔌 Connectors<br/>full crawl + delta / webhooks / CDC"]
    CON --> NORM["🧹 Parse + normalize<br/>text, layout, metadata"]
    NORM --> SENS["🛡️ Sensitivity + PII<br/>labels, redaction, exclusions"]
    SENS --> CH["✂️ Chunk + embed"]
    CH --> IDX[("🗄️ Index<br/>chunks + ACL principals<br/>+ source, version, label")]
    IDP["🔐 Identity provider<br/>users, groups, nesting"] --> Q
    U["👤 User query"] --> Q["🔎 Retrieval<br/>filtered by the user's principals"]
    IDX --> Q

    style CON fill:#d8dfe8,stroke:#b0bac8
    style SENS fill:#e8e0d4,stroke:#c8b89a
    style IDX fill:#dde4dc,stroke:#b0c4b0
    style Q fill:#ddd8e4,stroke:#b8b0c8

Every chunk should carry, at minimum: a stable doc_id and chunk_id, the source system and URL (for citations), the source version or modified time, the ACL principals allowed to read it, its sensitivity label, and the ingest timestamp.

Connectors and Change Detection

Practical rules:

• Treat notifications as hints, not truth. Webhooks get dropped. Combine them with periodic delta syncs and a slower full reconciliation (compare source listings with the index) that catches anything missed.
• Respect source rate limits. Microsoft Graph and other SaaS APIs throttle with 429 and Retry-After; a full re-crawl of a large tenant can take days. Design connectors to resume from a checkpoint (Python for AI Engineering).
• Skip unchanged content with content hashes, so re-embedding happens only when text actually changes.
• Run connectors with their own scoped identity - read-only, limited to the sites or spaces in scope - never an administrator's account (Cloud Networking & IAM for AI).

Freshness and Deletes

An index that keeps serving deleted or outdated documents is a correctness and compliance problem, not just a quality one.

• Upserts keyed by doc_id: on change, delete the document's old chunks and insert the new ones in one idempotent operation, so retries never leave duplicates (RAG System Design).
• Propagate deletes as first-class events (tombstones). A document deleted or moved out of scope at the source must leave the index, and every cache derived from it, within a stated time.
• State a freshness SLO - for example, "99% of source changes searchable within 15 minutes; deletes and permission removals within 5 minutes" - and measure it by writing canary documents and timing how long they take to become searchable (or to disappear).
• Version the index. Large re-processing jobs (a new chunker or embedding model) build a new index version alongside the old one and switch over after evaluation (RAG in Production).

Permission-Aware Retrieval

The rule is simple: a user must never receive a chunk from a document they couldn't open in the source system. Getting it right takes three parts:

1. Mirror ACLs at ingest. Store the principals allowed to read each document - users, groups, "everyone in the organization", external guests - on every chunk.
2. Resolve the user at query time. Expand the signed-in user into all their principals, including nested group membership, from the identity provider (Entra ID, Okta, Google Workspace), and cache the result briefly.
3. Filter inside the search, not after it. Pass the principals as a filter to the vector and keyword queries (early binding); post-filtering top-k results returns too few results and risks leaks through scores, counts or caches.

Permissions change faster than content. When someone restricts a document, the index still holds the old ACL until the next sync - the ACL lag. For highly sensitive sources, add late binding: re-check candidate documents against the source (or a fresh permissions cache) before using them. A runnable sketch of both:

from dataclasses import dataclass

# Identity provider view: user -> direct groups, group -> parent groups (nested membership).
USER_GROUPS = {"maria": {"finance-emea"}, "tom": {"engineering"}}
GROUP_PARENTS = {"finance-emea": {"finance"}, "finance": {"all-staff"}, "engineering": {"all-staff"}}


def principals_for(user: str) -> frozenset[str]:
    """Everything a user can be matched on: the user id plus all groups, following nesting."""
    if user not in USER_GROUPS:
        raise PermissionError(f"cannot resolve identity for {user!r}")   # fail closed
    seen, stack = set(), list(USER_GROUPS[user])
    while stack:
        g = stack.pop()
        if g not in seen:
            seen.add(g)
            stack.extend(GROUP_PARENTS.get(g, ()))
    return frozenset({f"user:{user}"} | {f"group:{g}" for g in seen})


@dataclass(frozen=True)
class Chunk:
    doc_id: str
    text: str
    allowed: frozenset[str]          # ACL principals mirrored from the source at ingest
    acl_version: int


INDEX = [
    Chunk("budget-2027", "EMEA travel budget is 1.2M", frozenset({"group:finance"}), 7),
    Chunk("handbook", "Expense claims are due within 30 days", frozenset({"group:all-staff"}), 3),
    Chunk("salaries", "Salary bands for 2027", frozenset({"user:cfo"}), 2),
]


def source_still_allows(doc_id: str, principals: frozenset[str]) -> bool:
    """Late-binding check against the system of record (stubbed): the budget doc was just restricted."""
    if doc_id == "budget-2027":
        return "group:finance-leads" in principals
    return True


def retrieve(user: str, query: str, recheck: bool = True) -> list[str]:
    principals = principals_for(user)
    # Early binding: the filter goes INTO the vector/keyword query, so unauthorized chunks never come back.
    hits = [c for c in INDEX if c.allowed & principals]          # stands in for search(query, filter=...)
    # Optional late binding for sensitive sources: confirm with the source before use.
    if recheck:
        hits = [c for c in hits if source_still_allows(c.doc_id, principals)]
    return [c.doc_id for c in hits]


print("maria principals:", sorted(principals_for("maria")))
print("maria, index ACLs only:", retrieve("maria", "travel budget", recheck=False))
print("maria, with source re-check:", retrieve("maria", "travel budget"))
print("tom:", retrieve("tom", "travel budget"))
try:
    retrieve("unknown-user", "anything")
except PermissionError as e:
    print("unknown user:", e)

Output:

maria principals: ['group:all-staff', 'group:finance', 'group:finance-emea', 'user:maria']
maria, index ACLs only: ['budget-2027', 'handbook']
maria, with source re-check: ['handbook']
tom: ['handbook']
unknown user: cannot resolve identity for 'unknown-user'

Maria reaches the finance budget through nested groups (finance-emea → finance), but the document was restricted after the last ACL sync: the index alone would still return it, and only the late-binding check removes it. And when identity can't be resolved, the system returns nothing - it fails closed.

Sync permission changes on a faster path than content changes, and make ACL removal - like deletion - a high-priority event.

The Oversharing Problem

Permission-aware retrieval faithfully enforces source permissions - including the bad ones. Organizations are full of files shared with "everyone" by accident, and search-based assistants make them findable in seconds. Microsoft's documentation for Microsoft 365 Copilot puts it plainly: Copilot only surfaces data a user has at least view permission for - so overshared data is surfaced too, which is why Microsoft offers controls such as Restricted Content Discovery to keep specific SharePoint sites out of Copilot and organization-wide search while their permissions are reviewed.

Before connecting a source:

• Audit sharing on the highest-risk content (HR, finance, legal, M&A) and fix permissions at the source.
• Exclude sites, spaces or folders that shouldn't be in scope, even if permissions are technically correct.
• Honour sensitivity labels (for example Microsoft Purview labels) - exclude or restrict "Highly Confidential" content from the index.
• Pilot with a small group and red-team retrieval with accounts from different departments (Safety Evaluation & Red-Teaming).

Sensitive Data, Lineage and Erasure

• PII detection and redaction. Tools such as Microsoft Presidio detect names, emails, phone numbers and identifiers. Decide per source whether to redact at ingest (the index never holds it), mask at output, or exclude the source - and remember redaction is imperfect, so it complements access control rather than replacing it.
• Embeddings are data. Research has shown text can be substantially reconstructed from embeddings (vec2text), and OWASP's 2025 LLM Top 10 lists vector and embedding weaknesses - including cross-tenant leakage - as a risk category. Protect vector stores like the source documents.
• Lineage. Each chunk carries its source URI and version; answers cite them; ingestion runs record what was processed (OpenLineage-style metadata). When a source is found to be wrong or out of scope, lineage tells you exactly which chunks and cached answers to purge.
• Erasure. A right-to-erasure request (GDPR Article 17) or a legal deletion must reach every copy: the index, chunk stores, semantic and answer caches, evaluation sets built from real queries, and logs and traces with retrieved content. Design deletion by doc_id and by subject identifier from the start; retrofitting it is painful.
• Audit logging. Record who retrieved which documents for which question - essential for investigating a leak and for regulated industries.

Structured Data and Live Systems

Not everything should be copied into a vector index. Numbers that change by the minute, records with row-level security, and transactional systems are often better queried live:

Live queries carry the user's own permissions and are always fresh, at the cost of latency and dependence on the source system's availability.

Build or Buy

Managed knowledge services - Amazon Bedrock Knowledge Bases and Amazon Q Business connectors, Google's Gemini Enterprise and Vertex AI Search, Microsoft Foundry IQ and Microsoft 365 Copilot connectors, and dedicated enterprise-search vendors - ship dozens of connectors with ACL sync built in (Managed RAG on Cloud Platforms). Connectors are expensive to build and maintain - APIs change, permissions models are intricate, throttling is unforgiving - so buy them for mainstream SaaS sources unless you have a specific reason not to. Build when a source is in-house, the permissions model is unusual, or you need control over parsing and chunking that the service doesn't give. Either way, test permission enforcement yourself with accounts from different groups before launch.

Check Yourself

Exercises

Study Notes

Must-know:

• Chunks carry doc_id, source URL, version, ACL principals, sensitivity label, ingest time
• Change detection: Graph delta queries, Drive changes API, webhooks, CDC (Debezium); notifications are hints - add delta syncs and reconciliation
• Connectors: scoped read-only identity, throttling-aware, resumable, content hashing
• Freshness SLO measured with canaries; idempotent upserts by doc_id; deletes as first-class events; versioned index rebuilds
• Permission-aware retrieval: mirror ACLs, expand nested groups at query time, filter inside the search, fail closed; ACL lag → faster permission sync and late binding for sensitive sources
• Oversharing: fix source permissions, exclude sites, honour sensitivity labels, red-team before rollout
• PII redaction complements access control; embeddings are sensitive (vec2text, OWASP LLM08); lineage enables purges; erasure must reach index, caches, eval sets and logs
• Live data via text-to-SQL with database row-level security or tools/MCP with the user's identity; buy mainstream connectors, test permissions yourself

References

• Microsoft, Use delta query to track changes in Microsoft Graph data (2026) and Microsoft Graph throttling guidance (2026)
• Google, Retrieve changes - Google Drive API (2026)
• Debezium documentation (2026) - change data capture
• Microsoft, Data, Privacy, and Security for Microsoft 365 Copilot (2026) and Restricted Content Discovery (2026); Learn about sensitivity labels (2026)
• AWS, Amazon Q Business connector concepts (2026)
• OWASP, LLM08:2025 Vector and Embedding Weaknesses (2025); Morris et al., Text Embeddings Reveal (Almost) As Much As Text (2023)
• Microsoft Presidio (2026); OpenLineage (2026); GDPR Article 17 - Right to erasure (2016)

Last reviewed: 2026-10