Karakoram Technologies

1. System Architecture

Enterprise RAG demands sub-second latency from query to token generation. To achieve this, we completely bypass cold-start penalties by executing the entire retrieval and LLM orchestration layer on the Vercel Edge Network.

graph TD
    User([User Request]) --> Edge[Next.js Edge Route Handler]
    
    subgraph Vercel Edge Network
        Edge --> Embed[Generate Embedding]
        Embed --> VercelAI[Vercel AI SDK Stream]
    end
    
    subgraph Infrastructure
        Embed -.-> OpenAI[OpenAI API / text-embedding-3-small]
        Edge -.-> Pinecone[(Pinecone Vector DB)]
        VercelAI -.-> GPT4[OpenAI GPT-4o]
    end
    
    Pinecone -.->|Top K Vectors| Edge
    VercelAI -->|Streaming Response| User

2. Sub-500ms Latency Optimizations

Traditional RAG pipelines chain synchronous HTTP calls (Embed -> DB -> LLM), resulting in TTFB (Time to First Byte) latency exceeding 2 seconds. By utilizing the Vercel AI SDK and Edge runtime, we stream the response directly to the client the millisecond the LLM begins generation.

app/api/chat/route.ts

import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { Pinecone } from '@pinecone-database/pinecone';

export const runtime = 'edge'; // Crucial for low TTFB

export async function POST(req: Request) {
  const { messages } = await req.json();
  const latestMessage = messages[messages.length - 1].content;

  // 1. Generate Embedding
  const { embedding } = await embed({
    model: openai.embedding('text-embedding-3-small'),
    value: latestMessage,
  });

  // 2. Query Vector DB (Pinecone Serverless)
  const pc = new Pinecone();
  const index = pc.index('enterprise-rag');
  const results = await index.query({
    topK: 5,
    vector: embedding,
    includeMetadata: true,
  });

  // 3. Stream Response
  const result = await streamText({
    model: openai('gpt-4o'),
    messages: [
      { role: 'system', content: buildContext(results) },
      ...messages
    ]
  });

  return result.toDataStreamResponse();
}

3. Hallucination Safeguards (Semantic Thresholding)

A common failure mode in RAG is injecting irrelevant context when a user asks an out-of-domain question, forcing the LLM to hallucinate connections. We implement Semantic Thresholding: if the Pinecone similarity score drops below 0.75, we intercept the query and fallback to a deterministic refusal.

utils/rag-safeguards.ts

const SEMANTIC_THRESHOLD = 0.75;

// Filter out irrelevant context before it hits the LLM
const validMatches = queryResult.matches.filter(
  match => match.score > SEMANTIC_THRESHOLD
);

if (validMatches.length === 0) {
  // Prevent LLM Hallucination by short-circuiting
  return new Response(
    "I do not have enough specific context in my knowledge base to answer this question accurately.", 
    { status: 200 }
  );
}

Enterprise RAG Stack

1. System Architecture

2. Sub-500ms Latency Optimizations

3. Hallucination Safeguards (Semantic Thresholding)

Karakoram AI