Model Evaluation & Tuning: Custom Embeddings vs Finetuning Foundation Models

Deciding the Core Architecture: Context Enrichment vs Private Models

In the architectural design of enterprise generative AI systems, tech leaders face a critical decision-making threshold: should they enrich the context of a general-purpose foundation model at runtime, or should they invest in fine-tuning a private foundation model? This dilemma, often simplified as Retrieval-Augmented Generation (RAG) versus Fine-Tuning, has profound implications for an organisation's capital expenditure, operational latency, security posture, and system accuracy. Making the wrong architectural bet can result in millions of pounds of wasted compute spend or, conversely, highly rigid systems that fail to adapt to live business dynamics. To build a robust AI-ready enterprise, architects must understand the precise boundary between these two paradigms.

Context Enrichment, or RAG, operates on the principle of dynamic knowledge retrieval. Instead of expecting the large language model to retain all corporate knowledge within its neural weights, relevant facts, customer histories, and product catalogues are stored in external databases (such as a vector database or semantic index). When a user query is received, the system queries the vector database, extracts the most semantically relevant text chunks, and injects them directly into the LLM's prompt window. This approach is highly efficient for volatile, frequently updated datasets because updating system knowledge is as simple as updating database records. Furthermore, context enrichment maintains clear lineage and auditability: the model can cite the exact source document used to generate its response, drastically reducing hallucination rates and simplifying compliance audits.

Fine-tuning, on the other hand, involves updating the neural network weights of an existing foundation model using domain-specific training datasets. This method does not introduce real-time information; instead, it standardises the model's behaviour, terminology, and stylistic output. Fine-tuning is indispensable when the application requires strict adherence to specialised industry formats (such as generating medical coding records, producing complex legal documents, or conforming to custom JSON schemas). It is also highly effective for training smaller, highly efficient models (e.g. 8-billion parameter models) to perform specialised tasks with the same accuracy as expensive 70-billion parameter models, thereby optimising long-term operational costs. However, fine-tuning represents static knowledge: the moment the training run finishes, the model’s knowledge is frozen, and any subsequent business changes require a new training pipeline.

Constructing Custom Semantic Indices in Salesforce Data Cloud

To implement an effective context enrichment architecture, organisations must construct highly reliable semantic indices. Within the Salesforce ecosystem, Salesforce Data Cloud serves as the foundational engine for this capability, enabling real-time data ingestion, transformation, and vector search. Data Cloud allows architects to ingest unstructured data sources—such as customer support transcripts, knowledge articles, product manuals, and internal PDF documents—and transform them into queryable vector embeddings. These embeddings represent the semantic meaning of the text, enabling search queries to find conceptually similar matches even when exact keywords do not align.

The technical implementation pipeline in Data Cloud involves three sequential phases: ingestion, chunking, and embedding generation. During ingestion, raw text data is mapped to Data Model Objects (DMOs). The text fields are then processed by a chunking engine. Because LLMs have finite context windows and embedding models have input limitations, large documents must be broken down into smaller, digestible chunks. Architects must carefully design the chunking strategy: fixed-size chunking (e.g. 512 tokens) is simple but can split critical sentences, whereas recursive character chunking respects paragraph boundaries. Introducing a sliding window overlap (e.g. 10% to 20% overlap between adjacent chunks) ensures that contextual continuity is preserved across chunk boundaries.

Once chunked, the text segments are sent to an embedding model (such as OpenAI's text-embedding-3-small or AWS Bedrock's Cohere Embed v3) via direct integration. The model maps each chunk into a high-dimensional vector space (e.g. 1536 dimensions) and stores the resulting coordinate coordinates back in the Data Cloud vector database. When a user queries Einstein, the search phrase is dynamically converted into a vector embedding using the same model, and a semantic search is executed. Below is a SQL representation of a semantic similarity vector search query executed within Salesforce Data Cloud, demonstrating how chunks are retrieved based on vector distance:

SELECT Chunk_Text__c, Score__c 
FROM VECTOR_SEARCH(
    TABLE(Case_Summary_Embedding__dlm), 
    'How do I reset my security token?', 
    'openai-text-embedding-3', 
    5
);

The Engineering and Commercial Math of Fine-Tuning Off-Platform Foundation Models

While context enrichment is highly effective for fact-based retrieval, organisations often require private, fine-tuned models to execute specialised cognitive tasks. However, entering the domain of model fine-tuning requires rigorous engineering and commercial evaluation. Fine-tuning foundation models (such as Llama 3 or Mistral) requires massive computational resources, specialised engineering talent, and a deep understanding of training hardware costs. Architects must master the commercial math of model training to justify these investments to the executive suite and prevent substantial budget overruns.

To estimate compute requirements, we must analyse the parameters being trained and the size of the dataset. Full parameter fine-tuning modifies all weights in the network, requiring massive GPU clusters and long training runs. To optimise compute efficiency, standard enterprise engineering leverages Parameter-Efficient Fine-Tuning (PEFT) methods, with Low-Rank Adaptation (LoRA) being the industry standard. LoRA freezes the original model weights and injects small trainable rank decomposition matrices into each transformer block. This reduces the number of active training parameters by over 99%, lowering VRAM consumption and making it possible to train large models on single or small clusters of GPUs. To estimate the compute cost of a training run, architects use the following commercial formula:

Assume we have a training dataset consisting of 10,000 highly curated customer service transcripts. Each transcript averages 2,000 tokens, resulting in a total dataset size of 20,000,000 tokens. We plan to train the model for 3 epochs to prevent overfitting. We lease an on-demand cluster of 8x NVIDIA H100 GPUs (80GB VRAM each), where each H100 GPU costs approximately £3.50 per hour. Across our 8-GPU cluster, our distributed training framework achieves a throughput of 20,000 tokens per second. Substituting these values into our equation:

Total Tokens Processed = 20,000,000 tokens * 3 epochs = 60,000,000 tokens
Training Time (Seconds) = 60,000,000 tokens / 20,000 tokens/sec = 3,000 seconds
Training Time (Hours) = 3,000 seconds / 3,600 seconds/hour = 0.83 hours
Total Training Cost = 8 GPUs * £3.50/hour * 0.83 hours = £23.24

Exposing Private Models Securely via BYO-LLM and Multi-Tenant Endpoints

Once a private foundation model has been successfully fine-tuned on off-platform infrastructure (such as Amazon Bedrock, Google Cloud Vertex AI, or Microsoft Azure ML), it must be exposed to Salesforce applications. Rather than attempting to host these computationally heavy models directly within the core CRM environment, Salesforce provides a standard architectural pattern known as Bring Your Own LLM (BYO-LLM). Through the Salesforce Model Builder interface, architects can securely register external LLM endpoints and expose them as native generation services within standard Prompt Builder flows.

The secure connection between Salesforce and the external model enclave is governed by the Einstein Trust Boundary. This framework ensures that no customer data is compromised during model execution. When a prompt template triggers an external LLM callout, the Salesforce orchestrator routes the request through the Trust Boundary gateway. Here, real-time data masking engines scan the prompt payload to identify and redact sensitive information (such as credit card numbers, national insurance numbers, and names) before the packet leaves the Salesforce network. Additionally, the communication channel is secured using robust enterprise authentication methods, typically utilising OAuth 2.0 Client Credentials or mutual TLS (mTLS) to verify tenant identity. Below is a structured JSON representation of an Einstein Model Builder configuration payload, showing how a secure external BYO-LLM endpoint is registered:

{
  "modelName": "custom-finetuned-llama3-70b",
  "endpointUrl": "https://api.secure-enclave.organisation.com/v1/chat/completions",
  "authType": "OAuth2ClientCredentials",
  "clientId": "sf_einstein_gateway_prod_client",
  "clientSecret": "SECURE_SECRET_REDACTED",
  "parameters": {
    "temperature": 0.1,
    "max_tokens": 1024,
    "top_p": 0.9,
    "stop_sequences": ["\nUser:", "\nAssistant:"]
  }
}

Once the external model processes the prompt, it returns the generated text response to the Einstein Gateway. Before the response is injected back into the CRM workflow, the Trust Boundary performs a second security pass: it executes toxicity filtering to block harmful content and uses its data-mapping registry to automatically de-mask the sensitive fields, seamlessly returning the personalised summary to the business user. This asynchronous, double-masked pipeline allows organisations to leverage the power of highly specialised, off-platform models while maintaining complete compliance with data protection laws and corporate governance standards.

Comparative Evaluation Matrix for Cost, Latency, and Hallucination Control

Choosing the optimal model adaptation strategy requires a balanced trade-off across multiple, competing operational parameters. No single approach is superior in every context. For instance, while RAG is highly cost-effective and adapts to real-time changes instantly, it can suffer from higher retrieval latencies. Conversely, a fine-tuned model delivers lightning-fast responses with highly customised formatting but requires substantial upfront capital to train and lacks real-time knowledge. Architects must evaluate these factors side-by-side to match the tuning strategy with their application’s specific SLA and performance requirements.

To assist in this decision-making process, the CoE must standardise a multidimensional evaluation framework. This framework scores each adaptation pattern against core metrics: Initial CapEx (training costs), Ongoing OpEx (inference token costs), Latency (time-to-first-token), Hallucination Control, and Language/Tone Customisation. By grading each approach, organisations can establish standard deployment templates (for example, utilizing RAG for standard customer queries and hybrid models for complex automated contract writing).

Below is a comparative matrix designed to guide enterprise architects in selecting the correct tuning strategy based on their specific operational bounds: