Continued Pre-Training: Domain Adaptation for Large Language Models

Continued pre-training (CPT) extends a pretrained model by training it further on a large corpus of domain-specific text using the standard next-token prediction objective. Unlike SFT, which teaches the model to follow instructions using structured input-output pairs, CPT injects raw knowledge into the model by exposing it to large volumes of unstructured domain text.

Think of CPT as extending the model pretraining phase with your own data. The model learns domain vocabulary, relationships between domain concepts, and the style of domain-specific writing. After CPT, the model has a stronger foundation in your domain, which makes subsequent SFT and alignment stages more effective.

Use CPT when the domain knowledge your model needs was underrepresented in the original pretraining data. General-purpose LLMs are trained primarily on web text, which means they have shallow coverage of specialized domains like underwriting rules, patent law, quantitative finance, or proprietary internal documentation.

If the model already understands your domain but needs to follow specific instructions or produce particular output formats, SFT alone is sufficient. But if the model lacks fundamental domain knowledge (it confuses claims terminology, misapplies legal precedents, or produces factually incorrect domain content), CPT is the right starting point.

CPT shows up most often in four domains:

• Insurance: train on policy wordings, claims notes, and underwriting guidelines to build models that understand coverage language and claims reasoning patterns.
• Legal: train on case law, statutes, contracts, and regulatory filings to produce models that can analyze legal documents and reason about legal concepts.
• Financial: train on earnings reports, SEC filings, market analysis, and financial news to build models that understand financial instruments, valuation methods, and market dynamics.
• Code: train on specialized codebases, internal libraries, and documentation to build models that understand your tech stack and coding conventions.

CPT datasets are raw text files, with no instruction-response formatting needed. The text should be clean, well-structured, and representative of the domain knowledge you want the model to absorb. Common sources include: research papers, technical documentation, internal wikis, textbooks, standards documents, and curated web crawls filtered for domain relevance.

Quality matters more than quantity. A well-curated 1GB dataset of high-quality domain text often produces better results than 10GB of noisy web scrapes. Deduplicate your data, remove boilerplate (headers, footers, navigation text), and ensure the text is properly decoded (no encoding artifacts or broken characters).

Select Continued Pre-Training as the training method in the Run BiOS configurator. Upload your domain text as a plain text or JSONL file (one document per line). Key hyperparameters include learning rate (typically 1e-5 to 5e-5 for CPT, lower than SFT), context length, and number of training epochs.

The Run BiOS training wizard configures CPT-specific parameters:

CPT jobs are typically longer than SFT jobs because they process large text corpora. Monitor the perplexity on a held-out validation set in the Run BiOS dashboard. Decreasing perplexity indicates the model is learning the domain language patterns. Stop training when perplexity plateaus to avoid overfitting.

The recommended pipeline is: base model, then CPT on domain text, then SFT on instruction-response pairs, then optional alignment (DPO/SimPO/etc.) on preference data. Each stage builds on the previous one.

After CPT, the model understands your domain but does not yet follow instructions. The SFT stage teaches it to produce structured outputs in response to user queries. The alignment stage then refines output quality to match human preferences. On Run BiOS, you can chain these as separate training jobs, using each job checkpoint as the starting model for the next stage.

When the gap is behavior, not knowledge. CPT teaches the model your domain's text — vocabulary, facts, conventions — but it does not teach it to follow your instructions. If the failure is format, tone, or task discipline, that is SFT's job, and pre-training on more documents will not fix it.

When the corpus is small. Continued pre-training is the hungriest method in the catalog; a few megabytes of domain text is not a pre-training dataset, it is context. Small, high-value knowledge bases are usually better served by retrieval, which updates by editing a document instead of running a training job.

And when the domain knowledge changes weekly. Pre-training bakes knowledge into weights, and weights freeze at the training date. Fast-moving facts — prices, inventories, policies — belong in a retrieval layer, where an update is an edit, not a retrain.

CPT results depend heavily on how you prepare and run the job. First, curate your corpus carefully. CPT is sensitive to data quality because the model absorbs patterns from every document in the training set. Include only high-quality, well-written domain text. Remove boilerplate content like headers, footers, navigation menus, copyright notices, and duplicate passages. If your source material includes tables, ensure they are formatted as readable text rather than raw HTML or fragmented columns.

Second, use a lower learning rate than you would for SFT. CPT modifies the model at a deeper level than instruction tuning, and aggressive learning rates can overwrite useful pre-trained knowledge. Start with 1e-5 and decrease to 5e-6 if you observe quality degradation on general-purpose benchmarks after CPT. The goal is to add domain knowledge without losing the model general capabilities.

Third, validate your CPT model before moving to the SFT stage. Generate responses to domain-specific questions and compare them against the base model. The CPT model should show improved domain vocabulary, more accurate domain-specific reasoning, and better familiarity with domain concepts. If the improvement is not visible, your dataset may be too small, too noisy, or too similar to what the model already learned during pre-training.

Fourth, do not skip the SFT stage after CPT. The CPT model has enhanced domain knowledge but has not been trained to follow instructions. Applying SFT after CPT produces significantly better results than either stage alone. On Run BiOS, you chain the stages by using the CPT checkpoint as the starting model for your SFT training run.

Continued pre-training begins with the corpus. Collect high-quality text from your domain sources: research papers, technical documentation, internal wikis, textbooks, or curated web crawls. Clean the text by removing boilerplate, deduplicating, and ensuring proper encoding. Format the corpus as a plain text file or JSONL file with one document per line.

Upload the corpus through the Run BiOS Datasets page. For CPT, the platform accepts plain text files and validates the encoding and formatting. Create a new training run, select Continued Pre-Training as the method, and choose your base model. Set the learning rate to 1e-5, configure the context length to match your document lengths (2048 or 4096 tokens is typical), and set the number of epochs based on your corpus size: 1 to 2 epochs for large corpora, 3 to 5 for smaller ones.

Launch the job and monitor the perplexity metric in the Run BiOS dashboard. Perplexity should decrease steadily during training, indicating the model is learning the domain language patterns. When perplexity plateaus, the model has absorbed the available signal from your corpus and additional training is unlikely to help. Stop the run at this point to avoid overfitting and wasting compute.

After CPT completes, use the checkpoint as the starting model for an SFT training run. This two-stage pipeline, CPT then SFT, produces domain-specialized models that both understand your domain deeply and follow instructions effectively.