Skip to content
Menu
Programme

Where knowledge lives

Much real work needs consistent judgement inside one domain, under that domain's own rules. This programme asks where specialised knowledge should live, in weights, in retrieval or in validators, and what a small specialist must show before it is worth building.

Where should what a model knows live, in its weights, in a store it consults or in checks on its output, and when does a small specialist judge better than a general model given the same knowledge?

Generated cyanotype print of a row of small woodworking planes and carving tools, each shaped for one task.Generated photograph of five different small white plaster models in a row (a bridge, a pipe junction, a seed pod, a bearing, a sluice gate), each with one vermilion pin.Generated render of a narrow dredged channel through a shoal, mapped in fine detail while its surroundings fade to black.
Plate ITools made for one job each.Five problems, five models, each pinned where it would fail.One channel, surveyed exactly. A model for a specific problem knows its route in full.Generated image in a cyanotype treatment. Far Akin art direction.Generated image. Far Akin art direction, Datum look.Generated image. Far Akin art direction, Swath look.

Why it matters

Much of the work models are asked to do is narrow and repeated: deciding which rule or procedure applies to a case, or checking an output against a domain's standard. Agentic systems in particular call models for a small number of specialised tasks with little variation, and one position paper argues that small models are capable enough, more suitable and cheaper for many such calls[1]. A model built for one problem could be smaller, easier to inspect and simpler to keep current than a general one. Whether it is also better at the problem is the open question.

The evidence points both ways. A second phase of pretraining on in-domain text improved task performance across four domains[2]. Instruction prompt tuning aligned a large general model to clinical questions, though human evaluation still found gaps[3]. Yet a general model steered by careful prompting, with no domain-specific training, outperformed specialist medical models on the benchmarks studied[4].

What is hard

The main difficulty is the comparison. A specialist has to beat a general model that has the same knowledge available at inference time. Retrieval-augmented generation combines a model's parametric memory with an external index, a response to the difficulty of giving provenance for, and updating, knowledge held in weights[5]. When fine-tuning and retrieval were compared as ways to inject knowledge, retrieval did consistently better, and models struggled to learn new facts through unsupervised fine-tuning[6]. Guided generation, using an index over the model's vocabulary, can guarantee that an output follows a regular expression or grammar[7]. A claim for a specialised model has to survive these baselines at matched cost.

A second difficulty is structure. Many domains already have their knowledge organised as ontologies, taxonomies or typed rules, and several lines of work feed that structure to models. Joint pretraining on text and knowledge-graph subgraphs improved multi-step reasoning[8]. Self-training against a medical ontology aligned a model with the ontology's concepts[9]. A small model tuned on WordNet handled several lexical-semantic tasks[10]. Graph-structured retrieval and plain retrieval each show distinct strengths depending on the task[11]. When structure belongs in the weights, and when it is better kept outside in retrieval and validators, is not settled.

A third difficulty is measurement. A model can learn to reproduce the form of a schema without the judgement the schema encodes. Telling schema adherence from domain reasoning needs held-out cases that pattern-matching cannot solve, and data splits that keep paraphrases of a test case out of training.

Current directions

  • Weights against scaffolding. Comparing a small trained specialist with a general model given raw retrieval, structured retrieval, deterministic validators and repair loops.
  • Easy before hard. Starting with classification and routing inside a domain, where answers can be checked exactly, before attempting open judgement.
  • Specialists as services. Asking whether a small specialist improves a general model's work when called as a tool, not only whether it beats the general model alone.

How the lab approaches it

In the weights or beside them starts from a sceptical default: a small model trained on ontology-structured domain material must show value beyond a general model equipped with structured retrieval, validators and repair. The win condition is better domain judgement, not schema mimicry. The design would compare the specialist with each of those baselines, and with a general model that calls the specialist, on one task family whose test cases are to be published. The held-out set is to be frozen before any training data is generated, and each test case is to have paraphrased variants that expose surface matching.

The design is written; none of its arms has run, and the inquiry has no public results. A null result would still be useful. If retrieval and validators close the gap, the domain does not need a specialist, and the design would record where that line falls for the task family tested.

The longer aim is a working account of when a problem deserves its own model, and of how such a model should be built and evaluated.

Open problems in this programme

References

  1. Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S., et al. (2025). Small Language Models are the Future of Agentic AI. arXiv preprint 2506.02153.
  2. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., et al. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), 8342–8360.
  3. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., et al. (2023). Large language models encode clinical knowledge. Nature 620(7972), 172–180.
  4. Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., et al. (2023). Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine. arXiv preprint 2311.16452.
  5. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS 2020).
  6. Ovadia, O., Brief, M., Mishaeli, M., Elisha, O. (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Proceedings of EMNLP 2024, 237–250.
  7. Willard, B. T., Louf, R. (2023). Efficient Guided Generation for Large Language Models. arXiv preprint 2307.09702.
  8. Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., et al. (2022). Deep Bidirectional Language-Knowledge Graph Pretraining. Advances in Neural Information Processing Systems (NeurIPS 2022); arXiv preprint 2210.09338.
  9. Liu, Z., Gan, C., Wang, J., Zhang, Y., Bo, Z., Sun, M., et al. (2025). OntoTune: Ontology-Driven Self-training for Aligning Large Language Models. Proceedings of the ACM Web Conference 2025, 119–133.
  10. Moskvoretskii, V., Neminova, E., Lobanova, A., Panchenko, A., Nikishina, I. (2024). TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2331–2350.
  11. Han, H., Ma, L., Wang, Y., Shomer, H., Lei, Y., Qi, Z., et al. (2025). RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv preprint 2502.11371.

Inquiries in this programme

  • Generated cyanotype print of a jeweller's loupe and a cut gemstone beside a large plain magnifying glass.Generated photograph of a detailed white plaster model of a river weir and sluice gate, with vermilion pins clustered along the one gate.Generated render of a single rock pinnacle mapped in fine detail and lit against an unmapped black sea floor.
    Designed

    In the weights or beside them

    On one family of software-architecture judgement tasks, does a small model trained on a domain's structure (its pattern catalogue, rules and validators) give better-judged answers than a frontier model given the same structure through retrieval, validators and a repair loop?

    Machine-learning foundations · Measurement and research methodology