At the Workshop on Memory for LLM-Based Agentic Systems during ICLR 2025 in Singapore, Apple researchers presented a paper titled LaCy. The study focuses on Small Language Models (SLMs), which struggle with world knowledge compression due to limited parameter sizes. Unlike larger models, SLMs often produce factually incorrect outputs because their pretrained knowledge is constrained by architecture limits.
The paper highlights a key challenge: pretraining can only embed so much information into a model's parameters. For SLMs, this upper bound is particularly restrictive. The research team tested whether giving SLMs access to external sources could solve this problem. Their approach allows models to query a larger knowledge base via a provided URL, effectively offloading factual recall to an external system.
During experiments, the team found that SLMs with external query access reduced errors by 34% compared to baseline models. This method suggests a practical workaround for the inherent limitations of smaller architectures. The paper does not propose replacing pretraining but rather augmenting it with targeted retrieval.
The findings were presented on May 7, 2025, at the ICLR workshop in Singapore. Apple's research team included lead authors from the company's AI research division. The paper is available on the Apple Machine Learning Research website under the title LaCy: What Small Language Models Can and Should Learn.
Source: machinelearning.apple.com