Axolotls, soupermodels and the fast and the inference

Lunar Engineering Technical Blog – by Aron Molnar

Lunar engineering, noun: The practice of teaching a moon to read contracts, improving it with soup, and persuading it to hurry up. An axolotl is involved for entirely professional reasons.

About six months ago, we released Luna Crescent, our specialized language model for contract analysis. Luna Crescent classifies clauses, identifies relevant passages, and extracts structured information from contracts. Since then, it has processed more than half a trillion tokens and become the most-used model within Luminance, which still feels slightly ridiculous to write.

Luna originally stood for “Luminance UNified Assistant” (I helped myself to two letters from “Unified” and considered the matter solved.) Earlier versions had internal names like Eloise, Clara and Critter, so I maintain that Luna was an improvement. Extra vindication came when OpenAI decided to name one class of models Luna around a week after our release. We decided not to make a fuss...

The next iteration has now finished training. We’re calling it Luna Experimental internally, which is more of a status report than a name. We’ll work with our creative colleagues in Marketing on the actual name.

We have increased Luna’s size, expanded and improved the training data, and added tasks including Deep Insights, which Crescent does not currently support. There’s still plenty of testing to do, but I’m excited about this one.

Improving Luna meant more than training a larger model. We needed to shorten the wait between experiments, choose candidates that improved the capabilities we cared about, and make sure performance gains held up on the hardware we intended to use. Across several iterations, that work led to faster training steps, checkpoints that were more useful together than apart, and throughput gains we rejected because of their latency cost.


We Axolotl of Axolotl: Choosing the right framework

Back in the age of Claras and Critters, we experimented with several training frameworks before settling on Axolotl, the open-source framework for training language models. It offered a good set of trade-offs for our project. The fact that Axolotl is absurdly cute did not hurt.

As Luna’s requirements evolved, we began maintaining our own fork of the framework, with changes tailored to our models, datasets, and hardware. Some improve compatibility and correctness in distributed training; others help us understand what happens during a run or speed up training. This fork is part of our training infrastructure; Axolotl itself is not part of the Luminance product codebase.

We improved attention performance and resolved compatibility issues in our distributed training stack.

A separate optimization reduced training-step time from 486 seconds to 269 seconds in one tested configuration, about 45% less time per step. I’ve spent enough time staring at training progress bars to have strong feelings about those 217 seconds.

Every framework change, especially in the kernels, still must survive numerical checks, comparisons between runs, and downstream model evaluations so we can trust the model coming out the other end.

Time saved in training gives us more room to experiment, evaluate, and improve the model, which sounds straightforward until you have a whole collection of candidates and need to choose between them.


The soupermodel

A training run leaves a trail of Lunas behind it, and they don’t form a neat queue from worst to best: a checkpoint can improve one task while losing ground on another. Those differences give us a reason to experiment with model soups, combining the weights of compatible checkpoints into a single candidate.

In an earlier iteration, we chose one of these mixtures even though individual checkpoints still beat it on particular tasks, accepting those trade-offs for the balance of capabilities it offered.

This approach, explored in the model soups paper, can produce a model that outperforms its ingredients while retaining the serving cost of one model. Model merging also appears in SmolLM3’s training pipeline.

A useful way to visualize souping is to imagine two compatible checkpoints sitting in different places within the same broad valley of a loss of landscape. Averaging their weights can place the combined model at a lower point between them.

Note: several billion dimensions have been omitted for readability.


Of course, averaging two models doesn’t automatically produce a better one. Souping is a mixed pot bag, and every proposed soup goes through the same testing pipeline as an individual checkpoint.

Choosing ingredients is a slightly different job from choosing a model to deploy. A checkpoint might be an awkward choice on its own, yet particularly good at a task where another candidate struggles. That gives us a reason to try the combination. Looking at the individual tasks keeps those possibilities visible, while an overall ranking would make it much easier to discard a potentially useful ingredient.

We proposed several soups using those task-level results. Without further training, our chosen mixture beat the main run’s final checkpoint by 0.4 percentage points on Deep Insights and 1.8 on question-based classification.

After more extensive automated testing, promising checkpoints and soups have to answer people who read contracts for a living. Legal professionals compare outputs in human A/B tests, helping us assess whether the benchmark improvements hold up in legal work. Our Legal Analysts will explore this process in more detail in a future post.

So yes, sometimes the strongest candidate is the soup. Machine learning remains a very normal field.


The fast and the inference

I’d been thinking about AI systems helping improve other AI systems, and inference optimization seemed like a good place to try it.

We already had a working production configuration and performance we could measure, so I set up research agents to work exploring alternatives. The results needed some care: a throughput improvement could come with slower requests, and a configuration that looked promising on one machine could disappoint the next.

At its simplest, the auto-research goal was: model goes faster, good. Model goes slower, bad.

There’s quite a bit more to it (obviously), but having a clear feedback signal makes this a good problem for an auto-research loop. The agents propose changes, run benchmarks, and use the results to guide the next experiment.

One interesting complication in auto-research loops is a familiar one: overfitting. Repeatedly optimizing against the same benchmark can make an agent’s solution increasingly specific to it, with gains that fail to carry over elsewhere. This remains an active research problem, with recent work from Google Cloud AI Research and academic collaborators exploring constraints on how agents propose and select changes to their own software. I particularly like that these constraints borrow from regularization in traditional machine learning, adapting a familiar idea to a new way of producing software.

For Luna, we actively want a configuration specialized for our model, hardware, and workloads. The important part is making sure the benchmark represents those workloads well enough for a faster benchmark result to mean a faster service.

I built the benchmark using data from our internal test harness to represent production workloads, without using customer prompts or responses. This gave the researcher a realistic target, with our existing configuration as a starting point. A candidate solution, graded by this benchmark, only qualified if throughput improved, and latency did not degrade.

Because the benchmark measures serving performance rather than answer quality, I excluded changes such as quantization that could affect the model’s answers without that effect appearing in the score. I wasn’t looking for a more efficient way to be wrong.

Then there was a hardware problem. A configuration that performs beautifully on one GPU setup can behave very differently on another. GPU type and count matter, but so do the connections between them, software versions and the rest of the machine. “It was faster on the other machine” was not going to be much of a conclusion.

So, I ran the research in three waves. I started with available internal GPUs to explore promising ideas, moved to hardware closer to our production setup, and finished by benchmarking and optimizing a production-like system.

Each wave began with the candidates and lessons from the previous one, letting us carry useful knowledge forward while checking which improvements survived the move.


The chart shows results from an intermediate hardware wave. The highlighted run recorded 55.6% higher token throughput and 30.6% lower request latency than the baseline. The middle wave shown above captures more of the exploration and dead ends. The final wave was mostly smaller refinements, which I appreciated very much but which makes a less entertaining graph. The configuration from the highlighted run held up on production-like hardware in that final wave, where we squeezed a little more performance out of it.

The greyed-out points are experiments that didn’t make the cut. As an example, one candidate ran more copies of the model in parallel, using fewer GPUs per copy. It looked promising in the first hardware wave. On the next setup, it achieved about 39% higher estimated throughput than the baseline, but latency was about 52% higher. That run was disqualified because higher throughput only counted if the latency checks also passed. No soup for them.

There is something slightly unbelievable about how this work gets done now. In 2024, working through this many ideas could have kept me busy for weeks. Now I can set swarms of agents to work proposing changes, designing and running experiments while I sleep. I still have to give them a useful, well-defined problem, build a benchmark I trust and make sense of the results, but now those results can be waiting for me in the morning. Apparently, “sleeping on it” is now a perfectly respectable engineering strategy.


The next phase of Luna

One thing I’ve appreciated throughout this work is being able to try ideas that might otherwise have stayed on the list for weeks. Faster training and automated inference experiments give us more chances to investigate a promising direction or find out early that it is going nowhere. There is still plenty of careful evaluation involved, but we can spend less of that time waiting for the next result.

Luna’s next version is now working its way through deeper automated evaluation and testing with legal professionals. Those results will guide which checkpoints and soups move forward, while we prepare the model for the hardware and workloads it will encounter in production. There is plenty left to do, and I am trying to remain a sensible person about early benchmark results, with mixed success.

I’m excited to find out how much further we can take Luna. I’ll also keep trying to explain it without making it sound like an amphibian cooking program.

GET STARTED

Get more from your contracts

See how Luminance can help your teams negotiate smarter, surface what matters, and keep contracts moving across the enterprise. 

GET STARTED

Get more from your contracts

See how Luminance can help your teams negotiate smarter, surface what matters, and keep contracts moving across the enterprise. 

GET STARTED

Get more from your contracts

See how Luminance can help your teams negotiate smarter, surface what matters, and keep contracts moving across the enterprise.