NLDL 2027 · Proceedings of Machine Learning Research 352

Meta-LSQ: A Closed-Form Sobolev Inner Solver for Few-Shot Meta-Learning

Antonin Sulc

Lawrence Berkeley National Laboratory

Paper Code BibTeX
Four panels: ten support points of one sinusoid task with values and slopes; the stacked linear system of value rows and slope rows; the fitted function and its derivative over the truth; test error against training time for MLP-MAML and Meta-LSQ.
One held-out sinusoid task. Ten values and ten slopes become the rows of one linear system, a single ridge solve gives the fit, and its derivative is available in closed form. Only σ, μ and w are meta-learned. On the right, Meta-LSQ with slopes after 0.5 s of training is below MLP-MAML after 50 s.

Gradient-based meta-learners such as MAML adapt to a new task with an inner loop of gradient descent that has to be differentiated through, and they meta-learn every network weight. Meta-LSQ fits each task with one regularised least-squares solve over a fixed random Fourier basis and meta-learns only three scalars: a bandwidth, a ridge and a derivative weight.

Since the derivative of a sinusoid is again a sinusoid, the same solve can also take derivative observations, and the fitted model has closed-form derivatives of any order. The method is meant for smooth, low-dimensional families, and the paper also reports where it loses.

3meta-learned scalars, against 1,761 to 8,513 weights for the MAML baselines
14 sof meta-training on 10-shot sinusoids, single CPU thread
5.5×lower test error than a FOMAML baseline that trains 36× longer
1.6 µmmedian error in simulated beam tuning at DESY's ARES, given model derivatives

How it works

  1. A frozen sine basis

    The model is a sum of N random sine waves. Their frequencies and phases are drawn once and never trained; only the amplitudes are fitted, which is a linear problem.

  2. One solve per task

    Every value and every derivative observed on a task is one linear equation in the amplitudes. Stacked, they are solved by a single ridge regression, with no inner loop.

  3. Three scalars across tasks

    The bandwidth σ, the ridge μ and the derivative weight w are learned with Adam on the query error, by differentiating through the solve.

β∗(σ, μ, w) = (A⊤A + μI)−1 A⊤t,   A = [ H ; √w Γ1 ; … ; √w Γd ]

H holds the features at the support points and Γk their derivatives along coordinate k, which are the same features shifted by a quarter period and scaled by the frequency. As a kernel method, the solve is gradient-enhanced kriging with random features, in which σ is a length-scale, μ a noise-to-prior ratio and w the ratio of value noise to derivative noise. The learned scalars follow these predictions in the experiments. Rows for differential operators and integrals are built the same way.

Results

10-shot sinusoid regression

200 test tasks; meta-iterations in parentheses. Meta-LSQ and standard MAML: mean ± std over 5 seeds; ALPaCA-style and MAML + Sobolev over 3; strong FOMAML is one run. The per-task GEK fits its kernel on every task by marginal likelihood. Training times are single-thread CPU and only indicative.
methodlearned parameterstraining (s)test MSE
MAML, standard (8k)1,761510.70 ± 0.13
FOMAML, strong (60k)8,5135120.104
MAML + Sobolev loss (8k)1,761780.216 ± 0.031
ALPaCA-style (8k)1,721120.026 ± 0.017
GEK, fitted per taskn/an/a0.013 ± 0.008
GEK, shared hyperparameters (2k)3n/a0.026 ± 0.014
Meta-LSQ, values only (2k)2110.060 ± 0.043
Meta-LSQ-Sobolev (2k)3140.019 ± 0.016

Without derivatives Meta-LSQ is still 12× better than standard MAML, and MAML given the same derivatives stays 11× worse than Meta-LSQ-Sobolev. An exact GP with shared, meta-learned hyperparameters reaches 0.026, so most of the gain over MAML comes from sharing one kernel across tasks and adapting with a closed-form solve. Measured by wall-clock time, Meta-LSQ-Sobolev after half a second of training is 8.6× more accurate than MLP-MAML after fifty seconds.

Derivative rows

Four panels on the effect of derivative rows: held-out value, gradient and Hessian error with and without them; error against the derivative weight; effective rank against bandwidth; function recovery against bandwidth.
At the learned bandwidth, ten derivative rows add about three independent directions to the ten from the values and lower the ratio of the largest to the tenth singular value of the design by more than two orders of magnitude. On held-out tasks the value, gradient and Hessian errors all fall, the Hessian error by 4.7× although nothing above first order is supervised.

Closed-loop beam tuning in simulation

The solve also works as the surrogate inside a tuning loop. On the ARES experimental area at DESY, three quadrupoles and two steerers set the position and size of the electron beam on a screen. We simulate the task with Cheetah on DESY's 300 evaluation trials, with their start setting and their budget of 150 measurements. One meta-learned model per beam parameter predicts the beam around the last setting, and the setting predicted closest to the target is measured next.

Error of the best setting after 150 measurements (µm) and measurements needed to remove 99% of the starting error (median, with the share of trials that get there). DESY's trained agent and their Bayesian optimisation run in our re-implementation of their protocol, with the agent's magnets held to their limits. One training run per method.
methodusesmedianmeanto 99%
DESY's RL agentvalues4.417.139 (62%)
DESY's Bayesian optimisationvalues30.738.0– (1%)
ALPaCA-stylevalues7.513.2– (46%)
Meta-LSQvalues7.813.2– (44%)
Gradient descent+ derivatives14.227.7– (37%)
ALPaCA-style+ derivatives1.99.820 (78%)
Meta-LSQ+ derivatives1.69.616 (79%)

From measurements alone the two surrogates are level, and about four times more accurate than the Bayesian optimisation, while the agent has the lowest median. Its mean is above the surrogates' because we hold its magnets to their limits, which DESY's simulation did not do; without that it is 11.8 µm. With derivatives from a model of the beamline, Meta-LSQ is slightly more accurate than ALPaCA-style and well ahead of gradient descent. On DESY's simulated grid of targets over the full range of the environment (beam sizes up to 2 mm), the values-only loops end at a median of 7.3 µm, against 14.4 for the agent and 41.9 for the Bayesian optimisation; the agent is level with the surrogates on targets below 0.5 mm.

The simulated screen for five methods after 1, 10, 20, 50 and 150 measurements on one trial, with the target drawn as a dashed ellipse and the final differences magnified five times.
The simulated screen on the trial where Meta-LSQ with derivatives ends closest to its median, at the best setting found so far, with the mean absolute error of the four beam parameters. The dashed ellipse is the target, and the last column shows the final beam against it with the differences magnified five times. All of this is simulation; no method here was run on the machine.

Where it does not work

Use it

pip install git+https://github.com/sulcantonin/metalsq-public
import torch
from metalsq import MetaLSQ, tasks

model = MetaLSQ(in_dim=1, n_features=200, sobolev=True)
model.meta_fit(tasks.sinusoid(K=10, K_q=10), steps=2000, lr=5e-3)   # about 4 s on a laptop CPU

# Adapting to a new task is one ridge solve.
task = tasks.sinusoid(K=10, K_q=50)(1, torch.Generator().manual_seed(0))[0]
f = model.adapt(task.x, task.y, task.dy)     # dy is optional (values only)
f(task.xq)                                   # predictions
f.gradient(task.xq)                          # first derivatives
f.derivative(task.xq, (3,))                  # any order, closed form

The repository also holds the research scripts, the stored results and the LaTeX source. One script recomputes every number printed in the paper from the stored results and exits with an error on any disagreement:

cd reproduce && python3 paper/nldl2027/check_numbers.py

Citation

@inproceedings{sulc2027metalsq,
  title     = {Meta-{LSQ}: A Closed-Form {S}obolev Inner Solver for Few-Shot Meta-Learning},
  author    = {Sulc, Antonin},
  booktitle = {Proceedings of the 8th Northern Lights Deep Learning Conference (NLDL)},
  series    = {Proceedings of Machine Learning Research},
  volume    = {352},
  publisher = {PMLR},
  year      = {2027}
}