Staged Factorial Screening for Budget-Constrained Micro-Pretraining
A paper tests staged fractional-factorial screens for tiny pretraining runs and finds they can identify strong factor directions quickly.
Intelligence analysis by GPT-5.4 Mini
Using 613 experiments across short screens, reruns, anchor checks, and longer continuations, the paper studies how to triage candidate training recipes on one GPU under tight budgets. It argues that designed screening can surface high-penalty settings early, but results remain sensitive to budget and host.
It is like testing lots of cake recipes with tiny bite-sized samples before baking the whole cake. The paper says short, planned tests can quickly show which ingredients hurt the result, but the best recipe can still change depending on the oven and how long it bakes.
Analysis
What the paper tests
The paper asks whether a staged fractional-factorial workflow can recover stable early effect structure in budget-constrained micro-pretraining. The setup uses a fixed autoresearch-derived single-GPU training loop and runs 613 experiments across several phases: pilot and follow-up screens at 2, 5, and 10 minutes; full 16-condition seeded reruns at 5 and 10 minutes; targeted seeded anchor checks; same-host greedy and matched-cost random baselines; a 60-minute bridge package; and bounded Windows A100 and Linux L40S anchor continuations through 24 hours.
Main findings
The strongest early penalties come from total batch, depth, and width, and those penalties are largest at short budgets before relaxing as budget increases. In the predeclared seeded full-screen families, D, A, B, and C keep non-zero estimates at 5 and 10 minutes after within-budget Benjamini-Hochberg correction, while E does not. The paper also reports that random search can find strong incumbents in the 32-condition space, but it tends to do so repeatedly in the same low-penalty region and without factor attribution.
The 60-minute bridge anchor has the lowest mean, though the author notes that this does not separate workflow refinement from the bridge model's larger capacity advantage. In bounded 12-hour and 24-hour three-anchor continuations on both hosts, the bridge remains lowest by sample mean, while the ordering among the non-bridge settings changes with host.
Takeaway
The paper’s recommendation is narrow and bounded: use short designed screens to identify high-penalty directions, confirm promising anchors with repeated runs, and refine locally inside the reduced space. The evidence supports a bridge-centered recommendation through 24 hours on the two tested hosts, but not a hardware-invariant ranking or a general claim of hyperparameter-optimization superiority.
Key points
- The paper studies staged fractional-factorial screening for micro-pretraining under tight compute budgets.
- It reports 613 experiments across short screens, reruns, baselines, and longer host-specific continuations.
- Total batch, depth, and width show the largest early penalties, especially at short budgets.
- The bridge anchor has the lowest mean in the tested longer runs, but the author says this is not proof of hardware-invariant superiority.
- The recommended use is narrow: screen early, confirm anchors, and refine within the reduced space.
If this approach works as intended, teams with small compute budgets could screen many training recipes cheaply and focus longer runs on the most promising ones. That could reduce wasted GPU time and make early-stage pretraining experiments more systematic.
The paper also shows that results depend on budget and host, so the same ranking may not hold across hardware or longer runs. It also warns that the best-performing bridge package may reflect a larger model capacity advantage, not just a better workflow.
