LabFactory: Building and Evaluating Executable AI Labs

LabFactory contributes a construction and delivery protocol, component-level accounts of constructed solvers and controller patterns, and execution-grounded case records linking delivered artifacts to host-side results.

Source: [1], [5], [21]

LabFactory describes a solver as a composite artifact of input processing, model parameters, knowledge and retrieval resources, executable tools, and a controller that invokes and combines those components; not every solver must contain every component.

Source: [13]

The delivery contract fixes a common entry-script interface for batch inputs and identifier-indexed predictions while leaving internal representations, model families, tools, and inference policies to task-specific construction.

Source: [2], [16]

Three recurring controller patterns are described: fixed procedures combining predictors with prescribed language-model support; LLM-driven loops that write and run code or queries against constructed tools; and retrieval-and-rank controllers that select among candidates returned by an index.

Source: [11], [17]

Independent evaluation runs the delivered solver on held-out inputs and computes the configured metric against host-side references, recording execution status, coverage, and applicable output checks; this outcome is distinct from builder development measurements.

Source: [3], [15]

Among the 28 delivered artifacts, 10 contain predictive models fitted during construction, one further case fits retrieval-ranking weights, and the remaining 17 primarily assemble tools, knowledge resources, and control procedures around the platform LLM.

Source: [6]

Builders work in a metered build VM with network access to public datasets, software, and model weights; resource use is recorded, but the framework does not impose a single fixed budget across tasks.

Source: [20]

The host keeps evaluation references outside the solver input interface and remints case identifiers and permutes input fields at evaluation; however, builders may acquire public resources during construction, and the interface boundary does not establish that those resources are disjoint from evaluation samples.

Source: [10]

An episode can end with an explicit builder submission or with operator closure when a builder stops or times out with a staged artifact; the staged artifact proceeds to independent evaluation in either case.

Source: [7]

The paper distinguishes the platform LLM from domain-specific components: the builder trains domain models and writes invocation code but does not fine-tune the platform LLM, and delivered artifacts contain no newly trained LLM weights.

Source: [18]

A typical construction episode inspects the brief and environment, acquires public data and software, develops models or tools, validates on development splits, packages the solver, and submits it; these stages are common rather than mandatory, and development feedback is separate from independent evaluation.

Source: [9]

The task specification represents a task by its scientific brief, input and output formats, available resources and delivery contract, and evaluation metric; the framework supplies working and evaluation environments, while representation, model family, tool design, and inference policy are chosen during construction.

Source: [2], [4]