Research note
Deep Learning Autoresearch with GPT
How GPT chooses experiments and interprets their results
We asked GPT to conduct data science research by designing and training multilayer perceptrons (MLPs) for given input features and targets. We implemented an autoresearch loop in which GPT can design up to three new experiments at each iteration by changing permitted parameters and inspecting past results in a database. A separate LLM judge, with its own instructions, provides feedback.
Across these runs, GPT was a weak autonomous researcher. It showed poor research taste, mostly made narrow changes and struggled to turn evaluation feedback into useful next experiments.
Experimental design
The researcher and judge follow separate instruction files and share results and feedback through the experiment database.
We provide GPT with input features and targets derived from historical price and volume data. The targets are future returns adjusted for market movements. GPT fits the neural network by changing the settings listed below, while the inputs, targets and training and validation periods stay fixed. The evaluation script scores each model on held-out validation data. Both the researcher and judge can inspect the validation scores and learning curves.
Before designing a new experiment, GPT can inspect the database of previous experiments and feedback for inspiration. Its proposed models are submitted as GPU jobs to a cluster. An evaluation script trains and scores each model, then the LLM judge reviews the results and stores its feedback in the database.
| What the agent can change | Examples |
|---|---|
| Model design | Architecture, width, depth, activation function |
| Optimisation and regularisation | Optimiser, learning rate, weight decay, gradient clipping |
| Training schedule | Batch size, learning rate schedule, early stopping |
We measure performance by the correlation between the output and target.
Results
The charts below show experiment results across multiple autoresearch steps. Performance does not improve at every iteration.
Green marks the best result at each iteration and grey the rest, while the labels describe changes from the winning experiment’s parent or starting configuration.
The lines show mean duration and model size across the two longer histories, with shading for ±1 SD.
These two plots track how long each autoresearch iteration takes and the size of the models it produces. Iteration duration includes time spent on experiment design, queueing, training and judging.
Qualitative analysis
GPT does not explore the hyperparameter space well
Comparing the hyperparameter configurations within each autoresearch iteration shows that the experiments are often very similar. Their cosine similarities are concentrated near 1, suggesting limited exploration of different configurations.
Cosine similarity compares scaled hyperparameter vectors; identical vectors score 1.
GPT has poor research taste
GPT often makes minimal parameter changes that introduce little conceptually new. Its search resembles brute-force refinement around a promising configuration, with little judgement about the value of each new experiment.
In the example below, GPT takes the strongest configuration from the previous round and proposes three experiments that change only its learning rate. Two proposals are identical; the third differs by just 0.00000072 (7.2 × 10⁻⁷). We would not expect these closely spaced proposals to yield meaningfully different research insights.
Three proposals from one iteration, all reducing the learning rate of the strongest model from the previous round by similar amounts.
GPT struggles to interpret results correctly
GPT struggles with research where outcomes are not simply good or bad and require intelligent judgement. In these examples, the experiments still indicate learning progress, but GPT describes their validation curves as flat.
Green shows the average of the last five scores with ±1 SD shading, alongside GPT’s description.