I Think that fully automating the scientific research process is impossible, as crucial decisions regarding resource allocation still require human intervention.

A new research paper titled “The AI Scientist” highlighted that the cost of producing a research paper using this automated system is around $15. While this low cost is beneficial for small-budget research projects, it may not be sufficient to achieve significant progress in any field.

Let’s discuss “The AI Scientist”: Does it truly automate the scientific research process?

The “AI Scientist” framework aims to fully automate the scientific research process. This includes generating new research ideas, suggesting ways to test them, writing code, conducting experiments, and writing complete scientific papers.

Firstly, humans provide the research topic to “The AI Scientist” as a starting point. Then, the AI Scientist uses large language models (LLMs) to generate research ideas. These ideas are evaluated using tools like Semantic Scholar to select the best ones based on novelty and other factors. The winning idea(s) move to the next stage.

In this stage, another language model, such as AIder, suggests a way to experiment with the idea and generates the code to test it. This code is then automatically executed to obtain results.

It’s important to note that humans provide a template or basic code to guide the LLM, ensuring it remains consistent with their intended approach and doesn’t deviate significantly from the programming solution they envision.

Finally, the process of writing the paper is handled by the LLM using a template, such as a NeurIPS conference template. Given the capabilities of LLMs, it’s not surprising that they can perform this task with great accuracy.

The “AI Scientist” has succeeded in generating several good research ideas, implementing them, and writing scientific papers with few errors, which may lead some to believe that the scientific research process can be fully automated by such tools.

Why can’t this process be fully automated?

As we have seen, human intervention is required at several points in The AI Scientist, to ensure it doesn’t go beyond its defined boundaries, whether in terms of the code it generates, the cost, or the training data it uses. Therefore, human intervention remains essential in decision-making, even if the process may seem otherwise.

Scientific research is a process that requires resource allocation and management, and this process cannot be performed by The AI Scientist, even if its work is limited to developing code ideas that can be executed on a computer. Letting it experiment with whatever ideas it deems appropriate would be economically costly and considered a waste of resources (imagine how many millions it costs to train large generative AI models). There has never been a shortage of research ideas, but rather a shortage of securing financial support to realize them.

Keep in mind that many research projects try to obtain funding, but only a few succeed. Similarly, while a few startup projects can obtain funding, only a few succeed.

Ideas are abundant, and the problem lies in the cost of implementing and testing them. Although The AI Scientist facilitates this process, accelerates it, and adds to the knowledge of its users, it does not solve the fundamental problem of automating the process of creating knowledge.

Nevertheless, such tools and the automation of the scientific research process remain highly valuable in helping human researchers accelerate their work and explore new ideas.