Autonomous robot plays with NanoLEGO

Rapid prototyping, the fast and cost-effective production of prototypes or models -- better known as 3D printing -- has long since established itself as an important tool for industry. "If this concept could be transferred to the nanoscale to allow individual molecules to be specifically put together or separated again just like LEGO bricks, the possibilities would be almost endless, given that there are around 1060 conceivable types of molecule," explains Dr. Christian Wagner, head of the ERC working group on molecular manipulation at Forschungszentrum Jülich.

There is one problem, however. Although the scanning tunnelling microscope is a useful tool for shifting individual molecules back and forth, a special custom "recipe" is always required in order to guide the tip of the microscope to arrange molecules spatially in a targeted manner. This recipe can neither be calculated, nor deduced by intuition -- the mechanics on the nanoscale are simply too variable and complex. After all, the tip of the microscope is ultimately not a flexible gripper, but rather a rigid cone. The molecules merely adhere lightly to the microscope tip and can only be put in the right place through sophisticated movement patterns.

"To date, such targeted movement of molecules has only been possible by hand, through trial and error. But with the help of a self-learning, autonomous software control system, we have now succeeded for the first time in finding a solution for this diversity and variability on the nanoscale, and in automating this process," says a delighted Prof. Dr. Stefan Tautz, head of Jülich's Quantum Nanoscience institute.

The key to this development lies in so-called reinforcement learning, a special variant of machine learning. "We do not prescribe a solution pathway for the software agent, but rather reward success and penalize failure," explains Prof. Dr. Klaus-Robert Müller, head of the Machine Learning department at TU Berlin. The algorithm repeatedly tries to solve the task at hand and learns from its experiences. The general public first became aware of reinforcement learning a few years ago through AlphaGo Zero. This artificial intelligence system autonomously developed strategies for winning the highly complex game of Go without studying human players -- and after just a few days, it was able to beat professional Go players.

"In our case, the agent was given the task of removing individual molecules from a layer in which they are held by a complex network of chemical bonds. To be precise, these were perylene molecules, such as those used in dyes and organic light-emitting diodes," explains Dr. Christian Wagner. The special challenge here is that the force required to move them must never exceed the strength of the bond with which the tip of the scanning tunnelling microscope attracts the molecule, since this bond would otherwise break. "The microscope tip therefore has to execute a special movement pattern, which we previously had to discover by hand, quite literally," Wagner adds. While the software agent initially performs completely random movement actions that break the bond between the tip of the microscope and the molecule, over time it develops rules as to which movement is the most promising for success in which situation and therefore gets better with each cycle.

However, the use of reinforcement learning in the nanoscopic range brings with it additional challenges. The metal atoms that make up the tip of the scanning tunnelling microscope can end up shifting slightly, which alters the bond strength to the molecule each time. "Every new attempt makes the risk of a change and thus the breakage of the bond between tip and molecule greater. The software agent is therefore forced to learn particularly quickly, since its experiences can become obsolete at any time," Prof. Dr. Stefan Tautz explains. "It's a little as if the road network, traffic laws, bodywork, and rules for operating the vehicle are constantly changing while driving autonomously." The researchers have overcome this challenge by making the software learn a simple model of the environment in which the manipulation takes place in parallel with the initial cycles. The agent then simultaneously trains both in reality and in its own model, which has the effect of significantly accelerating the learning process.

"This is the first time ever that we have succeeded in bringing together artificial intelligence and nanotechnology," emphasizes Klaus-Robert Müller. "Up until now, this has only been a 'proof of principle'," Tautz adds. "However, we are confident that our work will pave the way for the robot-assisted automated construction of functional supramolecular structures, such as molecular transistors, memory cells, or qubits -- with a speed, precision, and reliability far in excess of what is currently possible."