New tool automatically turns math into pictures

The tool enables users to create diagrams simply by typing an ordinary mathematical expression and letting the software do the drawing. Unlike a graphing calculator, these expressions aren't limited to basic functions, but can be complex relationships from any area of mathematics.

The researchers named it Penrose after the noted mathematician and physicist Roger Penrose, who is famous for using diagrams and other drawings to communicate complicated mathematical and scientific ideas.

"Some mathematicians have a talent for drawing beautiful diagrams by hand, but they vanish as soon as the chalkboard is erased," said Keenan Crane, an assistant professor of computer science and robotics. "We want to make this expressive power available to anyone."

Diagrams are often underused in technical communication, since producing high-quality, digital illustrations is beyond the skill of many researchers and requires a lot of tedious work.

Penrose addresses these challenges by enabling diagram-drawing experts to encode how they would do it in the system. Other users can then access this capability using familiar mathematical language, leaving the computer to do most of the grunt work.

The researchers will present Penrose at the SIGGRAPH 2020 Conference on Computer Graphics and Interactive Techniques, which will be held virtually this July because of the COVID-19 pandemic.

"We started off by asking: 'How do people translate mathematical ideas into pictures in their head?'" said Katherine Ye, a Ph.D. student in the Computer Science Department. "The secret sauce of our system is to empower people to easily 'explain' this translation process to the computer, so the computer can do all the hard work of actually making the picture."

Once the computer learns how the user wants to see mathematical objects visualized -- a vector represented by a little arrow, for instance, or a point represented as a dot -- it uses these rules to draw several candidate diagrams. The user can then select and edit the diagrams they want from a gallery of possibilities.

The research team developed a special programming language for this purpose that mathematicians should have no trouble learning, Crane said.

"Mathematicians can get very picky about notation," he explained. "We let them define whatever notation they want, so they can express themselves naturally."

An interdisciplinary team developed Penrose. In addition to Ye and Crane, the team included Nimo Ni and Jenna Wise, both Ph.D students in CMU's Institute for Software Research (ISR); Jonathan Aldrich, a professor in ISR; Joshua Sunshine, an ISR senior research fellow; cognitive science undergraduate Max Krieger; and Dor Ma'ayan, a former master's student at the Technion-Israel Institute of Technology.

"Our vision is to be able to dust off an old math textbook from the library, drop it into the computer and get a beautifully illustrated book -- that way more people understand," Crane said, noting that Penrose is a first step toward this goal.

The National Science Foundation, Defense Advanced Research Projects Agency, the Sloan Foundation, Microsoft Research and the Packard Foundation supported this research.