Professor Chyi-Yeu Lin and his colleagues at the National Taiwan University of Science and Technology’s Center for Intelligent Robot (NTUST, or Taiwan Tech) have programmed their robot to autonomously read music notes and sing the song lyrics (automatically converted to Pinyin) with a synthesized voice. Earlier this week New Scientist reported on a demonstration that used the lab’s 2nd generation female head (described in a 2011 research paper here), however the actual video clip is from March 2008. In this unabridged video, the singing begins at around the 1 minute mark.
If that falls too deeply into the uncanny valley for your tastes, then take a look at their two-wheeled self-balancing robot. It hasn’t been given a name as far as I can tell, but it is regularly shown alongside their bipedal robots (such as at TiiST 2008) and was the first robot they tested with their music reading software. Its body contains an Intel Core Duo 1.66 GHz CPU with 512 MB of RAM which runs the software, and maintains its balance using a gyro and tilt sensor. It is able to move forwards and backwards as well as make left and right turns.
It is equipped with a single Logitech QuickCam Sphere webcam for its vision needs, which is unable to focus, so you have to manually adjust the distance between it and the music notes. The camera takes a snapshot of the notes and then the software straightens out the image, reduces its color to black and white, and extracts the relevant characters. The software has a character recognition accuracy greater than 98%. At this point you may be wondering why printed music notes are used at all, since it would save some time and effort if the robot was simply programmed to sing specific songs. The researchers say that a little showmanship helps to make the performances feel more real to the audience. When it’s not singing, it can also play the drums and draw simple portraits. Listen to its rendition of “Twinkle, Twinkle Little Star” and see some of its other talents in the video below.
“Maybe it’s one of those things that a robot can do better than ordinary people,” Lin says. “It can read [the music] in 40 seconds and then close its eyes to sing.” Human singers take more time to read and memorize songs. “It can sing any song, as long as the lyrics are written with Roman spellings”, he added.
From a survey of 100 people stopped at random in Taipei, 50% agreed the robotic singing was entertaining, 40% found the voice quality “similar to a human voice”, while another 40% found it “not realistic, but acceptable”. This isn’t the only singing robot, of course, but it is the first to sing in Mandarin. The SONY QRIO had a speech synthesizer that was capable of singing, and since then the Vocaloid software has become hugely popular. AIST’s HRP-4C is probably the most famous example of a singing robot, and is able to visually track a singer’s facial expressions and duplicate them (with some success) in its own performances.
A few photos of the two-wheeled robot follow after the break.