Ever since music existed, people have been associating it with colors. This is a phenomenon that occurs regularly, and apps and studies have also been carried out into this [1]. However, turning shapes and colors into music is a lot less known to people. Because this is still an early stage, this project is going to do research into whether the phenomenon also corresponds to children. This data is processed and eventually developed as an interactive game for children. This allows them to play with shapes, colors, sizes, and placement that are transformed into music. This then creates a playful way for children to learn to recognize music elements like, pitches, and notes by making a beat.
Project decription
The main question of the project is as follows: How can shapes, colors, sizes, and placement be transformed into music in a way that ‘through an app or physical play method’ children learn music elements like, pitches, and notes through a beat?
At the start, a requirements package was created, which formulated and specified the customer’s requirements and preferences. From these requirements, several technical features emerged that the team had not previously utilized. Therefore, initial research was conducted to understand what they entailed, how they functioned, and to determine their accessibility and applicability for the project use. Further research was conducted on AI, vision technology, music in general, and its computer-generated composition. In addition to these technical investigations, a study was conducted with children, age 8-11 , to explore the aspects of music they associated with shapes and colors.
Project results
After the research phase, the initial goal was adjusted to develop an old movie camera that could be placed in front of a whiteboard, capturing compositions of shapes and colors and generating corresponding sound (see video below). Subsequently, this goal was realized and verified through testing the product with the children who conducted the research.
The hardware design utilizes 3D-printed components, with an life-sized 1930s movie camera as the chosen design. The camera incorporates LED strips, switches, buttons, speakers, and a large ENTER button to start the process This aligns perfect with the appealing and playfulness of the experience.
The camera is placed in front of a whiteboard. A composition is made consisting of a, by the user, chosen number of shapes placed on the whiteboard. The user can choose out of six different shapes of variating sizes, and each six different colors. The position, color, size, and shape of each object will added a note or accord to the resulting sound. Inside the film camera, there is a microcomputer called Jetson Nano, which analyzes the captured image and converts it into sound using custom software. This sound is then played back to the public through the speakers in the camera.
The system is able to make a corresponding sound, based on the research, of a composition.
Video
Acknowledgements
The BeCreative student team wants to thank the International School Eindhoven (ISE) for generously allowing us to conduct our research and for providing the necessary resources for the successful execution of this project. Also to Fontys Engineering for their technical support.