This is a Weka bird. Except it's been Wekinated, so it has scary red eyes.

The Wekinator: Software for using machine learning to build real-time interactive systems

Rebecca Fiebrink, Dan Trueman, and Perry Cook


The Wekinator is a free software package to facilitate rapid development of and experimentation with machine learning in live music performance and other real-time domains. The Wekinator allows users to build interactive systems by demonstrating human actions and computer responses, rather than by programming.

Example applications:

  • Creation of new musical instruments
    • Create mappings between gesture and computer sounds. Control a drum machine using your webcam! Play Ableton using a Kinect!
  • Creation of gesturally-controlled animations and games
    • Control interactive visual environments like Processing or Quartz Composer, or game engines like Unity, using gestures sensed from webcam, Kinect, Arduino, etc.
  • Creation of systems for gesture analysis and feedback
    • Build classifiers to detect which gesture a user is performing. Use the identified gesture to control the computer or to inform the user how he's doing.
  • Creation of real-time music information retrieval and audio analysis systems
    • Detect instrument, genre, pitch, rhythm, etc. of audio coming into the mic, and use this to control computer audio, visuals, etc.
  • Creation of other interactive systems in which the computer responds in real-time to some action performed by a human user (or users)
    • Anything that can output OSC can be used as a controller
    • Anything that can be controlled by OSC can be controlled by Wekinator

The Wekinator includes:

  • real-time feature extraction 
    • built-in feature extraction from audio input, USB game controllers, laptop motion sensors, and webcams
    • ability to plug-in other feature extractors that output OSC, for getting control data from Arduino, Kinect, Processing/OpenCV, iPhone/iPad running TouchOSC, or other programs written in Java/C++/Python/you-name it
  • Weka's implementations of discrete and continuous supervised learning methods
    • including neural networks, AdaBoost, support vector machines, k-nearest-neighbor, and decision trees
  • A GUI for managing input devices, feature extraction, selection and configuration of learning algorithms, creating training examples by demonstration, and the training and running of machine learning models 
    • allows users to interactively create machine learning models from scratch, without any programming
  • built-in support for ChucK sound synthesis so users can easily plug in their own ChucK code
  • support for controlling other sound synthesis, visuals, or other processes in other programs or environments that can receive OSC
    • e.g., control Processing, Max/MSP, PD, SuperCollider, Ableton, Quartz Composer, Unity 3D game engine, ... or custom programs written in Java/C++/Python/you-name-it.

get it & use it

Instructions for downloading, running, and implementing your own add-ons are here

ChucK version or later and Java version 5 or later

support & mailing lists

New user? Need help? Join the Wekinator users mailing list
Just want occasional announcements about new releases? Join the announcement list

introductory videos

Quick overview by Rebecca Fiebrink, using webcam and USB game controller to control ChucK and Max/MSP (September 2010)
Demo by Phoenix Perry, using Kinect to control Unity game engine (January 2011)

compositions using Wekinator

Clapping Machine Music Variations (CMMV)
by Dan Trueman
Wekinator is used to create various new musical instruments, including one that controls the blotar physical modeling synthesis algorithm using the Real World Golf USB controller
3 April 2010 - PLOrk Spring Concert: watch
5 June 2010 - Sideband @ International Computer Music Conference
25 July 2010 -
SO Percussion Summer Institute: watch

by Raymond Weitekamp
Laptop performers hit their laptops in different locations; based on the hit location detected by Wekinator, ChucK plays different sounds
3 April 2010 - PLOrk Spring Concert: watch
15 May 2010 - PLOrk final extravaganza

The Gentle Senses
by Michelle Nagai
The performer plays an instrument consisting of a piece of tree bark with embedded light sensors; the Wekinator translate light patterns into sound in Max/MSP
27 April 2010: Michelle Nagai at Princeton University

by Rebecca Fiebrink
A piece for laptop orchestra and optional shakuhachi, in which performers control sound in ChucK by gesturing with flashlights in front of laptop webcams
 19 April 2009: PLOrk at HASTAC, Chicago
16 May 2009 - PLOrk Spring Concert with special guest Riley Lee
10 October 2009 - PLOrk at MMiX NYC

by Rebecca Fiebrink
A real-time machine learning improvisation, in which performers create mappings from controllers to sound from scratch and evolve them throughout the piece
January 2009 - Princeton University: watch

Fiebrink, R., P. R. Cook, and D. Trueman. "Human model evaluation in interactive supervised learning." Proceedings of the SIGCHI Conference on Human-Computer Interaction (CHI'11), Vancouver, BC, May 7-12, 2011.

Fiebrink, R. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. PhD dissertation, Princeton University, 2011.

Fiebrink, R., D. Trueman, C. Britt, M. Nagai, K. Kaczmarek, M. Early, M.R. Daniel, A. Hege, and P. R. Cook. “Toward understanding human-computer interaction in composing the instrument.Proceedings of the International Computer Music Conference (ICMC), New York City, June 1–5, 2010.

Fiebrink, R., P. R. Cook, and D. Trueman. "Play-along mapping of musical controllers." Proceedings of the International Computer Music Conference (ICMC), Montreal, August 4–9, 2009.

Fiebrink, R., D. Trueman, and P. R. Cook. "A meta-instrument for interactive, on-the-fly machine learning." Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), Pittsburgh, June 4–6, 2009.

Wekinator Users Hall of Fame
Andrew McPherson
Anne Hege
Cameron Britt
Dan Trueman
Konrad Kaczmarek
Meg Schedel
Michelle Nagai
Michael Early
M. R. Daniel
Phoenix Perry
Raymond Weitekamp
Students of PLOrk 2010
Thanks to everyone above for making cool projects and sharing their feedback!

see also
Wekintor on Google Code
weka - from the University of Waikato

Rebecca Fiebrink's homepage