This chapter introduces the VirtualBoard framework for building vision-based Perceptual User Interfaces (PUI). While most vision-based Human Computer Interaction applications developed over the last decade focus on the technological aspects related to image processing and computer vision, our main effort is towards ease and naturalness of use, integrability and compatibility with the existing systems and software, portability and efficiency. VirtualBoard is based on a modular architecture which allows the implementation of several classes of gestural and vision-based human-computer interaction approaches it is extensible and portable and requires relatively few computational resources, thus also helping in reducing energy consumption and hardware costs. Particular attention is also devoted to robustness to environment conditions (such as illumination and noise level). We believe that current technologies …