CS3483 Multimodal Interface Design Assignment: Handpose model
CS3483 Multimodal Interface Design Assignment
Overview
In the assignment, you are required to use p5.js and the ml5.js Handpose model to develop an interface for viewing and interacting with an image. These actions are performed by using fingertip detection and keyboard operations.
Detailed Requirement
1. Initial Setup
Create a display window.
Load and display an image in the window.
The instant view of the camera should be displayed next to the image and be the same size as the image.
2. Index fingertip detection
A small circle should be drawn on the camera view to indicate the position of the detected index fingertip. When you move your index finger in front of the camera, the circle should keep following your fingertip. A corresponding position indicator should also be drawn on the image to follow the movement of your index fingertip.
3. Freehand drawing on the image
When the ‘f’ key is pressed and released, the interface should enter the freehand drawing mode. In this mode, a freehand trace should be drawn on the image according to the movement of your index fingertip.
4. Region selection and copying on the image
When the ‘s’ key is pressed and released, the interface should enter the region selection and copying mode. In this mode, the image should be displayed. A bounding box should be displayed between the tips of your thumb and index finger in the camera view, and its size can be adjusted according to the distance between the two fingers. A bounding box with
the same size should also be displayed in the corresponding location of the image.
In this mode, when the ‘c’ key is pressed and released, the image region enclosed by the bounding box should be captured. The bounding box can then be moved to a new location, and its size can be adjusted according to the distance between the tips of your thumb and index finger. When the ‘v’ key is pressed and released, the previously captured image region should be pasted onto the region enclosed by the bounding box at the new location.
This operation is required to be performed once only. After the operation is completed, the position indicators can be re-displayed.
5. Exiting the freehand drawing/region selection and copying mode
When the ‘e’ key is pressed and released, the interface should exit from the freehand drawing/region selection and copying mode, and return to the state specified in “2. Index fingertip detection” with the original image re-displayed.
Assignment Submission
Program (40%)
You should submit your p5.js program and the associated image file.
Report (60%)
You should summarize your work in the form of a report which should include:
-
A description of the design of the different sections of your program.
-
Screen captures of the output.
-
Limitations and possible improvements of the program.
