Welcome, guest | Sign In | My Account | Store | Cart

Control your mouse with the Microsoft Kinect using OpenKinect (Python recipe) by Alexander James Wallar
ActiveState Code (http://code.activestate.com/recipes/578082/)

This is a simple code that uses the Kinect in order to simulate the mouse on the screen. In order for this code to work you must have a Linux machine, the freenect module from OpenKinect, NumPy, and Xlib. The last three can be downloaded and installed by running sudo apt-get install {PROGRAM NAME HERE}. Freenect needs to be downloaded and installed for python using the terminal and there are good instructions here how to do it http://openkinect.org/wiki/Getting_Started. You will also need to install the python wrapper. This can be done by following the instructions on this site: http://openkinect.org/wiki/Python_Wrapper. To run the code you must first cd into the directory in which the code resides and then in the terminal run the command sudo python {THE NAME YOU STORED THE CODE BELOW AS}.py. This should run the code. The code needs superuser privileges. Also if you wonder why the mouse will stop tracking your hand, there are two reasons. 1) This code is not super sophisticated, all it does is find the minimum point in the depth image and scales the coordinates to the size of the screen and moves the mouse there so if your stomach is the closest thing to the screen it will track it and place the mouse there. The other reason the mouse might stop tracking is because there is a counter in the that will only let it run 10000 iterations. This is just a safety precaution in case you are not able to control the mouse and the computer freaks out. The last thing you need to know to run this code is that the main_mouse() function takes to arguments, the first is the x dimension of your screen and the second is the y dimension. THE CODE WILL NOT FUNCTION PROPERLY ON MULTIDISPLAY COMPUTERS. Thank You

Python, 75 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
#!/usr/bin/env python
"""
Use the Kinect to control your mouse

@Author   = Alexander James Wallar
@Date     = March 19, 2012
@Version  = 1.0
@Filename = Mouse.py
"""
from Xlib import X, display
import Xlib.XK
import Xlib.error
import Xlib.ext.xtest
from freenect import sync_get_depth as get_depth
import numpy as np

zeros = lambda length: [0 for _ in range(length)]

global depth #Makes the depth global

#Sets the size of the screen
xSize = 640
ySize = 480

#Mean filter caches
yList = zeros(35)
xList = zeros(35)

d = display.Display()

def mouse_move(x,y):
    s = d.screen()
    root = s.root
    root.warp_pointer(x,y)
    d.sync()

def mouse_click_down(button): 
    Xlib.ext.xtest.fake_input(d,X.ButtonPress, button)
    d.sync()
    
def mouse_click_up(button):
    Xlib.ext.xtest.fake_input(d,X.ButtonRelease, button)
    d.sync()

def get_min_pos_kinect():
    
    (depth,_) = get_depth()
        
    minVal = np.min(depth) #This is the minimum value from the depth image
    minPos = np.argmin(depth) #This is the raw index of the minimum value above
    xPos = np.mod(minPos, xSize) #This is the x component of the raw index
    yPos = minPos//xSize #This is the y component of the raw index
        
    xList.append(xPos)
    del xList[0]
    xPos = int(np.mean(xList))
    yList.append(yPos)
    del yList[0]
    yPos = int(np.mean(yList))
        
    return (xSize - xPos-10, yPos, minVal)
    
def main_mouse(screen_x = 1280, screen_y = 800):
    timer = 0
    while timer < 10000:
        x_min, y_min, min_val = get_min_pos_kinect()
        print min_val
        x_min = int((screen_x/630)*x_min)
        y_min = int(2*(screen_y/479)*y_min)
        mouse_move(x_min, y_min)
        if min_val < 600:
            mouse_click_down(1)
        else:
            mouse_click_up(1)
        timer +=1

How to use the code above:

You run it in the terminal with superuser privileges. It will track the minimum point on the screen. If that minimum point is smaller than 600 arbitrary units (These are the values being printed in the terminal), the mouse will click down at the scaled coordinates of the minimum point. If the minimum point is larger than 600 arbitrary units, the mouse will click up. Basically, you put your hand in front of the Kinect, you push in to click down and pull out to click up.

If you have any concerns please do not hesitate to email me at wallarelvo@hotmail.com. Also please no harsh comments, this is only a super-alpha.

Tags: computer, control, freenect, kinect, linux, microsoft, mouse, numpy, openkinect, vision, xlib