Controlling google earth using 3D Gesture MGC3130 Hillstar Dev Kit

       
           In this blog post I am just going to give overview about one of the application using 3D gesture. In fact the idea I am going to discuss came in mind when discussing about this new technology with my friends. And here is I am able to control Google earth using hand gestures in air. Just see video below to get idea, how does it work exactly

     

     At first I downloaded SDK and Aurea GUI on which I could test functionality of HillStar development Kit. You can find these resources herehttp://www.microchip.com/pagehandler/en-us/technology/gestic/gettingstarted.html.

      I tested Aurea GUI application which worked out of box with Hillstar development kit. Then I also tested SDK example codes which worked without problem.

      For the current application I used one of the sample code from the given SDK which gives data on command line. In this project I had to control Google earth, but there is not way to in C to control google earth, hence I needed to send this data from the MGC3130 Application to Python application that controls Google earth using Google earth API for python. I used socket programming in this and transmitted data over local network using TCP. I could also use shared memory in this. And finally I have a working model that controls google earth. 

       In fact, what I did is written on top of SDK demo code that can be found with SDK called "console". This demo code gives all gesture related data on console. E.g. x,y,z co-ordinates, circle gesture detection, left-right,top-bottom vice versa. Means, In order to use the gesture output to control Google earth, I could use the same console data. But here the main question was how can I control Google earth, obviously using Google API's. I preferred to use Google's Python APIs for same. Now, In this case, I need to send these data whatever I get on console to the program which will control Google earth using Python APIs. Now there can be two options to send this data from C program to Python is to use socket programming or using shared memory. I preferred socket programming and thought to transfer this data on TCP socket over local IP to Python program. So I need to write TCP client which sends the data from TCP client which is implemented in C within SDK provided to TCP Server which is basically python program running.

Code snippets -

Here, I am just going to give details about add on codes which I did to make this application, rather than providing complete code.

Following is the TCP Client implemented in C (You can find lot of examples online about TCP client implementation in C)

1. /* 
2.     Create a TCP socket 
3. */  
4.   
5. #include<stdio.h>  
6. #include<winsock2.h>  
7. #include "console.h"  
8.   
9.   
10. #pragma comment(lib,"ws2_32.lib") //Winsock Library  
11.   
12.  WSADATA wsa;  
13.  SOCKET s;  
14.  struct sockaddr_in server;  
15.  char *message;  
16.   
17.   
18. int tcpInit(void)  
19. {  
20.        printf("\nInitialising Winsock...");  
21.     if (WSAStartup(MAKEWORD(2,2),&wsa) != 0)  
22.     {  
23.         printf("Failed. Error Code : %d",WSAGetLastError());  
24.         return 1;  
25.     }  
26.        
27.     printf("Initialised.\n");  
28.        
29.     //Create a socket  
30.     if((s = socket(AF_INET , SOCK_STREAM , 0 )) == INVALID_SOCKET)  
31.     {  
32.         printf("Could not create socket : %d" , WSAGetLastError());  
33.     }  
34.   
35.     printf("Socket created.\n");  
36.     server.sin_addr.s_addr = inet_addr("127.0.0.1");  
37.     server.sin_family = AF_INET;  
38.     server.sin_port = htons(5005 );  
39.   
40.     //Connect to remote server  
41.     if (connect(s , (struct sockaddr *)&server , sizeof(server)) < 0)  
42.     {  
43.         puts("connect error");  
44.         return 1;  
45.     }  
46.   return 0;  
47. }  
48. int tcpSendData(char *uData)  
49. {  
50.      //while (1)  
51.      //{  
52.   //puts("Connected");  
53.   
54.   
55.   retry:    
56.   //Send some data  
57.   //message = "GET / HTTP/1.1\r\n\r\n";  
58.   if( send(s , uData , strlen(uData) , 0) < 0)  
59.   {  
60.   puts("Send failed");  
61.   return 1;  
62.   goto retry;  
63.   }  
64.   Sleep(1);  
65.   puts("Data Send\n");  
66.     //}  
67.   return 0;  
68. }  
69.  //   closesocket(s);  
70.    // WSACleanup();  
71.   //  return 0;  
72. //}  

As you can see the function tcpSendData is used to send data over TCP socket to local Ip over 5005 port. This function is called in gestic.c file, when gesture is detected. Following is the code snippet for same.

1.        if(data->gestic_gesture != NULL) {  
2.             /* Remember last gesture and reset it after 1s = 200 updates */  
3.             if(data->gestic_gesture->gesture > 0 && data->gestic_gesture->gesture <= 6)  
4.                 data->last_gesture = data->gestic_gesture->gesture;  
5.             else if(data->gestic_gesture->last_event > 1) //Shrenik  
6.                 data->last_gesture = 0;  
7.   
8.   
9.             printf("Gesture:          %s \n", gestures[data->last_gesture]);  
10.     
11. if(strcmp(gestures[data->last_gesture],"DNULL"))  
12.   {  
13.   tcpSendData(gestures[data->last_gesture]);  
14. }  
15.   
16.   
17.   }

Before writing program you need to install Google python API package which is attached with this blog. Install this package as you install normally python package "setup.py install". Following is the python code snippet I am giving which does following job

1. Initiates TCP server over localhost with port numbe 5005.

2. Initiate Google earth APIs default variable value.

3. Reads Gesture data which comes over TCP port 5005 on localhost.

4. Parse that data in order to take specific action to google earth movement according to gesture.

5. Keep doing step 3 and 4 continuously.

1. import socket     
2. import win32com.client, time  
3. googleEarth =  win32com.client.Dispatch("GoogleEarth.ApplicationGE")  
4. while not googleEarth.IsInitialized():  
5.     time.sleep(0.5)  
6.     print "waiting for Google Earth to initialize"  
7.     
8. latitude=41.487942                            # Latitude in degrees. Between -90 and 90.  
9. longitude=0#-81.6865                            # Longitude in degrees. Between -180 and 180. to rotate the earth  
10. altitude=1000    
11. # in meters  
12. tilt=0  # looking to the horizon=90, looking to the center of Earth=0  
13. azimuth=370  # looking North=0, East=90, South=180, West=270  
14. 
    
15. speed=0.5    
16.  # speed transition. must be >= 0, above 5.0 the transition is instantaneous  
17. 
    
18. range=0              # If not=0 camera will move backward from "range meters along the camera axis  
19. altMode=1  #Altitude mode that defines altitude reference origin (1=above ground, 2=absolute)  
20. focusDistance=10000000  #Zoom Value can be changed  
21. 
    
22. TCP_IP = '127.0.0.1'  
23. TCP_PORT = 5005  
24. BUFFER_SIZE = 20  # Normally 1024, but we want fast response  
25. s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  
26. s.bind((TCP_IP, TCP_PORT))  
27. s.listen(1)  
28. conn, addr = s.accept()  
29. print 'Connection address:', addr  
30. while 1:  
31. data = conn.recv(BUFFER_SIZE)  
32.  
33. if not data: break  
34. 
    
35. #print "received data:", data  
36.   
37. #focusDistance=0  
38.  
39. #data=0  
40.  
41. if data == "DRIGHT":  
42.  
43. print "Right Direction longitude = ", longitude  
44. 
    
45. if longitude <= 180:  
46. 
    
47. longitude = longitude-10  
48. 
    
49. if longitude == -180:  
50. longitude = 180  
51. 
    
52. elif data == "DLEFT":  
53. 
    
54. if longitude >= -180:  
55.  
56. longitude = longitude+10  
57.  
58. if longitude == 180:  
59.  
60. longitude = -180  
61.   
62. print "Left Direction longitude = ", longitude  
63.  
64. elif data == "DUP":  
65.  if tilt > 10:  
66.   
67. tilt = tilt - 10   
68.   
69. if tilt == 0:  
70.   
71. tilt = 90  
72.  
73. print "Up Direction tilt = ", tilt  
74.  
75. elif data == "DDOWN":  
76.   
77. if tilt < 90:  
78. 
    
79. tilt = tilt + 10   
80.  
81. if tilt == 90:  
82. tilt = 0  
83. print "Down Direction tilt = ", tilt  
84.  
85. elif data[0] == 'Z':  
86. 
    
87. print "Z data Direction"  
88. 
    
89. data = data[1:6]  
90. focusDistance = (int(data))  
91. print 'focus_distance z = ', focusDistance  
92.     #conn.send(data)  
93. #conn.close()  
94. googleEarth.SetCameraParams (latitude, longitude, altitude, altMode, focusDistance, tilt, azimuth, speed)  
95.  
96. cam=googleEarth.GetCamera(True)       # returns a ICameraInfoGE  object 

   This project I did as part of RoadTest from Element14. You can find project source here download.

I hope you enjoyed this project. Let me know if any querry/ comments.

Regards,

Shrenik.