First steps into Google Summer of Code
Well just to let you know, facing a blank page is hard!
As some of you may know by my twitter time-line, I've been accepted into the Google Summer of Code 2010 (GSoC) to develop a project for the Mozilla Foundation. The GSoC is a program in which Google funds students from all around the world to work on a project for an Open Source Organization in their Summer free time. In my case -and the same is for the ones who live in the Southern Hemisphere- is the Winter of Code, but is fun all the same!
My project is a Multitouch Simulation Framework for Firefox. As the name tells, the idea is to make an extension that lets you simulate multitouch input form firefox to test your web aplication. This will be cool, as it will let developers do multitouch interactions without actually having to own a multitouch device!
Initial Considerations
The idea is that you can write (and maybe later, record) a sequence of multitouch steps like:
Drag finger1 from (100x, 100y) to (150x, 100y)
Double-tap (150x, 100y)
Pinch Zoom 30px centered in (75x, 30y)
so one of the first issues I faced with this project was actually writing a protocol that let you do that. The first idea that came to my head was that the TUIO protocol would be a wonderful source of inspiration, with loads of useful info... Now I know it was a really naive thought.
The TUIO (Table-Top User Interfaces Objects) project has a protocol for sending multitouch events through a network, something very similar to what I needed. So I read all the protocol versions (1.0, 1.1 and 2.0) to understand it fully, and just came to the conclusion that the TUIO protocol was to me like a bazooka to kill a mosquito. It defines things in a much more deeper detail that the one I need, is a lot more generic -I just need to track fingers, TUIO is able to track even custom shapes in 3D- and has nothing about timings. So... no soup for you, come back one year!
So if I couldn't get anything from TUIO I had to write it all by myself, those were good and bad news. Good because I could make the spec so that the final files were written in JSON -something way more JavaScript friendly!- and bad because I couldn't take advantage from TUIO.
After a while thinking, I came up with what I think can be called the 0.1 version of my protocol. My idea is to do various iterations of the protocol while I implement the extension, that way I can tweak what I think is being troublesome to use, or implement suggestions. The most complicated thing I faced with my protocol is the timings and frame-rate.
Timings and Frame-Rate
The aim of this project is to replay actions, so the protocol should be focused on replaying more than mere message passing (as TUIO is) so timings are important. I don't know if it's the best decision, but I decided to put a global frame-rate property and to writean object defining each frame.
Taking something from TUIO the addition and deletion of fingers is not explicitly told, is deduced from the difference with the previous frame's active fingers. Having "frame-objects' can be a bit redundant sometimes, but is the most simple approach to a protocol to describe things for later replay: being explicit. Is like MPEG-1 and WebM, the former is heavier but simpler to read than the later optimized format.
Having a constant frame-rate makes easier to implement in the extension, so thas was the way to go. Although in some points it might make sense being able to temporarily reduce the framerate, or indicate that some frame-object spans for more than one frame, those are power-features that can be later considered.
Velocity
Other thing I had to consider was the velocity and acceleration of the finger movements. You may notice that when you move your mouse, or your finger on the trackpad, you don't make constant velocity movents. Usually you accelerate and then slow down to be more precise, doing the complete movement with constant velocity would seem like it was done by a robot... But this is hard to describe in a protocol, and when acceleration comes in timings are harder to implement, so the decision was to make all the movements with constant velocity first. In a later iteration accelerations could be implemented.
The Protocol
This would be a sample file written with this protocol:
{
"frames":[
{
"fingers":['1','2','3'],
"positions":[
{'1',0,0},
{'2',10,10},
{'3',20,20}
],
"events":[
{'1', 'DOUBLE_TAP'}
]
}
],
"fps":20
}
The rules are simple:
- frames
A collection of Frame-Objects - Frame-Object
An object that has three properties to define the status of each frame:- fingers
This is a collection of IDs to identify each finger. The IDs are strings - positions
A collection of objects containing the finger ID that the position belongs and x and y position: {ID, X_POS, Y_POS} - events
Collection of objects containing finger ID and Event Type. Event Types are to be designed, but inital ones are: TAP, DOUBLE_TAP, TAP_DOWN, TAP_UP
- fingers
- fps
The framerate that will be used to play the file
Possible Issues
Some issues may arise from the current protocol version, this are the ones I could think of
- If a user wants to have multiple fingers fixed and just a few moving, he has to copy several times the same info for the still fingers. This could be reduced, maybe
- To code pauses between movements, the final frame-object has to be copied fps x seconds times...
- Movements more complicated than just moving between two points in a straight line can be somewhat complicated to code manually...
Well, that's all that I have for now, next post will hopefully have some piece of code to actually try the extension and a discussion about my progress with coding it in Jetpack.
Please, share your thoughts in the comments!
Hernán