From the beginning I wanted to code the extension's modules in a way that would allow me to take them away from the Jetpack extension. But that was it, just a wish, I didn't write each single line thinking on portability. Hopefully things ended up better than I expected.
When it was time to take the code out, I just copied the events dispatcher and files parser modules, performed 2 or 3 global replaces because of conflicting names and... job done! You can see the results in here with Firefox Minefield.

I think this is a great feature, not only meaning that you can take your code out from a Jetpack extension, but also the possibility of putting in a Jetpack previously coded "for the web" JavaScript libraries!

If you know me a little,  you would guess that I couldn't leave that possibility without a try... And you would be quite right So I took some little bit of code that Marcio Galli showed me on twitter, a QR-Code maker done entirely in Canvas+Javascript (see here), and decided to stuff it in a Jetpack.
Again, a very effort-less thing with great results. I ported that code to Jetpack in a blink, and created a qr-code library for jetpack, and in 5 more minutes, I had this Selection-To-QRcode extension running.

So, in brief, these are some exciting new possibilities to expand and remix the web and your experience with it!

Notice
If you don't want to read the whole stuff and go right to the testing and trying fun stuff, just jump to the "Links! Links! Links!" section on the bottom

The new Recording features

Something that was very very needed was a more natural way to record the input files. Doing so programatically tends to be more dificoult and troublesome than moving your mouse around, so something had to be done. After some thinking and discussion with Felipe, I came up with a more comfortable way to record the files. I'll describe it briefly here, but you can alsgo skip it and see a demo after the break .

Basically the UI consists of a representation of the fingers on the screen. Moving them with the mouse represents the movement of the fingers on the multitouch surface.

First you click the Start Recording Button and choose where to save the final file. Once the file is specified, the number of fingers that will be on-screen has to be chosen. Pushing the Finger Button will add a finger to the screen, but beware because there's no way to remove a finger for now.

Doing the recording is pretty straight forward. All fingers are recorded in cycles, one at a time starting from the first one added and then going sequentially with the rest. The cycle starts again when the last finger is done.
Before starting to record the actual moves, you have to set the starting position of each finger. Doing so is pretty easy, drag the black-bordered (active) finger and then click on the Accept Button, then the next finger will be activated and so on. Once all the fingers are set, the recordings can start.

You will notice when the fingers are being recorded because a trace of the finger movement will appear on screen. If a mistake happens, the finger movement can be undone, and this can go as far as the first finger of the cycle. The recording cycles can be repeated until you decide to hit the save button.

When the cycle is finished (last finger moved) it can be played back, triggering the multitouch events on the page, with the Play button. This is in case some actions are needed in order to continue recording. Keep in mind that this is a replay of just the previous recorded cycle: you cannot do it twice and cycles don't accumulate.

When you decide the recording is done and click on the save button, the file will be closed and saved. Also, every element added for the recording will be removed from the screen.

Multitouch Landed on trunk!

For those who are wondering what does this means: Multitouch support is now on Minefield and will be included on the Firefox 4 Beta 3 that is going to be released soon. This also means that the custom build I had to upload on my previous post is no longer needed to try the extension

Links! Links! Links!

You can download the multitouch simulator 0.4 from AMO here. Remember to download the xpi and drag n' drop it on firefox, that will do for installation. In Firefox 3.6 you'll have to restart to see the efects, but in Firefox 4 betas the extension will load without restart If you find the bottom bar annoying, it can be hidden pressing Ctrl+Shift+U (or in OSX Cmnd+Shif+U).

If you just want to see this in action on a live demo, you can see the JavaScript library version here with the Firefox 4 beta 3 or greater. You don't need to install any addon, is a stand-alone version of the code running on a demo page

Also, here you have a list of available demos to try the multitouch support coded by Felipe Gomes:

• Image Crop
  Draw a rectangle to potentially "crop" an image with a pinch gesture
• Image Resize
  Shrink and Enlarge images with pinch gestures. With two simultaneous pinches you can resize two different images
• Touch Canvas
  Simple canvas that tracks your fingers
• Tracking Divs
  Two divs keep tracking of the fingers you have on the screen
• Fluids
  Fluid simulation following your fingers!

Protocol optimized

After some tinkering with the old protocol, Felipe and I thought it was over sized and a little repetitive to work with, so we put it on a diet. Now it's more compact and kind of easier to work with, I'll show you/

This would be a sample file written with the new version:

{
  "frames":[
     {
       "positions":[
            {"id":"1","x":300,"y":300}
        ],
       "events":[
           {"id":"1","type":"MozTouchDown"}
       ]
     },
     {
       "active_duration": 5
     },
     {
       "positions":[
           {"id":"1","x":400,"y":400}
       ],
       "events":[
          {"id":"1","type":"MozTouchRelease"}
       ]
     }
  ],
  "fps": 10
}

The new rules are even simpler:

• frames
  A collection of Frame-Objects
• Frame-Object
  An object that has two properties to define the status of each frame:
  • positions
    A collection of objects containing the finger ID that the position belongs and x and y position: {ID, X_POS, Y_POS}
  • events
    Collection of objects containing finger ID and Event Type. Event Types are to be designed, but inital ones are the same as the implementation:
    • MozTouchDown
    • MozTouchUp
    • MozTouchRelease
    • MozTouchMove
• Frame-Object Types
  There are three Frame-Object types now:
  • Snapshot
    This is the type in where you define positions and events
  • Spacer Active
    A Frame-Object is of this kind when it's only property is "active-duration". This indicates that between two snapshots objects an animation should be done, and the duration is the value of "activ-duration"
  • Spacer Inactive
    This has the same structure as the active one, but instead the property is "idle-duration". This indicates that no transition should be calculated between two snapshots, it's a waiting time.
• fps
  The framerate that will be used to play the file

Changes:

• Frame-objects no longer have a fingers collection. This is now deduced from the positions IDs
• Events now are the same as the ones implemented in firefox. This is to simplify naming, and probably they'll soon change to something more "human" and less "techy".
• The Frame-Objects have 3 subtypes now: Snapshot, Spacer Active, Spacer Idle.

Demo Available

As I promised last time, the code is ready for some demoing!

As the multitouch part is not yet available on firefox latest builds you'll have to use a special build of firefox to try this stuff. Here you'll find a compiled version for your system:

• Mac OSX (tested on 10.6, should also work on 10.5)
• Windows
• Linux (not here yet, still have to compile this)

Now that you have the correct build to test the extension is time to install it. I would strongly recommend you to create a new profile, it's a good practice when you're trying new stuff that may explode .

You can grab here the extension compiled as an XPI, and here some good sutff: you won't need to restart firefox to try it out! As a side-effect of coding the extension with Jetpack, and having to use the latest firefox build for the multitouch part, we end up having the best setup ever: restartless addons enabled! To install the addon, just download it and drag-n-drop it on firefox.

But we need some demos to try the addon, so here you have a list of available demos to try multitouch coded by Felipe Gomes:

• Image Crop
  Draw a rectangle to potentially "crop" an image with a pinch gesture
• Image Resize
  Shrink and Enlarge images with pinch gestures. With two simultaneous pinches you can resize two different images
• Marcio Drawing
  A demo that lets you draw with your fingers. Touching a color in the pallete, changes the drawing color.
• Pong (not much usefull in for this addon)
  A simple implementation of the classic Pong game.
• Touch Canvas
  Simple canvas that tracks your fingers
• Tracking Divs
  Two divs keep tracking of the fingers you have on the screen
• Fluids
  Fluid simulation following your fingers!

And here are some sample input files:

• Sample 1: A simple pinch gesture.
• Sample 2: Two simultaneous simple pinch gestures.
• Sample 3: Cross-drawing (this is designed for the Marcio Drawing Demo)

Using the simulator is pretty simple, it just has three buttons:

The first one lets you load an input file (as the samples I'm providing) to be processed. The second one opens a kind of "console" so you can write a input file on the fly and then run it with the third button, as you can see here:

Where to go now

The next steps I think will be the harder ones. What is pending to be added is a way to simulate multitouch input with just one pointer. That means, how to make the extension itself write the input files instead of doing it by hand. I still have to come up with a UI for that, and any suggestion/idea is welcome. If you think you can collaborate on that stuff, don't hesitate to contact me on twitter or leaving a comment.

Feedback

I would really appreciate if some (or all!) of you try this extension and reports issues/enhancements (you can do it here), that would be priceless. Of course any thing you want to ask you can reach me in #jetpack on irc.mozilla.org,  I go by the name of "peregrino", or by leaving a comment here.

As some of you may know by my twitter time-line, I've been accepted into the Google Summer of Code 2010 (GSoC) to develop a project for the Mozilla Foundation. The GSoC is a program in which Google funds students from all around the world to work on a project for an Open Source Organization in their Summer free time. In my case -and the same is for the ones who live in the Southern Hemisphere- is the Winter of Code, but is fun all the same!

My project is a Multitouch Simulation Framework for Firefox. As the name tells, the idea is to make an extension that lets you simulate multitouch input form firefox to test your web aplication. This will be cool, as it will let developers do multitouch interactions without actually having to own a multitouch device!

Initial Considerations

The idea is that you can write (and maybe later, record) a sequence of multitouch steps like:

Drag finger1 from (100x, 100y) to (150x, 100y)
Double-tap (150x, 100y)
Pinch Zoom 30px centered in (75x, 30y)

so one of the first issues I faced with this project was actually writing a protocol that let you do that. The first idea that came to my head was that the TUIO protocol would be a wonderful source of inspiration, with loads of useful info... Now I know it was a really naive thought.

The TUIO (Table-Top User Interfaces Objects) project has a protocol for sending multitouch events through a network, something very similar to what I needed. So I read all the protocol versions (1.0, 1.1 and 2.0) to understand it fully, and just came to the conclusion that the TUIO protocol was to me like a bazooka to kill a mosquito. It defines things in a much more deeper detail that the one I need, is a lot more generic -I just need to track fingers, TUIO is able to track even custom shapes in 3D- and has nothing about timings. So... no soup for you, come back one year!

So if I couldn't get anything from TUIO I had to write it all by myself, those were good and bad news. Good because I could make the spec so that the final files were written in JSON -something way more JavaScript friendly!- and bad because I couldn't take advantage from TUIO.

After a while thinking, I came up with what I think can be called the 0.1 version of my protocol.  My idea is to do various iterations of the protocol while I implement the extension, that way I can tweak what I think is being troublesome to use, or implement suggestions. The most complicated thing I faced with my protocol is the timings and frame-rate.

Timings and Frame-Rate

The aim of this project is to replay actions, so the protocol should be focused on replaying more than mere message passing (as TUIO is) so timings are important. I don't know if it's the best decision, but I decided to put a global frame-rate property and to writean object defining each frame.

Taking something from TUIO the addition and deletion of fingers is not explicitly told, is deduced from the difference with the previous frame's active fingers. Having "frame-objects' can be a bit redundant sometimes, but is the most simple approach to a protocol to describe things for later replay: being explicit. Is like MPEG-1 and WebM, the former is heavier but simpler to read than the later optimized format.

Having a constant frame-rate makes easier to implement in the extension, so thas was the way to go. Although in some points it might make sense being able to temporarily reduce the framerate, or indicate that some frame-object spans for more than one frame, those are power-features that can be later considered.

Velocity

Other thing I had to consider was the velocity and acceleration of the finger movements. You may notice that when you move your mouse, or your finger on the trackpad, you don't make constant velocity movents. Usually you accelerate and then slow down to be more precise, doing the complete movement with constant velocity would seem like it was done by a robot... But this is hard to describe in a protocol, and when acceleration comes in timings are harder to implement, so the decision was to make all the movements with constant velocity first. In a later iteration accelerations could be implemented.

The Protocol

This would be a sample file written with this protocol:

{
    "frames":[
    {
        "fingers":['1','2','3'],
        "positions":[
            {'1',0,0},
            {'2',10,10},
            {'3',20,20}
        ],
        "events":[
            {'1', 'DOUBLE_TAP'}
        ]
    }
    ],
    "fps":20
}

The rules are simple:

• frames
  A collection of Frame-Objects
• Frame-Object
  An object that has three properties to define the status of each frame:
  • fingers
    This is a collection of IDs to identify each finger. The IDs are strings
  • positions
    A collection of objects containing the finger ID that the position belongs and x and y position: {ID, X_POS, Y_POS}
  • events
    Collection of objects containing finger ID and Event Type. Event Types are to be designed, but inital ones are: TAP, DOUBLE_TAP, TAP_DOWN, TAP_UP
• fps
  The framerate that will be used to play the file

Possible Issues

Some issues may arise from the current protocol version, this are the ones I could think of

• If a user wants to have multiple fingers fixed and just a few moving, he has to copy several times the same info for the still fingers. This could be reduced, maybe
• To code pauses between movements, the final frame-object has to be copied fps x seconds times...
• Movements more complicated than just moving between two points in a straight line can be somewhat complicated to code manually...

Well, that's all that I have for now, next post will hopefully have some piece of code to actually try the extension and a discussion about my progress with coding it in Jetpack.

Please, share your thoughts in the comments!

Hernán

If you ever wondered how all those Firefox extensions are made, you probably discovered (maybe with some disappointment) that they're not that easy to code. You have to learn a language called XUL for the UI, assimilate a lot of new terms like Overlays, XPI, XPcom... But hey, I'm not telling this stuff is impossible to grasp (in fact, there's a lot of documentation and tutorials available), just that you have to learn something new.
So Mozilla thought about that for a while, and came up with a pretty cool idea: you are most likely to allready know JavaScript, CSS and HTML. Why you should have to learn something new? And thus born Jetpack, a new way to code Add-Ons in just a single file and entirely coded in JavaScript, CSS and HTML.

Firebug and Jetpack

And what has Jetpack to do with Firebug? Well, pretty much. As stated on Jetpack's site, "not wanting to reinvent the wheel" they used Firebug for debugging and logging purposes. So for now everything's great: a simple way to develop add-ons, and a nice debugger... what else could I ask for?
Maybe for them to work together out of the box?

As some people has noticed, the latest versions of Firebug and Jetpack won't work together right after installing them. This is because the current version of Jetpack (0.6.2) uses an old API call to communicate with Firebug that changed in version 1.4.3. It seems that all the Firebug interaction was developed using Firebug 1.4.2, so they never noticed the change in the API. But don't panic! There is a way to fix this, and is as simple as changing a single word on a single file.

Making them play nice together again

Important Notice:
As I write this article the bug opened for this issue is already Verified and Fixed, so that means that the fix will be on the next Jetpack release. Meanwhile, you'll have to do the following patch by hand to use Jetpack with Firebug 1.4.3 or later.

First of all, you have to locate your Firefox profile. This depends on your system, but this are some of the most common locations:

• Linux:
  ~/.mozilla/firefox/<profile folder>

• Mac OS X:
  ~/Library/Mozilla/Firefox/Profiles/<profile folder>
  OR
  ~/Library/Application Support/Firefox/Profiles/<profile folder>

• Windows XP:
  C:\Documents and Settings\<Windows login/user name>\Application Data\Mozilla\Firefox\Profiles\<profile folder>

• Windows 7:
  C:\Users\<Windows login/user name>\AppData\Roaming\Mozilla\Firefox\Profiles\<profile folder>

If you are using firefox 3.6, you can go to the address "about:support" and there's a link to your profile folder.

After finding your profile folder, you'll have to go to the following folder inside it.

extensions/jetpack@labs.mozilla.com/content/js/xul

and edit this file:

firebug-extension.js

Changing the line 19 from:

Firebug.URLSelector.watchBrowser(browser);

To:

Firebug.Activation.watchBrowser(browser);

Save that file, restart Firefox, and voilà, Firebug and Jetpack are working together now!

Testing if Firebug is now working

If you want to test if Firebug is working as expected, then paste the following into the develop area of Jetpack:

jetpack.future.import("menu");

jetpack.menu.context.page.add({
    label: "LogThis!",
    command: function(){
        console.log("Firebug is Working OK");
    }
});

Now hit the "Try out this code" button, enable Firebug and refresh the page. If everything went right, you should have a new entry on the context menu of the webpage, called "LogThis!". If you click on that entry and the text "Firebug is Working OK" appears on the console, you have it working.

In this version the UI has changed a little. Now multiple filtering rules can be set, although the system for setting them is a little archaic.  As can be seen in this screenshot:

Multiple Rules alowed

The activation mechanism stays the same, and so the quick search in the net tab. But now there are two buttons, to remove the applied filters (in case you want to view all the results) and one to apply the stored rules.

New buttons in the net panel

You can grab this release from the releases folder. Also keep in mind that I'm trying to set up an upgrade channel in this webpage, so you can install once and then get the updates automatically.

The aim of this extension is to let you add custom filters to the requests listed in the Net Panel. Right now this is the first... nightly. It isn't even an alpha, I just made it work, and then packed and uploaded it. So don't expect it to be a lot friendly, it just works.

To use this extension you should have Firebug 1.5.0a21, 1.5X.0a21 or the latest SVN revision, since it depends on a listener added in a21. Just go to releases and install the xpi.

How to use it

• Install the extension and make sure the dependencies are met
• Go to the Net Panel
• Press "Advanced Filters", a prompt should appear
• Enter the desired RegExp pattern against you want the request's URL to be tested.
• Reload the page, now as long as the "Advanced Filters" button is selected, your requests will be filtered. To disable the filtering, press again in "Advanced Filters".
• You can change the applied filter pattern by disabling and then enabling the advanced filtering. Each time you enable the filtering, a prompt will show up in wich you can enter the new pattern. Then it'll be applied to the monitored requests.