Newsfeed Module Update

This week, a lot of the work I'm doing is of the fixing/testing/documenting variety.

There are two new features that change the functionality of the newsfeed in notable ways.

One is PubSubHubbub support. It's working, and was surprisingly easy to get running.

It works with a hook in the newsfeed update process. Remember how the feed update process includes a sender (the updated entity) and a list of receivers? Because it's the feed of the receivers thats is being updated, the pubsub ping is made for each of these receivers.

First we get the feed URL for the receiver:

# resolve the URL of the entity's ATOM feed
entity_feed_url = news_feed.getFeedUrl()

Then a POST request is sent to a central hub that includes the feed_url:

headers = {'content-type': 'application/x-www-form-urlencoded'}
post_params = {
'hub.mode': 'publish',
'hub.url': entity_feed_url,
}
payload = urllib.urlencode(post_params)
try:
# can these be sent in a batch?
response = urlfetch.fetch(HUB_URL, method='POST', payload=payload)
except urlfetch.Error:
logging.exception('Failed to deliver publishing message to %s', HUB_URL)

Wow, that was easy, wasn't it?


The second feature is the subscribe-by-star pattern, which already appears in plenty of other Google products and should be useful for people who don't like/understand RSS feeds, since subscribing by the star signs you up for email updates.



Finally, I've posted questions that would be useful to ask Melange users in an optional survey.


The questions are on a survey on my staging server: http://jamstage.appspot.com/survey/take/program/google/gsoc2009/updatepreferences

GSoC09: Statistics module 8th weekly report: Refactoring, fixes... and jQuery 1.3!

So... the penultimate week of GSoC came and passed, being a great moment for maintenance tasks like refactoring, bug fixing and coding small features left behind during project track. Beside that, following a Madhusudan's proposal, we've finally upgraded to jQuery 1.3 (commit 5723f329d0) and to JQueryUI 1.7 (commit 4e2789b8e8)!

Here's the story: following Pawel's suggestions, I've made little changes to dashboard's code to transclude itself when encountering DIVs with class "melangeDashboard" instead of MELANGE tags with attribute type "statistic_dashboard". Furthermore, I removed the fancy stack menu (that was too different from the rest of the application style) and put only a "new chart" link in the upper side of the page. That should be a temporary solution, as more features will come into place we'll integrate another menu.

A couple of bug fixes:

• Before this week, if you changed the visualization type right after widget creation, it wouldn't be saved on the backend. It would have happened only if you changed it after one dashboard reloading. Now it's fixed.
• Fixed the bug that Daniel Hans mention in his post: now, when switching the virtual statistic, the visualization is properly refreshed.

I've focused also on some refactoring. First of all I've finally changed what Daniel Hans and Lennie suggested me when I've sent patches to the mailing list (mainly on Python code) :)

The main work, however, has been on the Javascript side, mainly to avoid code duplication (for example code for updating a widget in the backend was present three times) and to separate dashboard code and widget code better. For example, before the refactoring, code to move and delete widgets in the dashboard was written in the widget code, while these are functions more related to dashboard logic.

But while porting the logic to move widgets inside the dashboard has been an easy task, as the dashboard already contained all the code to make the widgets move (using jQuery sortable plugin and binding the function to update the backend to the "stop" event), widget deletion has not been so easy to accomplish. Why? Because the deletion button is properly created in the widget class, and the function to delete the widget in the backend is in the dashboard, so in another scope.

That could be easily workarounded if I made the deletion code a "privileged" function of the dashboard... but, to me, this is not correct design. Deleting a widget (and so the chart entity in the backend) is a crucial task, so the function that does this should not be callable from other code. The solution? Publish/Subscribe pattern! While this is a built in behavior in Dojo, this is not the case in jQuery namespaced events (they can be used as topics), but then it happens that using a "click.widgetdeletion" namespace, when it bubbles up through the DOM, it loses the "widgetdeletion" part (so appearing as a common "click" event). Furthermore, to have a clean code, I would have liked to pass a reference to the widget instance with the event, but no data could be passed along with the event in jQuery 1.2.6.

So I've made a quite ugly hack, using common event delegation, detecting the target object and calling the deletion function if it has the "remove" class (which happens to be the class of the remove button... but what if we want to call that class with some other name in the future?), and using jQuery traversing to detect the ID of the widget from the real id of the parent div DOM node (which has a pattern like widget-IDNUMBER, that is extracted using a simple regex) and then using it to retrieve the widget instance from the widget list array that is present as a variable in the dashboard. Something like:

var widget_id = jQuery((jQuery(event.originalTarget).parents(".widget"))[0]).attr("id");
var widget_instance = widgets_list[/^widget-(\d*)$/.exec(widget_id)[1]];

Pretty ugly, isn't it?

But then, after jQuery 1.3, and using the new Event object, all is smoother now:

This is what happens after confirming deletion:

jQuery(this).trigger({type: "widgetdeletion",widget: _self}).remove();

You can notice just an event triggering, with a specific type (as a publish/subscribe topic) and passing the widget instance along with the event. In the dashboard, catching the event is as easy as:

dashboard.bind("widgetdeletion",function (event) {updateOnDelete(event.widget);});

That is, binding the "widgetdeletion" event and calling the proper function, passing it the widget instance reference. Simpler, cleaner and not prone to be messed if something in the HTML code changes (as the classes assigned to a widget or to its remove button).

Some enhancements made during the week has been the live editing of widget titles and support for the new get_available_statistics function (made by Daniel Hans), that has proper rights checking and returns all statistics (GSoC/GHOP, 2008/2009..and so on) with their proper scope path and link ids.

For refactoring purposes again, I've tried to look for a suitable browser side templating engine. In Dojo, there's the faboulous dojox.dtl Django porting. But what about jQuery? There's a bunch of technologies out there (made as jQuery plugin or standalone), as:

Standalone
Google JSTemplate
Ajax Pages
Pure (this can be used also along with jQuery)
JBST
jQFRagment
JSHTML
twoBirds
JsonFX.NET

jQuery plugins/dependent
jTemplate
chain.js
Nano
John Resig's micro templating
Some other guidelines plus fixes to John Resig's micro templating
Custom made 1, custom made 2

It's really not easy to deal with all these different options, and to explore major advantages/drawbacks. I'll try to do my best during this week, because it's really a must-have for us, to avoid ugly mixing between JavaScript and HTML code.

GSoC: Statistic Module update

Here is my next status update of the eleventh week of Google Summer of Code 2009:

First of all I finished working on displaying multiple number of visualizations inside one widget. Let us consider the standard 'Students Per Degree' example. Now you create a new widget and change visualization between:
'Students Per Degree (all)',



'Students Per Degree (with projects)':



'Students Per Degree (without projects)':



'Students Per Degree (cumulative)':




The last one represents all three types in a single visualization.

This new option works pretty well on statistic 'visualize' page (you can get there by clicking 'visualize' link on the page which lists all statistics).
It does not perfectly work on the actual statistic dashboard. There is a bug: when you choose a new statistic to be displayed, the visualization does not change. Anyway, when you change a visualization type, a new visualization for the new statistic is displayed. To be fair, I have not found a solution for this issue yet. I tried to debug that JavaScript scripts for a couple of hours and even discovered what was wrong, but could not fix that. I asked Mario for some help and he said he would try to take care of that.

The next thing I did was preparing a new series of patches based on Sverre's suggestions. The patches that were already fine, I already pushed to the main repository.

Then I started to work on access issues. Each statistic should have information about who is allowed to see it. Some statistics should be visible only to program administrators, while some others can be seen also by organization admins. I added a new 'read_access' parameter to the statistic model. Currently frontend may retrieve a complete list of statistics available for the current user by calling /statistic/get_available_statistics request.

GSoC 2009: GHOP - Eleventh Week Status Update!

Hello everyone,
A week of frustrations come to an end. The goal for last week was to get the Tag system in place for GHOP. The goal included giving the Program Admin to Add/Delete and Edit Tags. And you might already know that we are using a modified version of Taggable-mixin for tagging in Melange.

Although Adding and Deleting tags looked very trivial to me, edit was not so trivial. The first reason being, the key name for the tag stored in the Datastore used the tag name. So every time you edit a tag a new key needs to be created, but all other properties must be retained, so tag must be copied to the new entity. Also over and above this, on the UI side I was not even sure how to allow Program Admin to Add/Delete and Edit a tag in the same interface. After a lot of struggling and battles within and frustrations, I thought for a GHOP Program number of types of Program defined tags are fixed, one is for Task Difficulty level and other is for Task Type. So I created 2 pages, one for each where one could do all the 3 actions(Add/Edit/Delete).

For accomplishing this per Task page, I have used a JQuery plugin called, in-place-edit. This allows us to edit a tag in-place. So one can click on Add button to add new tags, and make the contents of the tag empty to delete it or edit the tag name. The editing of the tag name automatically copies the old Tag entity to the newly created Tag entity. And each creation, edit and delete happens via AJAXy calls, so one need have to save the changes after editing the page completely. Program Admin can just exit the page.

Once I solved this problem, a new problem arose. Usually tags have to be sorted in some order, say for example for difficulty level tags, it must be sorted from easiest to hardest. But tags need not be created in the same order. For example, Program Admin might initially think, only Easy, Medium and Hard are sufficient, but might later realize a Trivial difficulty level is required. So this needs to be ordered in the order enforced by the Program Admin. Once again I had to goto JQuery for help. I used JQuery UI's sortable plugin already in Melange code base to drag and drop sort the list of tags. So ordering is implemented as well.

After this, the interface to Add Tags to create/edit tasks had to be added. For a difficulty level tag, it is a normal drop down. But a Task can be of multiple types, like Documentation and Translation. So for this I have used an HTML Multiselect box. Also there is a text box for adding arbitrary tags. At the moment, these two types of tags are merged and stored as the entities of same TaskTypeTag model in the datastore with mandatory property=True for Task type tags and False for arbitrary tags. But Lennie wanted this implementation to be changed and separate them into 2 models. This requires a good amount of re-work. I will do it this week.

After all these, I started working on Search for tasks and happily created multi selects for Organizations, Task status, Difficulty Level, Task Type and started generating queries for them. It was then Lennie, made me realize that there can be only 30 subqueries per query. So this kind of search mechanism won't work. I have currently stalled my work on it. I am thinking of work arounds to this. Once we get a fair idea of what to do, I will resume the work on Task search.

Amidst all these work, I have been fixing bugs, making some implementation changes like Task state transitions etc. This week, I will be mostly working on writing a starter manual, fixing bugs and student data model conversion to include whether student belongs to High School or University.

Last but not the least, all my work till now is available on my demo instance at, http://melange-madhusudancs.appspot.com