GSoC09: Statistics module final weekly update: Javascript fireworks and Code Swarm! :)

It's not a secret that I truly love Javascript language :)

So, as the last week of GSoC went by, I wanted to finish some of the work about Melange Javascript refactoring that I begun on May/June, just to give a real starting point and solid bases for a renewed Javascript layer for Melange.

So, first of all, I've learnt some advanced bash programming, thanks to the awesome Mendel's Cooper online book, "Advanced Bash-Scripting Guide" (PDF version). I found it very well written and really useful. Having learnt at least enough to deal with variables, loops, conditionals, exit statuses and functions, I've made a script to run shrinksafe over all our Javascript files during build and before deploying. In this way we can shrink our JS files to a half, and so it'll download faster once live.


Then, as we're using pylint to check proper style of Python files, I've made a bash script file to run JSLint over all our JS files, and also I've coded a big-mega patch to make all our JS files JSLint compliant! And here the final result..!


Furthermore, I've written JSDoc style comments in melange.js and melange.graph.js files, to let the community have an example on how to document JS files properly (it's very difficult to communicate semanthic of JS files to JSDoc Toolkit!). Running JSDoc Toolkit over those two files made documentation for private (developer) and public (only API) purposes. Following a sample excerpt of the final result.


Apart from that, I've also achieved this:


Seems not so exciting, huh? Well, what this alert box, behind the scenes, means is that Melange now has a facility to expose public JS API more or less like Google does (with Maps API and so on). Using this facility, I've made possible an up-to-date live embedding of statistics data to any web page, by inserting only a a script tag (and a div where the visualization needs to be shown).
There are two kind of exporting:

• LIVE: this connects the web page to a widget in the statistics dashboard. This means that if you change the visualization to a pie chart or the "virtual statistic" to another one, the visualization in your web page will be changed as well.
• FREEZED: this exports the current visualization in the widget, and it will never change.

So, more things can happen in the future with this new opportunity (manage a wiki of visualizations, for example), but also embedding statistics into document pages in Melange (this is very straightforward to achieve now). And, now, exposing JS APIs, we can think of many exciting new features (what about something to be embedded in the organizations' real home pages?). Along with JS refactoring and seed_db frontend these could be the next steps for my post-GSoC participation in Melange community.

During the beginning of this week I've also made two simple code swarm videos (coloured by programming language).
This one is for our main branch Melange repository:

And this one is for Melange statistics branch repository (which is merged with main repository).

Before ending the post, I would like to thank everyone involved in Melange community, my mentor Pawel Solyga, our project lead Lennard De Rijk, Sverre Rabbelier, my "in-place guide" James Crook (which I met several times here in Dublin, thank you again for everything!) the other GSoC students (Daniel Hans, with whom the statistics module has been possible, Madhusudan C.S. and James Alexander Levy), not forgetting the great Daniel Diniz.
And obviously Google for making GSoC possible, and Leslie Hawthorn and Ellen Ko for their hard work on it.

Also, last but not least, I would like to give a huge hug to OpenStreetMap community (and my former-year mentor Frederik Ramm), with which I've shared GSoC 2008, and with which I've had the opportunity to learn Javascript language in depth. Surely, as I've done during this year, I'm going to stick around and contribute to OSM community as well.

Twelfth Week NewsFeed Update

As the GSOC program wraps up, I'm finishing with refactoring and testing, and have added a couple new important features in the last week:

* Subscribe-By-Star. The subscribe-by-star pattern allows for an e-mail subscription for a particular entity to be added or removed by toggling a star on the news feed box on or off. Aside from the popup help message explaining how the feature work, the subscribe-by-star is working on http://jamstage.appspot.com.

The ajax itself is simple. After using jQuery.ajax() to make the call, I first process the POST data so that it is suitable for logic (converting byte strings to native types, mostly):


def edit_subscription(request, *args, **kwargs):
if request.GET.get('entity_key'):
entity_key = request.GET.get('entity_key')
else:
return http.HttpResponseServerError()
if request.GET.get('subscribe') == 'true':
subscribe = True
elif request.GET.get('subscribe') == 'false':
subscribe = False
else:
return http.HttpResponseServerError()
subscription_logic.editEntitySubscription(entity_key, subscribe)
return http.HttpResponse('OK')


I then call the editEntitySubscription subscription logic method:


entity_key = db.Key(entity_key)
user = user_logic.getForCurrentAccount()
subscriber = self.getSubscriberForUser(user)
if subscribe and entity_key not in subscriber.subscriptions:
subscriber.subscriptions.append(entity_key)
logging.info('added subscription for entity_key %s for user %s'
% (entity_key, user.key().name()))
elif not subscribe and entity_key in subscriber.subscriptions:
subscriber.subscriptions.remove(entity_key)
logging.info('removed subscription for entity_key %s for user %s'
% (entity_key, user.key().name()))
else: return
db.put(subscriber)


I've also added a model-based class method for determining auto-subscriptions, which will help for what I've found to be the hardest part of this project. While the goal is simply to get all users with at least read-access to an arbitrary Linkable entity, it is actually very difficult, and when possible it's only from manually creating logic. Essentially, this list of users should be sharded into its own relational entity (just a reference to the Linkable and a ListProperty of users). And I've also made various other rafactorings of the subscription code.


I look forward to committing code this week! At almost 2,000 lines, this will be a big commit, but worth the effort.

GSoC 2009: GHOP - The work begins now!

Hello everyone,
That was the last week of GSoC. Worked on very few things last week. Last week, I mostly worked on creating menus for easy accessibility of GHOP Tasks per organizations for general public and students. I also worked on Student data store model conversion to introduce School Type for students and wrote a Appengine task to convert the existing Student entities. I also added another field to hold the student grade if he is a High School Student. All other things like cleaners for these fields and views were already written during the beginning days of GSoC. Just merged them together.

Also made some modifications to access.py module to accommodate program logic as parameter than hard coded logic. Also have fixed the bugs Lennie had noted and made some changes to the subscription star toggling among other things accordingly. My demo instance is up and running at http://melange-madhusudancs.appspot.com/ and includes all the above said features.

Have also started writing the Starter Manual on how to use GHOP features for its users. I have setup a wiki page at http://code.google.com/p/soc/wiki/GHOPUserManual It is a work in progress. I will complete it soon.

And coming to the reason why I said "The work begins now!" is that, I really want to see this programme happen. If that happens I am quite sure there will be many feature requests for GHOP, which I am looking forward to implement. We want to listen to the community for the feedback on how we can improve GHOP and make it better and make it happen.

Last but not the least, I would like to thank Lennie a lot, my mentor in particular for extending his helping hand in every way possible and to have spent so much time for mentoring me. He was there every time I had a doubt or I had a question or I had a problem or I was feeling like pulling out my hair trying to resolve a bug. He spent much more time in mentoring me than I expected at the start of the programme. Thank you so much Lennie. Also I would like to thank Sverre and Pawel and everyone else on the Melange team who helped me to get this project going.

-- Thank you,
Madhusudan.C.S

P.S When can I call myself a Melange developer :P ?

Newsfeed Module Update

This week, a lot of the work I'm doing is of the fixing/testing/documenting variety.

There are two new features that change the functionality of the newsfeed in notable ways.

One is PubSubHubbub support. It's working, and was surprisingly easy to get running.

It works with a hook in the newsfeed update process. Remember how the feed update process includes a sender (the updated entity) and a list of receivers? Because it's the feed of the receivers thats is being updated, the pubsub ping is made for each of these receivers.

First we get the feed URL for the receiver:

# resolve the URL of the entity's ATOM feed
entity_feed_url = news_feed.getFeedUrl()

Then a POST request is sent to a central hub that includes the feed_url:

headers = {'content-type': 'application/x-www-form-urlencoded'}
post_params = {
'hub.mode': 'publish',
'hub.url': entity_feed_url,
}
payload = urllib.urlencode(post_params)
try:
# can these be sent in a batch?
response = urlfetch.fetch(HUB_URL, method='POST', payload=payload)
except urlfetch.Error:
logging.exception('Failed to deliver publishing message to %s', HUB_URL)

Wow, that was easy, wasn't it?


The second feature is the subscribe-by-star pattern, which already appears in plenty of other Google products and should be useful for people who don't like/understand RSS feeds, since subscribing by the star signs you up for email updates.



Finally, I've posted questions that would be useful to ask Melange users in an optional survey.


The questions are on a survey on my staging server: http://jamstage.appspot.com/survey/take/program/google/gsoc2009/updatepreferences

GSoC09: Statistics module 8th weekly report: Refactoring, fixes... and jQuery 1.3!

So... the penultimate week of GSoC came and passed, being a great moment for maintenance tasks like refactoring, bug fixing and coding small features left behind during project track. Beside that, following a Madhusudan's proposal, we've finally upgraded to jQuery 1.3 (commit 5723f329d0) and to JQueryUI 1.7 (commit 4e2789b8e8)!

Here's the story: following Pawel's suggestions, I've made little changes to dashboard's code to transclude itself when encountering DIVs with class "melangeDashboard" instead of MELANGE tags with attribute type "statistic_dashboard". Furthermore, I removed the fancy stack menu (that was too different from the rest of the application style) and put only a "new chart" link in the upper side of the page. That should be a temporary solution, as more features will come into place we'll integrate another menu.

A couple of bug fixes:

• Before this week, if you changed the visualization type right after widget creation, it wouldn't be saved on the backend. It would have happened only if you changed it after one dashboard reloading. Now it's fixed.
• Fixed the bug that Daniel Hans mention in his post: now, when switching the virtual statistic, the visualization is properly refreshed.

I've focused also on some refactoring. First of all I've finally changed what Daniel Hans and Lennie suggested me when I've sent patches to the mailing list (mainly on Python code) :)

The main work, however, has been on the Javascript side, mainly to avoid code duplication (for example code for updating a widget in the backend was present three times) and to separate dashboard code and widget code better. For example, before the refactoring, code to move and delete widgets in the dashboard was written in the widget code, while these are functions more related to dashboard logic.

But while porting the logic to move widgets inside the dashboard has been an easy task, as the dashboard already contained all the code to make the widgets move (using jQuery sortable plugin and binding the function to update the backend to the "stop" event), widget deletion has not been so easy to accomplish. Why? Because the deletion button is properly created in the widget class, and the function to delete the widget in the backend is in the dashboard, so in another scope.

That could be easily workarounded if I made the deletion code a "privileged" function of the dashboard... but, to me, this is not correct design. Deleting a widget (and so the chart entity in the backend) is a crucial task, so the function that does this should not be callable from other code. The solution? Publish/Subscribe pattern! While this is a built in behavior in Dojo, this is not the case in jQuery namespaced events (they can be used as topics), but then it happens that using a "click.widgetdeletion" namespace, when it bubbles up through the DOM, it loses the "widgetdeletion" part (so appearing as a common "click" event). Furthermore, to have a clean code, I would have liked to pass a reference to the widget instance with the event, but no data could be passed along with the event in jQuery 1.2.6.

So I've made a quite ugly hack, using common event delegation, detecting the target object and calling the deletion function if it has the "remove" class (which happens to be the class of the remove button... but what if we want to call that class with some other name in the future?), and using jQuery traversing to detect the ID of the widget from the real id of the parent div DOM node (which has a pattern like widget-IDNUMBER, that is extracted using a simple regex) and then using it to retrieve the widget instance from the widget list array that is present as a variable in the dashboard. Something like:

var widget_id = jQuery((jQuery(event.originalTarget).parents(".widget"))[0]).attr("id");
var widget_instance = widgets_list[/^widget-(\d*)$/.exec(widget_id)[1]];

Pretty ugly, isn't it?

But then, after jQuery 1.3, and using the new Event object, all is smoother now:

This is what happens after confirming deletion:

jQuery(this).trigger({type: "widgetdeletion",widget: _self}).remove();

You can notice just an event triggering, with a specific type (as a publish/subscribe topic) and passing the widget instance along with the event. In the dashboard, catching the event is as easy as:

dashboard.bind("widgetdeletion",function (event) {updateOnDelete(event.widget);});

That is, binding the "widgetdeletion" event and calling the proper function, passing it the widget instance reference. Simpler, cleaner and not prone to be messed if something in the HTML code changes (as the classes assigned to a widget or to its remove button).

Some enhancements made during the week has been the live editing of widget titles and support for the new get_available_statistics function (made by Daniel Hans), that has proper rights checking and returns all statistics (GSoC/GHOP, 2008/2009..and so on) with their proper scope path and link ids.

For refactoring purposes again, I've tried to look for a suitable browser side templating engine. In Dojo, there's the faboulous dojox.dtl Django porting. But what about jQuery? There's a bunch of technologies out there (made as jQuery plugin or standalone), as:

Standalone
Google JSTemplate
Ajax Pages
Pure (this can be used also along with jQuery)
JBST
jQFRagment
JSHTML
twoBirds
JsonFX.NET

jQuery plugins/dependent
jTemplate
chain.js
Nano
John Resig's micro templating
Some other guidelines plus fixes to John Resig's micro templating
Custom made 1, custom made 2

It's really not easy to deal with all these different options, and to explore major advantages/drawbacks. I'll try to do my best during this week, because it's really a must-have for us, to avoid ugly mixing between JavaScript and HTML code.

GSoC: Statistic Module update

Here is my next status update of the eleventh week of Google Summer of Code 2009:

First of all I finished working on displaying multiple number of visualizations inside one widget. Let us consider the standard 'Students Per Degree' example. Now you create a new widget and change visualization between:
'Students Per Degree (all)',



'Students Per Degree (with projects)':



'Students Per Degree (without projects)':



'Students Per Degree (cumulative)':




The last one represents all three types in a single visualization.

This new option works pretty well on statistic 'visualize' page (you can get there by clicking 'visualize' link on the page which lists all statistics).
It does not perfectly work on the actual statistic dashboard. There is a bug: when you choose a new statistic to be displayed, the visualization does not change. Anyway, when you change a visualization type, a new visualization for the new statistic is displayed. To be fair, I have not found a solution for this issue yet. I tried to debug that JavaScript scripts for a couple of hours and even discovered what was wrong, but could not fix that. I asked Mario for some help and he said he would try to take care of that.

The next thing I did was preparing a new series of patches based on Sverre's suggestions. The patches that were already fine, I already pushed to the main repository.

Then I started to work on access issues. Each statistic should have information about who is allowed to see it. Some statistics should be visible only to program administrators, while some others can be seen also by organization admins. I added a new 'read_access' parameter to the statistic model. Currently frontend may retrieve a complete list of statistics available for the current user by calling /statistic/get_available_statistics request.

GSoC 2009: GHOP - Eleventh Week Status Update!

Hello everyone,
A week of frustrations come to an end. The goal for last week was to get the Tag system in place for GHOP. The goal included giving the Program Admin to Add/Delete and Edit Tags. And you might already know that we are using a modified version of Taggable-mixin for tagging in Melange.

Although Adding and Deleting tags looked very trivial to me, edit was not so trivial. The first reason being, the key name for the tag stored in the Datastore used the tag name. So every time you edit a tag a new key needs to be created, but all other properties must be retained, so tag must be copied to the new entity. Also over and above this, on the UI side I was not even sure how to allow Program Admin to Add/Delete and Edit a tag in the same interface. After a lot of struggling and battles within and frustrations, I thought for a GHOP Program number of types of Program defined tags are fixed, one is for Task Difficulty level and other is for Task Type. So I created 2 pages, one for each where one could do all the 3 actions(Add/Edit/Delete).

For accomplishing this per Task page, I have used a JQuery plugin called, in-place-edit. This allows us to edit a tag in-place. So one can click on Add button to add new tags, and make the contents of the tag empty to delete it or edit the tag name. The editing of the tag name automatically copies the old Tag entity to the newly created Tag entity. And each creation, edit and delete happens via AJAXy calls, so one need have to save the changes after editing the page completely. Program Admin can just exit the page.

Once I solved this problem, a new problem arose. Usually tags have to be sorted in some order, say for example for difficulty level tags, it must be sorted from easiest to hardest. But tags need not be created in the same order. For example, Program Admin might initially think, only Easy, Medium and Hard are sufficient, but might later realize a Trivial difficulty level is required. So this needs to be ordered in the order enforced by the Program Admin. Once again I had to goto JQuery for help. I used JQuery UI's sortable plugin already in Melange code base to drag and drop sort the list of tags. So ordering is implemented as well.

After this, the interface to Add Tags to create/edit tasks had to be added. For a difficulty level tag, it is a normal drop down. But a Task can be of multiple types, like Documentation and Translation. So for this I have used an HTML Multiselect box. Also there is a text box for adding arbitrary tags. At the moment, these two types of tags are merged and stored as the entities of same TaskTypeTag model in the datastore with mandatory property=True for Task type tags and False for arbitrary tags. But Lennie wanted this implementation to be changed and separate them into 2 models. This requires a good amount of re-work. I will do it this week.

After all these, I started working on Search for tasks and happily created multi selects for Organizations, Task status, Difficulty Level, Task Type and started generating queries for them. It was then Lennie, made me realize that there can be only 30 subqueries per query. So this kind of search mechanism won't work. I have currently stalled my work on it. I am thinking of work arounds to this. Once we get a fair idea of what to do, I will resume the work on Task search.

Amidst all these work, I have been fixing bugs, making some implementation changes like Task state transitions etc. This week, I will be mostly working on writing a starter manual, fixing bugs and student data model conversion to include whether student belongs to High School or University.

Last but not the least, all my work till now is available on my demo instance at, http://melange-madhusudancs.appspot.com