GSoC 2011 GUI Overhaul: Redesigned Admin Dashboard

Hi everyone,

As I promised from last post, I'll talk about redesigned admin dashboard in this post. Currently, admin dashboard is just a page with list, without any information, to other pages. I've provided mockup for admin dashboard with iconic information. There will be also new functionalities such as manage organizations with another sub functionalities to manage org, proposal, mentors, and students, participant locations, etc. There are two approaches to render child page in admin dashboard. These approaches is discussed in the following section :

Admin Dashboard with AHAH for Child Page

From last conference, Sverre told that Carol needs page in scope of admin should be stay on same page with parent page, that's admin dashboard. He said, it could be achieved with AJAX request/response. I tried the AJAX approach by making a div container to hold the HTML response. It's better to call this AHAH rather than AJAX. So, this div container is a hidden one and will be displayed after the HTML response injected to the container. The container was placed below the iconic information. To make the request handled with a minimum HTML responded, I created another blank template, base_ahah.html which contains following code :

{% block main_content %}
{% if ds_write_disabled %}
<div id=user-message class=error>
<strong>ALERT: </strong>Google Appengine datastore write
capability has been disabled at the moment. You will
<strong>NOT</strong> be able to save your changes at this
time. We are sorry for the inconvenience caused. Please visit
<a href=http://code.google.com/status/appengine>
http://code.google.com/status/appengine</a> to check
Appengine datastore status.
</div>
{% endif %}
{% block page_content %}
{{ tmpl.render }}
{% endblock page_content %}
{% endblock main_content %}

Page requested via AHAH must extends this template. We also need to modify form_base.html to check whether it was requested via AHAH or regular request with full base layout. If a page requested via AHAH, the JS code will add another query string, ?ahah=1. In the AHAH base template, we don't request any CSS or JS asset. I don't know if there's a way for LABjs loads once for the same script, and skip injecting the head for subsequent script calls. To be safe, any assets should be provided first when admin dashboard page requested. The form submission should be changed with AJAX. This was easily done with jQuery.post function. Any response, either invalid or valid, will be injected again in the AHAH container. With this approach, any admin sub page that need to be added later must exclude the asset its needed to parent page, admin dashboard. This make any sub page coupled to its parent page, which is not good.

Admin Dashboard with Iframed Colorbox for Child Page

Colorbox is yet another thickbox-like jQuery plugin. Colorbox provides a way to hold the content of external page in iframed container. For iframed template we need to create another base layout again, say base_colorbox, with minimum markup and variabel template. In iframed container we could extend block stylesheet and add melange script dependencies. Here's what admin dashboard with colorbox looked alike :


Within iframe, we need to specify the width and height for colorbox. We can specify the width and height in URL query string. Colorbox approach seems the easy way to implement without coupling to the parent page.

In the next day I'll provide the live instance to experiment with the new dashboard, so stay tunned.

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 3

I've been developing generic methods to handle OAuth. When developing software it's important to create meaningful abstractions so a method can refer to these abstractions without need to understand low-level details what they do. For the OAuth part of my project a helper module handles this low-level details and caller that refers to these abstractions is an access_checker. It asks following questions:

• Get access token for user.
• Create a service for user.
• Generate a OAuth authorization URL.
• Process URL for OAuth verifications.

Low-level helper module does following operations for questions above:

Get access token for user:

• Generate a unique key consists of user's key and current host (different consumer key,secret so different services and access tokens are used for each hosts). 
• Ask memcache or a storage mechanism to fetch token corresponding to that key.
• Return token.
• If there isn't a token return None.

Create a service for user:

Service is a GDocsService object that is used to make API calls.

• Create a service for the current host. 
• Set service's input parematers to host's consumer_key and consumer_secret. 
• Set encryption method to sha or rsa.
• Return service.

Generate a OAuth authorization URL:

• Fetch a request token. 
• Save request token's secret to memcache for later use (when validating token).
• Generate a authorization URL from request token.
• Return URL.

Process URL for OAuth verifications:

• If request URL contains a verification key as GET parameter, parse it.
• Verify previously stored access token with parsed verification key.
• Store verified access token to memcache or datastore.

GSoC 2011 GUI Overhaul: Toggleable List

Hi everyone,

Today I'll talk about toggleable list for regular dashboard. To make you clear, there are two dashboards in Melange, one is regular dashboard and the other one is admin dashboard. The regular dashboard is the page that we see if we clicking the "My Dashboard" link on the left sidebar menu, or if you visit the url at gsoc/dashboard/(host)/(program_name). The admin dashboard is available in the left sidebar menu as "Admin Dashboard" if you have a role as a host. If you're login with account that listed as an administrator of melange instance, you have access to that admin dashboard. Admin dashboard can be visited at gsoc/admin/(host)/(program_name)

Currently, regular dashboard is loading a bunch of list at one time. Each list, or called component in dashboard view, in regular dashboard is fetching a lot of entities from datastore in the background. Each role gets different components. If someone has a role both as a host and a mentor, Carol's case, then she gets much components. Too much component will reduce usability. User needs to scroll hardly if she/he looks for component at the bottom. Fetching all list at one time is inefficient too. I've provided three ideas with mockups to handle this problem, one is with tab, two is with iconic page (the same as admin dashboard that I'll talk in the next post) and the last is with toggleable button. Carol agreed with toggleable button. The reason for toggleable button is user can toggle which list/component she/he wants to show without leaving the dashboard page.

I've provided the patch to melange-soc-dev. The implementation is quite simple. To make a component toggleable we just set toggleable and collapse to True inside the context. The list template will check this variable. If we ignore toggleable and collapse then the list will be requested when loading the page. We can set a component toggleable but doesn't collapse at first load by set the toggleable to True and collapse to False. With this approach, other pages with list, such as accepted_orgs, wouldn't be affected. In the next post I'll talk about admin dashboard that being redesigned right now.

GSoC 2011 Melange Testing Project : CP W3

Hi,

For quite a long time I had been working on the online test runner and could not make it work on a live instance. As I had discussed it in my last blog post about my implementation that I intended to exploit some plugins of Nose to run the tests, it failed because of the reason that Nose uses some libraries which are restricted by GAE. So Leo asked me to take my other approach of discovering all the test modules in the tests directory and then make a suite of tests using the unittest.loadTestsFromName  method. I used the os.walk() built-in function to first list all the modules with their absolute paths and then changed the paths into a dotted notation so that these paths could be easily used by unittest.loadTestsFromName to load the tests. I have used a lot of code from the gaeunit source specifically which is related to the browser interface and the test runner. The test runner had to be modified so as to fit in with melange code. I also had to include the tests directory, gaetestbed, nose, webtest and  mox libraries in the app folder so that tests can easily use them when running on a live instance.
    The online test runner runs on my local development environment but it shows errors that are due to the bugs in the runner itself. Below is the video of the melange tests running on my local machine through the browser.

I have stopped working on the runner as of now. But in the end, I discovered that I have extended some of the features of gaeunit. It has been 2 years since gaeunit was last released, so I was planning to contact the gaeunit developers if I can develop for them.

     During the last week conference call, Sverre asked me to treat the online runner as a side project and concentrate on my testing project now. So, I have started working on my project and wrote the test for app.soc.logic.accounts and submitted it for review. At the time of writing this post, I have also almost completed the test for app.soc.logic.dicts with a few methods left to test.

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 2

It's been two weeks since coding started, due to my heavy exams i haven't been working too much and had to skip first week's blog. I updated my timeline with target deliverables [0].

First thing to solve is how OAuth will be handled with a generic solution. As we wouldn't want to handle all jobs (e.g. token check, redirection, access token parsing) in every seperate view, there needs to be a generic handler (a 'token_required' decorator and a 'gdocs_service_object' middleware in Django's terminology) for views that needs to make GDocs api calls over OAuth.

I started to work on this diagram to discuss OAuth mechanism with community:

One of the particular problems when playing with gdata library was a not-self-defining error about unicodes. As GAE returns entity field values in unicode, a  method in built-in hmac library that expects 8-bit strings was crashing with this error message:
character mapping must return integer, None or unicode
Solution was simply encoding field value to utf-8: consumer_key.encode('utf-8'). I don't think same error would arise when using rsa instead of sha with authentication.

[0] -http://code.google.com/p/soc/wiki/GSoC2011IntegrationWithExternalAPIs#4._Project_Timeline

GSoC 2011 Melange Testing Project : CP W2

My task for this week was to implement an online test runner which enables us to run our unit tests online rather than on the local machine. I have written the necessary modules which provide a very basic functionality and features for now. More features can be added and improved with time. Leo had suggested me the gaeunit Open Source project to study and see how they do it.

I outline the basic process of how gaeunit does the magic :-

• gaeunit defines two main request handler classes - MainTestPageHandler and JsonTestRunHandler. The request to run the tests is handled by the MainTestPageHandler class which first validates the URI arguments and renders an html /plain report according to the 'format' argument. Our interest lies in the generation of the html report.
• When the request is to render an HTML report, gaeunit first creates a suite of all the tests found according to the 'name' parameter passed to it in the URIs. 'name' can be any package, module or class. This suite of tests is then converted into a JSON format with structure {"module_name": {
  "class_name": [list_of_methods]}} and passed with the HTML response to the browser.
• As soon as the response is loaded in the browser, it triggers a javascript whose task is to iterate through the JSON object and pass the (module_name, class_name, method_name) tuple to the server by generating an asynchronous XMLHttpRequest which then waits for the data to be sent by the server.
• The GAE server upon receiving the tuple runs the JsonTestRunner and the test result is collected in a unittest.TestResult object. This TestResult object is then again converted into a JSON object and returned to the browser. 
• The browser then sends the data to the appropriate fields in the web page

My Implementation:

My implementation does not work presently and it needs lot of debugging. I moved the tests folder to melange/app/ so that GAE finds it easily. I wrote two modules online_run.py and online_tester.py. online_run is colleague of tests.run.py with lot of code borrowed from it to implement the test runner for online testing.

      online_tester implements classes to handle and manage all the tasks right from initiating the tests to printing the test reports. I thought of exploiting the 'testid' and 'collect' plugins of nose. Running the tests with these plugins only collects the tests without actually running the tests and assigns an integer id to all of them. We can run a specific test next time by just passing the integer ids. So I have defined three classes MainTestPageHandler, PrepareForTests and RunTests. MainTestPageHandler just renders a view where users can select which tests to run. The arguments are passed as POST arguments to the PrepareForTests handler which is supposed to collect the tests and assign IDs to them. The number of collected tests is then passed to the browser with HTML source which contains a javascript . This javascript passes the integer IDs to RunTests which makes use of the runner in online_run module to execute the tests and returns the HTML output to response.out stream.

This implementation is not working  and I asked Madhu for help. He said may be I have messed with the appengine paths. I have to try again keeping the tests in app/ and making the necessary changes in my scripts.

Also the paths like '/home/praveen/testmelange/app/tests' which I have included in the python paths according to my system will not work on GAE because we do not know about their file systems.

You can take a look at the error reported by my implementation at my instance[1]
Just choose 'Run all tests' and hit Run.

[1] http://melangetesting.appspot.com/runtest
[2] online_run.py    http://tinypaste.com/da4b5e
[3] online_tester.py http://tinypaste.com/1f3b88

GCI 2011 GCI with New Architecture: Coding phase - Week 1

Hello all!

I'm sorry for the late post.This week has not been as expected, my internet was down for a couple of days and then my deadlines fell apart and I'm lagging a bit but I will try to cover it mostly by this next week.The patches that I have sent got reviewed and I re-sent the patches with the suggested changes.When moving the list generator from gsoc module to core I did not follow the import conventions and I was redirected to this wiki and I learnt from there.And I moved the common fields in GSoCProfile to soc.models.role.Profile to prevent code duplication.Also the grade property is added to the parent StudentInfo class as that contains the school_type and Finally I have modified the role_conversion script to convert GCIRoles and update references to GCITask and StudentRanking model.I was asked to move it to a seperate file in the GCI module rather than mixing up both in the same file, so I will do that now and after this I will finish up with RequestData so the from beginning of next week I can work with views.
In the mean time I have ordered Oreilly - Programming Google App Engine and started reading the book, it is a really nice book which is very helpful to understand App Engine better in detail.