Monday, June 6, 2011

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 2

It's been two weeks since coding started, due to my heavy exams i haven't been working too much and had to skip first week's blog. I updated my timeline with target deliverables [0].

First thing to solve is how OAuth will be handled with a generic solution. As we wouldn't want to handle all jobs (e.g. token check, redirection, access token parsing) in every seperate view, there needs to be a generic handler (a 'token_required' decorator and a 'gdocs_service_object' middleware in Django's terminology) for views that needs to make GDocs api calls over OAuth.

I started to work on this diagram to discuss OAuth mechanism with community:

One of the particular problems when playing with gdata library was a not-self-defining error about unicodes. As GAE returns entity field values in unicode, a  method in built-in hmac library that expects 8-bit strings was crashing with this error message:
character mapping must return integer, None or unicode
Solution was simply encoding field value to utf-8: consumer_key.encode('utf-8'). I don't think same error would arise when using rsa instead of sha with authentication.

[0] -

GSoC 2011 Melange Testing Project : CP W2

My task for this week was to implement an online test runner which enables us to run our unit tests online rather than on the local machine. I have written the necessary modules which provide a very basic functionality and features for now. More features can be added and improved with time. Leo had suggested me the gaeunit Open Source project to study and see how they do it.

I outline the basic process of how gaeunit does the magic :-
  • gaeunit defines two main request handler classes - MainTestPageHandler and JsonTestRunHandler. The request to run the tests is handled by the MainTestPageHandler class which first validates the URI arguments and renders an html /plain report according to the 'format' argument. Our interest lies in the generation of the html report.
  • When the request is to render an HTML report, gaeunit first creates a suite of all the tests found according to the 'name' parameter passed to it in the URIs. 'name' can be any package, module or class. This suite of tests is then converted into a JSON format with structure {"module_name": {
    "class_name": [list_of_methods]}} and passed with the HTML response to the browser.
  • As soon as the response is loaded in the browser, it triggers a javascript whose task is to iterate through the JSON object and pass the (module_name, class_name, method_name) tuple to the server by generating an asynchronous XMLHttpRequest which then waits for the data to be sent by the server.
  • The GAE server upon receiving the tuple runs the JsonTestRunner and the test result is collected in a unittest.TestResult object. This TestResult object is then again converted into a JSON object and returned to the browser. 
  • The browser then sends the data to the appropriate fields in the web page
My Implementation:
My implementation does not work presently and it needs lot of debugging. I moved the tests folder to melange/app/ so that GAE finds it easily. I wrote two modules and online_run is colleague of with lot of code borrowed from it to implement the test runner for online testing.
      online_tester implements classes to handle and manage all the tasks right from initiating the tests to printing the test reports. I thought of exploiting the 'testid' and 'collect' plugins of nose. Running the tests with these plugins only collects the tests without actually running the tests and assigns an integer id to all of them. We can run a specific test next time by just passing the integer ids. So I have defined three classes MainTestPageHandler, PrepareForTests and RunTests. MainTestPageHandler just renders a view where users can select which tests to run. The arguments are passed as POST arguments to the PrepareForTests handler which is supposed to collect the tests and assign IDs to them. The number of collected tests is then passed to the browser with HTML source which contains a javascript . This javascript passes the integer IDs to RunTests which makes use of the runner in online_run module to execute the tests and returns the HTML output to response.out stream.

This implementation is not working  and I asked Madhu for help. He said may be I have messed with the appengine paths. I have to try again keeping the tests in app/ and making the necessary changes in my scripts.

Also the paths like '/home/praveen/testmelange/app/tests' which I have included in the python paths according to my system will not work on GAE because we do not know about their file systems.

You can take a look at the error reported by my implementation at my instance[1]
Just choose 'Run all tests' and hit Run.


Sunday, June 5, 2011

GCI 2011 GCI with New Architecture: Coding phase - Week 1

Hello all!

I'm sorry for the late post.This week has not been as expected, my internet was down for a couple of days and then my deadlines fell apart and I'm lagging a bit but I will try to cover it mostly by this next week.The patches that I have sent got reviewed and I re-sent the patches with the suggested changes.When moving the list generator from gsoc module to core I did not follow the import conventions and I was redirected to this wiki and I learnt from there.And I moved the common fields in GSoCProfile to soc.models.role.Profile to prevent code duplication.Also the grade property is added to the parent StudentInfo class as that contains the school_type and Finally I have modified the role_conversion script to convert GCIRoles and update references to GCITask and StudentRanking model.I was asked to move it to a seperate file in the GCI module rather than mixing up both in the same file, so I will do that now and after this I will finish up with RequestData so the from beginning of next week I can work with views.
In the mean time I have ordered Oreilly - Programming Google App Engine and started reading the book, it is a really nice book which is very helpful to understand App Engine better in detail.