GSoC 2011 Melange Testing Project : CP W2

My task for this week was to implement an online test runner which enables us to run our unit tests online rather than on the local machine. I have written the necessary modules which provide a very basic functionality and features for now. More features can be added and improved with time. Leo had suggested me the gaeunit Open Source project to study and see how they do it.

I outline the basic process of how gaeunit does the magic :-

• gaeunit defines two main request handler classes - MainTestPageHandler and JsonTestRunHandler. The request to run the tests is handled by the MainTestPageHandler class which first validates the URI arguments and renders an html /plain report according to the 'format' argument. Our interest lies in the generation of the html report.
• When the request is to render an HTML report, gaeunit first creates a suite of all the tests found according to the 'name' parameter passed to it in the URIs. 'name' can be any package, module or class. This suite of tests is then converted into a JSON format with structure {"module_name": {
  "class_name": [list_of_methods]}} and passed with the HTML response to the browser.
• As soon as the response is loaded in the browser, it triggers a javascript whose task is to iterate through the JSON object and pass the (module_name, class_name, method_name) tuple to the server by generating an asynchronous XMLHttpRequest which then waits for the data to be sent by the server.
• The GAE server upon receiving the tuple runs the JsonTestRunner and the test result is collected in a unittest.TestResult object. This TestResult object is then again converted into a JSON object and returned to the browser. 
• The browser then sends the data to the appropriate fields in the web page

My Implementation:

My implementation does not work presently and it needs lot of debugging. I moved the tests folder to melange/app/ so that GAE finds it easily. I wrote two modules online_run.py and online_tester.py. online_run is colleague of tests.run.py with lot of code borrowed from it to implement the test runner for online testing.

      online_tester implements classes to handle and manage all the tasks right from initiating the tests to printing the test reports. I thought of exploiting the 'testid' and 'collect' plugins of nose. Running the tests with these plugins only collects the tests without actually running the tests and assigns an integer id to all of them. We can run a specific test next time by just passing the integer ids. So I have defined three classes MainTestPageHandler, PrepareForTests and RunTests. MainTestPageHandler just renders a view where users can select which tests to run. The arguments are passed as POST arguments to the PrepareForTests handler which is supposed to collect the tests and assign IDs to them. The number of collected tests is then passed to the browser with HTML source which contains a javascript . This javascript passes the integer IDs to RunTests which makes use of the runner in online_run module to execute the tests and returns the HTML output to response.out stream.

This implementation is not working  and I asked Madhu for help. He said may be I have messed with the appengine paths. I have to try again keeping the tests in app/ and making the necessary changes in my scripts.

Also the paths like '/home/praveen/testmelange/app/tests' which I have included in the python paths according to my system will not work on GAE because we do not know about their file systems.

You can take a look at the error reported by my implementation at my instance[1]
Just choose 'Run all tests' and hit Run.

[1] http://melangetesting.appspot.com/runtest
[2] online_run.py    http://tinypaste.com/da4b5e
[3] online_tester.py http://tinypaste.com/1f3b88