Melange Functional Testing

Hi,

This week i worked on developing test scripts for testing Profile Page and Search Page but i spent major part of this week on developing a script for loading and running these test scripts one by one. Initially i tried to run them with bin/run-tests in soc but i was not successful and faced a lot of issues so i made a new functional-run.py script for executing these test cases one by one in my local environment and the way i want them to run.

Options that i provided in this functional-run.py script:
(1) -r  for report format, whether you want result on console(default) or you want a html based report. If you give the option like this $ python functional-run.py -r "html", a result folder is created in the functional folder where all the test scripts are saved and then it pick up all the files whose name start with "test" and executes them one by one and finally a html based report is saved in functional/results folder.

(2) -v for virtual display, if you want to execute all the scripts in virtual display. when this option is provided script.  All the scripts are executed virtually and you will not see thier execution in firefox. you will only see the final result either on console or in result report depending upon the report format you had chosen.

Currently this script is only for executing functional tests in my local environment.

GSoC 2011 Melange Testing Project : Community Bonding Period

Its been two days since the GSoC 2011 results were declared and I am still unable to tell myself that I am now a Google Summer of Code student. :) . A vote of thanks and gratitude from the very depth of my heart to the Melange developers.

Python is awesome and I am now in love with it. My project is on testing and frankly speaking I have never studied about software testing before. The only idea about testing which I had was during solving exercise problems proposed in Google's Python Class videos. The coding problems had to be solved and then their output tested.

I explored about the Melange testing project more and more. I did not understand many of the terms like 'Test Driven Development', 'Continuous Integration' etc mentioned on the Melange's ideas page, but I googled and learnt about them, followed online tutorials and resources linked on the ideas page. I joined the melange-soc-dev mailing list and asked Sverre if I understood correctly what the project is aimed at. He said 'Sounds about right :) ' and then he pointed me to the 'Getting Started' guide. Since then there was no turning back and I devoted most of my time getting familiar with the framework and the testing modules. I explored the Melange's blog and read the posts of previous GSoC students. Leo and Savitha very well explained how they tried to understand the codebase. I did not know about the tools like tree and pylint.pyreverse before.

Then I decided to write a test for one of the modules in app.soc.logic and chose the 'validate' module. The test was simple but difficult to write if one does not know about the organisation of the modules and how the functions were used. I took me two days to get a feel of the code and get familiar with the directory structure. I finished the test, learnt and fixed the style issues and finally signed the CLA and the module was integrated into the codebase. That was my first contribution to Melange as a pre-GSoC exercise. This gave a boost to my confidence to contribute more to Melange.

I knew there were many students competing for the project and I decided to write my proposal differently so as to have an edge over other students. I decided to show a graph of the failures and errors that occurred everyday, the data which I collected everyday by running tests over a period of ten days. I also thought of showing a test code vs production code analysis. April 8 was the last date of submitting the proposals and I was busy with exams till April 5. Melange was the only organisation on which I had concentrated since the beginning and I knew I had to write a good proposal to get selected and dedicated all the 3 days in writing the proposal. The 'graph' and 'the code analysis' do not reveal anything groundbreaking but that was what my mind could think of then.

I regularly followed all the discussions on IRC, melange-soc-dev and gsoc lists. There were criticisms about the tinyMce editor and asked Sverre if the editor size can be increased because it seemed too cluttered. He asked me to do it. I read the tinyMce documentation and found many more helpful buttons can be added to the toolbar. Mario helped me in writing the configurations for the 'basic' and 'advanced' editor settings. I had never learnt jQuery before and I believed that I should not say " I don't know jQuery, so I can not do it". Instead I followed some tutorials and then it was easy. I also thought "Hey! you applied for the testing project and you are working on the editor."

It was a great experience working hard to bag the project. It is the first time I am contributing to an Open Source project in a major way and to see my name in the Author's List, Mario's Code Swarm video and the commit messages fills me with a feeling that I find difficult to describe.

Yesterday I attended my first conference call with the Melange developers and heard the sweet voice of Carol :P . I thought the meeting would be very formal but all of them were very jovial. Sadly, I could not introduce myself to other members as the mic was not working. Hope to fix it soon so that I can communicate during the meeting on Monday. I have exams starting Tuesday next week, so will discuss about the project plan with Leo after 15 May 2011. I will get a week of time before May 23 to read about GAE, Django and Software testing.

Melange community is really the 'awesome-st ' community and I wish to meet the developers and have fun with them at least once in my life. :) :)

GSoC 2010 Melange Testing Project: Week 7 of the Coding Period

Last week, I found that Python coverage does not track the properties definition of Melange models which makes the coverage of models still quite low. So, I first tried to exclude models from coverage report. This is done by stubbing out nose.plugins.cover.Coverage.wantModuleCoverage with a method which offers the option to skip packages for testing coverage report through pymox stubout. Later, however, I remembered that coverage should be able to track class variables and module variables. I reckoned that the problem may be that these Melange models have been loaded before the coverage starts tracking. So, I further used pymox stubout to stub out nose.plugins.cover.Coverage.begin so that it loads Melange after the testing coverage starts. It works! All the class variables and module variables of Melange models, logic and views can be tracked by coverage and the code coverage now is over 50%.

Then I further worked on various views, e.g. sponsor, program, student, organization, etc and aim to increase the overall code coverage to over 60% next week.

GSoC 2010 Melange Testing Project: Week 6 of the Coding Period

In this week, I first tried to add test coverage to Melange models, i.e. did model tests. However, I found that Python coverage does not track the properties definition of models which makes the coverage of models still quite low. When I discussed with Sverre, he suggested me to exclude models in the coverage report and start to test other modules with lowest coverage. After rechecking the coverage report, I decided to do view tests for sponsor, org, org_admin, mentors, student and so on.

Then my hard drive was corrupt:(. Fortunately, I have the online clone on Google servers, so I have not lost much work. It just took me some time to reinstall the operating system and resetup the environments. Due to the problem, I have only finished most parts of the sponsor view tests and some parts of org and student tests. Next week, I will definitely work harder to catch up.

GSoC 2010 Melange Testing Project: Week 5 of the Coding Period

As I had planned last week, I added content to the wiki page GettingStartedOnTesting [0] with the aim to provide a guideline on writing tests for Melange this week. Sverre has had a look at it and thinks that it should be extended, though it looks good.

I have also added test cases for a couple of logic modules this week and am currently working on testing the views of sponsor, program and student. Sverre thought the test coverage is still low and suggests me to focus the modules with low coverage next week, e.g. do some tests on the models. So, I will move my focus on testing the models next week.

[0] http://code.google.com/p/soc/wiki/GettingStartedOnTesting

GSoC 2010 Melange Testing Project: Week 4 of the Coding Period

In the last week, I further added more test cases to the user view test to test the views for administrator users. In order to test the operations of administrator users, an XSRF middleware token has to be manually generated and added to the post data so that the operation can pass authentication. Then, I added more test cases to the user logic test. This week, I will try to add test cases for more logic and view pairs.

One big thing for Melange next month is that each commit will be accompanied with tests. In order to contribute to the movement, I am preparing for a wiki page explaining the current testing environment (e.g. datastore), test runners (e.g. how to only run a single test case), available tools (e.g. pymox and Django test client), utilities (e.g. StuboutHelper), and so on.

GSoC 2010 Melange Testing Project: Week 3 of the Coding Period

Last week, I finally got Django test client working with the help of Sverre (my mentor). In this week, however, another problem arose. For some urls, e.g. /cron/poke, the response returned by the test client is a Django http response object; for other urls, e.g. /user/show/a_user, however, the response is a dict. After failing to figure out what had caused the problem, I asked Sverre for help again. Sverre suggested that it might be due to the stubout code (pymox) in tests/app/soc/views/models/test_base.py. After I commented the code, however, the problem still did not go away. Later, Sverre found that the problem was due to that the fact that the stubout in tests/app/soc/modules/ghop/views/models/test_task.py has not been unset. So, he added the unset code to tearDown methods and also helped me reviewed/improved and merged my code of refactoring the stubout code into a helper class. Thank you very much, Sverre!

After solving these problems, I started to do view tests for soc.views.models.user using Django test client. I have almost finished it. Yesterday, I pushed the code into my online clone. Today, I asked Sverre to have a quick look to see if it is alright and he was satisfied with it. Though I am still behind my original schedule, I am confident that I can catch up soon. This is because I have solved all major obstructions and have finished adding tests for one logic (base) and one view (user) so that I can follow similar patterns to add tests for other logic and views very quickly.

This week, I also came across the trace module of python when I tried to find another problem I met. It can be used either in command line with python -m option or in your program. This module save you time to code a call back function for sys.settrace by offering some common functions. One useful function is to trace the execution flow of your program. It traces every statement, so be aware the output could be quite large. For example, the output of tracing a single statement of Django test client in Melange testing environment is more than 100MB. Another disadvantage is that the output only shows the module name without package path which could confuse you when you have many modules with the same name or you have a big project and do not know the package of every module. Another useful function is to trace calls. The output of this one is quite small compared with the first function and also has full package path. This function, however, is not perfect either. The problem is that the calling functions are ordered by their module name rather than by their calling order which is arguably more useful. And you have no options to change the order. Besides, you can only use it in command line.

GSoC 2010 Melange Testing Project: Week 2 of the Coding Period

In this week, I had planned to add test cases for app.soc.views.models.base.View as well as app.soc.logic.models.User.Logic and app.soc.views.models.User.View, app.soc.logic.models.Role.Logic and app.soc.views.models.Role.View.

As I mentioned in my project proposal, I want to use Django test client which acts as a dummy Web browser to test views of Melange because of its ease of use and good integration with Django (no surprise since it was developed by the Django team). So, I first tried to get Django test client working.

After many attempts, however, I still could not make it work. Then, I found that I could not even run Melange locally suddenly due to the error (google.appengine.api.yaml_errors.EmptyConfigurationFile). I first tired to solve the problem by checking the tracktrace and code. Unfortunately, however, I could not find the problem. I thought that I might have accidentally changed or removed some important files. So, I rebuilt my local Melange but it still did not work. I also tried to re-clone and rebuild another Melange, but it was still in a vain. At that time, I realized that I had to fall back on my last resort, asking my mentor Sverre :-) . Awesomely, Sverre found that the problem was due to the corrupted .appcfg_nag file and he then solved the problem by removing it.

After this problem was solved, I tried the test client again. However, it still did not work. So, I asked Sverre again and found that it is because I had used the trailing slash at the end of the url following Django's tradition while Melange uses a url pattern without the trailing slash. Finally, Django test client works!!! Hope I had asked Sverre earlier :-).

Because of these problems I encountered, I have not finished this week's work as planned. Fortunately, however, I have got Django test client working. In the next week, I will try to speed up my work, finish this week's work as soon as possible and then start next week's work load.

GSoC 2010 Melange Testing Project: Week 1 of the Coding Period

The Coding Period of GSoC 2010 has officially started! In this week, I continued to add more test cases for the app.soc.logic.models.base.Logic.getForFields() method, started to commit and push the code to my clone. On this weekend, I will start to add test cases for other methods of the class and hope to complete it before next Monday. Another task on this weekend is to add a tentative project timeline to my project wiki, which becomes clearer after I discussed with Sverre tonight. Next week, I hope to complete the job of adding test cases for app.soc.views.models.base.View and some other pairs of logic and view, e.g. role and user. Anyway, time flies. Therefore I need to fly as well so that I can catch it or at least not fall far behind :-).

In this week, I also came across a nice book titled Programming Google App Engine from Dan Sanderson who has excellently explained the difference between GAE runtime environment and traditional web hosting or self-managed servers, the difference between GAE datastore and traditional relational databases, and pointed out the special consideration and strategies for building a scalable, responsive and optimized app upon GAE due to the difference. In addition, the book also covers basic things about GAE, e.g. how to set it up and how to build an app step by step as well as more advanced tools, e.g. the URL Fetch service, the Mail service and the XMPP service. So, it is highly recommonded for both beginners and intermediate users who want to make best use of GAE.

GSoC 2010 Melange Testing Project: the last week of Community Bonding - starting to code

In this week, I started to add more test cases to tests.app.soc.logic.models.base and tests.app.soc.views.models.base. I first learned the code of the corresponding code base and the existing test cases in more detail in order to understand their functions and logic. Then I tried to design some new test cases. Due to the fact that I am still not very familiar with the implementation detail of Melange and met problems when setting up the coding environment, e.g. could not get the new version of ctags to work well with Vim and Python, the process is a bit slow. However, these preparations have built a good foundation for the official coding stage, which will start next week. Hope that during that period I can code fast and leave more time to improve and clean up my code as well as do some extra work for the project.

GSoC 2010 Melange Testing Project: Week 1 - learning the code base

It has been a very exciting week. In this week, I made a clone for the testing project, deployed Melange to my GAE instance, got the posting right for melange-dev blog, created a wiki page for the project, signed a Contributor License Agreement (CLA), submitted tax forms, got an ACM account (the courtesy of Google), had the first conference meeting with other developers of Melange, helped some Melange users to solve their log-in problems, and had meetings with my mentor (Sverre). What a week!

My main job in this week, however, was to learn the code base of Melange since my project is to add tests to the code base. In order to help me better understand the code base, I decided to visualize the organization of the code base and the relaionship of modules/classes. Felix' diagrams are very interesting. However, they are only for Melange data models whereas views and logic are more relavant for my project. So, I used a tree tool to generate the module organization of Melange (click the picture below to view the searchable full HTML file)

and module organization of Melange tests (click the picture below to view the searchable full HTML file)

and used pylint.pyreverse to generate the package diagram of Melange (click the picture below to view the searchable full PDF file)

and the class diagram of Melange (click the picture below to view the searchable full PDF file)

These diagrams have helped me a lot to learn the code base. Hope they are also helpful to others who want to learn the code base.

During my meeting with Sverre, he suggested me to start with the testing of the modules which are the most important and used most, e.g. soc.logic.models.base and soc.views.models.base. Therefore, my next week's plan is to start coding, specifically add more test cases to the existing tests of these modules, e.g. tests.app.soc.logic.models.base and tests.app.soc.views.models.base.

Following is a brief introduction to the testing project:

This project is to add unit test suites/cases for the logic and views of the existing modules. These test suites/cases will enable developers to refactor the existing code at any time while at the same time to ensure that they still work correctly by passing all the tests (i.e. regression testing). This project will also serve as a basis for the Melange team to switch the development methodology to test driven development.

Cheers,

Leo (Chong Liu)

Newsfeed Status Update

Working on the news_feed feature has required work in all the different sections of the codebase (Model, logic, caching, view, template) and has required updates to about a dozen existing modules. As you can imagine, merging upstream is a fun exercise in getting aquainted with some of the more advanced features of version control.

The mechanics behind the news feed are very simple, at least to start with. Whenever an entity - a document, for instance - is created, updated, or deleted, it gathers a list of receivers. The implementation right now is limited to the scope of linkable models, but luckily that covers almost all of the use-cases where we'd want the news feed to be updated.

A new FeedItem entity is created for each receiver - since the receiver in the current implementation is the scope, the receiver is only one entity, but I'm working on a method to propagate news feed updates like an event bubbling up through a hierarchy.


A news feed is constructed for the receiver entities - such as Program, Organization, Club, etc. - and aggressively cached. The template for the news feed itself is very similar to how blogs are formatted in Melange.

One part of the template has been a big headache - the link redirecting to the sender entity.

Right now, this is what the link HTML template looks like -

href="/{{ item.sender.kind.lower }}/show/{{ item.sender.key.name }}"

Yes, it's hacky. But it works...most of the time. When the sender's kind name is "Document", then this translates to /document/, which is what we want. But when the sender's kind name is "StudentProject", this is rendered as /studentproject/, while it should be /student_project/. I'm still figuring out how to properly linkify an arbitrary entity...and I tried using the redirect views helper, but I was hoping to do something that wouldn't require any additional server-side code.

The next week has a lot of challenging tasks ahead. Now that the basic implementation is working, I'm going to work on automation and testing. I'm also going to code up an ATOM feed that can be used to export news feed updates. (This shouldn't be a security issue, since news feed updates only include very basic information and a link to the entity for further information)

Surveys also need to have their ACL component finished so they can be taken for midterms...I've struggled with this when attempting it before, but GSOC needs this feature working, so I'm going to have to figure out a way to do it.

Finally, I've also updated logic for surveys and documents so that based on the prefix of the document, it creates a scope property. This way, the scope doesn't have to be retrieved at runtime and the document model is more in sync with the rest of Linkable entities.