GSoC 2010 Melange Testing Project: the last week of Community Bonding - starting to code

In this week, I started to add more test cases to tests.app.soc.logic.models.base and tests.app.soc.views.models.base. I first learned the code of the corresponding code base and the existing test cases in more detail in order to understand their functions and logic. Then I tried to design some new test cases. Due to the fact that I am still not very familiar with the implementation detail of Melange and met problems when setting up the coding environment, e.g. could not get the new version of ctags to work well with Vim and Python, the process is a bit slow. However, these preparations have built a good foundation for the official coding stage, which will start next week. Hope that during that period I can code fast and leave more time to improve and clean up my code as well as do some extra work for the project.

GSoC 2010 Data Seeder - Project design

Another week of the community bonding period has gone by as we are getting closer and closer to actually having some code written :D

So here are some of the features that have been decided upon so far:

• The configuration for the seeding operation will be customizable by the user using a web interface which will conclude with the creation of a JSON configuration sheet. This configuration fill will be usable both locally and on-site and can later be edited using the same web interface.
• The configuration sheet is meant to be very easily customizable and extendable. Different scenarios for different needs can be saved in predefined configuration files and then executed at any times. Examples include saved states in different phases of a GSoC program (proposal phase, midterm survey etc.).
• There will be an option to export data to Python fixture files, either directly by the use of a script along with a JSON configuration file, or by saving the state of a running instance.

You can find more details on my wiki page: http://code.google.com/p/soc/wiki/GSoC2010DataSeeder#Design_documents

The script that I have used to generate the diagrams from my previous post has been committed to my Melange clone. You can try it out anytime, no further setup is required. Here it is: http://code.google.com/r/sttwister-gsoc2010/source/browse/scripts/model_graph.py

Week 3: Getting Ready to Code

The community bonding period is drawing to an end this week, which means we begin the actual coding work on our projects starting next week! I am quite excited to start writing code :).

So far, I have finalized most of the design and workflow for the two features I am developing for my project. There are still a few loose ends with regards to design hanging around, which will be tied up soon. My project wiki is now updated with the design details and awaiting the approval of my mentor, Madhu :).

The main additions and modifications to the design this week were:

• Some changes to the user_page.py data model. The multi-valued properties job.py and education.py data models were scrapped to be replaced by a simple biography text field where users can write whatever they would like to share with the community.
• Detailed workflows for Calendars and Maps feature (addition/request new events). Here are the usecase diagrams.
• Both events and user_page data models have a tags property which is basically a comma-separated string with tags related to the user or event(e.g. C#, Java, Django). Users can also be given to the option to subscribe for feeds to be notified of events with certain tags.
• I have also done a detailed timeline, with a list of deliverables for every week up until the mid-term to keep be oriented and directed with aims for every week.

Coming up next week:

• Coding, coding all the way!
• Follow timeline closely to reach the weeks' goals.
• Resolve any bugs/issues, if assigned.

New Document Editor: HTML diffs

One of the goals of my project is to create revision control for Melange documents. Each RC framework must provide some kind of diff engine. The engine should find changes in the document and represent them to the user. It is pretty simple with text, but Melange stores documents as HTML.

Talking about HTML diffs, we should consider two cases:

1. HTML as text. A bunch of tags, attributes, values and content.
2. HTML as a rendered document (e.g. image)

Another question is: what causes representation changes in a rendered HTML document?

1. Tags (through browser's default CSS or applied CSS).
2. Attributes class and id (through applied CSS)
3. Applied CSS (server-side, in-document or in-line).
4. Style modification with DOM.

Let's consider first case - HTML as text. This case seems pretty obvious, cause changes can be tracked with textual diff engines. But it's not that simple. There are several cases of changes to HTML which don't influence representation:

1. Tags are changed, but CSS is the same:
   <h1>Hello, world!</h1>
   is changed to
   <h2>Hello, world!</h2>
   but the CSS is
   h1, h2 {font-size: 12px; font-style: normal;}
2. Some pieces of HTML are rendered the same:
   <div class="alert">Hello, world!</div>
   and
   <div class="alert">
<p>Hello, world!</p>
</div>
3. Class or id is changed but CSS is the same:
   <div class="original"></div>
   is changed to
   <div class="changed"></div>

The second case - HTML as an image. I mean HTML with applied CSS which is displayed to the user. Tracking changes to images is the correct way of handling HTML diffs. It can be performed with several tools. One of them is convert utility from ImageMagick. This approach is, however, a little tricky and CPU consuming.

Let's return to HTML as text. If we can guarantee that CSS represents different tags different and there are no class or id changes, then everything seems right. We can focus on textual diffs for HTML. Considering TinyMCE (which is a defaul editor for Melange) all representation changes are made with tags. If there is no appropriate tag, then the style is applied with <span> tag or a chain of <span> tags.

Textual HTML diffs can be generated by several tools. One of them is HTML diff for Python. I'm now thinking about using it as a skeleton and try to make more intellectual engine with Beautiful Soup

Week 2: Working on Project Design

A second week has gone by and I am happy to say that I am now better acquainted with the Melange code-base. During the week, I managed to create a clone of the source and have now deployed to my very own GAE instance :).

Update on my bug-fixing progress:
I managed to submit my very first patch for issue 890. I also started looking into issue 884, but as it requires some re-design of existing templates, discussion is ongoing as to how this issue can be resolved.

Update on my Project:
As my project is to appear as a new module in the soc.modules section, I am tentatively naming it as social.

I have started on the design work for the ‘Tabbed User Pages’ which is one of the two main features I have proposed.

Data Models:
I have come up with additional data models to contain information on users which will be displayed on their user pages. The proposed additional models are as follows. These are not finalized as discussion is still ongoing as to the inclusion and exclusion of some of them.

soc.modules.social.models.user_page.py
soc.modules.social.models.education.py
soc.modules.social.models.project_details.py

The user_page.py model will serve as the master model which will contain all the attributes of the user model which will appear on the user profile page. The details of education and past and current GHoP/GSoC projects can have multiple values for a given user and hence are extracted out to separate data models by themselves.

Thus far, I have envisioned the User Pages to have a tabbed view with three tabs, the ‘Profile’, ‘Project’ and ‘My Stuff’ tabs. The ‘Profile’ and ‘Project’ tabs are self-explanatory, the ‘My Stuff’ tab will be used for users to show their personal blog feeds using the RSS widget for now. Should more widgets be developed (i.e. Twitter widget J), they can be added in this tab.

Users will be able to edit the data in their user pages using a single form to edit the data shown in all the tabs, using the 'Edit User Page' link. This link will only be visible if the user is viewing their own user page.

Here is a simple use-case diagram to elaborate on the workflow.

Views:

soc.modules.social.views.models.user_page.py
Since the user_page feature serves more or less the same purpose for all the different user roles in Melange, a single view will be created.

Templates:
soc.templates.modules.social.user_page.view_user_page.html
Since the user pages will be tabbed, the entire page will not reload when users switch between tabs. With the help of JavaScript, this can be achieved with a single template with different html div and id tags defining the data to be displayed in different tabs :).

soc.templates.modules.social.user_page.edit_user_page.html
A different template with the html form will be created for the 'Edit User Page' function which will be used to update the user_page.py data model.

Logic:
soc.modules.social.logic.models.user_page.py
To handle the edit/update of the user data a logic is created which will be used to update the user_page.py data model.

In the coming week:

• Check for updates on resolving Issue 884
  
• Finalize the Design for Feature 1 & Feature 2: Maps + Calendars
• Finalize Project Timeline and list code deliverables for mid-term as well as weekly deliverables
• Decide on the specific APIs and libraries to be used for the project.
  

GSoC 2010 Melange Testing Project: Week 1 - learning the code base

It has been a very exciting week. In this week, I made a clone for the testing project, deployed Melange to my GAE instance, got the posting right for melange-dev blog, created a wiki page for the project, signed a Contributor License Agreement (CLA), submitted tax forms, got an ACM account (the courtesy of Google), had the first conference meeting with other developers of Melange, helped some Melange users to solve their log-in problems, and had meetings with my mentor (Sverre). What a week!

My main job in this week, however, was to learn the code base of Melange since my project is to add tests to the code base. In order to help me better understand the code base, I decided to visualize the organization of the code base and the relaionship of modules/classes. Felix' diagrams are very interesting. However, they are only for Melange data models whereas views and logic are more relavant for my project. So, I used a tree tool to generate the module organization of Melange (click the picture below to view the searchable full HTML file)

and module organization of Melange tests (click the picture below to view the searchable full HTML file)

and used pylint.pyreverse to generate the package diagram of Melange (click the picture below to view the searchable full PDF file)

and the class diagram of Melange (click the picture below to view the searchable full PDF file)

These diagrams have helped me a lot to learn the code base. Hope they are also helpful to others who want to learn the code base.

During my meeting with Sverre, he suggested me to start with the testing of the modules which are the most important and used most, e.g. soc.logic.models.base and soc.views.models.base. Therefore, my next week's plan is to start coding, specifically add more test cases to the existing tests of these modules, e.g. tests.app.soc.logic.models.base and tests.app.soc.views.models.base.

Following is a brief introduction to the testing project:

This project is to add unit test suites/cases for the logic and views of the existing modules. These test suites/cases will enable developers to refactor the existing code at any time while at the same time to ensure that they still work correctly by passing all the tests (i.e. regression testing). This project will also serve as a basis for the Melange team to switch the development methodology to test driven development.

Cheers,

Leo (Chong Liu)

GSoC 2010 Data Seeder - Project start off

I'm posting this to let you all know how my first week with the Melange community has been.

Bonding with the community has gone very smoothly, the community is very close together and I love the idea of having weekly conference calls. Sounds like a really fun summer ahead of me and I hope everybody will enjoy it.

The first thing I needed to do for my project was to really understand the data model behind Melange. Since there are quite a few entities involved and keeping in mind that models might perhaps change somewhere in the future, I thought the best way to accomplish this is to build an automated tool that draws diagrams from only the source code. There are two kinds of diagrams I wanted to have: inheritance diagram, showing how classes derive from each other and an entity relationship diagram, showing the relations between entities.

The code that I have produced is inspired from an open source project that does mostly the same thing for Django models. It is part of django-command-extensions.

I hope I can integrate my script with the Melange code base pretty soon, so that anyone can generate graphs from their own copy if anything has changed in the data model.

Here are my results so far:

Inheritance

Inheritance, separated by module

Entity relationship

Entity relationship, with attributes

Entity relationship, separated by module

Some conclusions:

Boy, are there a lot of relations out there!

Personally, I didn't expect to see that many relations between the models. This is definitely going to influence the way I'll design things around, especially the web interface, in order to make it as easy as possible to configure relations in the seeded data.

That's about it for this week, I'll now start actually doing those design documents! :)

Cheers,

sttwister (Felix Kerekes)