Social Features Project: Week 12

The final week of GSoC has rolled in and my blog post is incredibly late too as I spent much of my time working on my code.

My final version is upload on my app engine instance here. It consists of User Pages, Calendars and a Map View. The Calendars feature is functional and works as per the workflow I had originally planned. The past week had some last minute bugs which popped up only when I ran my code on AppEngine as they were not detected while I was testing on local host. The bug was that I was storing a universal session authtoken for the entire application instead of one for each user. This did not show on my own machine as I always used it under my own google account! Hence, I had to spend time to rectify this and ensure that each User model stores the session token and later the services uses the set token when the user decides to use the calendars feature.
This issue has been fixed now. However, I believe it will take a bit more work to integrate it and will have to spend time with my mentor in revising the finer details before actual code integration with the main code base. I look forward to doing this in the week after next.

My latest code is uploaded to my bit bucket account here.

Some problems still awaiting to be solved are nasty server error being cause by issues in the Google Calendars API, which I thought I resolved however, they still seem to pop up randomly. The issue has been taken up by others using the API here, although there does not seem to be a good fix for this yet.

Another thing that needs to be settled is the scalability of the Maps View. As GSoC has about a 1000 students, 1000 mentors and then org_admins etc. Right now, as I am having to manually populate the data, the map view has hardly 40-50 pointers on it. Hence how will the JS handle a large number of objects is to be seen as yet.

Finally, it has been a great experience with loads of learning and coding. It has been great working with the Melange team, and specially my mentor Madhu. :)
I am looking forward to staying with Melange and contributing to the project as best as I can in the future too.

GSoC 2010 Data Seeder: Wrapping up for end of GSoC

So GSoC is coming to an end, it's time to wrap up the project to a pretty much fully functional and usable state, while there could still be much to do. So, my latest changes mostly include usability improvements like being able to test the configuration for a data provider and get some sample values, include provider and parameters descriptions, being able to validate, download or upload a configuration sheet.

Other improvements include some visual styling, although I personally find my work very ugly, I'm hoping someone with better visual appeal might help me with this part in the future.

As a final proof of concept, I will provide a number of configuration sheets that will demonstrate the full power of the module: being able to generate mass amounts of "real world" data very easily (although doing a first-time configuration might be a little time consuming).

I have also encountered some unexpected problems. It seems that GAE (or maybe the Mapper API, I'm not sure) limits the amount of data that a task can receive as a parameter. Configuration sheets can get pretty big (I won't be surprised to see some reach tens of KBs) so I needed to find a workaround. My solution involves storing the actual configuration sheet in a model, and then each task would read the configuration sheet from the datastore. While this solution speeds down the process by quite a bit, it might prove more scalable and also provide some historical value, should the configuration sheets remain in the datastore.

Enough about the technical stuff... It's been a real joy working with everyone around the Melange community and I'm sorry if I haven't been all that serious all the time during GSoC. Many thanks to my mentor Mario who has provided me with priceless information all the time and who has made all my work so far possible. Also, thanks to all the other developers (Lennard, Sverre, Madhusudan and Daniel), you are simply great and it's been a pleasure to work with you all. Cheers to all other GSoC students (Leo, Pavel and Savitha), I hope we'll see each other in the future as well. Having said that, it's time to continue working on the project, there's still a lot of things that can be improved.

Cheers!

GSoC 2010 Melange Testing Project: Week 12 (Final) of the Coding Period

In this week, I have cleaned up, commited and pushed all the tasks testing code to my online clone. I also worked on adding test cases for soc.logic.cleaning with suggestion of Sverre. Besides, I have also uncovered a couple of bugs in the soc.modules.ghop.tasks.task_update module and sent a patch to the mailing list. Lennard was very kind to help me merge the patch. I have also uncovered one issue with my test code for soc.modules.gsoc.tasks.accept_proposals (set the timelimit parameter too small) and then corrected it. In addition, I have also investigated the diference between the console and html coverage report. It seems that the console one counts statements while the html one counts lines.

When I tried to move the the gaetestbed and pymox modules to buildout dependencies, however, I found the easy_installed versions of both modules have problems (easy_install is used during Melange buildout). For gaetestbed, I found that the gaetestbed on PyPi is an old version which cannot check the html body of an email. So, I emailed the author to ask him to put the latest version onto PyPI. The author (JJ Geewax) was very kind to have helped me do it. For pymox, the easy_installed version available is a quite old version (0.5.0) which has a bug fixed in the later versions. Unfortunately, these newer versions are not available on PyPi, so not easy_install searchable. I reported the issue to the project issue tracker. One of its authors (Steve) was also very kind to helped me to put a newer version (0.5.3) to PyPi. After all these problems have been sorted out, there remains one issue that gaetestbed currently depends on webtest even though you do not need this part of function due to the design of gaetestbed, which the author agreed to resolve in the near future. After consulting Sverre, I finally moved the the gaetestbed and pymox modules to buildout dependenciesi and then commited and pushed the change to my online clone.

In the next week, I will add the experience of how to do mail and taskqueue related tests with gaetestbed to the testing wiki page, make a patch for removing the cron/job related modules and then send it to the list as well as add more test cases for existing Melange modules.

This is the last week of GSoC2010. It was a quite exciting and enjoyable summer. My project is very interesting, my mentor (Sverre), other developers (Lennard, Daniel, Madhusudan, Mario) and GSoC administrator (Carol) are very helpful, and my fellow students (Felix, Pavel and Savitha) are also very nice people to work with. Specially thank my wonderful mentor, Sverre, who is always ready to help me out when I got stuck and without whom I may not complete my project. During this period of time, I have also learned a lot about GAE and Melange, I have also come up with something useful and merged to the trunk, and uncovered several issues. Though the program has ended, it does not mean the end of my project or my involement in Melange. On the contrary, I will continue to work on the testing project after GSoC, i.e. add more test cases for more modules, improve the testing design and code, integrate Felix' data seeder, and so on, which have not been done because of the time limit of GSoC or dependency. I also hope to work for Melange beyond the testing project and hopefully become one core developer of Melange in the future. I also wish to come up some ideas to improve Melange so that I can serve as a mentor for Melange in the next GSoC and experience the excitement of these guys about the upcoming mentor summit :-).

Document Editor: Last week of GSoC coding

Last week of GSoC coding was devoted to... everything :)

I spent 4 days on form saving framework. First of all I had to fix JS code, which sends dirty fields to the server. The problem was as follows: framework was saving only dirty fields of the form. If you loaded the form for the second time, initial state might have been rewritten. This issue was fixed and now form states are consistent. Then I've implemented back-end. I needed to create model which stores temporary fields as a JSON dumped dict. Some logic class which manipulates all the stuff has been implemented. I also had to change createGet, createPost, editGet and editPost methods of the soc.views.models.base.View class to support temporary form saving. Temporary form saving for the model is enabled via "form_autosave" key of the params dict.

After form saving I started testing the code. An issue happened immediately. There was a problem with home documents for different models which derive from soc.models.presence.Presence. Views and logic were not aware of the revision control for the document, so exceptions were raised. This issue is fixed now. I also had to fix soc.models.seed_db.seed function to seed documents in the right way.

I've also refactored diff template, so it can now show arbitrary number of diffs for different revisable fields of the model.

At last I worked at the UI. I've added "Show revisions" button on the edit page for the model. Models that support RC are now distinguished by the "revisable" key of the params dict.

I've also rearranged fields of the Document and StudentProposal forms, so they look as supposed.

Final version of my GSoC contribution was uploaded to the appspot instance http://melange-editor.appspot.com

GSoC coding has come to it's end. And I must admit that it was quite an experience. I've already participated in large software projects and earn my living with development, but GSoC is really memorable. It was cool to participate in such a big and complex open-source project! I want to thank Melange dev team, Carol, fellow GSoCers and my mentor Daniel. They were very helpful and opened. And, you know what? I wish they let me continue contributing :) Come on, guys, I love it!

Document Editor: Week 11

The goals for the last week were to finish RC for StudentProposal model and start working on forms auto-saving. Revision control for StudentProposal is fully functional now. I also made some changes to decouple RC from the model's code.

After that I started working on form auto-saving. The main architectural concern was "what to store". The decision was made to store only changed fields of the form. Then I tried to find some existing solution to figure out dirty form fields. Mario proposed dirtyFields plugin for jQuery. It looked cool, but I found out that it triggers user-defined event for each field change. Furthermore it doesn't provide any means to retrieve collection of changed fields with new values. That's why I decided to develop my own script based on dirtyFields. It's now ready and pushed to my repository clone.

This week I will finish auto-saving for forms, write tests and make code clean-up.

Week 11: Social Features Project

This week I spent mostly on doing the Map View which basically involved writing mostly JS code to retrieve and then filter data based on search terms that are entered in the text input box. Initially, I had planned to allow users to filter based on location, projects etc, later I decided to just let them enter keywords. Internally, I check this keyword against tags as well as locations of users and events to filter.

This way, users do not have to specify what kind of search they want, they can just type any keyword such as 'Python' or 'Australia' to get the relevant filters.

The Google Maps API gave me a bit of problem as it took sometime and debugging to realize some method names have changed completely such as for adding/removing markers etc between V2 and V3, and I had to accordingly change my code.

The latest version with all three features is on my instance.

I have quite a bit of things left to do in the final week of GSoC:

• write tests
  
• write code to catch the exceptions(Application Error) being thrown by Calendars API and re-send requests.
• test the functionality and workflow of user pages, calendars and maps view for bugs and fix them.
  

GSoC 2010 Melange Testing Project: Week 11 of the Coding Period

In this week, I have uncovered a couple of bugs of existing Melange modules with the help of related test cases and sent pathes to the list, which again proves the importance of testing. I have also resolved the error "raise self.__TranslateError(e.application_error, e.error_detail) UnknownQueueError" by adding the path of taskqueue yaml setting file to the instantiation arguments list (root_path) of taskqueue_stub.TaskQueueServiceStub. Besides, I was also looking for a good way to test the mailing and taskqueueing functions in order to test that the correct mail has been sent out and the correct task has been scheduled. During my search, I found gaetestbed which is very simple but serves the purpose well. Although it is not hard to make an equivalent one myself, why not reinvent the wheel if you can make use of an existing one. Its licence is also very generous (GNU V2). So, I added it to Melange tests and also extended its MailTestCase and TaskQueueTestCase and then put them into tests.util module. For MailTestCase, I overrided its assertEmailSent method to add two functions

1. It will print out all sent messages to facilitate debug in case of failure.
2. It accepts an optional argument n which is used to assert exactly n messages satisfying the criteria are sent out.

Later, I plan to add the functions of checking the CC field, allowing to partly match the subject field or even supporting regular expression. For TaskQueueTestCase, I have not added any functions yet. I further used them in my mail and taskqueue related tests, and cleaned up some of my existing testing code and then commited and pushed them to my online clone. In addition, I have also done some new tests on gsoc tasks and statistics etc. and managed to increase the testing coverage of Melange to 59% and the coverage of almost all modules except one (with 39%) over 40%. In fact, the coverage report of the html version which apparently uses a different algorithm from the console one shows the overal testing coverage is already over 70%.

In the next week (the last week of GSoC2010), I will clean up code and commit and push all uncommited testing code, increase the overall coverage printed in console to over 60%, and make the coverage of most modules over 50%. In addition, I will also move the the gaetestbed and pymox modules to buildout dependency and add the experience of how to do mail and taskqueue related tests with gaetestbed to the testing wiki page.

GSoC 2010 Melange Testing Project: Week 10 of the Coding Period

In this week, I worked on some ghop modules with low coverage, e.g. soc.modules.ghop.tasks.task_update and soc.modules.ghop.logic.models.task and managed to increase the testing coverage of Melange to 57%. About the problem that there is no api proxy stub available for service "taskqueue", I found that there is one taskqueue_stub in google.appengine.api.labs.taskqueue, so I registered it to 'taskqueue' service with apiproxy_stub_map.apiproxy.RegisterStub in the testing environment. However, it (taskqueue.py) raises another error, "raise self.__TranslateError(e.application_error, e.error_detail) UnknownQueueError" which is a bit strange. This is the reason why I have not yet checked in the task testing code.

In the next week, in addition to solve the above error and increase the testing coverage of Melange to over 60%, I will also try to find a good way to test the mailing and taskqueueing functions and make sure the correct mail has been sent out and the correct task has been scheduled.

Document Editor: Week 10

As I've already mentioned in the previous post, there was a problem with versioning more then one property of the model. Daniel proposed to store necessary fields as a JSON-formatted dictionary. On tuesday I reimplemented RevisionContent model and several logic methods to support new architecture. Document model, logic and view were refactored too.

Then I started to implement revision control for StudentProposal model. There were several minor issues, but everything is fine now. Revision control for student proposal is almost fully functional now (diff template and delete logic are left). New version of Melange was uploaded to my GAE instance.

On tuesday I plan to finish revisions for student proposals and start implementing auto-saving feature.

Week 10: Social Features Project

This week I spent on integrating CalVis calendars UI with the current Calendars features as well as adding the maps functionality. The new calendars UI can be seen here. Clicking on events brings out a pop-up widget with event details. The Calendars feature is done. However, the two errors being thrown the by the Data apis earlier posted to the dev list still seem to persist. :( Need to spend some time and also consult Madhu to find a way around this.

I will be starting on the final Maps feature/view this weekend and hoping that this feature will not take long to code as it does not involve much creation/manipulation of data models, it mainly involves querying the data (user pages and events) and updating the map.

In the coming week:

• start work on Maps feature
• test the current code, esp. UI to check in case of any bugs

GSoC 2010 Data Seeder - Seeding using MapReduce framework

The seeder is currently fully functional running on a live instance and hopefully should be able to seed any provided configuration sheet. The final implementation uses a mapreduce framework[0] for GAE and is highly parallelizable.

I also did some benchmarking on the cloud. I tried seeding 10,100 simple Linkable entities, including a string and a reference property. Running on a 50 tasks / second and having the mapreduce framework set to process 100 tasks / second (default value) finished successfully after around 1 min 40 seconds. That's pretty fast in my opinion. This resulted in about 0.45 CPU hours being consumed in the cloud. This was possible because mapreduce runs multiple map operation in a single task. Perhaps even better speed/efficiency can be used by using mutation pools to do batch updates in the datastore.

The way the current solution is implemented is by starting a new mapreduce job for related models (back-referencing). I believe this is also faster and easier to implement, the downside being that a lot of jobs appear in the status interface. I think this can be fixed by using some sort of smart sharding (already supported in mapreduce) instead of starting new jobs, I'll look into it.

So now that the seeder is working, it's time to get back to working on the web interface. I will add the ability to import an existing configuration sheet, better navigation options and better control when configuring references (like being able to specify a subclass of a specific model etc.) during the next week. If time permits, I'll also try to do add some styling.

[0] http://code.google.com/p/appengine-mapreduce/

Week 9: Social Features Project

The past week of GSoC was spent mostly on winding up the Calendars features, such as finalizing workflow and using the JS GMaps package in Melange for specifying event locations. This is primarily used so that it can later be used in the Maps widget provided by CalVis Calendar UI. The latest is at my GAE.

I am currently working on the CSS designs for CalVis UI, and hoping to integrate the design and UI into my GAE by today or tomorrow at the latest.
The rest of the current week, I plan to spend on testing the Calendars and User Pages feature and also starting work on the Google Maps view/feature for filtering users and events based on users/event location and tags.

I lost the last weekend as I had to make a trip to India for some family obligations, hence the slowness in progress.

In the next week:

• Complete CSS design for CalVis and integrate with current feature
• Write and test calendars feature
• Get started on Maps view/feature for users and events
  

Document Editor: Week 9

I spent Tuesday on fixing code issues which were pointed by Daniel. Then I started working on revisions for the student proposal. It was pretty simple, but there was a problem. StudentProposal model contains abstract text property, which in my opinion should also be revisable. But current implementation of the RevisionInfo and RevisionContent classes are not able to store two content fields. This issue is to be resolved.

At the end of the week Mario implemented client-side redirects for the lists framework. I used this feature to re-implement "Show diff" button, Now it looks and behaves right. New version of the "Manage revisions" view was uploaded to my GAE instance.

GSoC 2010 Melange Testing Project: Week 9 of the Coding Period

In this week, I mainly worked on low coverage modules, e.g. soc.tasks.surveys, soc.tasks.grading_survey_group and etc. With these work, I have managed to increase the testing coverage of Melange to over 55%. I am cleaning up the code and will commit and push them today or tomorrow at the latest. One problem is that there is no api proxy stub available for service "taskqueue" yet, so it raises an error "ERROR:root:No api proxy found for service "taskqueue". I am also working on it.

In the next two weeks, I aim to increase the testing coverage of Melange to over 60% and increase the testing coverage of those modules below 50% (40% at least) to over 50%.

GSoC 2010 Data Seeder: Web Interface

The web interface should be pretty much functional right now. It's very ugly, I admit, I just added some expand/collapse tree view functionality for easier navigation, but it's just temporary. More user-friendly navigation features and some fancier display is scheduled for the future, but right now it works for defining any configuration sheet as was planned in the initial design and it will stay this way until the server-side code is fully functional. You can check it out by visiting my GAE instance: http://sttwister-melange.appspot.com/seeder/home.

Make sure to force reload the page in your browser so that it fetches the most recent JavaScript files.

For the weekend, I'm planning to add tests for the server-side views that have been implemented so far and then try to keep up with the test coverage in the future. After that, I will start fixing the server-side seeder code to make sure it works correctly in all cases and then port it to GAE Task Queue API.

That's it for now.

Document Editor: Week 8

The main goal for the last week was to finish revision control for Document model. I had four tasks to complete:

1. Deploy HTML diff engine into Melange
2. Create view that shows diff between two revisions
3. Update "Manage revisions" view to support diffs
4. Code clean

All these tasks were completed successfully. HtmlDiff class from difflib was used as diff engine. I've actually tried several python diff engines from this page, but none of them appeared to work stable and fast.

I wanted URLpattern for "show diff" to be used as API. That's why I wanted it to be "pretty". But there was a problem with jqGrid buttons. Neither "redirect_simple", nor "redirect_custom" methods can create URLs with two different parameters. That's why I had to create separate button and trigger onClick event for it.

Full-featured revision control for Document model was uploaded to my appspot instance http://melange-editor.appspot.com

GSoC 2010 Melange Testing Project: Week 8 of the Coding Period

With the suggestion of Daniel, I first tried Google Wave in this week. It seems very interesting that everyone can contribute/add anywhere and edit posts. Then, I have further put this week's meeting notes on it and also added our Melange group to the participant list so that all members are able to view, add and edit the waves.

Then I finished adding test cases to GSoC organization view, but I noticed unfortunately that it had not been run by nose tests. After some investigation, I found that the problem was because I had not added __init__.py in the GSoC test packages. Then, I further worked on other view tests. This week, however, I have not achieved my original target, i.e. increase the overall code coverage to over 60%. I will work hard to make it this week.

Week 8: Social Features Project

This week went by in doing more work with Calendars, such as sorting out the issue on the User Authorization for Google data apis.
Earlier in the week, I implemented the workflow of Calendars feature with fuctionality such as:

• creation/edit of events by users for chosen Calendars.
• the addition of org_admins userid's to the Access Control Lists of their org's Google Calendar
  
• sending notifications to Calendar owners with event request message and consequent approval by org_admin was also implemented.
• uplading of approved events into the respective Google Calendar
  

The latest is at my GAE. All users with user roles and user pages will be able to see the "Add Event", "Events" and "List Event" options. "Add Event" will result in a notification being sent to the respective owner of the Calendar for approval, hence one may not be able to see the event pop up directly on the respective Google Calendar.

After some discussions with my mentor, it was decided that it would be an acceptable trade-off to use the CalVis API to display Calendars. As there was no very definetive decision reached in the dev-list discussion, at the same time, there was also not a strong opposition to CalVis, my mentor and I decided to go with the CalVis. The UI of CalVis can be manipulated with CSS to a decent extent and hence, it can be designed to look better. Although it may never look as good as Google's :-(, it is a trade-off we get for having the cool Google Maps plus Youtube widget pop-up on Calvis. I am currently working on the CSS designs for the Calendars as well as the User Pages(i was supposed to have done this a while back!).

For the next week:

• prepare cool designs for the Calendars UI and User pages
• find out how and write cron jobs for keeping Melange-side and Google Calendar-side events in sync
• spend some time to refactor code to ensure it sticks to Melange styles and conventions
  

GSoC 2010 Data Seeder - Client-side HTML rendering

I have been much more busy than usual lately and it's very clear that this has affected my time spent developing for Melange. I'm pretty much behind schedule right now and I'll have to try a lot harder in the future to make up for the lost time.

I have been facing some problems choosing a right JavaScript templating engine for rendering HTML. The solution that I was going to adopt was actually a jQuery plugin, but had some problems rendering some templates causing an infinite loop in the browser. Failing to find a proper solution for quite some time, I have decided to switch to pure. I was preferring tempest at first because it was using the Django templating format, but now I see the benefits that the pure separation between templates, directive and data offers (a directive is a structure that binds data to the templates).

The switch to pure is now complete and you can check it out at http://sttwister-melange.appspot.com/seeder/home

There's support for seeding models and defining providers for all properties. There's limited support for references at this time, only being able to bind a model to another random model.

See you next time!

Document Editor: Week 7

Last week I finished implementing "revert to revision" action. Now, if you list revisions for the entity, you'll notice check boxes and a button "revert to revision". This button is activated when only one revision is checked. If you check some revision and press this button, back-end will delete all the subsequent revisions and will make checked revision current.

I also started working on the diff engine for revisions, but it is still work-in-progress.

In fact, I have to implement two buttons for revisions list: "revert to revision" and "show diff". I didn't manage to configure two different buttons for jqgrid, but Mario helped me a lot. I was able to understand the meaning of the "bounds" list and everything is perfect now. Thanks Mario!

GSoC 2010 Melange Testing Project: Week 7 of the Coding Period

Last week, I found that Python coverage does not track the properties definition of Melange models which makes the coverage of models still quite low. So, I first tried to exclude models from coverage report. This is done by stubbing out nose.plugins.cover.Coverage.wantModuleCoverage with a method which offers the option to skip packages for testing coverage report through pymox stubout. Later, however, I remembered that coverage should be able to track class variables and module variables. I reckoned that the problem may be that these Melange models have been loaded before the coverage starts tracking. So, I further used pymox stubout to stub out nose.plugins.cover.Coverage.begin so that it loads Melange after the testing coverage starts. It works! All the class variables and module variables of Melange models, logic and views can be tracked by coverage and the code coverage now is over 50%.

Then I further worked on various views, e.g. sponsor, program, student, organization, etc and aim to increase the overall code coverage to over 60% next week.

Week 7: Social Features Project

This week I worked on Calendars feature. I created data models for events to store event entities as they are created as well as simultaneously created event entities on the selected Google Calendar. Also I created a template to view all the Calendars(program as well as org using a drop down box to select).

I am currently working on the program_host "activating" calendars for all organizations once the accepted organizations are announced.Org_admins will in turn choose to "activate" their respective calendars after which their user ids will be added to their org calendar with "owner" rights. I am hoping to have a working version of this feature by the end of the weekend.

In the next week:

• finish rest of Calendars feature (requests to add events)
• Testing and revising of User Pages
  

Document Editor: Week 6

Last week I was implementing views to support revision management.

First of all there is a view that renders document revision with the given number. For example, if you want to view second revision of the "user/parser/test1" document, you should visit http://melange-editor.appspot.com/document/revision/user/parser/test1/2. Permissions and range control now work properly.

The second goal was to implement revision management view. It is not finished yet. You can get list of all the revisions for current document via the "revisions" link. For example, all revisions for "user/parser/test1" can be viewed here

http://melange-editor.appspot.com/document/revisions/user/parser/test1

This view is a draft. I need to make several fixes, add buttons, checkboxes and behavior. It was pretty hard for me to get a grip of the new lists in Melange. But Sverre and Mario helped me and I hope to finish this view in a day or two.

GSoC 2010 Melange Testing Project: Week 6 of the Coding Period

In this week, I first tried to add test coverage to Melange models, i.e. did model tests. However, I found that Python coverage does not track the properties definition of models which makes the coverage of models still quite low. When I discussed with Sverre, he suggested me to exclude models in the coverage report and start to test other modules with lowest coverage. After rechecking the coverage report, I decided to do view tests for sponsor, org, org_admin, mentors, student and so on.

Then my hard drive was corrupt:(. Fortunately, I have the online clone on Google servers, so I have not lost much work. It just took me some time to reinstall the operating system and resetup the environments. Due to the problem, I have only finished most parts of the sponsor view tests and some parts of org and student tests. Next week, I will definitely work harder to catch up.

GSoC 2010 Data Seeder - Model references in the UI

This week is all about beginning to handle references. Right now, direct and back references information is included in the JSON data, and a random reference data provider has been implemented and tested. It's also working!

While working to add easy manipulation of references in the web UI, the current local JavaScript templating solution (John Resig's microtemplating) has proven to be a bit hard to maintain when getting to more advanced manipulation. So, I'm currently working on finding and integrating different templating solution. I'm investigating pure [0] and jQuery tempest plugin [1].

[0] http://beebole.com/pure/

[1] http://plugins.jquery.com/project/tempest

Week 6: Social Features Project

This week was productive as I finally moved on to working on the next feature, which is Calendars + Events for Melange. I am using Google Data APIs for this, particularly the Calendars API. It is new and exciting, and I am also creating a new events data model in Melange to store the local copies of the events created by various users as well as for making requests to add events.

I am hoping to upload a bare essentials version of "adding events" into my GAE by the end of the weekend.

The User Pages feature is done minus "image upload" which is in the process of being resolved with Madhu's help :). Also, for the "add another job" and "add another education", I created additional data models education.py and job.py for storing this for each user. I am expecting there will some more changes to make here and there based on everyone's feedback.

The latest has been uploaded to my GAE. The brief workflow which I also posted in an email to the list is as follows:

Student Roles:

1. Log In
2. Apply for a Student role by registering as a student.
3. Upon completion of registeration, click the read "Create User Page" option on the sidebar.
4. Fill in form.
5. Click on 'View User Page to see how user page looks. or Browse User pages by clickin on "List User Pages" on the sidebar.

Mentor/OrgAdmin Roles:

1. Log In
2. Apply for a Mentor/OrgAdmin roles.
3. Upon being accepted as mentor/orgadmin, click the "Create User Page" option on the sidebar.
4. Fill in form.
5. Click on 'View User Page to see how user page looks. or Browse User pages by clickin on "List User Pages" on the sidebar.

The Week Ahead:

• Develop Calendars+events feature upto near-completion
• Revise User Pages based on feedback from development team
• Improve the datamodel structures of jobs and education to create them as App Engine entity groups with the User Page as parent.
  

Social Feature Project: Coding Week 5

This week saw a lot of delays and lost time for me. I got stuck with figuring out how to handle the images (profile picture) for the User Pages. I followed the direction given at here for uploading/resizing the images. Also, it took me a lot of time on this as I did not have an "image upload/resize" example already in the code-base to refer to. However, the upload does not seem to happen as I get a NotImageError. More on this on a mail to the dev list.

After that I integrated the Tags property for User Pages, so that users may search for other users based on user tags. After this, I made the remaining modifications based on comments given in the previous conference call. "Add Job" and "Add Education" required more effort as they had to be their own entities and would share a many:1 relationship with each User Page entity.

Last but not least, as the weekend approached I could not access any of the "appspot.com" web pages, especially my own instance. Hence, I was not able to upload and test my latest changes on the live instance. I tried to go through a proxy, but always got stuck after the "Google Account Login" as it would not redirect to Melange after login. However, this is fine now and I am able to access my instance successfully and will be uploading the latest to my instance soon and sending a mail to the dev list so that everyone can try creating their own user pages.

For this week, I will be working very hard to catch up with my schedule which says I should be working on Calendars feature already. I intend to wrap up User Page (minus images) by tonight so that I will get to start on Calendars from tomorrow(wednesday).

GSoC 2010 Melange Testing Project: Week 5 of the Coding Period

As I had planned last week, I added content to the wiki page GettingStartedOnTesting [0] with the aim to provide a guideline on writing tests for Melange this week. Sverre has had a look at it and thinks that it should be extended, though it looks good.

I have also added test cases for a couple of logic modules this week and am currently working on testing the views of sponsor, program and student. Sverre thought the test coverage is still low and suggests me to focus the modules with low coverage next week, e.g. do some tests on the models. So, I will move my focus on testing the models next week.

[0] http://code.google.com/p/soc/wiki/GettingStartedOnTesting

Document Editor: Week 5

According to the plan, I had to finish revision control this week. But I'm a little behind the schedule now.

Last week I fixed issues with document listing. It was a pretty tough stuff to implement, cause JSON-based list view supports field prefetching. But it works now and fixed instance has been deployed to appspot. I also started implementation of the view which helps to work with revisions (revert and show diff), but it is still in progress.

This week I plan to finish revision control and start working on diffs.

GSoC 2010 Data Seeder - Web Interface

A very, very, veeery early form of the web interface is ready. It's very ugly in the current form, I haven't applied any styling to it yet, but at least it's functional. You can check it out here.

One problem that's pretty obvious at first is the space management. For most useful models, there are a lot of properties which take up a lot of space, but only a few of them might actually be used. My solution for this would be to compact them into collapsible/foldable panels so that properties that are not useful and properties that have already been configured don't take up space. Expect this to change in the future.

The web interface currently produces a valid JSON configuration sheet, only including model and properties configuration (without references) and also has the ability to send it to the server backend, but that's it for now. If you want to take a look at the resulting JSON, there's a dump in the JavaScript debug console.

Stay tuned for more in the future!

GSoC 2010 Melange Testing Project: Week 4 of the Coding Period

In the last week, I further added more test cases to the user view test to test the views for administrator users. In order to test the operations of administrator users, an XSRF middleware token has to be manually generated and added to the post data so that the operation can pass authentication. Then, I added more test cases to the user logic test. This week, I will try to add test cases for more logic and view pairs.

One big thing for Melange next month is that each commit will be accompanied with tests. In order to contribute to the movement, I am preparing for a wiki page explaining the current testing environment (e.g. datastore), test runners (e.g. how to only run a single test case), available tools (e.g. pymox and Django test client), utilities (e.g. StuboutHelper), and so on.

Document Editor: Week 4

I planned to implement revision control for Document model this week. New revision is now created on document creation. Revision creation on edit and view that shows all the revisions for current document are left.

To implement revision control for document I had to do the following stuff:

1. Implement Revision, RevisionInfo and RevisionContent classes in soc.models.revision module
2. Implement Logic class for revisions in soc.logic.models.revision. It overrides soc.logic.models.base.Logic.updateOrCreateFromFields and defines method to create new revision.
3. Change forms configuration in soc.views.models.document.View

It was pretty hard to figure out all the dependencies between classes and methods in Melange. I didn't want to break existing code base, inheritance is so useful here :)

UPD: New version of Melange that supports revisions is available at appspot.

Week 4: Final touches to User Pages

It is going to be almost a month since GSoC has started. I have made decent progress. The almost-finished version of User-Page is now at http://melange-social.appspot.com/.

I have created a dummy program,org,org_admin,mentor, a couple of students. I am still working on adding the "past projects" data for some of my dummy students and mentor roles so everyone can take a look soon.

I have enabled mentor sign up and student sign up currently. So everyone can sign up and meddle around with it. There are bound to be bugs turning up, please let me know if you find something weird happening. Also, suggestions and feedback is welcome.

I am facing a teeny weeny delay with the User tags and profile_picture uploads which will be integrated into my GAE over the weekend.

Over the weekend:

• wind up remaining functionality issues
• do some testing

Next Week:

• Start work on the Calendars
  

GSoC 2010 Data Seeder - Views

My project drifted away a bit from the original timeline due to some unforeseen complications, so I'm now almost a week late. Melange has been built from the ground up heavily supporting the View-Logic-Model layers and in this way reusing a lot of code. However, so far, all views and logic classes have been tied up to a specific entity which dictates all this behavior.

Unfortunately, the data seeder project doesn't involve any specific model in order to build functionality around it, but rather it collects data regarding all other models in the application. So there's now a need for a new type of view that is not centered around any model. Initially, I thought that setting up a new general-purpose view is as simple as registering some patterns and some methods, but this doesn't seem to be the case. Instead, some of the code in soc.views.models.base had to be adapted to a new type of view (soc.views.base) that is not model-centric. I have used this new type of view to create the backend side of the AJAX API implementation. An example of the JSON data that is collected by the backend can be seen here [0]. The current version might seem a bit too big (mainly because the list of possible countriy choices is repeated for several models), but that might change.

Next stop: Build a basic web interface

[0] http://sttwister-melange.appspot.com/seeder/get_data

GSoC 2010 Melange Testing Project: Week 3 of the Coding Period

Last week, I finally got Django test client working with the help of Sverre (my mentor). In this week, however, another problem arose. For some urls, e.g. /cron/poke, the response returned by the test client is a Django http response object; for other urls, e.g. /user/show/a_user, however, the response is a dict. After failing to figure out what had caused the problem, I asked Sverre for help again. Sverre suggested that it might be due to the stubout code (pymox) in tests/app/soc/views/models/test_base.py. After I commented the code, however, the problem still did not go away. Later, Sverre found that the problem was due to that the fact that the stubout in tests/app/soc/modules/ghop/views/models/test_task.py has not been unset. So, he added the unset code to tearDown methods and also helped me reviewed/improved and merged my code of refactoring the stubout code into a helper class. Thank you very much, Sverre!

After solving these problems, I started to do view tests for soc.views.models.user using Django test client. I have almost finished it. Yesterday, I pushed the code into my online clone. Today, I asked Sverre to have a quick look to see if it is alright and he was satisfied with it. Though I am still behind my original schedule, I am confident that I can catch up soon. This is because I have solved all major obstructions and have finished adding tests for one logic (base) and one view (user) so that I can follow similar patterns to add tests for other logic and views very quickly.

This week, I also came across the trace module of python when I tried to find another problem I met. It can be used either in command line with python -m option or in your program. This module save you time to code a call back function for sys.settrace by offering some common functions. One useful function is to trace the execution flow of your program. It traces every statement, so be aware the output could be quite large. For example, the output of tracing a single statement of Django test client in Melange testing environment is more than 100MB. Another disadvantage is that the output only shows the module name without package path which could confuse you when you have many modules with the same name or you have a big project and do not know the package of every module. Another useful function is to trace calls. The output of this one is quite small compared with the first function and also has full package path. This function, however, is not perfect either. The problem is that the calling functions are ordered by their module name rather than by their calling order which is arguably more useful. And you have no options to change the order. Besides, you can only use it in command line.

Document Editor: Week 3

This week I worked on revision implementation. As proposed by Lennard I tried to refactor soc.models.work.Work, but found out that Work is also the base class for Quiz, Proposal, Documentation, Question and Survey. Daniel also proposed to implement revisions as three different models. That's why I decided to leave Work for models that doesn't support revisions and implement separate revision-related models.

In a couple of days I plan to implement logic for revisable models, refactor create function and implement view to list all the revisions.

Coding Week 3

A third week of coding has gone by. The User Page status is at an "almost-there" status. The latest is uploaded to my GAE instance.
To have a user page, all users must have role assigned to them such as student,mentor,org_admin or program_admin. Only then do they get the option to Create a User Page on their sidebar. On the current instance, all users can Register as Student and then go ahead to create the User Pages.
However, there are still some things left to wind up with regards to User Pages, before I move onto extensive testing of this feature and then move on Calendars, which I have to start this week according to my timeline!

Things left to wind up:

• listing all User Pages
• Pulling present/past project data for student/mentors/org_admins
• getting images api to upload profile picture(currently using blobstore to upload, but the picture doesn't seem to show up)
• polishing CSS and layout of User pages

Upcoming Week:

• Finish up with User Pages feature and start on testing.
• Start coding work on Calendars
  

GSoC 2010 Melange Testing Project: Week 2 of the Coding Period

In this week, I had planned to add test cases for app.soc.views.models.base.View as well as app.soc.logic.models.User.Logic and app.soc.views.models.User.View, app.soc.logic.models.Role.Logic and app.soc.views.models.Role.View.

As I mentioned in my project proposal, I want to use Django test client which acts as a dummy Web browser to test views of Melange because of its ease of use and good integration with Django (no surprise since it was developed by the Django team). So, I first tried to get Django test client working.

After many attempts, however, I still could not make it work. Then, I found that I could not even run Melange locally suddenly due to the error (google.appengine.api.yaml_errors.EmptyConfigurationFile). I first tired to solve the problem by checking the tracktrace and code. Unfortunately, however, I could not find the problem. I thought that I might have accidentally changed or removed some important files. So, I rebuilt my local Melange but it still did not work. I also tried to re-clone and rebuild another Melange, but it was still in a vain. At that time, I realized that I had to fall back on my last resort, asking my mentor Sverre :-) . Awesomely, Sverre found that the problem was due to the corrupted .appcfg_nag file and he then solved the problem by removing it.

After this problem was solved, I tried the test client again. However, it still did not work. So, I asked Sverre again and found that it is because I had used the trailing slash at the end of the url following Django's tradition while Melange uses a url pattern without the trailing slash. Finally, Django test client works!!! Hope I had asked Sverre earlier :-).

Because of these problems I encountered, I have not finished this week's work as planned. Fortunately, however, I have got Django test client working. In the next week, I will try to speed up my work, finish this week's work as soon as possible and then start next week's work load.

GSoC 2010 Data Seeder - Data providers

This week was all about implementing data providers. I haven't got much time to work this week, it was the last week of my university semester and I had a lot of unfinished work to... finish, and also had to prepare for a Google internship interview.

What I've done so far includes the base classes for data providers and implementations for string data providers. Other providers should be easier to implement now that I have a functioning base, I hope I can finish them over the weekend as I've got a bit more time on my hands.

After an IRC meeting with Mario, we have decided that the AJAX API needs some refactoring which I will update soon.

Coding Week 2

The week 2 of coding has gone by. And I have made progress with my code.
I have written the View and templates for the User Page with most of the functions such as "view_page", "edit_userpage", "create_userpage", "view_calendar" etc.

The prototype of the User Pages, Calendars up on my GAE, plus I have created some forms using Melange's inbuilt forms framework for editing data in the User Pages using the base.py View methods such as create, edit, editPost, editGet, etc.

There are some issues with the creation/saving of the forms and entity data which still need to be resolved. Aiming to resolve it before the weekend is over.

Finally, I need to incorporate the ListView for User Page entities using the list functions in Melange's base.py View framework.

Next week:

• I should have completed User Page functionality, and should have my code reviewed by my mentor, before pushing the changes to my clone.
• I should update project wiki with a complete timeline, taking into account my current progress on the project.
• I should start working on the Calendars feature.
  

New Document Editor. First week of coding.

Strangely, there were not much coding this week :) All I've created was same kind of "Hello, world" app for Django-GAE. But now I got a good idea about how these things actually work together.

Most of the time was spent on studying the GAE datastore. I had to get a firm grip on this topic before hacking out Melange's models. First of all, I've looked through a wonderful book on GAE. The architecture of the datastore seemed quite unusual compared to the Django ORM. But it definitely has some cool features like associativity and Expando (sounds great!) class.

My main concern is the architecture of the revision control for Melange. There are two main approaches:

1. Add fields that store revision information to the existing models and convert the whole bunch of existing documents.
2. Create model that stores information about revisions and link existing entities to it.

I hope this issue to be resolved soon for me to start actual coding.

P.S. Here is the thread in the development list.

GSoC 2010 Data Seeder - First week of coding

So, coding for Google Summer of Code 2010 has officially started. This week I have acquainted a bit more with the codebase and I have successfully managed to set up Eclipse and all the necessary related tools (pylint, code completion, Google App Engine integration etc.), so coding should go smoothly from this point on.

After discovering a small problem with the new GAE version 1.3.4 in the Mercurial repository, I have actually made my first commit related to the data seeder module, yey :D.

Next, I should update my proposed timeline in the wiki to include the second part of the project, and then I'll try to stick to the plan :).

A draft specification of the AJAX communication API between the web-interface and the server-side data seeder module should be ready by the Monday conference call.

GSoC 2010 Melange Testing Project: Week 1 of the Coding Period

The Coding Period of GSoC 2010 has officially started! In this week, I continued to add more test cases for the app.soc.logic.models.base.Logic.getForFields() method, started to commit and push the code to my clone. On this weekend, I will start to add test cases for other methods of the class and hope to complete it before next Monday. Another task on this weekend is to add a tentative project timeline to my project wiki, which becomes clearer after I discussed with Sverre tonight. Next week, I hope to complete the job of adding test cases for app.soc.views.models.base.View and some other pairs of logic and view, e.g. role and user. Anyway, time flies. Therefore I need to fly as well so that I can catch it or at least not fall far behind :-).

In this week, I also came across a nice book titled Programming Google App Engine from Dan Sanderson who has excellently explained the difference between GAE runtime environment and traditional web hosting or self-managed servers, the difference between GAE datastore and traditional relational databases, and pointed out the special consideration and strategies for building a scalable, responsive and optimized app upon GAE due to the difference. In addition, the book also covers basic things about GAE, e.g. how to set it up and how to build an app step by step as well as more advanced tools, e.g. the URL Fetch service, the Mail service and the XMPP service. So, it is highly recommonded for both beginners and intermediate users who want to make best use of GAE.

Week 4: Code-base and Coding

The first official week of GSoC has gone by, and I have officially started coding. I have started off with the User Pages feature. I created a directory structure as my project is to appear as a module in the soc.modules folder. I started coding the data model(soc.modules.social.user_page) and View(soc.modules.social.views.models.user_page) for the User Page. However, I realized I need to get a better grip on the code-base and how things work with regards to code before I can go ahead with the actual coding for my project. Hence, over the weekend and the week after, I am going to acquaint myself with the code-base and get as best an understanding as possible and then proceed on with the actual coding of completing the View and then the templates etc.

This of course, sets me behind time with regards to my timeline :-(
However, I have decided to put in extra time and work extra hard this weekend and next week to strive and catch up with my User Page feature so that by my next post, I will have progressed to having a better, working User Page feature :-).

GSoC 2010 Melange Testing Project: the last week of Community Bonding - starting to code

In this week, I started to add more test cases to tests.app.soc.logic.models.base and tests.app.soc.views.models.base. I first learned the code of the corresponding code base and the existing test cases in more detail in order to understand their functions and logic. Then I tried to design some new test cases. Due to the fact that I am still not very familiar with the implementation detail of Melange and met problems when setting up the coding environment, e.g. could not get the new version of ctags to work well with Vim and Python, the process is a bit slow. However, these preparations have built a good foundation for the official coding stage, which will start next week. Hope that during that period I can code fast and leave more time to improve and clean up my code as well as do some extra work for the project.

GSoC 2010 Data Seeder - Project design

Another week of the community bonding period has gone by as we are getting closer and closer to actually having some code written :D

So here are some of the features that have been decided upon so far:

• The configuration for the seeding operation will be customizable by the user using a web interface which will conclude with the creation of a JSON configuration sheet. This configuration fill will be usable both locally and on-site and can later be edited using the same web interface.
• The configuration sheet is meant to be very easily customizable and extendable. Different scenarios for different needs can be saved in predefined configuration files and then executed at any times. Examples include saved states in different phases of a GSoC program (proposal phase, midterm survey etc.).
• There will be an option to export data to Python fixture files, either directly by the use of a script along with a JSON configuration file, or by saving the state of a running instance.

You can find more details on my wiki page: http://code.google.com/p/soc/wiki/GSoC2010DataSeeder#Design_documents

The script that I have used to generate the diagrams from my previous post has been committed to my Melange clone. You can try it out anytime, no further setup is required. Here it is: http://code.google.com/r/sttwister-gsoc2010/source/browse/scripts/model_graph.py

Week 3: Getting Ready to Code

The community bonding period is drawing to an end this week, which means we begin the actual coding work on our projects starting next week! I am quite excited to start writing code :).

So far, I have finalized most of the design and workflow for the two features I am developing for my project. There are still a few loose ends with regards to design hanging around, which will be tied up soon. My project wiki is now updated with the design details and awaiting the approval of my mentor, Madhu :).

The main additions and modifications to the design this week were:

• Some changes to the user_page.py data model. The multi-valued properties job.py and education.py data models were scrapped to be replaced by a simple biography text field where users can write whatever they would like to share with the community.
• Detailed workflows for Calendars and Maps feature (addition/request new events). Here are the usecase diagrams.
• Both events and user_page data models have a tags property which is basically a comma-separated string with tags related to the user or event(e.g. C#, Java, Django). Users can also be given to the option to subscribe for feeds to be notified of events with certain tags.
• I have also done a detailed timeline, with a list of deliverables for every week up until the mid-term to keep be oriented and directed with aims for every week.

Coming up next week:

• Coding, coding all the way!
• Follow timeline closely to reach the weeks' goals.
• Resolve any bugs/issues, if assigned.

New Document Editor: HTML diffs

One of the goals of my project is to create revision control for Melange documents. Each RC framework must provide some kind of diff engine. The engine should find changes in the document and represent them to the user. It is pretty simple with text, but Melange stores documents as HTML.

Talking about HTML diffs, we should consider two cases:

1. HTML as text. A bunch of tags, attributes, values and content.
2. HTML as a rendered document (e.g. image)

Another question is: what causes representation changes in a rendered HTML document?

1. Tags (through browser's default CSS or applied CSS).
2. Attributes class and id (through applied CSS)
3. Applied CSS (server-side, in-document or in-line).
4. Style modification with DOM.

Let's consider first case - HTML as text. This case seems pretty obvious, cause changes can be tracked with textual diff engines. But it's not that simple. There are several cases of changes to HTML which don't influence representation:

1. Tags are changed, but CSS is the same:
   <h1>Hello, world!</h1>
   is changed to
   <h2>Hello, world!</h2>
   but the CSS is
   h1, h2 {font-size: 12px; font-style: normal;}
2. Some pieces of HTML are rendered the same:
   <div class="alert">Hello, world!</div>
   and
   <div class="alert">
<p>Hello, world!</p>
</div>
3. Class or id is changed but CSS is the same:
   <div class="original"></div>
   is changed to
   <div class="changed"></div>

The second case - HTML as an image. I mean HTML with applied CSS which is displayed to the user. Tracking changes to images is the correct way of handling HTML diffs. It can be performed with several tools. One of them is convert utility from ImageMagick. This approach is, however, a little tricky and CPU consuming.

Let's return to HTML as text. If we can guarantee that CSS represents different tags different and there are no class or id changes, then everything seems right. We can focus on textual diffs for HTML. Considering TinyMCE (which is a defaul editor for Melange) all representation changes are made with tags. If there is no appropriate tag, then the style is applied with <span> tag or a chain of <span> tags.

Textual HTML diffs can be generated by several tools. One of them is HTML diff for Python. I'm now thinking about using it as a skeleton and try to make more intellectual engine with Beautiful Soup

Week 2: Working on Project Design

A second week has gone by and I am happy to say that I am now better acquainted with the Melange code-base. During the week, I managed to create a clone of the source and have now deployed to my very own GAE instance :).

Update on my bug-fixing progress:
I managed to submit my very first patch for issue 890. I also started looking into issue 884, but as it requires some re-design of existing templates, discussion is ongoing as to how this issue can be resolved.

Update on my Project:
As my project is to appear as a new module in the soc.modules section, I am tentatively naming it as social.

I have started on the design work for the ‘Tabbed User Pages’ which is one of the two main features I have proposed.

Data Models:
I have come up with additional data models to contain information on users which will be displayed on their user pages. The proposed additional models are as follows. These are not finalized as discussion is still ongoing as to the inclusion and exclusion of some of them.

soc.modules.social.models.user_page.py
soc.modules.social.models.education.py
soc.modules.social.models.project_details.py

The user_page.py model will serve as the master model which will contain all the attributes of the user model which will appear on the user profile page. The details of education and past and current GHoP/GSoC projects can have multiple values for a given user and hence are extracted out to separate data models by themselves.

Thus far, I have envisioned the User Pages to have a tabbed view with three tabs, the ‘Profile’, ‘Project’ and ‘My Stuff’ tabs. The ‘Profile’ and ‘Project’ tabs are self-explanatory, the ‘My Stuff’ tab will be used for users to show their personal blog feeds using the RSS widget for now. Should more widgets be developed (i.e. Twitter widget J), they can be added in this tab.

Users will be able to edit the data in their user pages using a single form to edit the data shown in all the tabs, using the 'Edit User Page' link. This link will only be visible if the user is viewing their own user page.

Here is a simple use-case diagram to elaborate on the workflow.

Views:

soc.modules.social.views.models.user_page.py
Since the user_page feature serves more or less the same purpose for all the different user roles in Melange, a single view will be created.

Templates:
soc.templates.modules.social.user_page.view_user_page.html
Since the user pages will be tabbed, the entire page will not reload when users switch between tabs. With the help of JavaScript, this can be achieved with a single template with different html div and id tags defining the data to be displayed in different tabs :).

soc.templates.modules.social.user_page.edit_user_page.html
A different template with the html form will be created for the 'Edit User Page' function which will be used to update the user_page.py data model.

Logic:
soc.modules.social.logic.models.user_page.py
To handle the edit/update of the user data a logic is created which will be used to update the user_page.py data model.

In the coming week:

• Check for updates on resolving Issue 884
  
• Finalize the Design for Feature 1 & Feature 2: Maps + Calendars
• Finalize Project Timeline and list code deliverables for mid-term as well as weekly deliverables
• Decide on the specific APIs and libraries to be used for the project.
  

GSoC 2010 Melange Testing Project: Week 1 - learning the code base

It has been a very exciting week. In this week, I made a clone for the testing project, deployed Melange to my GAE instance, got the posting right for melange-dev blog, created a wiki page for the project, signed a Contributor License Agreement (CLA), submitted tax forms, got an ACM account (the courtesy of Google), had the first conference meeting with other developers of Melange, helped some Melange users to solve their log-in problems, and had meetings with my mentor (Sverre). What a week!

My main job in this week, however, was to learn the code base of Melange since my project is to add tests to the code base. In order to help me better understand the code base, I decided to visualize the organization of the code base and the relaionship of modules/classes. Felix' diagrams are very interesting. However, they are only for Melange data models whereas views and logic are more relavant for my project. So, I used a tree tool to generate the module organization of Melange (click the picture below to view the searchable full HTML file)

and module organization of Melange tests (click the picture below to view the searchable full HTML file)

and used pylint.pyreverse to generate the package diagram of Melange (click the picture below to view the searchable full PDF file)

and the class diagram of Melange (click the picture below to view the searchable full PDF file)

These diagrams have helped me a lot to learn the code base. Hope they are also helpful to others who want to learn the code base.

During my meeting with Sverre, he suggested me to start with the testing of the modules which are the most important and used most, e.g. soc.logic.models.base and soc.views.models.base. Therefore, my next week's plan is to start coding, specifically add more test cases to the existing tests of these modules, e.g. tests.app.soc.logic.models.base and tests.app.soc.views.models.base.

Following is a brief introduction to the testing project:

This project is to add unit test suites/cases for the logic and views of the existing modules. These test suites/cases will enable developers to refactor the existing code at any time while at the same time to ensure that they still work correctly by passing all the tests (i.e. regression testing). This project will also serve as a basis for the Melange team to switch the development methodology to test driven development.

Cheers,

Leo (Chong Liu)

GSoC 2010 Data Seeder - Project start off

I'm posting this to let you all know how my first week with the Melange community has been.

Bonding with the community has gone very smoothly, the community is very close together and I love the idea of having weekly conference calls. Sounds like a really fun summer ahead of me and I hope everybody will enjoy it.

The first thing I needed to do for my project was to really understand the data model behind Melange. Since there are quite a few entities involved and keeping in mind that models might perhaps change somewhere in the future, I thought the best way to accomplish this is to build an automated tool that draws diagrams from only the source code. There are two kinds of diagrams I wanted to have: inheritance diagram, showing how classes derive from each other and an entity relationship diagram, showing the relations between entities.

The code that I have produced is inspired from an open source project that does mostly the same thing for Django models. It is part of django-command-extensions.

I hope I can integrate my script with the Melange code base pretty soon, so that anyone can generate graphs from their own copy if anything has changed in the data model.

Here are my results so far:

Inheritance

Inheritance, separated by module

Entity relationship

Entity relationship, with attributes

Entity relationship, separated by module

Some conclusions:

Boy, are there a lot of relations out there!

Personally, I didn't expect to see that many relations between the models. This is definitely going to influence the way I'll design things around, especially the web interface, in order to make it as easy as possible to configure relations in the seeded data.

That's about it for this week, I'll now start actually doing those design documents! :)

Cheers,

sttwister (Felix Kerekes)

GSoC 2010 : Week 1(Community Bonding)

Hello Everyone!
My first week as a GSoC student for Melange has gone by :-). I had the first few meetings with my mentor (Madhusudan) and am right now feeling very excited about my project. Here is a short summary of the week and updates on my GSoC project:

I had a couple of meetings with my mentor where we discussed about my project design and other high-level details as well as some technical stuff. Here is a brief overview:


About my Project:
To develop and integrate a social features module into the current soc framework. The purpose of the module is to help users participating in the programs (such as GSoC) to get to interact and find out more about their fellow GSoC'ers and where they are from, what their project is about and anything else which they want to share with the rest of the community such as events they are organizing or would like to attend etc.

More Specifically:
As of now, there are two main features we have decided on at this stage:

• Google Maps + Calendar
  This feature allows users to find on a global map users, events, meets etc. based on location (hopefully reducing the quantity of "i-am-from" emails on gsoc-students-mailing list), organization, type of project etc. The Calendars will be used to show mainly details of events and meetings occurring within a certain date range for a certain organization etc.
  
  
• User Pages
  Once you locate users on the global map mentioned above, then more information can be found about them through their user pages. Users can use this page to share their personal/professional profiles, details and updates on their projects, their blog feeds.

The above is very brief and a more detailed version is available here.


Venturing into the Melange code-base...
To get started on the task of getting to know the code, my mentor suggested I could take up an issue and fix it. So I asked the team for the most suitable one and took up one (issue 890). After a good amount of trudging around in the code (and some tips from my mentor :-) ), I am on the verge of fixing it! And it has definitely helped me to get familiar with the architecture and code.

In the coming week:

• Work on a more formal specification of the project design, features and work flow (to be added to wiki) and frameworks, APIs to be used.
• Fix another bug (issue 884)
  
• Start thinking about actual implementation details (idea of how the code might be structured etc.)
  
• More meetings with my mentor :)