Melange Development Blog

GSoC 2011 Integration with External APIs: GSoC ends with great experiences

2011-08-23T06:41:00.000-07:00

Hi,

We came into the end of Summer of Code 2011. It was one of the best collaborative development environments i've ever been before. I learned lots of stuff especially about Google Data APIs, OAuth, Javascript, Rest structure . I improved my code writing skills for more beautiful Python code. I worked with very nice and smart people. I tried to look problems like from their perspectives which led me better reasoning against problems. As a result, it was a long but short summer that I spent my most of my time in front of my computer, but that was really fun. I own special thanks to my mentor Mario and co-mentor Madhusudan for spending quite amount of their time with my problems, reviews and newbie questions.

Last week was a little rush for me. I worked on Carol's issue about shipment trackings. Program administrator wants to share some spreadsheets that holds shipment tracking numbers, notes, address for students. A task for this spreadsheet is going to run and data will be synced to Melange datastore so student will be able to see the shipments sent for them in the dashboard page. We hope it will be used for the final shipments this year.

That's a video i captured for its initial state:

GSoC 2011 GUI Overhaul: Generic Iconic Dashboard

2011-08-17T10:33:00.000-07:00

Hi everyone,

This week I need to post remaining patches, especially the generic iconic dashboard. I just posted early so I can have much time iterating the review process and hopefully got the patches submitted this week. The generic dashboard class is now placed in app/soc/views/dashboard.py. It contains the underlying view base for iconic dashboard. Every nested-iconic-dashboard should be inherited from Dashboard class. You can take a loot at admin.py, dashboard.py and org_admin_dashboard.py views to get the idea how to use Dashboard class. Now all related base dashboard templates were placed in v2/soc/dashboard directory. It contains the base.html and _user_action.html. All stylesheet rules should be defined inside v2/gsoc/dashboard.css.

There were changes on the melange.list and melange.dependency JS after refactoring the dashboard. The changes were to hold the lists to be queued and executed when the lists were rendered inside the dashboard. After applying the changes, I run the test and it broke in the test_admin.py. I figure out the case immediately, should be the assertTemplate. Yeah, there were some templates removed and added. After fixing the test, I post all the patches and ping Daniel. I hope I can run fast for the review process ;)

GSoc 2011 : Melange Testing Project CP W12 Last Week

2011-08-15T03:42:00.000-07:00

Hi,

Only a week is left for GSoC to end and I need to finish testing all the important views as soon as possible. I had been working on views.org_profile, profile_show, accepted_orgs, proposal_review, duplicates and proposal_review this week. Testing views is turning out to be a little more involved and complicated task as compared to testing the logic. I got stuck at some places and so could not submit them for review. I had thought that I would be able to finish the views much earlier and start functional testing but I was not fast enough and also I could not devote 10+ hours per day during the weekends as I had promised in the proposal due to heavy course work, classes and personal issues.

I will be trying to commit as much tests as possible in this week and work on the remaining tests after GSoC. I aim to work on functional testing and buildbot after GSoC. Lets hope for the best :).

GSoC 2011 GUI Overhaul: What's Left

2011-08-15T02:24:00.000-07:00

Hi everyone,

The GSoC 2011 program will end shortly. Last week I was working on the account management, assign host, adding evaluation group and dashboard. There's a problem with Django test client that I couldn't figure out. The handler always treat self.data.POST as regular dict for every requests (in my case via POST) made from Django test client. Since in validate method there's a line that expect QueryDict instance, by calling getlist method self.data.POST.getlist('assign_program'), the test always fail. Maybe I missed something here, so I read the client.py code from Django that comes along with Google App Engine SDK. It took me two days or more to trace this problem. I don't think we should remove getlist from the handler nor hacking the django test client. So, I left the test ATM and move to another to do list, that's it, the dashboard.

The dashboard will be moved to soc/views/helpers/dashboard.py and the code itself is self explanatory:

class Dashboard(Template):

"""Base dashboard for admin page

"""



def __init__(self, request, data, subpages=None):

 """Initializes the dashboard.



 Args:

   request: The HTTPRequest object

   data: The RequestData object

   subpages: Subpages of current dashboard

 """

 self.request = request

 self.data = data

 self.subpages = subpages



def getSubpagesLink(self):

 """Returns the link to other dashboard that appears

 on top of the dashboard.

 """

 return self.subpages



def templatePath(self):

 """Returns the path to the template that should be used in render()

 """

 return 'v2/modules/gsoc/dashboard/dashboard.html'



def _divideSubPages(self, subpages):

 """Returns the subpages divided into two columns.

 """

 middle_ceil = int(math.ceil(float(len(subpages))/2))



 return [

     subpages[:middle_ceil],

     subpages[middle_ceil:],

 ]

The templates is improved to be able to handle list components that we have seen from regular dashboard. I hope the patch will be posted for reviewing process lately this week and pushed before the program ends.

GSoC 2011 Melange Testing Project: CP W11

2011-08-08T09:51:00.000-07:00

Hi,

I worked I tested gsoc.views.duplicates module. I have not submitted my tests which I wrote for org_profile, profile_show and student_forms for review as there is a need of some access checks to be defined new for the respective module in the codebase. I was rechecking my tests for org_profile and I felt that the timeline should also be checked if a program is active or not. I put this in the mailing list and Lennard suggested that isProgramActive be renamed to isProgramVisible and a new method isProgramRunning be introduced. Those pages which remain active after gsoc can include isProgramVisible in their access checks and which are current program specific can include isProgramRunning. I submitted the patch for the access checks and Lennard and Sverre reviewed them. I will be accommodating the changes soon.

There was also a problem with our coverage module which is outdated and so it does not work with the latest Nose 1.1.2. I reported the issue with Nose and they suggested that we use coverage > 3.0. Leo tried including it in the buildout dependencies so that the coverage module is always the latest but it too did not work. Just replacing the old coverage module with the latest coverage modules works fine.

GSoC 2011 Integration with External APIs: Proxy Url and Exporting Lists - W11

2011-08-08T08:39:00.000-07:00

My final decision about cross-domain requests: They can be headache. Especially if your requests are bound on some headers. There are new recommendations for this like "Cross Origin Resource Sharing". But this is so new to apply because it makes your application browser dependent. There are still many users on internet using ancient browsers and we can't force them to use latest version of specific browsers. Also server we're trying to access must support this. Because we actually asking server if it accepts some parameters, headers etc. Besides this, jsonp and script methods may be enough for lots of purposes. But to make oauth authenticated requests to gdata server, it wasn't. So i changed my approach. I saw oauth gadgets use the same approach: providing a proxy url on same domain that serves as a gateway and removes all cross-domain concerns.

That proxy url doesn't care about what request does. It just takes method, gdata server, url, data, headers as parameters and makes a request to gdata server and returns response directly to the client.

After resolving cross-domain request problem, i started to develop "list exporting" feature. For now i'm able to create an empty spreadsheet when user clicks "export to spreadsheet" button. Here are some screenshots of current state:

Button for exporting:

Uploading may take time, so it's stressed in a dialog window.

After upload complete, user may click to see the document:

Document is exported with the name of the list:

GSoC 2011 GUI Overhaul: More Handlers for PA

2011-08-07T15:27:00.000-07:00

Hi everyone,

GSoC is near to the final evaluation and I in a run to catch all my TODO. Last week, I wrote some handlers for PA to manage account and assign a user to be a host. PA can assign programs to be hosted by particular user in edit user page. Here's how it looks:

The code is somewhat similar to assign_mentors.py and project_details.py. To set a user as a host, PA needs:

Given an account to be set as a host, go to the Lookup page in the admin dashboard.
If the account is found then click the "Edit user" link
In the edit user form, there's a box at the bottom (see picture above) that contains program selection. To set multiple programs to be hosted by the user, click "Add new" link. If done with the selection, click the "SET AS A HOST" button

Actually there's some features that I've in mind such as add / edit sponsor, programs list that contains list of program name and its hosts as cols, page to view readonly program that in inactive state, and of course the page to create new program. I'll hold thus things for a moment because the dashboard is still being refactored and I promised to finish it by Aug 6th. I am late two days here.

GSoC 2011 Integration with External APIs: Same Origin Policy - W10

2011-08-02T02:15:00.000-07:00

Browsers have some restrictions for cross-domain Javascript requests due to some security issues. For "List exporting" from client side, we need to make cross-domain requests to Google from user. Before this jsonp was only standart that i know as a solution. Jsonp allows loading a json from a remote web server. But i knew GData protocol supports XML format data exchange. I tried to find a solution to load a cross-domain XML source. No solution was neat enough to apply.

Script Approach
Then i thought GData should support JSON of course, and found related document :) We should just have to add a "alt=json" parameter to url GET params. Same page also was providing a method to access resources from a cross-domain with JavaScript. We should call a special script that calls our callback with related json as parameter when page is loaded. Here is an example script:

<script src="http://www.google.com/calendar/feeds/developer-calendar@google.com/public/full?alt=json-in-script&callback=listEvents"></script>

which returns a javascript response with a form in which json response is embed.

listEvents({"version":"1.0" ..... })

This provides another way for making cross-domain requests. In this solution server must support this method with a special effort to form a javascript response instead of just interpretting request as a regular json request.

Http Access Control
This approach was good but unfortunately not enough for our purpose because we need to set Authentication header with request. So i returned back to jsonp. I thought it should be possible to make a jsonp request and add a "Authentication" header with request like that:

jQuery.ajax({url: a_cross_domain_url+"&callback=?", data: data, headers: {"Authentication": auth_header}, success: a_callback)

This was almost working until i encountered with another cross-domain restriction. Sending headers with a cross-domain request was limited to another standard. I read this document about "Cross-Origin Resource Sharing". As far as i understand from document, For sending headers with request, this is defining a way for browser to first ask to server if it allows given headers, then makes actual request. This sounds good but because it's a new recommendation and some browsers may not support it (and perhaps GData server too) , i skipped this. Firefox supports this after 3.5 version.

At the end
I learned some stuff but still was not happy as there wasn't a way sending OAuth header with request. GData documentation contains AuthSub authentication over Javascript but i couldn't find anything related OAuth with JavaScript in which we will provide OAuth header. Then, while i was hopelessly playing with OAuth playground, saw that choice in form:

Put oauth params in: Authorization header URL as params

~~This was just i was looking for. We were able to pass oauth params with url instead of header. This was something not documented in GData OAuth documentation. I was lucky to find this out.~~

~~At the and my approach was shaped: Make a jsonp request, put oauth params inside url.~~

Note: This didn't work either, to see my final solution to make Javascript requests see next post by me.

GSoC 2011 GUI Overhaul: Document List for Admin

2011-08-01T00:38:00.001-07:00

Hi everyone,

Lets talk about document list. Carol asked this on issue 1257. To implement such list, I borrowed the view from accepted_orgs.py. Lets name the list "DocumentList" and the page "DocumentListPage". I only added two visible cols, the title and shortname, for DocumentList. When the row is clicked, it will go to the edit document page view. I also created the test for document list inside test_document.py. The test will make sure, the list will be shown to the host (based on checkAccess method). Okay here's the class for DocumentList and DocumentListPage:

class DocumentList(Template):
 """Template for list of documents.
 """

 def __init__(self, request, data):
   self.request = request
   self.data = data
   r = data.redirect

   list_config = lists.ListConfiguration()
   list_config.addColumn('title', 'Title',
       lambda e, *args: e.title.strip())
   list_config.addSimpleColumn('link_id', 'Link ID', hidden=True)
   list_config.addColumn('short_name', 'Short Name',
       lambda e, *args: e.short_name.strip())
   list_config.setRowAction(
       lambda e, *args: r.document(e).urlOf('edit_gsoc_document'))

   list_config.setDefaultPagination(False)
   list_config.setDefaultSort('title')

   self._list_config = list_config

 def context(self):
   description = 'List of documents for %s' % (
           self.data.program.name)

   list = lists.ListConfigurationResponse(
       self.data, self._list_config, 0, description)

   return {
       'lists': [list],
   }

 def getListData(self):
   idx = lists.getListIndex(self.request)
   if idx == 0:
     fields = {'scope': self.data.program}
     response_builder = lists.QueryContentResponseBuilder(
         self.request, self._list_config, document_logic, fields)
     return response_builder.build()
   else:
     return None

 def templatePath(self):
   return 'v2/modules/gsoc/document/_document_list.html'


class DocumentListPage(RequestHandler):
 """View for the list documents page.
 """

 def templatePath(self):
   return 'v2/modules/gsoc/document/document_list.html'

 def djangoURLPatterns(self):
   return [
       url(r'documents/%s$' % url_patterns.PROGRAM, self,
           name='list_gsoc_documents'),
   ]

 def checkAccess(self):
   self.check.isHost()

 def jsonContext(self):
   list_content = DocumentList(self.request, self.data).getListData()

   if not list_content:
     raise AccessViolation(
         'You do not have access to this data')
   return list_content.content()

 def context(self):
   return {
       'page_name': "Documents for %s" % self.data.program.name,
       'document_list': DocumentList(self.request, self.data),
       'program_select': ProgramSelect(self.data, 'list_gsoc_documents'),
   }

And here's the test for document list:

class ListDocumentTest(DjangoTestCase):
 """Test document list page.
 """

 def setUp(self):
   self.init()
   self.data.createHost()

 def testListDocument(self):
   url = '/gsoc/documents/' + self.gsoc.key().name()
   response = self.client.get(url)
   self.assertGSoCTemplatesUsed(response)

   response = self.getListResponse(url, 0)
   self.assertIsJsonResponse(response)
   data = response.context['data']['']
   self.assertEqual(1, len(data))

GSoC 2011 Integration with External APIs: Popup Blocking Policy - W10

2011-07-31T15:51:00.000-07:00

This week, unfortunately i was not productive. I had quite problems mainly based on "same origin policy". Tonight, story of the week seems ending nice, because i finally resolved how to make cross-domain requests with JavaScript. I will write two blog posts this week, this one for another problem "popup blocking problem" and next one for "same origin policy". If you don't have extra time for reading two blog posts, i suggest you read next one, because it's the main story of the week for me.

Popup Blocker Policy
I want to mention this before "same origin policy" problem. We had to use a window popup in OAuth authentication. Popup was used to open immediately after user clicks an element whom click event is connected to an authentication required function. This authentication required function was either opening popup, or running its own logic if user was already authenticated. This was how it was working:

User (before page loads):
- User requests a page which uses melange.gdata.core.js on user side. This script provides gdata login methods and gdata API requests for JS.

Server:
- Page indicates melange.gdata.core.js dependecy in its template by using our dependecy *mechanism*.
- Page also needs to init melange.gdata.core.js with some context values those tell script some initial values: gdata_is_logged_in, oauth_redirect_url.

User (after page loads):
- If a page specific Javascript function has a logic that needs authentication to GData, for example for the proposal page:

In proposal/base.js:

function sync() {
//code that nees authentication
}
authenticated_sync = melange.gdata.core.createAuthorizedFunction(sync);
button.click(authenticated_sync)

This is working in peace for proposal page which adds melange.gdata.core.js to dependencies in it's template and inits it with some context value which is produced by a context helper (A helper which creates gdata_is_logged_in and oauth_redirect_url values). But for the "List Exporting" this was quite redundant workforce to init melange.gdata.core.js in every template because a big per cent of Melange pages contain lists. We did not want any extra configuration for every list page. Indicating list dependecy should be enough. So we tried to skip initialization of melange.gdata.core.js until user expresses he wants to use an authorization required feature. We tried to make this initilaziation with a JSON request. For example when user clicks "Export" button, it asks server for initialization values and with respect to response, performs required authentication process if required. Every thing seemed fine until i realized tha, request time delay was causing popup window to be blocked by browser's popup blocker. If we want to open a popup it should be immediately after a click event occured, but we can't know if we need to open a popup without asking it to server (so without waiting).

So we needed to change this mechanism. Script melange.gdata.core.js must be aware of gdata_is_logged_in value before click event. We solved this by moving gdata initialization into Melange's general javascript config initialization which works for every page. So with little redundancy (because we don't need to calculate is_logged_in value for every page), we were able to init it without extra initialization code for every page that uses melange.gdata.core.js. If we can assume %90 of Melange pages contains lists, this is not a big CPU usage redundacy.

I don't know if this popup blocking policy is browser dependent or not. But from my experiences i had the conclusion that most guarantee way of opening a popup is doing immediately after a click event is occured.

GSoC 2011 Melange Testing Project: CP W9 & W10

2011-07-30T23:04:00.000-07:00

Hi,

I could not write a blog post for the work I did during last week because of the
the university work and some other factors. This post summarises the work I did in last two weeks.

Last week, I fixed some tests which were failing due to the recent change done to the mentor property. The tests for soc.modules.gsoc.tasks.test_survey_conversion failed because the models were seeded(Model.put()) by explicitly defining the model properties. This method is definitely not a robust one since if we happen to add other required properties in future, the tests will fail again due to the missing properties while seeding. The solution is to use the data seeder which randomly seeds the properties not mentioned in the properties parameter. If there is no data provider for a property, we can patch the data provider to seed such properties and the tests would run fine. So, I edited the tests for gsoc.tasks.survey_conversion and made them to use data seeder to seed the models.

     I also fixed the profile_utils.createStudent() method which had been bugging me while testing gsoc.views.student_forms. My tests failed 1 out every 7-8 test runs. It was because of the failing access checks which resulted because of the number_of_projects property being seeded other than 0 for a student without a project. This problem bugged me for quite sometime and Madhu helped me to figure out that this was actually the problem. Also, I submitted some patches to remove the unused imports in some modules in gsoc.views.

   This week I tested a couple of views namely views.org_profile, views.profile_show and submitted them for review. Leo has reviewed them and I will be fixing them today. I also added createInactiveProfile() and createInactiveStudent() method to profile_utils which I needed while testing views.org_profile. I discovered a defect in DownloadForm request handler in student_forms and I filed a issue for that. The defect was that a student without a project should not be allowed to download the enrollment and tax forms and in the present case a 500 error is returned which should be 403 instead. As told by Sverre, I need to add an access check to fix this.

    Since I get very less time during the week days to work on the project, I will be working on Friday, Saturday and Sunday as I don't have classes on these days.

GSoC 2011 GUI Overhaul: Uploadable Logo with Area Selection

2011-07-28T10:32:00.000-07:00

Hi everyone,

When I discovered a problem with the logo url rendered in the homepage, I believe blobstore for the organization logo will be fine. Yeah, until I took pains with redirect response. So here's the specification to implement image area selection in blobstore, so you can crop the uploaded image:

We need coordinate fields, x1 as a horizontally starting-point where the area selection begins, x2 as horizontally end-point, y1 as vertically starting-point where the area selection begins and y2 as vertically end-point. Since we're using blobstore, we need to set BlobReferenceProperty field.
Create the form similar to the tax or enrollment form, but only set the BlobReferenceProperty field. To make it clear, the uploaded file will be handled by Blobstore API and handover to our written handler afterward. In our handler we can set initial values for coordinate fields and redirect again to upload form.
When the blob key is available, we need another handler to request the blob.
To render the uploaded image in the form, we can follow how Google Map is rendered in the profile form.
Image area selection was implemented using jQuery imgAreaSelect. The documentation is pretty obvious.
There should be coordinate fields update handler which is posted when we made selection over the image. There's onSelectEnd paramater that we can passed over when instantiate imgAreaSelect.
To serve the image with area selection, we need to create a handler that reads the blob based on provided blob key. Resized and cropped image will be manipulated here by using Image API.

I can't give you live demo for uploadable logo. I don't have a dashboard in billing-mode. Here's the screenshot if you're wondering:

GSoC 2011 Integration with External APIs: Melange Lists and GData API Protocol - W9

2011-07-25T04:17:00.000-07:00

Hi,
Last week, I started to work on "Melange list exporting to GDocs" that will be second deliverable of my project. I've been searcing through GData Protocol to use JavaScript for Spreadsheets API. This API is used required to create a spreadsheet and manage it. Unfortunately there isn't a JS API for this provided by Google. But GData APIs are built on REST like protocols over HTTP. This gives possibilty to anyone to use them with any language as long as correct HTTP requests are provided.

Well, we want to use Spreadsheets API with JS, but i haven't tell the reason yet. Reason is we can not trust how big the data is. For example it's quite big enough for accepted stundents page. GAE has a limitatation, max 30 seconds per request. Melange loads some of the lists with multiple requests, but when we want to create and upload a list we can't make it with multiple request (or that would be really harder). So we need to export data on client side.

This also saves time preventing reprocessing of query. Another benefit is that, it allows uploading modified list on client-side. For example, user may filter list for a parameter, or sort it on client side. With this approach we present user a feature like, "what you see is what you export" that is exactly what user wants.

GSoC 2011 Integration with External APIs: Proposal Sync - W8

2011-07-19T15:37:00.000-07:00

Sorry for late blog post. First evaluation results of GSoC 2011 was announced. I passed first evaluation and I'm very happy for it :) My first deliverable "proposal sync" is still in review process. But my guess is it's mainly complete and just needs code design fixes. I'm busy with fixing my patches for a few days. My patches are mostly contains JS code and I wasn't writing quality JavaScript code before but just ones that saved my days. Not suprisingly Mario pointed out lots of bad design choices in my code. I want to mention a few of i learned :

* Always use === and !== instead of == and !=, These operators also checks if operands are same type besides their values are equal.

* Do not rely on content of DOM elements. Do not use them as variables. Create global variables instead.

* Using if and else without braces if they have only one statement is tricky. This later may cause someone not to notice there are no braces, and put a statment outside of else scope, while thinking it's in the scope.

* Melange JavaScript Guide also has good tips, for readibilty and quality of source code.

From now on, i will also start to work on Melange lists. We basicly want all Melange lists to have upload option to Google Docs. Melange lists are built-on jqgrid library which allows easily create and manage interactive HTML tables. We need a generic button that will work for every list without extra configuration, so someone creating a new list will not take care about how it forms it's data and uploads itself to GDocs.

That's all about me. See you next week and cheers :)

GSoC 2011 Melange Testing Project: CP W8 Mid-term

2011-07-17T10:50:00.000-07:00

Hi,
I have completed the logic tests and the task for post mid-term period is to write the tests for all the gsoc.views modules. Leo and I had a discussion after the last conference call and we decided to prioritise which modules should be tested first. A survey of the modules which have tests and which needs to be tested can be found at [0]. Accordingly, I have to first test gsoc.views and gsoc.tasks and then go on to write the functional tests. I am presently working on gsoc.views.student_forms and stopped working on soc.views.helper.access_checker.
I have no experience with Django and I spent some time studying the basics and then asked Madhu some doubts that I had got while studying the view's code. I will develop pace as I get experience with the views.
This week was also assigned for mid-term evaluations and Leo has passed me :). My classes are starting from 20th July and I need to manage both gsoc and classes. The last two days went in travelling to the university and the next two days will go in registration and shifting to the new hostel room.

[0] https://spreadsheets.google.com/spreadsheet/ccc?key=0AuENNrqFbxWEdDJMclZySzFUalpsUTd6bzNJTjJGVlE&hl=en_US#gid=0

GSoC 2011 GUI Overhaul: Things that I've been working on last week

2011-07-17T10:39:00.000-07:00

Hi everyone,

As we already knew, mid term evaluation already passed. I contacted my mentor, Daniel, about the result of my mid term evaluation. Daniel told me to focus on the remaining works that need to be done. If you're following my discussion on the mailing list, there are obviously requested features in accordance with my project scope. Here's the list of thins I've been working on from the last week :

Account management. Sverre posted on mailing list about the problem he encountered by manually modify accounts using stats.py. Sverre gave some possible use cases:

People have their profile migrated, their email address stays the
same but their user_id changes, the User.user_id property needs to be
set to None.
People have a @gmail.com or @googlemail.com address attached to
their @some.domain.com address, and their User.account needs to be set
to the new address.
Lennie added his suggestion for ban user feature.

Regular dashboard using iconic information. Daniel said to me give a shot for iconic dashboard for regular users (students and mentors). This means there will be two options to vote later, toggleable list and iconic dashboard. Toggleable list has been reviewed by Mario and I've made some adjustments (not posted yet).

Leo gave some review on my test for withdraw projects view. I need to fix the test.
Lennie asked me to make the evaluation links 3 level deep. I've sent the patch, no response yet.
Participant locations statistic.

I will post the patch for the above features on Monday, or lately Tuesday. After that there will be one week to review all the patches before going to be pushed.

GCI 2011 GCI with New Architecture - W7

2011-07-11T09:54:00.000-07:00

Hello all,

This week I started with the Public Task view which first showed all the static fields and then I have created a Form to publish comments for the task and to retrieve the work submissions on that view.In the meanwhile my redirects patch got reviewed and I refactored the GSoC Redirects module but due to my mistake, it broke the existing code, I sincerely apologize for that.I have again written the GCI Redirects module avoiding these mistakes.I then wrote the GCI Timeline Helper class and moved some common code to the core.This got reviewed and will be committing this after the previous patches gets approved.After this I wrote the views to create and edit a GCI Task and have learnt some new things in this process and I sent this one for review too and then as advised by Madhu I moved onto to the task workflow, as the first step I have written a form with a drop down that has the list of actions that are available to the Org Admin, Mentor or Student depending upon the task status.I will be sending this to review now in bit.This week I have spent much more time with my project and in communicating with Madhu which helped me clear many doubts and work better, I feel I have this week's progress is better than the previous ones and I'm happy about it.I will be increasing my time on the project this coming week and will try my best to cover the lag.

GSoC 2011 GUI Overhaul: Push the code and some tasks need to be done

2011-07-11T00:48:00.000-07:00

Hi everyone,

Last week I've been working to fix my patches with Daniel. Today I've pushed it into repository. It took a time to convert my commits into chronological patch with Mq. When I sent my patches without Mq, I used the following command to output the patch :

hg export -r started_revision_of_admin_dashboard:tip > patch.diff

Meanwhile the upstream repo changes rapidly with Madhu's changes, so importing the patches that have different tip with my own tip may abort the import. Mq is totally helpful if you're working on many patches while there are rapidly changes in the upstream repo. All you need to do is push your patches into queue and refresh it while receiving feedback from review. And then do hg qfinish to convert the patch into permanent changeset. With the awesomeness of Mq and review/feedback from Daniel and Praveen I just pushed my code a couple hours ago.

The admin dashboard isn't finish yet, there are some functionalities need to be done for PA, such as statistic that Daniel has been working on. I also promised to create MapClusterer to display participant in a single map like the following one:

After finishing the MapClusterer and integrate the statistic module into admin dashboard, I'll working on uploadable org's logo with cropping feature. I've made silly app that prove how Image service from GAE was capable to implement this feature.

GSoC 2011 Melange Testing Project: CP W7

2011-07-10T14:37:00.000-07:00

Hi

The task for this week was to correct my previous tests and finish the remaining tests in soc.logic and soc.modules.gsoc.logic. I have corrected my previous tests and submitted them again for the review. I have tested all the logic functions which are used at least once in the code base. I have not tested the soc.logic.allocations, soc.logic.lists and soc.logic.helper.notifications because these are not in my priority list for now as I need more familiarity with the code base to test them correctly and efficiently. I will come back at some time in future to test them.

I have started attempting to write the view tests. I have never written any view tests earlier and my experience with Django is 0.1, but sometimes we have to learn the hard way. I studied many of the new tests written by Sverre. Testing a view seems to be a complicated task and I did not understand many things which I had to ask Madhu. Just before writing this post, I had a discussion with Madhu and I asked him what 'Context' is, what tests_utils.StuboutHelper is for, what soc.views.helper.access_checker.Mutator is used for and a load of other topics. At the time of writing this blog post, I was testing the soc.views.helper.access_checker. It will consume more time to test because there are more than 1100 lines of code to test. After testing this, I will test the gsoc.views.helper.access_checker module.

I have the target of completing all the view tests before the final evaluations. It will require a huge amount of effort which I am pouring in. Lets hope for the best. :)

GSoC 2011 Integration with External APIs: Proposal Sync - W7

2011-07-10T13:11:00.000-07:00

Approaching to midterm, i partially finished my first deliverable and sent my patches for review. I'm really For now there is one way sync from GDocs to Melange. We are still discussing to allow students "sync back" their proposals to GDocs.

I was struggled with some problems that i want to mention during this blog post. First when using GData libraries on App Engine infrastructure, exporting documents is not directly possible. The reason is that App Engine uses a distrubuted system and this does not allow to write to filesystem. To export anything, GData restricts its API to save output into a file instead of returning exported content. You need little change on gdata library to make it possible:

diff -r 84a930b76e63 -r e65b82d9ad71 app/gdata/docs/service.py

--- a/app/gdata/docs/service.py Sat Jul 09 03:58:23 2011 +0530

+++ b/app/gdata/docs/service.py Sun Jul 10 11:27:08 2011 +0300

@@ -106,6 +106,11 @@

server=server, additional_headers=additional_headers, **kwargs)

self.ssl = True

+ # Variables used to hack-in Export function to use it

+ # with a file handler instead of file path name.

+ self.file_handler = None

+ self.use_file_handler = False

def _MakeKindCategory(self, label):

if label is None:

return None

@@ -183,10 +188,17 @@

raise gdata.service.RequestError, {'status': server_response.status,

'reason': server_response.reason,

'body': response_body}

- f = open(file_path, 'wb')

- f.write(response_body)

- f.flush()

- f.close()

+ def writeResponseToFile(f, close=True):

+ f.write(response_body)

+ f.flush()

+ if close:

+ f.close()

+ if self.use_file_handler:

+ writeResponseToFile(self.file_handler, close=False)

+ else:

+ writeResponseToFile(open(file_path, 'wb'))

def MoveIntoFolder(self, source_entry, folder_entry):

"""Moves a document into a folder in the Document List Feed.

@@ -346,7 +358,7 @@

self._DownloadFile(url, file_path)

- def Export(self, entry_or_id_or_url, file_path, gid=None, extra_params=None):

+ def Export(self, entry_or_id_or_url, file_path, gid=None, extra_params=None, file_handler=None):

"""Downloads a document from the Document List in a different format.

Args:

@@ -361,6 +373,13 @@

Raises:

RequestError if the service does not respond with success

"""

+ if file_handler:

+ self.file_handler = file_handler

+ self.use_file_handler = True

+ else:

+ self.use_file_handler = False

ext = None

match = self.__FILE_EXT_PATTERN.match(file_path)

if match:

With this you can use a memory file (StringIO) with Export function to write content to file handler directly instead of giving a file name to function for opening.

Second problem was about OAuth authentication page. We were willing to put OAuth page inside a frame in a jQuery dialog instead of a popup window. For strange reasons, page did not show up in the frame. I started an issue to learn the reason behind that: http://code.google.com/p/gdata-issues/issues/detail?id=2651

You can see demo of proposal sync from here with username:testmelange@gmail.com , password:melange2011 and type to find "Test Document 1" that is in GDocs account of testmelange user.

GCI 2011 GCI with New Architecture

2011-07-04T10:06:00.000-07:00

Hello all,

Apologies for the late blog post.This week I have started off with the url patterns for GCI and Lennard suggested that I move the common functions to the core and leave the GCI related constants in that module and I have done that.I then went onto the GCITask view but before that I had to add another method to the access_checker to check if the task is published or not but looks like, I still need to modify this method a bit to include other cases too.I still have to retrieve work submissions for the Task view and form to edit the task and to subscribe the task.And me and Madhu had a small discussion about the templates and we decided to use the gsoc templates for now and make modifications where needed and use gsoc's css and JS till we get the designs.Meanwhile my other patches have been reviewed.First I had to refactor the gsoc redirects module, as many of the redirects are moved to the core and some methods have to be overridden to set the _url_name.And I was asked to make a little change to the gsoc access_checker as it's DeveloperAccessChecker should also inherit from AccessChecker, and the same was done to the gci access_checker too.Made required changes to the other patches as mentioned.I haven't spent enough time last week, as I had to help a Professor with his coursework and I really could not avoid it, I discussed with him today and took permission till the end of August, So I will cover up this week and will spend extra time on my project.

GSoC 2011 Melange Testing Project: CP W6

2011-07-04T03:58:00.000-07:00

Hi,
In the last conference call, I reported my problems to Sverre that most of my time is spent on reading the code itself and that I had problems in deciding about the use cases when writing tests. My productivity has been very low till now and I have written atmost two tests per week. This week I worked hard and I have written tests for all the modules in gsoc.logic and soc.logic except three modules. Though the tests are under review and they need drastic improvements, I have now become comfortable in deciding about the use cases and writing tests. I had a discussion with Leo and he told that I was unnecessarily using DjangoTestCase. I can use seeder_logic and inherit my test classes from unittest.TestCase and this will improve the performance and speed. I was using DjangoTestCase to avoid instantiating the GSoCOrganization, GSoCTimeline and other entities which I could use directly when inherited from DjangoTestCase. But seeder_logic is more useful and its also used in DjangoTestCase. So, I corrected the test for soc.logic.host and used seeder_logic to seed the entities and it was working fine.

I tested a total of 8 modules in this week and the coverage has increased by 4%. The present coverage of the tests in my local copy is 70%. The coverage will increase more when I write tests for the soc.views.

My work for this week is to correct the previous tests, test the remaining three modules in gsoc.logic and soc.logic and then start the view tests.I hope to finish all the logic tests before the mid-term evaluations.

PS: Mercurial queues are AWESOME. Salute and gratitude to the guy who wrote that extension. :)

GSoC 2011 GUI Overhaul: Typical PA Workflow and Admin Dashboard Usage

2011-07-03T22:56:00.000-07:00

Hi everyone,

A week ago I asked Madhu about the typical workflow for PA (Program Admin). Madhu was so kind to explain the workflow for PA. I'll summarize the workflow and usage of admin dashboard by the PA. The followings are ordered things that should be done by PA to organize the GSoC program:

Initialize the program settings such as program name, document page, or timeline of the program. These settings already provided by site.py and timeline.py views.
Invite organizations to participate.
Create the organization application survey. I believe this is related to what Madhu's did for survey with form builder. I can easily add the url inside the new admin dashboard to refer to the form builder.
PA and her team discuss the organization selection internally.
Accepted organization announced. I don't know yet if there's a view or task to accept organization.
Student signup period. Students begin submitting proposals to organizations.
PA can assign slot in this time or before the accepted projects announced. There's a view for slot assignment.
PA can check the duplicate proposal. There's a view to trigger the duplicate proposals' task queue. This queue will iterate each proposals and filter the proposals by the same student that would be accepted by two or more organizations. The resolution was done manually by gathering a meeting.
PA can check slot transfer by organization and accept/ignore the request of transfer by org admin. There's a view for this.
After all organizations solve the duplication status and agree with the given slots. PA can trigger the accept_proposals' task queue. There's a view to trigger accept_proposals queue.
PA creates survey for mentors and students. There's a survey builder committed by Madhu.
PA freeze the program after the final term.

I don't know yet what the task needed by the PA after creating the survey. But the typical workflow should be like the list I mentioned above.

GSoC 2011 Integration with External APIs: Learning new tools - W5

2011-06-26T15:49:00.000-07:00

This week i was busy with fixing my patches. Madhusudan suggested me a tool to create mutable changesets : Mercurial Queues (aka MQ). It was not possible to work on multiple commits without actually commiting them. This great extension makes this possible. You can create a patch, edit it, then create new one and return back to previous patch to edit it. (Patch is a mutable changeset) I benefit from this mostly in two ways : First as i am new to community my commits need to be reviewed. When i send a patch for review, i can start to work on another while first one is in review and mutating. Second reason which also will make me use MQ even after i'm good enough to send my commits without review is it helps divide and code! I can divide a job into parts which are slowly growing (each part corresponds to a commit). So i have a working job which is not complete but each parts are still growing and making working job better. Instead of finishing each part in order and trying to make them perfect as i dont have chance to go back, i'm working on every task at the same time synchronously. I also wrote on my Turkish blog about MQ :)

Another tool i was introduced was {{less}} which helps writing CSS in a hierarchical style, without requiring to write full selector path for each block. It also allows defining variables which prevents redundancy when using same values multiple times. It's clearly more programmers way to CSS.

While being little late for proposal syncing feautre. I still could find some time to think on it and send a mail to community to discuss on it. There is no harm to copy/paste my current idea here :)

When user goes to '.../proposal/submit...', represent a sub-page with two buttons : Edit HTML, Sync with GDocs
Go to a different page for each selection, each represents different form. (e.g. '/proposal/submit/gdocs', '/proposal/submit/default') One page requires to login GDocs and a gdocs flag will be True for proposal entity after saved.
For updating part, 'edit proposal' link's destination will be changed with regards to proposal's gdocs flag (either go to '/proposal/update/gdocs/id' or '/proposal/update/default/id').
Widget 'GDocs Link' will have a 'sync now' button near it. When user pastes link to field and presses "sync now" button, an ajax request will be executed. Text area 'Content' (which is not editable) will be updated with result of that AJAX request. So actually, in server-side, we won't process document link again and treat form like its content edited by hand.

GCI 2011 GCI with New Architecture

2011-06-26T11:58:00.000-07:00

Hello all !!

This week I have worked on the views after completing the conversions tasks.I first started off with RedirectHelper for constructing redirects in GCI.Each of the methods in this helper returns the RedirectHelper object after setting certain parameters like _no_url, url_name, _url, args and kwargs.The clear method clears the current internal state and urlOf method returns the resolved url for name using current internal state for args and kwargs.At this stage certain field from GSoCProgram like about_page, events_page, events_frame_url, connect_with_us_page etc are required for GCI too and hence they are moved up to soc.models.program.Program.The RequestData patch for GCI got reviewed and I am about to commit it now after making a couple of changes.
I then proceeded to the Access Checker for GCI.For this not much is needed at the moment and more methods will be added as we move on with the views.For now, I have to written a couple of methods one is access_checker.Mutator.taskFromKwargs which sets the task entity in the RequestData object and the other is access_checker.AccessChecker.isTaskInURLValid to check if the task in the url exitst or not.After acess_checker I have moved onto site menus for GCI.I was told that I can use the same site menus as that of GSoC for now.So I went ahead and have written a smiliar site menus for GCI which contains siteMenuContext to generate URL links for the hard-coded GCI site menu items and Header, MainMenu and Footer templates.I have then finally moved onto RequestHandler for GCI.The render method of the soc.views.base.RequestHandler is overridden to extend the context object with header, mainmenu and footer values which are rendered Templates for the current RequestData object.I have now started with the Task view.

GSoC 2011 GUI Overhaul: The Admin View's Design

2011-06-26T07:31:00.000-07:00

Hi everyone,

From my previous post I've talked about colorbox approach for admin dashboard. If you want to try, visit my demo instance at http://melange-new-dashboard.appspot.com/ using the following account:

username: melangetestdashboard@gmail.com

password: melangetestdashboardpassword

There are some changes in the JS, CSS and template. Some of these were just new files. I'll talk about code that make a sense to review. Before that, I'll divide the topic into two sections so you can skip a topic that you don't want to read.

Changes in admin's view

If you've tried the admin dashboard from the link that I gave above. You should try to toggle the colorbox or click the "Program settings" link. If you click the "Program settings" link, the dashboard will change its content. The case is the same for "Manage organizations" link. If you inspect the page with firebug, you'll notice that it just switching between div elements. Let's call this a sub dashboard.

So, a link that have sub links at the bottom of its description will point to the div that hidden at that moment, that's a sub dashboard. When we clicked "Program settings", the div with id "program-settings-dashboard" will be shown, the "disabled" class was removed. And the "main-dashboard" will be hidden. I figured out what's the best solution to reorder the links or reorganizing sub dashboard without coupled too many files. So I came up with the idea that these links and sub dashboards should be managed inside soc.modules.gsoc.views.admin. I managed to create a new class named Dashboard that inherits Template class:

class Dashboard(Template):
"""Base dashboard for admin page
"""

def __init__(self, request, data, subpages=None):
"""Initializes the dashboard.

Args:
 request: The HTTPRequest object
 data: The RequestData object
 subpages: Subpages of current dashboard
"""
self.request = request
self.data = data
self.subpages = subpages

def getSubpagesLink(self):
"""Returns the link to other dashboard that appears
on top of the dashboard.
"""
return self.subpages

def templatePath(self):
"""Returns the path to the template that should be used in render()
"""
return 'v2/modules/gsoc/admin/dashboard.html'

def _divideSubPages(self, subpages):
"""Returns the subpages divided into two columns.
"""
middle_ceil = int(math.ceil(float(len(subpages))/2))

return [
   subpages[:middle_ceil],
   subpages[middle_ceil:],
]

Currently there are three dashboards that inherits Dashboard class, they are: MainDashboard, ProgramSettigsDashboard and ManageOrganizationsDashboard. To get you the clear idea, here's the MainDashboard and ProgramSettigsDashboard classes:

class MainDashboard(Dashboard):
"""Dashboard for admin's main-dashboard
"""

def __init__(self, request, data):
 """Initializes the dashboard.

 Args:
   request: The HTTPRequest object
   data: The RequestData object
 """
 super(MainDashboard, self).__init__(request, data)

def context(self):
 """Returns the context of main dashboard.
 """
 r = self.data.redirect
 r.program()

 manage_orgs = ManageOrganizationsDashboard(self.request, self.data)
 program_settings = ProgramSettigsDashboard(self.request, self.data)

 subpages = [
     {
         'name': 'lookup_profile',
         'description': ugettext(
             'Lookup profile of mentor or student from various program.'),
         'title': 'Lookup profile',
         'link': r.urlOf('lookup_gsoc_profile')
     },
     {
         'name': 'allocate_slots',
         'description': ugettext(
             'Allocate slots (number of acceptable projects) per '
             'organization'),
         'title': 'Allocate slots',
         'link': r.urlOf('gsoc_slots')
     },
     {
         'name': 'slots_transfer',
         'description': ugettext(
             'Transfer slots for organizations'),
         'title': 'Slots transfer',
         'link': r.urlOf('gsoc_admin_slots_transfer')
     },
     {
         'name': 'duplicates',
         'description': ugettext(
             'Calculate how many duplicate proposals'),
         'title': 'Duplicates',
         'link': r.urlOf('gsoc_view_duplicates')
     },
     {
         'name': 'manage_organizations',
         'description': ugettext(
             'Manage organizations from active program. You can allocate '
             'slots for organizations, manage invitations for '
             'org admin/mentors, and withdraw/accept students/mentors '
             'from various organizations'),
         'title': 'Manage organizations',
         'link': '',
         'subpage_links': manage_orgs.getSubpagesLink(),
     },
     {
         'name': 'reminder_emails',
         'description': ugettext(
             'Send reminder emails for evaluations.'),
         'title': 'Send reminder emails for evaluations',
         'link': r.urlOf('gsoc_survey_reminder_admin')
     },
     {
         'name': 'program_settings',
         'description': ugettext(
             'Edit program settings and timeline'),
         'title': 'Program settings',
         'link': '',
         'subpage_links': program_settings.getSubpagesLink(),
     },
     {
         'name': 'participant_locations',
         'description': ugettext(
             'Show all participants (students and mentors) in a '
             'clusterer map. You can also view a single profile map '
             'after clicking the marker.'),
         'title': 'Participant Locations',
         'link': '#'
     },
     {
         'name': 'report_statistic',
         'description': ugettext(
             'Reports and statistics of program'),
         'title': 'Report/statistic',
         'link': '#'
     },
 ]

 return {
     'title': 'Admin Dashboard',
     'name': 'main',
     'subpages': self._divideSubPages(subpages),
     'enabled': True
 }


class ProgramSettigsDashboard(Dashboard):
"""Dashboard for admin's program-settings-dashboard
"""

def __init__(self, request, data):
 """Initializes the dashboard.

 Args:
   request: The HTTPRequest object
   data: The RequestData object
 """
 r = data.redirect
 r.program()

 subpages = [
     {
         'name': 'edit_program',
         'description': ugettext(
             'Edit your program settings such as information, slots, '
             'documents, etc.'),
         'title': 'Edit program',
         'link': r.urlOf('edit_gsoc_program')
     },
     {
         'name': 'edit_timeline',
         'description': ugettext(
             'Edit your program timeline such as program start/end date, '
             'student signup start/end date, etc.'),
         'title': 'Edit timeline',
         'link': r.urlOf('edit_gsoc_timeline')
     },
 ]

 super(ProgramSettigsDashboard, self).__init__(request, data, subpages)

def context(self):
 """Returns the context of manage organizations dashboard.
 """
 subpages = self._divideSubPages(self.subpages)

 return {
     'title': 'Program Settings',
     'name': 'program_settings',
     'backlink': {
         'to': 'main',
         'title': 'Admin dashboard'
     },
     'subpages': subpages
 }

To set the links, we need to set it into subpages context. If you want a link to another dashboard inside MainDashboard, such as "Program Settings", just create another class that inherits Dashboard, supply the backlink to refer back to the parent's dashboard and set the subpages of its own dashboard inside the new class. In the parent dashboard's subpages context supply the subpage_links to indicate this is a dashboard link. How about the icon itself? This is where the CSS rules. The name you supply for each link inside the subpages content will be rendered as a class name of the link's container. Everything's that related to the style of admin dashboard is defined inside v2/gsoc/admin.css file. I got the icons from Tango Desktop Project which is licensed under GPL.

The next thing that you might need to know is the addition of colorbox base layout. A colorbox base layout is minimalistic template without sidebarmenu and header. The rest is the same. To make any page rendered in colorbox layout, you need to pass cbox query string. The RequestHandler will take care the base layout. Each page that extends the base.html template need to change the extends tag to base_layout, {% extends base_layout %}. The rest of changes inside soc.modules.gsoc.views.admin were new handlers with list.

What's done and what needs to be done or in-progress

I promised to finish the admin dashboard before the mid-term. This means the code it self should be reviewed lately in this week. The scope for the deadline before the mid-term was a fully functional admin dashboard that the mockup already given before. Okay, here are what's done in the admin dashboard:

New admin's view architecture that I've discussed in the previous section
Handling colorbox template
A few handlers inside soc.modules.gsos.views.admin

Here are what needs to be done and in-progress:

Submit my code for review, possibly after writing this post.
The test_slots was failed because I created new template and changed the list template for accepted organizations. I refactor the list for accepted organizations inside admin's view so other handlers can use it. I'll fix the test after my code being reviewed.
Adding test for soc.modules.gsoc.admin. By looking at other view tests, it just a test case to make sure the template is rendered with the right context.
A participant locations to mark all participant into a single map using MarkerClusterer. This is not a critical requirement and can be skipped.
Refined toggleable list patch as suggested by Mario.

GSoC 2011 Melange Testing Project: CP W5

2011-06-25T11:30:00.000-07:00

Hi,

This was quite a busy week. Learned a lot and worked a lot. I worked on writing the tests for soc.logic.tags and soc.logic.models.base. I had attempted to test soc.logic.tags during the pre-GSoC era but failed because I did not understand properly what 'scope' and 'fields' parameters were. I asked Madhu what a 'scope' is and he explained it to me that it was a solution adapted by Melange during the early days of development to define a relation such as a GSoC project belongs to a Student. So, a GSoC project entity was scoped under a Student profile. But, as the developers understood more about the Transactionality and other stuff in GAE, they switched to using 'parent' argument of an entity to define the scope. But, scoping will stay for tagging.

Earlier during the week, after Leo reviewed my patch for soc.logic.accounts, I pushed the code without first updating my local repo. So, I first pushed the patch and then pulled. But, then I could not find the test I had written in the online repository. So, I pushed it again and as a result to this, I ended up creating two heads in the repository. Sverre asked me to take care of the extra head I had created. I did not know how to do that. So, I googled and asked some friends. Some suggested using the 'strip' extension, 'merge' and some 'mercurial queues'.
Ultimately, I asked Madhu how to revert what I had done. He looked at the case and asked me if I do not rebase my patches after pulling. rebase!!!???What is it??. I then asked him what rebase here refers to. He gave a very nice tutorial on the concepts behind rebasing and the two schools of thought which I will never forget, the 'Purists' and the 'Whatever-ists'. :). I like Madhu's way of explaining things. Things get so easily printed in the mind.

What I learnt about the 'parent' argument from the discussion with Madhu, helped me to extend the old tests for soc.logic.models.base authored by Sverre and Leo. The tests reported errors when I ran because it could not find test_model module which I also thought does not exists and may be it has been removed. So, I defined my own dummy class TestModel extended from soc.models.base.modelWithAttributes, added some more tests to the module and submitted them for review. After this, when I was looking for other module to test, I saw that there was a test_model in old_app.soc.logic.models and this was the reason why it gave ImportError. So, lets see what Leo says :).

I now plan to re-use the old tests which Leo had written last year. The goal of my project has been re-scoped to write the unit tests until mid-term evaluations. I had written in my proposal that I will start writing functional tests in June. But, there are modules which have to be unit tested before. So, the target until mid-terms is to add as much unit tests as I can.

GCI 2011 GCI with New Architecture

2011-06-20T09:50:00.000-07:00

Hi all !

This week I have committed the GCIProfile and GCIStudentInfo models.And currently I'm working on the script to convert GCIStudent data to GCIStudentInfo and as I had some troubling understanding certain things I spent a couple of days in going through the Oreilly - Programming Google Apps Engine book.I've spoken to my mentors sometime back and got a better idea as to what to do.Instead of querying for the profile that each student entity points while creating the GCIStudentInfo entity, it can be created when the Profile for each entity is created,the parent-child relationship can be obtained easily like this.Besides this I have also written a basic RequestData object that will be created for each request in GCI and will add more properties and fields as we go ahead in the project.The patch is under review along with the script to convert roles and update references in the models.This week I haven't worked upto the mark.I will finish the conversion works and start with views at the earliest this week.

GSoC 2011 Integration with External APIs: OAuth Demo - Week 4

2011-06-20T08:07:00.000-07:00

This week i'm finishing Open Auth integration and will start to work on proposal syncing with Google Documents next week.

A demo view that uses Google Docs Service is available here. I made a few modifications for OAuth workflow diagram that you can see here.

After login process is complete, Google documents of user are listed.

Currently, tokens are stored by memcache. Access tokens are long-lived. So i marked a TODO to store access tokens in datastore (with Sverre's suggestion). Memcache is not a convenient way to store a token for such a long period. After talking to Mario (my mentor), we decided to put a 'remember me' checkbox with login button, so when user logs out access token will be preserved for next login ( that is also current behaviour). These are changes that i can work asynchronously while i work on proposal syncing feature.

Proposal syncing feature is the first deliverable of my project. There are few potential problems awaiting:

Integrating a full-html document which also has seperate image, css files.
To sync proposal without user signs to site, a task must be implemented to sync proposals when user is offline.

GSoC 2011 Melange Testing Project : CP W4

2011-06-20T05:59:00.000-07:00

Hi!
I had submitted two patches last week: one for test of app.soc.logic.dicts and the other for app.soc.logic.accounts. There were lot of mistakes and issues pointed out in the review. I have corrected soc.logic.accounts and submitted it again for review today. I also missed some functions to test in soc.logic.dicts and will do it this week.
I wrote the test for app.soc.logic.system this week and I had a long discussion with Madhu whether to use DjangoTestCase or unittest.DjangoTestCase. Madhu suggested using the DjangoTestCase and I insisted that I could test the functions with the help of unittest.TestCase itself. I had to test the soc.logic.system.getHostname(data=None) function and for this I needed a data object which had a 'site.hostname' attribute. After some grepping and exploration, I came to the conclusion that it should be a soc.views.helper.request_data.RequestData() object and the 'site' should be a soc.models.site.Site() entity which Sverre told me. But then I had a problem in instantiating the Site entity and the traceback reported that a link_id property is missing. So, after struggling for some time I told Madhu the problem and he referred me to test_utils.DjangoTestCase which had a site and data attribute which he said I use in my tests. He also said that RequestData entity should be populated and seeded to efficiently test it. My point was that all the entities of RequestData object are set to either False or None and also I do not need all of its attributes in testing. I only need the data.site and data.site.hostname attributes and this I could accomplish without using the DjangoTestCase and directly handling the RequestData and Site objects. Finally he asked me to go ahead and write the test with unittest.TestCase.
85% of my time is spent on reading codes and there are multiple windows that remain open almost all the time. I have to hibernate my system to avoid opening so many windows again.
I did not do much coding this week but hope to do more this week. Me and a friend of mine are planning to participate in ACM-ICPC regionals this year. So, we spent most of the time in studying for ICPC only.

GSoC 2011 GUI Overhaul: Redesigned Admin Dashboard

2011-06-17T20:59:00.000-07:00

Hi everyone,

As I promised from last post, I'll talk about redesigned admin dashboard in this post. Currently, admin dashboard is just a page with list, without any information, to other pages. I've provided mockup for admin dashboard with iconic information. There will be also new functionalities such as manage organizations with another sub functionalities to manage org, proposal, mentors, and students, participant locations, etc. There are two approaches to render child page in admin dashboard. These approaches is discussed in the following section :

Admin Dashboard with AHAH for Child Page

From last conference, Sverre told that Carol needs page in scope of admin should be stay on same page with parent page, that's admin dashboard. He said, it could be achieved with AJAX request/response. I tried the AJAX approach by making a div container to hold the HTML response. It's better to call this AHAH rather than AJAX. So, this div container is a hidden one and will be displayed after the HTML response injected to the container. The container was placed below the iconic information. To make the request handled with a minimum HTML responded, I created another blank template, base_ahah.html which contains following code :

{% block main_content %}
{% if ds_write_disabled %}
<div id=user-message class=error>
<strong>ALERT: </strong>Google Appengine datastore write
capability has been disabled at the moment. You will
<strong>NOT</strong> be able to save your changes at this
time. We are sorry for the inconvenience caused. Please visit
<a href=http://code.google.com/status/appengine>
http://code.google.com/status/appengine</a> to check
Appengine datastore status.
</div>
{% endif %}
{% block page_content %}
{{ tmpl.render }}
{% endblock page_content %}
{% endblock main_content %}

Page requested via AHAH must extends this template. We also need to modify form_base.html to check whether it was requested via AHAH or regular request with full base layout. If a page requested via AHAH, the JS code will add another query string, ?ahah=1. In the AHAH base template, we don't request any CSS or JS asset. I don't know if there's a way for LABjs loads once for the same script, and skip injecting the head for subsequent script calls. To be safe, any assets should be provided first when admin dashboard page requested. The form submission should be changed with AJAX. This was easily done with jQuery.post function. Any response, either invalid or valid, will be injected again in the AHAH container. With this approach, any admin sub page that need to be added later must exclude the asset its needed to parent page, admin dashboard. This make any sub page coupled to its parent page, which is not good.

Admin Dashboard with Iframed Colorbox for Child Page

Colorbox is yet another thickbox-like jQuery plugin. Colorbox provides a way to hold the content of external page in iframed container. For iframed template we need to create another base layout again, say base_colorbox, with minimum markup and variabel template. In iframed container we could extend block stylesheet and add melange script dependencies. Here's what admin dashboard with colorbox looked alike :

Within iframe, we need to specify the width and height for colorbox. We can specify the width and height in URL query string. Colorbox approach seems the easy way to implement without coupling to the parent page.

In the next day I'll provide the live instance to experiment with the new dashboard, so stay tunned.

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 3

2011-06-14T13:54:00.000-07:00

I've been developing generic methods to handle OAuth. When developing software it's important to create meaningful abstractions so a method can refer to these abstractions without need to understand low-level details what they do. For the OAuth part of my project a helper module handles this low-level details and caller that refers to these abstractions is an access_checker. It asks following questions:

Get access token for user.
Create a service for user.
Generate a OAuth authorization URL.
Process URL for OAuth verifications.

Low-level helper module does following operations for questions above:

Get access token for user:

Generate a unique key consists of user's key and current host (different consumer key,secret so different services and access tokens are used for each hosts).
Ask memcache or a storage mechanism to fetch token corresponding to that key.
Return token.
If there isn't a token return None.

Create a service for user:

Service is a GDocsService object that is used to make API calls.

Create a service for the current host.
Set service's input parematers to host's consumer_key and consumer_secret.
Set encryption method to sha or rsa.
Return service.

Generate a OAuth authorization URL:

Fetch a request token.
Save request token's secret to memcache for later use (when validating token).
Generate a authorization URL from request token.
Return URL.

Process URL for OAuth verifications:

If request URL contains a verification key as GET parameter, parse it.
Verify previously stored access token with parsed verification key.
Store verified access token to memcache or datastore.

GSoC 2011 GUI Overhaul: Toggleable List

2011-06-13T08:33:00.000-07:00

Hi everyone,

Today I'll talk about toggleable list for regular dashboard. To make you clear, there are two dashboards in Melange, one is regular dashboard and the other one is admin dashboard. The regular dashboard is the page that we see if we clicking the "My Dashboard" link on the left sidebar menu, or if you visit the url at gsoc/dashboard/(host)/(program_name). The admin dashboard is available in the left sidebar menu as "Admin Dashboard" if you have a role as a host. If you're login with account that listed as an administrator of melange instance, you have access to that admin dashboard. Admin dashboard can be visited at gsoc/admin/(host)/(program_name)

Currently, regular dashboard is loading a bunch of list at one time. Each list, or called component in dashboard view, in regular dashboard is fetching a lot of entities from datastore in the background. Each role gets different components. If someone has a role both as a host and a mentor, Carol's case, then she gets much components. Too much component will reduce usability. User needs to scroll hardly if she/he looks for component at the bottom. Fetching all list at one time is inefficient too. I've provided three ideas with mockups to handle this problem, one is with tab, two is with iconic page (the same as admin dashboard that I'll talk in the next post) and the last is with toggleable button. Carol agreed with toggleable button. The reason for toggleable button is user can toggle which list/component she/he wants to show without leaving the dashboard page.

I've provided the patch to melange-soc-dev. The implementation is quite simple. To make a component toggleable we just set toggleable and collapse to True inside the context. The list template will check this variable. If we ignore toggleable and collapse then the list will be requested when loading the page. We can set a component toggleable but doesn't collapse at first load by set the toggleable to True and collapse to False. With this approach, other pages with list, such as accepted_orgs, wouldn't be affected. In the next post I'll talk about admin dashboard that being redesigned right now.

GSoC 2011 Melange Testing Project : CP W3

2011-06-13T04:36:00.000-07:00

Hi,

For quite a long time I had been working on the online test runner and could not make it work on a live instance. As I had discussed it in my last blog post about my implementation that I intended to exploit some plugins of Nose to run the tests, it failed because of the reason that Nose uses some libraries which are restricted by GAE. So Leo asked me to take my other approach of discovering all the test modules in the tests directory and then make a suite of tests using the unittest.loadTestsFromName method. I used the os.walk() built-in function to first list all the modules with their absolute paths and then changed the paths into a dotted notation so that these paths could be easily used by unittest.loadTestsFromName to load the tests. I have used a lot of code from the gaeunit source specifically which is related to the browser interface and the test runner. The test runner had to be modified so as to fit in with melange code. I also had to include the tests directory, gaetestbed, nose, webtest and mox libraries in the app folder so that tests can easily use them when running on a live instance.
The online test runner runs on my local development environment but it shows errors that are due to the bugs in the runner itself. Below is the video of the melange tests running on my local machine through the browser.

I have stopped working on the runner as of now. But in the end, I discovered that I have extended some of the features of gaeunit. It has been 2 years since gaeunit was last released, so I was planning to contact the gaeunit developers if I can develop for them.

During the last week conference call, Sverre asked me to treat the online runner as a side project and concentrate on my testing project now. So, I have started working on my project and wrote the test for app.soc.logic.accounts and submitted it for review. At the time of writing this post, I have also almost completed the test for app.soc.logic.dicts with a few methods left to test.

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 2

2011-06-06T12:18:00.000-07:00

It's been two weeks since coding started, due to my heavy exams i haven't been working too much and had to skip first week's blog. I updated my timeline with target deliverables [0].

First thing to solve is how OAuth will be handled with a generic solution. As we wouldn't want to handle all jobs (e.g. token check, redirection, access token parsing) in every seperate view, there needs to be a generic handler (a 'token_required' decorator and a 'gdocs_service_object' middleware in Django's terminology) for views that needs to make GDocs api calls over OAuth.

I started to work on this diagram to discuss OAuth mechanism with community:

One of the particular problems when playing with gdata library was a not-self-defining error about unicodes. As GAE returns entity field values in unicode, a method in built-in hmac library that expects 8-bit strings was crashing with this error message:
character mapping must return integer, None or unicode
Solution was simply encoding field value to utf-8: consumer_key.encode('utf-8'). I don't think same error would arise when using rsa instead of sha with authentication.

[0] -http://code.google.com/p/soc/wiki/GSoC2011IntegrationWithExternalAPIs#4._Project_Timeline

GSoC 2011 Melange Testing Project : CP W2

2011-06-06T06:57:00.000-07:00

My task for this week was to implement an online test runner which enables us to run our unit tests online rather than on the local machine. I have written the necessary modules which provide a very basic functionality and features for now. More features can be added and improved with time. Leo had suggested me the gaeunit Open Source project to study and see how they do it.

I outline the basic process of how gaeunit does the magic :-

gaeunit defines two main request handler classes - MainTestPageHandler and JsonTestRunHandler. The request to run the tests is handled by the MainTestPageHandler class which first validates the URI arguments and renders an html /plain report according to the 'format' argument. Our interest lies in the generation of the html report.
When the request is to render an HTML report, gaeunit first creates a suite of all the tests found according to the 'name' parameter passed to it in the URIs. 'name' can be any package, module or class. This suite of tests is then converted into a JSON format with structure {"module_name": {
"class_name": [list_of_methods]}} and passed with the HTML response to the browser.
As soon as the response is loaded in the browser, it triggers a javascript whose task is to iterate through the JSON object and pass the (module_name, class_name, method_name) tuple to the server by generating an asynchronous XMLHttpRequest which then waits for the data to be sent by the server.
The GAE server upon receiving the tuple runs the JsonTestRunner and the test result is collected in a unittest.TestResult object. This TestResult object is then again converted into a JSON object and returned to the browser.
The browser then sends the data to the appropriate fields in the web page

My Implementation:

My implementation does not work presently and it needs lot of debugging. I moved the tests folder to melange/app/ so that GAE finds it easily. I wrote two modules online_run.py and online_tester.py. online_run is colleague of tests.run.py with lot of code borrowed from it to implement the test runner for online testing.

online_tester implements classes to handle and manage all the tasks right from initiating the tests to printing the test reports. I thought of exploiting the 'testid' and 'collect' plugins of nose. Running the tests with these plugins only collects the tests without actually running the tests and assigns an integer id to all of them. We can run a specific test next time by just passing the integer ids. So I have defined three classes MainTestPageHandler, PrepareForTests and RunTests. MainTestPageHandler just renders a view where users can select which tests to run. The arguments are passed as POST arguments to the PrepareForTests handler which is supposed to collect the tests and assign IDs to them. The number of collected tests is then passed to the browser with HTML source which contains a javascript . This javascript passes the integer IDs to RunTests which makes use of the runner in online_run module to execute the tests and returns the HTML output to response.out stream.

This implementation is not working and I asked Madhu for help. He said may be I have messed with the appengine paths. I have to try again keeping the tests in app/ and making the necessary changes in my scripts.

Also the paths like '/home/praveen/testmelange/app/tests' which I have included in the python paths according to my system will not work on GAE because we do not know about their file systems.

You can take a look at the error reported by my implementation at my instance[1]
Just choose 'Run all tests' and hit Run.

[1] http://melangetesting.appspot.com/runtest
[2] online_run.py http://tinypaste.com/da4b5e
[3] online_tester.py http://tinypaste.com/1f3b88

GCI 2011 GCI with New Architecture: Coding phase - Week 1

2011-06-05T00:40:00.000-07:00

Hello all!

I'm sorry for the late post.This week has not been as expected, my internet was down for a couple of days and then my deadlines fell apart and I'm lagging a bit but I will try to cover it mostly by this next week.The patches that I have sent got reviewed and I re-sent the patches with the suggested changes.When moving the list generator from gsoc module to core I did not follow the import conventions and I was redirected to this wiki and I learnt from there.And I moved the common fields in GSoCProfile to soc.models.role.Profile to prevent code duplication.Also the grade property is added to the parent StudentInfo class as that contains the school_type and Finally I have modified the role_conversion script to convert GCIRoles and update references to GCITask and StudentRanking model.I was asked to move it to a seperate file in the GCI module rather than mixing up both in the same file, so I will do that now and after this I will finish up with RequestData so the from beginning of next week I can work with views.
In the mean time I have ordered Oreilly - Programming Google App Engine and started reading the book, it is a really nice book which is very helpful to understand App Engine better in detail.

GSoC 2011 Melange Testing Project : Coding Period Week 1

2011-05-26T10:09:00.000-07:00

The coding period has officially started and I have been rather slow and falling behind the schedule. I still have not started writing a test but I will try to give more time to the project in a couple of days.
It was quite an eventful week. I have been waiting for the welcome package by google and it was already May 23, so I just asked on the GSoC students mailing lists if anyone has already received it. Some of the students were very furious about starting such a thread as Carol had stated the deadline to be May 27. What I believed was that its not necessary that everyone will receive the package after 23. I also made a remark of why he bothered to read and reply if he was not interested. I was surprised to see that only one or two previous GSoC students from the 4k members strong mailing list care to answer constructively. It was encouraging to read Phillip Herron's comment which I quote below:

"Just be patient they _will_ come from last years GSoC it just takes
time if you consider the fact they have to individually create like 1000
of these things and get them each shipped to each of our address's
just be patient. Its a great feeling when it comes :) "

This is the reason why I love Melange. You are never restricted but re-directed if you are going wrong. I still remember when Mario told me that its bad collaboration when I commented a little offensively on the efforts of an aspiring student during the pre-GSoC time. I am also amazed by Carol's patience at answering the same set of questions again and again asked by students on the gsoc mailing list. Anyway, I learnt from the criticism that I should be patient and keep good relations and collaboration with fellow students.

    I am going slow at the project and falling behind the schedule. I have not written a single line of usable code till now. My project is concerned with writing tests for the logic and views and as I stated in my last post that I shall start with app.soc.logic modules, so I have generated a directory structure of app.soc.logic which is basically a tree like structure of the files in the directory. All the previous tests have been moved into tests/old_app directory as they conflicted with the new framework. My aim is to re-use the old tests and make the necessary changes with respect to the new framework. I will start with the app.soc.logic.allocations.py module.

I had a meeting with my mentor Leo yesterday i.e. on 25 May '11 and I discussed with him a few concepts related to testing.I asked hime what should the basic approach be while attempting to write the tests. Basically, we should first understand what a specific function does, think about all its use cases and then we test each of the use cases.
   I also have to implement a view to run the tests online rather than running the tests locally. I did not exactly know how to implement this. But Leo suggested that there are already some open source projects like gaeunit which do that. So, I referred to the gaeunit project and its just a single module which implements that. My approach was to run the tests using the python environment of GAE and somehow capture the test results but there is already a TestResult class in unittest module which returns an object for such manipulation. So, I need to make use of this class and tasks to implement my view.
   So, my goal for this week is to write the view and implement the test for app.soc.logic.allocations.py module at least.

GSoC 2011 GUI Overhaul Project: Start Coding

2011-05-26T09:34:00.000-07:00

Hi everyone, sorry for my late post :). In this post I'll talk about what things have been done by me to accomplish my project. A week ago I had conference with Carol and Daniel. We were talked about what admin, Carol, needs to improve the usability of administration dashboard. Actually before the meeting, I've been working to add breadcrumb and refactor melange.map.js. So, I'll talk about the breadcrumb and melange.map.js first. If you don't have an idea what breadcrumb looked alike, take a look at the following picture.

The breadcrumb is something like path navigation that shows site hierarchy. Jacob Nielsen has a nice written summary of why breadcrumb navigation increasingly useful. I thought that the way breadcrumb rendered would be the same as Header, Footer or MainMenu in base_templates.py. My first thought was not so good. I defined a global map of named URL patterns with theirs hierarchies, something like this one :

...
BREADCRUMB = {
'gsoc_homepage': {
'name': 'Home',
'parents': [],
'url': 'homepage'
},
'search_gsoc': {
'name': 'Search',
'parents': ['gsoc_homepage'],
'url': 'searchpage'
},
'gsoc_accepted_orgs': {
'name': 'Accepted Organizations',
'parents': ['gsoc_homepage'],
'url': 'allProjects'
},
'edit_gsoc_profile': {
'name': 'My Profile',
'parents': ['gsoc_homepage']
},
'create_gsoc_profile': {
'name': 'My Profile',
'parents': ['gsoc_homepage']
},
'gsoc_dashboard': {
'name': 'My Dashboard',
'parents': ['gsoc_homepage']
},
'gsoc_tax_form': {
'name': 'Tax Form',
'parents': ['gsoc_homepage', 'gsoc_dashboard']
},
'gsoc_enrollment_form': {
'name': 'Enrollment form',
'parents': ['gsoc_homepage', 'gsoc_dashboard']
},
'gsoc_admin_dashboard': {
'name': 'Admin Dashboard',
'parents': ['gsoc_homepage']
},
'lookup_gsoc_profile': {
'name': 'Lookup profile',
'parents': ['gsoc_homepage', 'gsoc_admin_dashboard']
}
}

...

class Breadcrumb(Template):
"""Breadcrumb template
"""

def __init__(self, data, current_view):
self.data = data
self.current_view = current_view

def context(self):
"""breadcrumb_path context will hold breadcrumb path of current view
which consists list of tuples containg page_name and url
"""

from django.core.urlresolvers import reverse
context = {'breadcrumb_path': []}
for view in self.current_view:
if BREADCRUMB.get(view, None):
for parent in BREADCRUMB[view]['parents']:
 if BREADCRUMB.get(parent, None):
   href = None
   if BREADCRUMB.get(parent).get('url'):
     func =
self.data.redirect.__getattribute__(BREADCRUMB[parent]['url'])
     href = func().url()
   elif self.data.profile:
     href = self.data.redirect.urlOf(parent)

   context['breadcrumb_path'].append((BREADCRUMB[parent]['name'], href,))
context['breadcrumb_path'].append((BREADCRUMB[view]['name'], None,))
break

return context

def templatePath(self):
return "v2/modules/gsoc/breadcrumb.html"

...

Sverre suggested me avoid the global map, and define a method that returns the breadcrumb navigation, more or less like this one :

class ProfilePage(RequestHandler):
"""View for the participant profile.
"""

parentView(self):
return (self.redirect.urlOf('homepage'), "Home")

I agree with Sverre's suggestion, I'll refactor my code :).

After the breadcrumb, I also plan to refactor melange.map.js and combine the map in profile.js into one. All map functionalities should be placed into melange.map.js. I've worked on the profile_readonly.js before. Basically profile.js and profile_readonly.js have the same code, except the event handler to update the marker was removed in profile_readonly.js. As you may see, every time I need a map on page, I create a mypage_with_map.js and copy the code from profile.js or melange.map.js, hack a little bit to suit the need and tadda I betray the DRY principle. That's why we need to refactor the melange.map.js. The MAP API also need to be updated to v3, same as the map we see in organization homepage.

Back again to our discussion with Carol. Here's what I noted after the meeting :

If we're login as a host and view the dashboard (My Dashboard), we will see the page overloaded by Jqgrid list. Carol wants the list is grouped. Akeda has suggestion to group the list by tabs.
When students with project announced, there might be a student consider himself ineligible to participate. If Carol rejects that student, there will be available slot to accept another student that already rejected. Carol needs this feature to withdraw / accept student in her dashboard.
Carol needs the Admin dashboard to be enhanced.

My first thought was Carol wants the Dashboard ("My Dashboard" I think) need to be cleaned up, since it contains a lot of list components. So far I am working with the data from seed_db and didn't realize that in the live instance might contains hundred of entities. I posted my mockup with the tabs in milist :

GCI 2011 GCI with New Architecture: Beginning of the coding phase

2011-05-25T10:30:00.000-07:00

The coding phase has finally begun and I started my project with the GCI Profile model.Currently, GCI has a different notification system where a user X can subscribe to the notification to a particular task by clicking a star on the task and can un-subscribe the task by clicking on the same star again, org_admin,mentor and the student claiming are automatically subscribed. In addition to this, we will now have Profile based notification settings where a user will be given an option to whether subscribe for the notifications automatically or not, for example a student at the time of profile creation can check/uncheck automatically subscribe to tasks I claim.The same can be applied for mentor and org_admin.The GCIStudentInfo model having the following additional fields specific to GCI was written next, number_of_tasks_completed , consent_form, student_id_form, grade.

The next step was to convert the roles itself.Basically for each student,mentor or org_admin entity I had to create a GCIProfile.And then I had to update all these references in the models containing them. I have used deferred library which lets us retrieve entites in batches and update their references easily.

My wiki page is at [0] and my instance is at [1]
[0] : http://code.google.com/p/soc/wiki/GCINewArchitectureGSoC2011
[1] : http://melange-slingshot316.appspot.com/

Week before the coding phase

2011-05-24T02:53:00.001-07:00

I am very late on this particular blogpost, apologies for that.A week prior to the coding phase, I was working on asynchronous file uploads (for the student forms), I have used the jquery-fileupload plugin for this task and I have some problem with sending the request and so will look into it sometime later.But in this process, I have learnt how to add any javascript files to Melange and as for this, I first had to put the plugin file in app.jquery and add it's corresponding entry in soc.content.melange-dependency and run bin/paver closure to create minified version of it.I learnt that doing all that logic inside the function in the dependency array is not really the way it is meant to be.Ideally, one should create all the logic for a form in a template which is called using the tc() facility[0], which loads the javascript
template file and put variables (for example an id called #foo) in the context of the template as we can get the variables in the context (after we use the tc() facility in the django template) by using var _self = this; and then _self.context..I also learnt that Whenever we put a "null" in the array, LAB.js will wait for the above scripts to be downloaded, evaluated and ran before evaluating and then run the others.My mentor Madhusudhan gave me a lot of support and taught me many new things and thanks to Mario for advice on using javascript in Melange.

[0] http://code.google.com/p/soc/wiki/JavascriptCSSDependencies

GSoC 2011 GCI with New Architecture: Week before the coding phase

2011-05-24T02:53:00.000-07:00

GSoC 2011 Melange Testing Project : CBP

2011-05-19T09:58:00.000-07:00

About one week is left for the start of the official coding period. I have been unable to concentrate much on my project as there were end-semester exams till 13th May and after that I was busy in packaging all the stuff as we have to empty our rooms before going for the summer vacations. I played with my ACM account and explored all the facilities that I could make use of. I found many good books related to programming in the Online Books section. It would have been great if there were more video tutorials available for the students from those which are available for professional members.
I had a meeting with my mentor Leo on 16th May and we discussed about the project and I have set coding plans for the first three weeks during the official coding period. I will start with the tests for soc.logic which require a basic knowledge of Django and GAE. Leo suggested reusing the old tests and making changes according to the new framework.
I have not much to write as I have not done anything significant. I have settled down now and seriously started working on the project today. My next post would be more concrete and specific.

Integration with external APIs - Getting Ready for Coding

2011-05-17T07:27:00.000-07:00

There is about one week for official coding starts period and i'm preparing myself for it. As i stated at my timeline i won't be able to work on project or rarely work between May 25 - June 10 due to my final exams at university. Then i will be free to work day n night at summer.

I made my first commit to Melange - added a new unit tests for "accept_proposals" task - despite some trouble while commiting it. This was exciting :) Last week i started to work on my project. Created a playground instance [0], created a wiki page for tracking progress [1]. Live instance for now just have a model for storing exported documents' IDs and names. I need to create a view right now to list them and another one to add some test data through Data Seeder. These will help me to learn Melange's jqgrid lists and Data Seeder.

This weeks plan for me until next Thursday:

Create a view that uses Data Seeder to add sample data, or integrate it with /seed_db
Create a view that lists these documents through Melange lists.
Plan and achieve a meeting with my mentor (Mario).
Get ready to start coding.

[0] http://orcuns-melange-playground.appspot.com

[1] http://code.google.com/p/soc/wiki/GSoC2011IntegrationWithExternalAPIs

GSoC 2011: Google Code-In with new architecture

2011-05-01T02:27:00.001-07:00

Hello Everyone!
I'm the last one to introduce myself here, had exams last week so sorry about that.And I'm Selwyn Jacob from India, one of the Summer of Code Student's for Melange this year.I'm doing my Computer Science Engineering in GITAM University and I have always been Inspired and Amazed by the Free and Open Source World.Google Summer of Code has been my dream and I just cant believe that I'm living it, I'm just too happy to put it into words and this is just the start and I have a long way ahead to go.

My Project is to port the Google Code-In module to the new Architecture.Google Code-In is a contest to introduce pre-university students to the many kinds of contributions like writing Code,Documentation,Outreach etc that make open source software development possible. The GSoC module has undergone a major re-write this year, the UI and the Views layer have completely been changed and the same has to be reflected for the Google Code-In module too and this is the focus of this project.I would like to thank Madhu and Sverre for helping me out right from the first day and welcoming me to the community.They have answered all my noob-ish questions and guided me with patience, so many thanks to them.This week I will be going through the current GCI code, the wiki pages and discuss the plan with my Mentor Madhu and Co-Mentor Lenny.
Looking forward for a great learning and contributing experience this summer and be a part of Melange Community :)

GSoC 2011: Integration with External APIs

2011-04-30T07:08:00.000-07:00

I'm Orçun Avşar. Yet another Melange student for GSoC 2011. I'm from Çanakkale Onsekiz Mart University, Çanakkale, Turkey. I study Computer Engineering. Summer of Code is a dream for most students and i'm very happy and proud of myself about taking first step on this. Since begining of university, I always loved the philosopy of Free Software. Devoloping software for everyone not for groups gives me enthusiasm on this.

We've chatted and made a Skype conference call with Melange team, we were added as commiter to project. My project aims to integrate External APIs ( for now Google APIs) with Melange. I will first start from Google spreadsheet API. then discuss other APIs like Calendar, XMPP messaging to see what else can be integrated.

My timeline till 13 May consist of theese:
- Read Melange source-code
- Read documentation and wiki pages
- Read old important discussions, issues
- Read new tests
- Get familiar with tools that Melange use (like zc.buildout)
- Get familiar with Google Data Apis and OAuth
- Submit small patches to warm up.

As well as studying melange code, i will start a playground GAE project to play with OpenOuth. You can see my proposal from here.

Mario is my mentor on that project. Also special thanks to Sverre for answering my newbie questions on #IRC without getting bored. That helped me a lot.

GSoC 2011 GUI Overhaul Project

2011-04-28T08:57:00.000-07:00

Hi, everyone. This blog post is about self introduction by me, Akeda Bagus, after my proposal, GUI Overhaul (Backend and Frontend), got accepted. For those who don't know what Melange is, you can read its project page and try the getting started page — yes, Melange is open source project that powers Google Summer of Code (GSoC) and Google Highly Open Participant (GHOP) contest. Melange was written in Python, using Django framework, that runs on Google App Engine (GAE). So, if you want to dig into Melange codebase you should have basic knowledge of Python, probably Django framework too, and GAE. Don't worry, Django and GAE have good documentation available on their sites. When I cloned Melange's repository for the first time and read its codebase it was little bit confusing because at that time Melange's fronted is being renovated. Some new contributors didn't recognize which one is the new codebase (V2) and which one is the old one. But it doesn't matter actually if we know how Melange system dispatch the request and how django template renders a response. Maybe in the next post I'll talk about the internal working of Melange. But for now this post is just about what "GUI Overhaul (Backend and Frontend)" project talks about.

As the name suggests, "GUI Overhaul (Backend and Frontend)" project has a goal to improve the newly Melange's UI both from backend (administration) and frontend (public, student, mentor or organization admin). The idea of administration interface improvement is described at Melange's GSoC 2011 Ideas Page. The rest of the timeline during GSoC period will be devoted to improve the frontend, of course, after administration interface is finished. This project fits my skill as my previous experience on GSoC 2010 also to enhance GUI side — It was "Improve GUI Configuration" project. But this time is little bit different than the last year — where I defined the spec of GUI and Validaton by myself. In Melange, I don't use administration interface regularly, neither know what kind of features wanted mostly. So, I need to gather requirements from users that use Melange in regular basis. From the requirement, we build a spec and then implement it. Of course, there will be much iterations over the spec and implementation until we get the "hey, this is awesome" or "we don't need that". I googling a bit about requirement gathering and found articles written by Joel Spolsky. The article has four parts and worth reading it. Recently, I found a nice quote that related to spec and implementation :

Implementations and specifications have to do a delicate dance together. You don't want implementations to happen before the specification is finished, because people start depending on the details of implementations and that constrains the specification. However, you also don't want the specification to be finished before there are implementations and author experience with those implementations, because you need the feedback. There is unavoidable tension here, but we just have to muddle on through ... I think we're doing OK.

GSoC 2011 Melange Testing Project : Community Bonding Period

2011-04-28T01:23:00.000-07:00

Its been two days since the GSoC 2011 results were declared and I am still unable to tell myself that I am now a Google Summer of Code student. :) . A vote of thanks and gratitude from the very depth of my heart to the Melange developers.

Python is awesome and I am now in love with it. My project is on testing and frankly speaking I have never studied about software testing before. The only idea about testing which I had was during solving exercise problems proposed in Google's Python Class videos. The coding problems had to be solved and then their output tested.

I explored about the Melange testing project more and more. I did not understand many of the terms like 'Test Driven Development', 'Continuous Integration' etc mentioned on the Melange's ideas page, but I googled and learnt about them, followed online tutorials and resources linked on the ideas page. I joined the melange-soc-dev mailing list and asked Sverre if I understood correctly what the project is aimed at. He said 'Sounds about right :) ' and then he pointed me to the 'Getting Started' guide. Since then there was no turning back and I devoted most of my time getting familiar with the framework and the testing modules. I explored the Melange's blog and read the posts of previous GSoC students. Leo and Savitha very well explained how they tried to understand the codebase. I did not know about the tools like tree and pylint.pyreverse before.

Then I decided to write a test for one of the modules in app.soc.logic and chose the 'validate' module. The test was simple but difficult to write if one does not know about the organisation of the modules and how the functions were used. I took me two days to get a feel of the code and get familiar with the directory structure. I finished the test, learnt and fixed the style issues and finally signed the CLA and the module was integrated into the codebase. That was my first contribution to Melange as a pre-GSoC exercise. This gave a boost to my confidence to contribute more to Melange.

I knew there were many students competing for the project and I decided to write my proposal differently so as to have an edge over other students. I decided to show a graph of the failures and errors that occurred everyday, the data which I collected everyday by running tests over a period of ten days. I also thought of showing a test code vs production code analysis. April 8 was the last date of submitting the proposals and I was busy with exams till April 5. Melange was the only organisation on which I had concentrated since the beginning and I knew I had to write a good proposal to get selected and dedicated all the 3 days in writing the proposal. The 'graph' and 'the code analysis' do not reveal anything groundbreaking but that was what my mind could think of then.

I regularly followed all the discussions on IRC, melange-soc-dev and gsoc lists. There were criticisms about the tinyMce editor and asked Sverre if the editor size can be increased because it seemed too cluttered. He asked me to do it. I read the tinyMce documentation and found many more helpful buttons can be added to the toolbar. Mario helped me in writing the configurations for the 'basic' and 'advanced' editor settings. I had never learnt jQuery before and I believed that I should not say " I don't know jQuery, so I can not do it". Instead I followed some tutorials and then it was easy. I also thought "Hey! you applied for the testing project and you are working on the editor."

It was a great experience working hard to bag the project. It is the first time I am contributing to an Open Source project in a major way and to see my name in the Author's List, Mario's Code Swarm video and the commit messages fills me with a feeling that I find difficult to describe.

Yesterday I attended my first conference call with the Melange developers and heard the sweet voice of Carol :P . I thought the meeting would be very formal but all of them were very jovial. Sadly, I could not introduce myself to other members as the mic was not working. Hope to fix it soon so that I can communicate during the meeting on Monday. I have exams starting Tuesday next week, so will discuss about the project plan with Leo after 15 May 2011. I will get a week of time before May 23 to read about GAE, Django and Software testing.

Melange community is really the 'awesome-st ' community and I wish to meet the developers and have fun with them at least once in my life. :) :)

Social Features Project: Week 12

2010-08-16T08:45:00.000-07:00

The final week of GSoC has rolled in and my blog post is incredibly late too as I spent much of my time working on my code.

My final version is upload on my app engine instance here. It consists of User Pages, Calendars and a Map View. The Calendars feature is functional and works as per the workflow I had originally planned. The past week had some last minute bugs which popped up only when I ran my code on AppEngine as they were not detected while I was testing on local host. The bug was that I was storing a universal session authtoken for the entire application instead of one for each user. This did not show on my own machine as I always used it under my own google account! Hence, I had to spend time to rectify this and ensure that each User model stores the session token and later the services uses the set token when the user decides to use the calendars feature.
This issue has been fixed now. However, I believe it will take a bit more work to integrate it and will have to spend time with my mentor in revising the finer details before actual code integration with the main code base. I look forward to doing this in the week after next.

My latest code is uploaded to my bit bucket account here.

Some problems still awaiting to be solved are nasty server error being cause by issues in the Google Calendars API, which I thought I resolved however, they still seem to pop up randomly. The issue has been taken up by others using the API here, although there does not seem to be a good fix for this yet.

Another thing that needs to be settled is the scalability of the Maps View. As GSoC has about a 1000 students, 1000 mentors and then org_admins etc. Right now, as I am having to manually populate the data, the map view has hardly 40-50 pointers on it. Hence how will the JS handle a large number of objects is to be seen as yet.

Finally, it has been a great experience with loads of learning and coding. It has been great working with the Melange team, and specially my mentor Madhu. :)
I am looking forward to staying with Melange and contributing to the project as best as I can in the future too.

GSoC 2010 Data Seeder: Wrapping up for end of GSoC

2010-08-16T08:00:00.000-07:00

So GSoC is coming to an end, it's time to wrap up the project to a pretty much fully functional and usable state, while there could still be much to do. So, my latest changes mostly include usability improvements like being able to test the configuration for a data provider and get some sample values, include provider and parameters descriptions, being able to validate, download or upload a configuration sheet.

Other improvements include some visual styling, although I personally find my work very ugly, I'm hoping someone with better visual appeal might help me with this part in the future.

As a final proof of concept, I will provide a number of configuration sheets that will demonstrate the full power of the module: being able to generate mass amounts of "real world" data very easily (although doing a first-time configuration might be a little time consuming).

I have also encountered some unexpected problems. It seems that GAE (or maybe the Mapper API, I'm not sure) limits the amount of data that a task can receive as a parameter. Configuration sheets can get pretty big (I won't be surprised to see some reach tens of KBs) so I needed to find a workaround. My solution involves storing the actual configuration sheet in a model, and then each task would read the configuration sheet from the datastore. While this solution speeds down the process by quite a bit, it might prove more scalable and also provide some historical value, should the configuration sheets remain in the datastore.

Enough about the technical stuff... It's been a real joy working with everyone around the Melange community and I'm sorry if I haven't been all that serious all the time during GSoC. Many thanks to my mentor Mario who has provided me with priceless information all the time and who has made all my work so far possible. Also, thanks to all the other developers (Lennard, Sverre, Madhusudan and Daniel), you are simply great and it's been a pleasure to work with you all. Cheers to all other GSoC students (Leo, Pavel and Savitha), I hope we'll see each other in the future as well. Having said that, it's time to continue working on the project, there's still a lot of things that can be improved.

Cheers!

GSoC 2010 Melange Testing Project: Week 12 (Final) of the Coding Period

2010-08-16T05:09:00.000-07:00

In this week, I have cleaned up, commited and pushed all the tasks testing code to my online clone. I also worked on adding test cases for soc.logic.cleaning with suggestion of Sverre. Besides, I have also uncovered a couple of bugs in the soc.modules.ghop.tasks.task_update module and sent a patch to the mailing list. Lennard was very kind to help me merge the patch. I have also uncovered one issue with my test code for soc.modules.gsoc.tasks.accept_proposals (set the timelimit parameter too small) and then corrected it. In addition, I have also investigated the diference between the console and html coverage report. It seems that the console one counts statements while the html one counts lines.

When I tried to move the the gaetestbed and pymox modules to buildout dependencies, however, I found the easy_installed versions of both modules have problems (easy_install is used during Melange buildout). For gaetestbed, I found that the gaetestbed on PyPi is an old version which cannot check the html body of an email. So, I emailed the author to ask him to put the latest version onto PyPI. The author (JJ Geewax) was very kind to have helped me do it. For pymox, the easy_installed version available is a quite old version (0.5.0) which has a bug fixed in the later versions. Unfortunately, these newer versions are not available on PyPi, so not easy_install searchable. I reported the issue to the project issue tracker. One of its authors (Steve) was also very kind to helped me to put a newer version (0.5.3) to PyPi. After all these problems have been sorted out, there remains one issue that gaetestbed currently depends on webtest even though you do not need this part of function due to the design of gaetestbed, which the author agreed to resolve in the near future. After consulting Sverre, I finally moved the the gaetestbed and pymox modules to buildout dependenciesi and then commited and pushed the change to my online clone.

In the next week, I will add the experience of how to do mail and taskqueue related tests with gaetestbed to the testing wiki page, make a patch for removing the cron/job related modules and then send it to the list as well as add more test cases for existing Melange modules.

This is the last week of GSoC2010. It was a quite exciting and enjoyable summer. My project is very interesting, my mentor (Sverre), other developers (Lennard, Daniel, Madhusudan, Mario) and GSoC administrator (Carol) are very helpful, and my fellow students (Felix, Pavel and Savitha) are also very nice people to work with. Specially thank my wonderful mentor, Sverre, who is always ready to help me out when I got stuck and without whom I may not complete my project. During this period of time, I have also learned a lot about GAE and Melange, I have also come up with something useful and merged to the trunk, and uncovered several issues. Though the program has ended, it does not mean the end of my project or my involement in Melange. On the contrary, I will continue to work on the testing project after GSoC, i.e. add more test cases for more modules, improve the testing design and code, integrate Felix' data seeder, and so on, which have not been done because of the time limit of GSoC or dependency. I also hope to work for Melange beyond the testing project and hopefully become one core developer of Melange in the future. I also wish to come up some ideas to improve Melange so that I can serve as a mentor for Melange in the next GSoC and experience the excitement of these guys about the upcoming mentor summit :-).

Document Editor: Last week of GSoC coding

2010-08-16T04:57:00.000-07:00

Last week of GSoC coding was devoted to... everything :)

I spent 4 days on form saving framework. First of all I had to fix JS code, which sends dirty fields to the server. The problem was as follows: framework was saving only dirty fields of the form. If you loaded the form for the second time, initial state might have been rewritten. This issue was fixed and now form states are consistent. Then I've implemented back-end. I needed to create model which stores temporary fields as a JSON dumped dict. Some logic class which manipulates all the stuff has been implemented. I also had to change createGet, createPost, editGet and editPost methods of the soc.views.models.base.View class to support temporary form saving. Temporary form saving for the model is enabled via "form_autosave" key of the params dict.

After form saving I started testing the code. An issue happened immediately. There was a problem with home documents for different models which derive from soc.models.presence.Presence. Views and logic were not aware of the revision control for the document, so exceptions were raised. This issue is fixed now. I also had to fix soc.models.seed_db.seed function to seed documents in the right way.

I've also refactored diff template, so it can now show arbitrary number of diffs for different revisable fields of the model.

At last I worked at the UI. I've added "Show revisions" button on the edit page for the model. Models that support RC are now distinguished by the "revisable" key of the params dict.

I've also rearranged fields of the Document and StudentProposal forms, so they look as supposed.

Final version of my GSoC contribution was uploaded to the appspot instance http://melange-editor.appspot.com

GSoC coding has come to it's end. And I must admit that it was quite an experience. I've already participated in large software projects and earn my living with development, but GSoC is really memorable. It was cool to participate in such a big and complex open-source project! I want to thank Melange dev team, Carol, fellow GSoCers and my mentor Daniel. They were very helpful and opened. And, you know what? I wish they let me continue contributing :) Come on, guys, I love it!

Document Editor: Week 11

2010-08-09T05:48:00.000-07:00

The goals for the last week were to finish RC for StudentProposal model and start working on forms auto-saving. Revision control for StudentProposal is fully functional now. I also made some changes to decouple RC from the model's code.

After that I started working on form auto-saving. The main architectural concern was "what to store". The decision was made to store only changed fields of the form. Then I tried to find some existing solution to figure out dirty form fields. Mario proposed dirtyFields plugin for jQuery. It looked cool, but I found out that it triggers user-defined event for each field change. Furthermore it doesn't provide any means to retrieve collection of changed fields with new values. That's why I decided to develop my own script based on dirtyFields. It's now ready and pushed to my repository clone.

This week I will finish auto-saving for forms, write tests and make code clean-up.

Week 11: Social Features Project

2010-08-09T05:21:00.000-07:00

This week I spent mostly on doing the Map View which basically involved writing mostly JS code to retrieve and then filter data based on search terms that are entered in the text input box. Initially, I had planned to allow users to filter based on location, projects etc, later I decided to just let them enter keywords. Internally, I check this keyword against tags as well as locations of users and events to filter.

This way, users do not have to specify what kind of search they want, they can just type any keyword such as 'Python' or 'Australia' to get the relevant filters.

The Google Maps API gave me a bit of problem as it took sometime and debugging to realize some method names have changed completely such as for adding/removing markers etc between V2 and V3, and I had to accordingly change my code.

The latest version with all three features is on my instance.

I have quite a bit of things left to do in the final week of GSoC:

write tests
write code to catch the exceptions(Application Error) being thrown by Calendars API and re-send requests.
test the functionality and workflow of user pages, calendars and maps view for bugs and fix them.

GSoC 2010 Melange Testing Project: Week 11 of the Coding Period

2010-08-09T03:12:00.000-07:00

In this week, I have uncovered a couple of bugs of existing Melange modules with the help of related test cases and sent pathes to the list, which again proves the importance of testing. I have also resolved the error "raise self.__TranslateError(e.application_error, e.error_detail) UnknownQueueError" by adding the path of taskqueue yaml setting file to the instantiation arguments list (root_path) of taskqueue_stub.TaskQueueServiceStub. Besides, I was also looking for a good way to test the mailing and taskqueueing functions in order to test that the correct mail has been sent out and the correct task has been scheduled. During my search, I found gaetestbed which is very simple but serves the purpose well. Although it is not hard to make an equivalent one myself, why not reinvent the wheel if you can make use of an existing one. Its licence is also very generous (GNU V2). So, I added it to Melange tests and also extended its MailTestCase and TaskQueueTestCase and then put them into tests.util module. For MailTestCase, I overrided its assertEmailSent method to add two functions

It will print out all sent messages to facilitate debug in case of failure.
It accepts an optional argument n which is used to assert exactly n messages satisfying the criteria are sent out.

Later, I plan to add the functions of checking the CC field, allowing to partly match the subject field or even supporting regular expression. For TaskQueueTestCase, I have not added any functions yet. I further used them in my mail and taskqueue related tests, and cleaned up some of my existing testing code and then commited and pushed them to my online clone. In addition, I have also done some new tests on gsoc tasks and statistics etc. and managed to increase the testing coverage of Melange to 59% and the coverage of almost all modules except one (with 39%) over 40%. In fact, the coverage report of the html version which apparently uses a different algorithm from the console one shows the overal testing coverage is already over 70%.

In the next week (the last week of GSoC2010), I will clean up code and commit and push all uncommited testing code, increase the overall coverage printed in console to over 60%, and make the coverage of most modules over 50%. In addition, I will also move the the gaetestbed and pymox modules to buildout dependency and add the experience of how to do mail and taskqueue related tests with gaetestbed to the testing wiki page.

GSoC 2010 Melange Testing Project: Week 10 of the Coding Period

2010-08-02T10:27:00.000-07:00

In this week, I worked on some ghop modules with low coverage, e.g. soc.modules.ghop.tasks.task_update and soc.modules.ghop.logic.models.task and managed to increase the testing coverage of Melange to 57%. About the problem that there is no api proxy stub available for service "taskqueue", I found that there is one taskqueue_stub in google.appengine.api.labs.taskqueue, so I registered it to 'taskqueue' service with apiproxy_stub_map.apiproxy.RegisterStub in the testing environment. However, it (taskqueue.py) raises another error, "raise self.__TranslateError(e.application_error, e.error_detail) UnknownQueueError" which is a bit strange. This is the reason why I have not yet checked in the task testing code.

In the next week, in addition to solve the above error and increase the testing coverage of Melange to over 60%, I will also try to find a good way to test the mailing and taskqueueing functions and make sure the correct mail has been sent out and the correct task has been scheduled.

Document Editor: Week 10

2010-08-02T08:46:00.001-07:00

As I've already mentioned in the previous post, there was a problem with versioning more then one property of the model. Daniel proposed to store necessary fields as a JSON-formatted dictionary. On tuesday I reimplemented RevisionContent model and several logic methods to support new architecture. Document model, logic and view were refactored too.

Then I started to implement revision control for StudentProposal model. There were several minor issues, but everything is fine now. Revision control for student proposal is almost fully functional now (diff template and delete logic are left). New version of Melange was uploaded to my GAE instance.

On tuesday I plan to finish revisions for student proposals and start implementing auto-saving feature.

Week 10: Social Features Project

2010-07-31T08:42:00.001-07:00

This week I spent on integrating CalVis calendars UI with the current Calendars features as well as adding the maps functionality. The new calendars UI can be seen here. Clicking on events brings out a pop-up widget with event details. The Calendars feature is done. However, the two errors being thrown the by the Data apis earlier posted to the dev list still seem to persist. :( Need to spend some time and also consult Madhu to find a way around this.

I will be starting on the final Maps feature/view this weekend and hoping that this feature will not take long to code as it does not involve much creation/manipulation of data models, it mainly involves querying the data (user pages and events) and updating the map.

In the coming week:

start work on Maps feature
test the current code, esp. UI to check in case of any bugs

GSoC 2010 Data Seeder - Seeding using MapReduce framework

2010-07-30T10:24:00.000-07:00

The seeder is currently fully functional running on a live instance and hopefully should be able to seed any provided configuration sheet. The final implementation uses a mapreduce framework[0] for GAE and is highly parallelizable.

I also did some benchmarking on the cloud. I tried seeding 10,100 simple Linkable entities, including a string and a reference property. Running on a 50 tasks / second and having the mapreduce framework set to process 100 tasks / second (default value) finished successfully after around 1 min 40 seconds. That's pretty fast in my opinion. This resulted in about 0.45 CPU hours being consumed in the cloud. This was possible because mapreduce runs multiple map operation in a single task. Perhaps even better speed/efficiency can be used by using mutation pools to do batch updates in the datastore.

The way the current solution is implemented is by starting a new mapreduce job for related models (back-referencing). I believe this is also faster and easier to implement, the downside being that a lot of jobs appear in the status interface. I think this can be fixed by using some sort of smart sharding (already supported in mapreduce) instead of starting new jobs, I'll look into it.

So now that the seeder is working, it's time to get back to working on the web interface. I will add the ability to import an existing configuration sheet, better navigation options and better control when configuring references (like being able to specify a subclass of a specific model etc.) during the next week. If time permits, I'll also try to do add some styling.

[0] http://code.google.com/p/appengine-mapreduce/

Week 9: Social Features Project

2010-07-27T01:58:00.000-07:00

The past week of GSoC was spent mostly on winding up the Calendars features, such as finalizing workflow and using the JS GMaps package in Melange for specifying event locations. This is primarily used so that it can later be used in the Maps widget provided by CalVis Calendar UI. The latest is at my GAE.

I am currently working on the CSS designs for CalVis UI, and hoping to integrate the design and UI into my GAE by today or tomorrow at the latest.
The rest of the current week, I plan to spend on testing the Calendars and User Pages feature and also starting work on the Google Maps view/feature for filtering users and events based on users/event location and tags.

I lost the last weekend as I had to make a trip to India for some family obligations, hence the slowness in progress.

In the next week:

Complete CSS design for CalVis and integrate with current feature
Write and test calendars feature
Get started on Maps view/feature for users and events

Document Editor: Week 9

2010-07-26T05:03:00.000-07:00

I spent Tuesday on fixing code issues which were pointed by Daniel. Then I started working on revisions for the student proposal. It was pretty simple, but there was a problem. StudentProposal model contains abstract text property, which in my opinion should also be revisable. But current implementation of the RevisionInfo and RevisionContent classes are not able to store two content fields. This issue is to be resolved.

At the end of the week Mario implemented client-side redirects for the lists framework. I used this feature to re-implement "Show diff" button, Now it looks and behaves right. New version of the "Manage revisions" view was uploaded to my GAE instance.

GSoC 2010 Melange Testing Project: Week 9 of the Coding Period

2010-07-25T23:16:00.000-07:00

In this week, I mainly worked on low coverage modules, e.g. soc.tasks.surveys, soc.tasks.grading_survey_group and etc. With these work, I have managed to increase the testing coverage of Melange to over 55%. I am cleaning up the code and will commit and push them today or tomorrow at the latest. One problem is that there is no api proxy stub available for service "taskqueue" yet, so it raises an error "ERROR:root:No api proxy found for service "taskqueue". I am also working on it.

In the next two weeks, I aim to increase the testing coverage of Melange to over 60% and increase the testing coverage of those modules below 50% (40% at least) to over 50%.

GSoC 2010 Data Seeder: Web Interface

2010-07-23T05:54:00.000-07:00

The web interface should be pretty much functional right now. It's very ugly, I admit, I just added some expand/collapse tree view functionality for easier navigation, but it's just temporary. More user-friendly navigation features and some fancier display is scheduled for the future, but right now it works for defining any configuration sheet as was planned in the initial design and it will stay this way until the server-side code is fully functional. You can check it out by visiting my GAE instance: http://sttwister-melange.appspot.com/seeder/home.

Make sure to force reload the page in your browser so that it fetches the most recent JavaScript files.

For the weekend, I'm planning to add tests for the server-side views that have been implemented so far and then try to keep up with the test coverage in the future. After that, I will start fixing the server-side seeder code to make sure it works correctly in all cases and then port it to GAE Task Queue API.

That's it for now.

Document Editor: Week 8

2010-07-20T01:20:00.000-07:00

The main goal for the last week was to finish revision control for Document model. I had four tasks to complete:

Deploy HTML diff engine into Melange
Create view that shows diff between two revisions
Update "Manage revisions" view to support diffs
Code clean

All these tasks were completed successfully. HtmlDiff class from difflib was used as diff engine. I've actually tried several python diff engines from this page, but none of them appeared to work stable and fast.

I wanted URLpattern for "show diff" to be used as API. That's why I wanted it to be "pretty". But there was a problem with jqGrid buttons. Neither "redirect_simple", nor "redirect_custom" methods can create URLs with two different parameters. That's why I had to create separate button and trigger onClick event for it.

Full-featured revision control for Document model was uploaded to my appspot instance http://melange-editor.appspot.com

GSoC 2010 Melange Testing Project: Week 8 of the Coding Period

2010-07-18T23:17:00.000-07:00

With the suggestion of Daniel, I first tried Google Wave in this week. It seems very interesting that everyone can contribute/add anywhere and edit posts. Then, I have further put this week's meeting notes on it and also added our Melange group to the participant list so that all members are able to view, add and edit the waves.

Then I finished adding test cases to GSoC organization view, but I noticed unfortunately that it had not been run by nose tests. After some investigation, I found that the problem was because I had not added __init__.py in the GSoC test packages. Then, I further worked on other view tests. This week, however, I have not achieved my original target, i.e. increase the overall code coverage to over 60%. I will work hard to make it this week.

Week 8: Social Features Project

2010-07-16T22:44:00.000-07:00

This week went by in doing more work with Calendars, such as sorting out the issue on the User Authorization for Google data apis.
Earlier in the week, I implemented the workflow of Calendars feature with fuctionality such as:

creation/edit of events by users for chosen Calendars.
the addition of org_admins userid's to the Access Control Lists of their org's Google Calendar
sending notifications to Calendar owners with event request message and consequent approval by org_admin was also implemented.
uplading of approved events into the respective Google Calendar

The latest is at my GAE. All users with user roles and user pages will be able to see the "Add Event", "Events" and "List Event" options. "Add Event" will result in a notification being sent to the respective owner of the Calendar for approval, hence one may not be able to see the event pop up directly on the respective Google Calendar.

After some discussions with my mentor, it was decided that it would be an acceptable trade-off to use the CalVis API to display Calendars. As there was no very definetive decision reached in the dev-list discussion, at the same time, there was also not a strong opposition to CalVis, my mentor and I decided to go with the CalVis. The UI of CalVis can be manipulated with CSS to a decent extent and hence, it can be designed to look better. Although it may never look as good as Google's :-(, it is a trade-off we get for having the cool Google Maps plus Youtube widget pop-up on Calvis. I am currently working on the CSS designs for the Calendars as well as the User Pages(i was supposed to have done this a while back!).

For the next week:

prepare cool designs for the Calendars UI and User pages
find out how and write cron jobs for keeping Melange-side and Google Calendar-side events in sync
spend some time to refactor code to ensure it sticks to Melange styles and conventions

GSoC 2010 Data Seeder - Client-side HTML rendering

2010-07-16T09:11:00.000-07:00

I have been much more busy than usual lately and it's very clear that this has affected my time spent developing for Melange. I'm pretty much behind schedule right now and I'll have to try a lot harder in the future to make up for the lost time.

I have been facing some problems choosing a right JavaScript templating engine for rendering HTML. The solution that I was going to adopt was actually a jQuery plugin, but had some problems rendering some templates causing an infinite loop in the browser. Failing to find a proper solution for quite some time, I have decided to switch to pure. I was preferring tempest at first because it was using the Django templating format, but now I see the benefits that the pure separation between templates, directive and data offers (a directive is a structure that binds data to the templates).

The switch to pure is now complete and you can check it out at http://sttwister-melange.appspot.com/seeder/home

There's support for seeding models and defining providers for all properties. There's limited support for references at this time, only being able to bind a model to another random model.

See you next time!

Document Editor: Week 7

2010-07-12T03:33:00.000-07:00

Last week I finished implementing "revert to revision" action. Now, if you list revisions for the entity, you'll notice check boxes and a button "revert to revision". This button is activated when only one revision is checked. If you check some revision and press this button, back-end will delete all the subsequent revisions and will make checked revision current.

I also started working on the diff engine for revisions, but it is still work-in-progress.

In fact, I have to implement two buttons for revisions list: "revert to revision" and "show diff". I didn't manage to configure two different buttons for jqgrid, but Mario helped me a lot. I was able to understand the meaning of the "bounds" list and everything is perfect now. Thanks Mario!

GSoC 2010 Melange Testing Project: Week 7 of the Coding Period

2010-07-11T15:08:00.000-07:00

Last week, I found that Python coverage does not track the properties definition of Melange models which makes the coverage of models still quite low. So, I first tried to exclude models from coverage report. This is done by stubbing out nose.plugins.cover.Coverage.wantModuleCoverage with a method which offers the option to skip packages for testing coverage report through pymox stubout. Later, however, I remembered that coverage should be able to track class variables and module variables. I reckoned that the problem may be that these Melange models have been loaded before the coverage starts tracking. So, I further used pymox stubout to stub out nose.plugins.cover.Coverage.begin so that it loads Melange after the testing coverage starts. It works! All the class variables and module variables of Melange models, logic and views can be tracked by coverage and the code coverage now is over 50%.

Then I further worked on various views, e.g. sponsor, program, student, organization, etc and aim to increase the overall code coverage to over 60% next week.

Week 7: Social Features Project

2010-07-10T07:18:00.000-07:00

This week I worked on Calendars feature. I created data models for events to store event entities as they are created as well as simultaneously created event entities on the selected Google Calendar. Also I created a template to view all the Calendars(program as well as org using a drop down box to select).

I am currently working on the program_host "activating" calendars for all organizations once the accepted organizations are announced.Org_admins will in turn choose to "activate" their respective calendars after which their user ids will be added to their org calendar with "owner" rights. I am hoping to have a working version of this feature by the end of the weekend.

In the next week:

finish rest of Calendars feature (requests to add events)
Testing and revising of User Pages

Document Editor: Week 6

2010-07-05T08:35:00.000-07:00

Last week I was implementing views to support revision management.

First of all there is a view that renders document revision with the given number. For example, if you want to view second revision of the "user/parser/test1" document, you should visit http://melange-editor.appspot.com/document/revision/user/parser/test1/2. Permissions and range control now work properly.

The second goal was to implement revision management view. It is not finished yet. You can get list of all the revisions for current document via the "revisions" link. For example, all revisions for "user/parser/test1" can be viewed here

http://melange-editor.appspot.com/document/revisions/user/parser/test1

This view is a draft. I need to make several fixes, add buttons, checkboxes and behavior. It was pretty hard for me to get a grip of the new lists in Melange. But Sverre and Mario helped me and I hope to finish this view in a day or two.

GSoC 2010 Melange Testing Project: Week 6 of the Coding Period

2010-07-04T23:10:00.000-07:00

In this week, I first tried to add test coverage to Melange models, i.e. did model tests. However, I found that Python coverage does not track the properties definition of models which makes the coverage of models still quite low. When I discussed with Sverre, he suggested me to exclude models in the coverage report and start to test other modules with lowest coverage. After rechecking the coverage report, I decided to do view tests for sponsor, org, org_admin, mentors, student and so on.

Then my hard drive was corrupt:(. Fortunately, I have the online clone on Google servers, so I have not lost much work. It just took me some time to reinstall the operating system and resetup the environments. Due to the problem, I have only finished most parts of the sponsor view tests and some parts of org and student tests. Next week, I will definitely work harder to catch up.

GSoC 2010 Data Seeder - Model references in the UI

2010-07-03T08:52:00.000-07:00

This week is all about beginning to handle references. Right now, direct and back references information is included in the JSON data, and a random reference data provider has been implemented and tested. It's also working!

While working to add easy manipulation of references in the web UI, the current local JavaScript templating solution (John Resig's microtemplating) has proven to be a bit hard to maintain when getting to more advanced manipulation. So, I'm currently working on finding and integrating different templating solution. I'm investigating pure [0] and jQuery tempest plugin [1].

[0] http://beebole.com/pure/

[1] http://plugins.jquery.com/project/tempest

Week 6: Social Features Project

2010-07-03T01:52:00.000-07:00

This week was productive as I finally moved on to working on the next feature, which is Calendars + Events for Melange. I am using Google Data APIs for this, particularly the Calendars API. It is new and exciting, and I am also creating a new events data model in Melange to store the local copies of the events created by various users as well as for making requests to add events.

I am hoping to upload a bare essentials version of "adding events" into my GAE by the end of the weekend.

The User Pages feature is done minus "image upload" which is in the process of being resolved with Madhu's help :). Also, for the "add another job" and "add another education", I created additional data models education.py and job.py for storing this for each user. I am expecting there will some more changes to make here and there based on everyone's feedback.

The latest has been uploaded to my GAE. The brief workflow which I also posted in an email to the list is as follows:

Student Roles:

Log In
Apply for a Student role by registering as a student.
Upon completion of registeration, click the read "Create User Page" option on the sidebar.
Fill in form.
Click on 'View User Page to see how user page looks. or Browse User pages by clickin on "List User Pages" on the sidebar.

Mentor/OrgAdmin Roles:

Log In
Apply for a Mentor/OrgAdmin roles.
Upon being accepted as mentor/orgadmin, click the "Create User Page" option on the sidebar.
Fill in form.
Click on 'View User Page to see how user page looks. or Browse User pages by clickin on "List User Pages" on the sidebar.

The Week Ahead:

Develop Calendars+events feature upto near-completion
Revise User Pages based on feedback from development team
Improve the datamodel structures of jobs and education to create them as App Engine entity groups with the User Page as parent.

Social Feature Project: Coding Week 5

2010-06-28T17:40:00.000-07:00

This week saw a lot of delays and lost time for me. I got stuck with figuring out how to handle the images (profile picture) for the User Pages. I followed the direction given at here for uploading/resizing the images. Also, it took me a lot of time on this as I did not have an "image upload/resize" example already in the code-base to refer to. However, the upload does not seem to happen as I get a NotImageError. More on this on a mail to the dev list.

After that I integrated the Tags property for User Pages, so that users may search for other users based on user tags. After this, I made the remaining modifications based on comments given in the previous conference call. "Add Job" and "Add Education" required more effort as they had to be their own entities and would share a many:1 relationship with each User Page entity.

Last but not least, as the weekend approached I could not access any of the "appspot.com" web pages, especially my own instance. Hence, I was not able to upload and test my latest changes on the live instance. I tried to go through a proxy, but always got stuck after the "Google Account Login" as it would not redirect to Melange after login. However, this is fine now and I am able to access my instance successfully and will be uploading the latest to my instance soon and sending a mail to the dev list so that everyone can try creating their own user pages.

For this week, I will be working very hard to catch up with my schedule which says I should be working on Calendars feature already. I intend to wrap up User Page (minus images) by tonight so that I will get to start on Calendars from tomorrow(wednesday).

GSoC 2010 Melange Testing Project: Week 5 of the Coding Period

2010-06-28T12:26:00.000-07:00

As I had planned last week, I added content to the wiki page GettingStartedOnTesting [0] with the aim to provide a guideline on writing tests for Melange this week. Sverre has had a look at it and thinks that it should be extended, though it looks good.

I have also added test cases for a couple of logic modules this week and am currently working on testing the views of sponsor, program and student. Sverre thought the test coverage is still low and suggests me to focus the modules with low coverage next week, e.g. do some tests on the models. So, I will move my focus on testing the models next week.

[0] http://code.google.com/p/soc/wiki/GettingStartedOnTesting

Document Editor: Week 5

2010-06-28T06:27:00.000-07:00

According to the plan, I had to finish revision control this week. But I'm a little behind the schedule now.

Last week I fixed issues with document listing. It was a pretty tough stuff to implement, cause JSON-based list view supports field prefetching. But it works now and fixed instance has been deployed to appspot. I also started implementation of the view which helps to work with revisions (revert and show diff), but it is still in progress.

This week I plan to finish revision control and start working on diffs.

GSoC 2010 Data Seeder - Web Interface

2010-06-26T03:36:00.000-07:00

A very, very, veeery early form of the web interface is ready. It's very ugly in the current form, I haven't applied any styling to it yet, but at least it's functional. You can check it out here.

One problem that's pretty obvious at first is the space management. For most useful models, there are a lot of properties which take up a lot of space, but only a few of them might actually be used. My solution for this would be to compact them into collapsible/foldable panels so that properties that are not useful and properties that have already been configured don't take up space. Expect this to change in the future.

The web interface currently produces a valid JSON configuration sheet, only including model and properties configuration (without references) and also has the ability to send it to the server backend, but that's it for now. If you want to take a look at the resulting JSON, there's a dump in the JavaScript debug console.

Stay tuned for more in the future!

GSoC 2010 Melange Testing Project: Week 4 of the Coding Period

2010-06-21T05:52:00.000-07:00

In the last week, I further added more test cases to the user view test to test the views for administrator users. In order to test the operations of administrator users, an XSRF middleware token has to be manually generated and added to the post data so that the operation can pass authentication. Then, I added more test cases to the user logic test. This week, I will try to add test cases for more logic and view pairs.

One big thing for Melange next month is that each commit will be accompanied with tests. In order to contribute to the movement, I am preparing for a wiki page explaining the current testing environment (e.g. datastore), test runners (e.g. how to only run a single test case), available tools (e.g. pymox and Django test client), utilities (e.g. StuboutHelper), and so on.

Document Editor: Week 4

2010-06-20T09:44:00.000-07:00

I planned to implement revision control for Document model this week. New revision is now created on document creation. Revision creation on edit and view that shows all the revisions for current document are left.

To implement revision control for document I had to do the following stuff:

Implement Revision, RevisionInfo and RevisionContent classes in soc.models.revision module
Implement Logic class for revisions in soc.logic.models.revision. It overrides soc.logic.models.base.Logic.updateOrCreateFromFields and defines method to create new revision.
Change forms configuration in soc.views.models.document.View

It was pretty hard to figure out all the dependencies between classes and methods in Melange. I didn't want to break existing code base, inheritance is so useful here :)

UPD: New version of Melange that supports revisions is available at appspot.

Week 4: Final touches to User Pages

2010-06-18T09:42:00.000-07:00

It is going to be almost a month since GSoC has started. I have made decent progress. The almost-finished version of User-Page is now at http://melange-social.appspot.com/.

I have created a dummy program,org,org_admin,mentor, a couple of students. I am still working on adding the "past projects" data for some of my dummy students and mentor roles so everyone can take a look soon.

I have enabled mentor sign up and student sign up currently. So everyone can sign up and meddle around with it. There are bound to be bugs turning up, please let me know if you find something weird happening. Also, suggestions and feedback is welcome.

I am facing a teeny weeny delay with the User tags and profile_picture uploads which will be integrated into my GAE over the weekend.

Over the weekend:

wind up remaining functionality issues
do some testing

Next Week:

Start work on the Calendars

GSoC 2010 Data Seeder - Views

2010-06-18T09:07:00.000-07:00

My project drifted away a bit from the original timeline due to some unforeseen complications, so I'm now almost a week late. Melange has been built from the ground up heavily supporting the View-Logic-Model layers and in this way reusing a lot of code. However, so far, all views and logic classes have been tied up to a specific entity which dictates all this behavior.

Unfortunately, the data seeder project doesn't involve any specific model in order to build functionality around it, but rather it collects data regarding all other models in the application. So there's now a need for a new type of view that is not centered around any model. Initially, I thought that setting up a new general-purpose view is as simple as registering some patterns and some methods, but this doesn't seem to be the case. Instead, some of the code in soc.views.models.base had to be adapted to a new type of view (soc.views.base) that is not model-centric. I have used this new type of view to create the backend side of the AJAX API implementation. An example of the JSON data that is collected by the backend can be seen here [0]. The current version might seem a bit too big (mainly because the list of possible countriy choices is repeated for several models), but that might change.

Next stop: Build a basic web interface

[0] http://sttwister-melange.appspot.com/seeder/get_data

GSoC 2010 Melange Testing Project: Week 3 of the Coding Period

2010-06-13T16:05:00.000-07:00

Last week, I finally got Django test client working with the help of Sverre (my mentor). In this week, however, another problem arose. For some urls, e.g. /cron/poke, the response returned by the test client is a Django http response object; for other urls, e.g. /user/show/a_user, however, the response is a dict. After failing to figure out what had caused the problem, I asked Sverre for help again. Sverre suggested that it might be due to the stubout code (pymox) in tests/app/soc/views/models/test_base.py. After I commented the code, however, the problem still did not go away. Later, Sverre found that the problem was due to that the fact that the stubout in tests/app/soc/modules/ghop/views/models/test_task.py has not been unset. So, he added the unset code to tearDown methods and also helped me reviewed/improved and merged my code of refactoring the stubout code into a helper class. Thank you very much, Sverre!

After solving these problems, I started to do view tests for soc.views.models.user using Django test client. I have almost finished it. Yesterday, I pushed the code into my online clone. Today, I asked Sverre to have a quick look to see if it is alright and he was satisfied with it. Though I am still behind my original schedule, I am confident that I can catch up soon. This is because I have solved all major obstructions and have finished adding tests for one logic (base) and one view (user) so that I can follow similar patterns to add tests for other logic and views very quickly.

This week, I also came across the trace module of python when I tried to find another problem I met. It can be used either in command line with python -m option or in your program. This module save you time to code a call back function for sys.settrace by offering some common functions. One useful function is to trace the execution flow of your program. It traces every statement, so be aware the output could be quite large. For example, the output of tracing a single statement of Django test client in Melange testing environment is more than 100MB. Another disadvantage is that the output only shows the module name without package path which could confuse you when you have many modules with the same name or you have a big project and do not know the package of every module. Another useful function is to trace calls. The output of this one is quite small compared with the first function and also has full package path. This function, however, is not perfect either. The problem is that the calling functions are ordered by their module name rather than by their calling order which is arguably more useful. And you have no options to change the order. Besides, you can only use it in command line.

Document Editor: Week 3

2010-06-13T06:31:00.000-07:00

This week I worked on revision implementation. As proposed by Lennard I tried to refactor soc.models.work.Work, but found out that Work is also the base class for Quiz, Proposal, Documentation, Question and Survey. Daniel also proposed to implement revisions as three different models. That's why I decided to leave Work for models that doesn't support revisions and implement separate revision-related models.

In a couple of days I plan to implement logic for revisable models, refactor create function and implement view to list all the revisions.

Coding Week 3

2010-06-11T18:56:00.000-07:00

A third week of coding has gone by. The User Page status is at an "almost-there" status. The latest is uploaded to my GAE instance.
To have a user page, all users must have role assigned to them such as student,mentor,org_admin or program_admin. Only then do they get the option to Create a User Page on their sidebar. On the current instance, all users can Register as Student and then go ahead to create the User Pages.
However, there are still some things left to wind up with regards to User Pages, before I move onto extensive testing of this feature and then move on Calendars, which I have to start this week according to my timeline!

Things left to wind up:

listing all User Pages
Pulling present/past project data for student/mentors/org_admins
getting images api to upload profile picture(currently using blobstore to upload, but the picture doesn't seem to show up)
polishing CSS and layout of User pages

Upcoming Week:

Finish up with User Pages feature and start on testing.
Start coding work on Calendars

GSoC 2010 Melange Testing Project: Week 2 of the Coding Period

2010-06-06T15:54:00.000-07:00

In this week, I had planned to add test cases for app.soc.views.models.base.View as well as app.soc.logic.models.User.Logic and app.soc.views.models.User.View, app.soc.logic.models.Role.Logic and app.soc.views.models.Role.View.

As I mentioned in my project proposal, I want to use Django test client which acts as a dummy Web browser to test views of Melange because of its ease of use and good integration with Django (no surprise since it was developed by the Django team). So, I first tried to get Django test client working.

After many attempts, however, I still could not make it work. Then, I found that I could not even run Melange locally suddenly due to the error (google.appengine.api.yaml_errors.EmptyConfigurationFile). I first tired to solve the problem by checking the tracktrace and code. Unfortunately, however, I could not find the problem. I thought that I might have accidentally changed or removed some important files. So, I rebuilt my local Melange but it still did not work. I also tried to re-clone and rebuild another Melange, but it was still in a vain. At that time, I realized that I had to fall back on my last resort, asking my mentor Sverre :-) . Awesomely, Sverre found that the problem was due to the corrupted .appcfg_nag file and he then solved the problem by removing it.

After this problem was solved, I tried the test client again. However, it still did not work. So, I asked Sverre again and found that it is because I had used the trailing slash at the end of the url following Django's tradition while Melange uses a url pattern without the trailing slash. Finally, Django test client works!!! Hope I had asked Sverre earlier :-).

Because of these problems I encountered, I have not finished this week's work as planned. Fortunately, however, I have got Django test client working. In the next week, I will try to speed up my work, finish this week's work as soon as possible and then start next week's work load.

GSoC 2010 Data Seeder - Data providers

2010-06-05T07:49:00.000-07:00

This week was all about implementing data providers. I haven't got much time to work this week, it was the last week of my university semester and I had a lot of unfinished work to... finish, and also had to prepare for a Google internship interview.

What I've done so far includes the base classes for data providers and implementations for string data providers. Other providers should be easier to implement now that I have a functioning base, I hope I can finish them over the weekend as I've got a bit more time on my hands.

After an IRC meeting with Mario, we have decided that the AJAX API needs some refactoring which I will update soon.

Coding Week 2

2010-06-04T18:09:00.000-07:00

The week 2 of coding has gone by. And I have made progress with my code.
I have written the View and templates for the User Page with most of the functions such as "view_page", "edit_userpage", "create_userpage", "view_calendar" etc.

The prototype of the User Pages, Calendars up on my GAE, plus I have created some forms using Melange's inbuilt forms framework for editing data in the User Pages using the base.py View methods such as create, edit, editPost, editGet, etc.

There are some issues with the creation/saving of the forms and entity data which still need to be resolved. Aiming to resolve it before the weekend is over.

Finally, I need to incorporate the ListView for User Page entities using the list functions in Melange's base.py View framework.

Next week:

I should have completed User Page functionality, and should have my code reviewed by my mentor, before pushing the changes to my clone.
I should update project wiki with a complete timeline, taking into account my current progress on the project.
I should start working on the Calendars feature.

New Document Editor. First week of coding.

2010-05-30T11:15:00.000-07:00

Strangely, there were not much coding this week :) All I've created was same kind of "Hello, world" app for Django-GAE. But now I got a good idea about how these things actually work together.

Most of the time was spent on studying the GAE datastore. I had to get a firm grip on this topic before hacking out Melange's models. First of all, I've looked through a wonderful book on GAE. The architecture of the datastore seemed quite unusual compared to the Django ORM. But it definitely has some cool features like associativity and Expando (sounds great!) class.

My main concern is the architecture of the revision control for Melange. There are two main approaches:

Add fields that store revision information to the existing models and convert the whole bunch of existing documents.
Create model that stores information about revisions and link existing entities to it.

I hope this issue to be resolved soon for me to start actual coding.

P.S. Here is the thread in the development list.

GSoC 2010 Data Seeder - First week of coding

2010-05-29T10:39:00.000-07:00

So, coding for Google Summer of Code 2010 has officially started. This week I have acquainted a bit more with the codebase and I have successfully managed to set up Eclipse and all the necessary related tools (pylint, code completion, Google App Engine integration etc.), so coding should go smoothly from this point on.

After discovering a small problem with the new GAE version 1.3.4 in the Mercurial repository, I have actually made my first commit related to the data seeder module, yey :D.

Next, I should update my proposed timeline in the wiki to include the second part of the project, and then I'll try to stick to the plan :).

A draft specification of the AJAX communication API between the web-interface and the server-side data seeder module should be ready by the Monday conference call.

GSoC 2010 Melange Testing Project: Week 1 of the Coding Period

2010-05-28T15:41:00.000-07:00

The Coding Period of GSoC 2010 has officially started! In this week, I continued to add more test cases for the app.soc.logic.models.base.Logic.getForFields() method, started to commit and push the code to my clone. On this weekend, I will start to add test cases for other methods of the class and hope to complete it before next Monday. Another task on this weekend is to add a tentative project timeline to my project wiki, which becomes clearer after I discussed with Sverre tonight. Next week, I hope to complete the job of adding test cases for app.soc.views.models.base.View and some other pairs of logic and view, e.g. role and user. Anyway, time flies. Therefore I need to fly as well so that I can catch it or at least not fall far behind :-).

In this week, I also came across a nice book titled Programming Google App Engine from Dan Sanderson who has excellently explained the difference between GAE runtime environment and traditional web hosting or self-managed servers, the difference between GAE datastore and traditional relational databases, and pointed out the special consideration and strategies for building a scalable, responsive and optimized app upon GAE due to the difference. In addition, the book also covers basic things about GAE, e.g. how to set it up and how to build an app step by step as well as more advanced tools, e.g. the URL Fetch service, the Mail service and the XMPP service. So, it is highly recommonded for both beginners and intermediate users who want to make best use of GAE.

Week 4: Code-base and Coding

2010-05-28T07:20:00.000-07:00

The first official week of GSoC has gone by, and I have officially started coding. I have started off with the User Pages feature. I created a directory structure as my project is to appear as a module in the soc.modules folder. I started coding the data model(soc.modules.social.user_page) and View(soc.modules.social.views.models.user_page) for the User Page. However, I realized I need to get a better grip on the code-base and how things work with regards to code before I can go ahead with the actual coding for my project. Hence, over the weekend and the week after, I am going to acquaint myself with the code-base and get as best an understanding as possible and then proceed on with the actual coding of completing the View and then the templates etc.

This of course, sets me behind time with regards to my timeline :-(
However, I have decided to put in extra time and work extra hard this weekend and next week to strive and catch up with my User Page feature so that by my next post, I will have progressed to having a better, working User Page feature :-).

GSoC 2010 Melange Testing Project: the last week of Community Bonding - starting to code

2010-05-21T21:42:00.000-07:00

In this week, I started to add more test cases to tests.app.soc.logic.models.base and tests.app.soc.views.models.base. I first learned the code of the corresponding code base and the existing test cases in more detail in order to understand their functions and logic. Then I tried to design some new test cases. Due to the fact that I am still not very familiar with the implementation detail of Melange and met problems when setting up the coding environment, e.g. could not get the new version of ctags to work well with Vim and Python, the process is a bit slow. However, these preparations have built a good foundation for the official coding stage, which will start next week. Hope that during that period I can code fast and leave more time to improve and clean up my code as well as do some extra work for the project.

GSoC 2010 Data Seeder - Project design

2010-05-21T10:11:00.000-07:00

Another week of the community bonding period has gone by as we are getting closer and closer to actually having some code written :D

So here are some of the features that have been decided upon so far:

The configuration for the seeding operation will be customizable by the user using a web interface which will conclude with the creation of a JSON configuration sheet. This configuration fill will be usable both locally and on-site and can later be edited using the same web interface.
The configuration sheet is meant to be very easily customizable and extendable. Different scenarios for different needs can be saved in predefined configuration files and then executed at any times. Examples include saved states in different phases of a GSoC program (proposal phase, midterm survey etc.).
There will be an option to export data to Python fixture files, either directly by the use of a script along with a JSON configuration file, or by saving the state of a running instance.

You can find more details on my wiki page: http://code.google.com/p/soc/wiki/GSoC2010DataSeeder#Design_documents

The script that I have used to generate the diagrams from my previous post has been committed to my Melange clone. You can try it out anytime, no further setup is required. Here it is: http://code.google.com/r/sttwister-gsoc2010/source/browse/scripts/model_graph.py

Week 3: Getting Ready to Code

2010-05-21T05:00:00.000-07:00

The community bonding period is drawing to an end this week, which means we begin the actual coding work on our projects starting next week! I am quite excited to start writing code :).

So far, I have finalized most of the design and workflow for the two features I am developing for my project. There are still a few loose ends with regards to design hanging around, which will be tied up soon. My project wiki is now updated with the design details and awaiting the approval of my mentor, Madhu :).

The main additions and modifications to the design this week were:

Some changes to the user_page.py data model. The multi-valued properties job.py and education.py data models were scrapped to be replaced by a simple biography text field where users can write whatever they would like to share with the community.
Detailed workflows for Calendars and Maps feature (addition/request new events). Here are the usecase diagrams.
Both events and user_page data models have a tags property which is basically a comma-separated string with tags related to the user or event(e.g. C#, Java, Django). Users can also be given to the option to subscribe for feeds to be notified of events with certain tags.
I have also done a detailed timeline, with a list of deliverables for every week up until the mid-term to keep be oriented and directed with aims for every week.

Coming up next week:

Coding, coding all the way!
Follow timeline closely to reach the weeks' goals.
Resolve any bugs/issues, if assigned.

New Document Editor: HTML diffs

2010-05-20T01:37:00.000-07:00

One of the goals of my project is to create revision control for Melange documents. Each RC framework must provide some kind of diff engine. The engine should find changes in the document and represent them to the user. It is pretty simple with text, but Melange stores documents as HTML.

Talking about HTML diffs, we should consider two cases:

HTML as text. A bunch of tags, attributes, values and content.
HTML as a rendered document (e.g. image)

Another question is: what causes representation changes in a rendered HTML document?

Tags (through browser's default CSS or applied CSS).
Attributes class and id (through applied CSS)
Applied CSS (server-side, in-document or in-line).
Style modification with DOM.

Let's consider first case - HTML as text. This case seems pretty obvious, cause changes can be tracked with textual diff engines. But it's not that simple. There are several cases of changes to HTML which don't influence representation:

Tags are changed, but CSS is the same:
<h1>Hello, world!</h1>
is changed to
<h2>Hello, world!</h2>
but the CSS is
h1, h2 {font-size: 12px; font-style: normal;}
Some pieces of HTML are rendered the same:
<div class="alert">Hello, world!</div>
and
<div class="alert"> <p>Hello, world!</p> </div>
Class or id is changed but CSS is the same:
<div class="original"></div>
is changed to
<div class="changed"></div>

The second case - HTML as an image. I mean HTML with applied CSS which is displayed to the user. Tracking changes to images is the correct way of handling HTML diffs. It can be performed with several tools. One of them is convert utility from ImageMagick. This approach is, however, a little tricky and CPU consuming.

Let's return to HTML as text. If we can guarantee that CSS represents different tags different and there are no class or id changes, then everything seems right. We can focus on textual diffs for HTML. Considering TinyMCE (which is a defaul editor for Melange) all representation changes are made with tags. If there is no appropriate tag, then the style is applied with <span> tag or a chain of <span> tags.

Textual HTML diffs can be generated by several tools. One of them is HTML diff for Python. I'm now thinking about using it as a skeleton and try to make more intellectual engine with Beautiful Soup

Week 2: Working on Project Design

2010-05-15T21:33:00.000-07:00

A second week has gone by and I am happy to say that I am now better acquainted with the Melange code-base. During the week, I managed to create a clone of the source and have now deployed to my very own GAE instance :).

Update on my bug-fixing progress:
I managed to submit my very first patch for issue 890. I also started looking into issue 884, but as it requires some re-design of existing templates, discussion is ongoing as to how this issue can be resolved.

Update on my Project:
As my project is to appear as a new module in the soc.modules section, I am tentatively naming it as social.

I have started on the design work for the ‘Tabbed User Pages’ which is one of the two main features I have proposed.

Data Models:
I have come up with additional data models to contain information on users which will be displayed on their user pages. The proposed additional models are as follows. These are not finalized as discussion is still ongoing as to the inclusion and exclusion of some of them.

soc.modules.social.models.user_page.py
soc.modules.social.models.education.py
soc.modules.social.models.project_details.py

The user_page.py model will serve as the master model which will contain all the attributes of the user model which will appear on the user profile page. The details of education and past and current GHoP/GSoC projects can have multiple values for a given user and hence are extracted out to separate data models by themselves.

Thus far, I have envisioned the User Pages to have a tabbed view with three tabs, the ‘Profile’, ‘Project’ and ‘My Stuff’ tabs. The ‘Profile’ and ‘Project’ tabs are self-explanatory, the ‘My Stuff’ tab will be used for users to show their personal blog feeds using the RSS widget for now. Should more widgets be developed (i.e. Twitter widget J), they can be added in this tab.

Users will be able to edit the data in their user pages using a single form to edit the data shown in all the tabs, using the 'Edit User Page' link. This link will only be visible if the user is viewing their own user page.

Here is a simple use-case diagram to elaborate on the workflow.

Views:

soc.modules.social.views.models.user_page.py
Since the user_page feature serves more or less the same purpose for all the different user roles in Melange, a single view will be created.

Templates:
soc.templates.modules.social.user_page.view_user_page.html
Since the user pages will be tabbed, the entire page will not reload when users switch between tabs. With the help of JavaScript, this can be achieved with a single template with different html div and id tags defining the data to be displayed in different tabs :).

soc.templates.modules.social.user_page.edit_user_page.html
A different template with the html form will be created for the 'Edit User Page' function which will be used to update the user_page.py data model.

Logic:
soc.modules.social.logic.models.user_page.py
To handle the edit/update of the user data a logic is created which will be used to update the user_page.py data model.

In the coming week:

Check for updates on resolving Issue 884
Finalize the Design for Feature 1 & Feature 2: Maps + Calendars
Finalize Project Timeline and list code deliverables for mid-term as well as weekly deliverables
Decide on the specific APIs and libraries to be used for the project.

GSoC 2010 Melange Testing Project: Week 1 - learning the code base

2010-05-14T15:05:00.000-07:00

It has been a very exciting week. In this week, I made a clone for the testing project, deployed Melange to my GAE instance, got the posting right for melange-dev blog, created a wiki page for the project, signed a Contributor License Agreement (CLA), submitted tax forms, got an ACM account (the courtesy of Google), had the first conference meeting with other developers of Melange, helped some Melange users to solve their log-in problems, and had meetings with my mentor (Sverre). What a week!

My main job in this week, however, was to learn the code base of Melange since my project is to add tests to the code base. In order to help me better understand the code base, I decided to visualize the organization of the code base and the relaionship of modules/classes. Felix' diagrams are very interesting. However, they are only for Melange data models whereas views and logic are more relavant for my project. So, I used a tree tool to generate the module organization of Melange (click the picture below to view the searchable full HTML file)

and module organization of Melange tests (click the picture below to view the searchable full HTML file)

and used pylint.pyreverse to generate the package diagram of Melange (click the picture below to view the searchable full PDF file)

and the class diagram of Melange (click the picture below to view the searchable full PDF file)

These diagrams have helped me a lot to learn the code base. Hope they are also helpful to others who want to learn the code base.

During my meeting with Sverre, he suggested me to start with the testing of the modules which are the most important and used most, e.g. soc.logic.models.base and soc.views.models.base. Therefore, my next week's plan is to start coding, specifically add more test cases to the existing tests of these modules, e.g. tests.app.soc.logic.models.base and tests.app.soc.views.models.base.

Following is a brief introduction to the testing project:

This project is to add unit test suites/cases for the logic and views of the existing modules. These test suites/cases will enable developers to refactor the existing code at any time while at the same time to ensure that they still work correctly by passing all the tests (i.e. regression testing). This project will also serve as a basis for the Melange team to switch the development methodology to test driven development.

Cheers,

Leo (Chong Liu)

GSoC 2010 Data Seeder - Project start off

2010-05-11T06:21:00.000-07:00

I'm posting this to let you all know how my first week with the Melange community has been.

Bonding with the community has gone very smoothly, the community is very close together and I love the idea of having weekly conference calls. Sounds like a really fun summer ahead of me and I hope everybody will enjoy it.

The first thing I needed to do for my project was to really understand the data model behind Melange. Since there are quite a few entities involved and keeping in mind that models might perhaps change somewhere in the future, I thought the best way to accomplish this is to build an automated tool that draws diagrams from only the source code. There are two kinds of diagrams I wanted to have: inheritance diagram, showing how classes derive from each other and an entity relationship diagram, showing the relations between entities.

The code that I have produced is inspired from an open source project that does mostly the same thing for Django models. It is part of django-command-extensions.

I hope I can integrate my script with the Melange code base pretty soon, so that anyone can generate graphs from their own copy if anything has changed in the data model.

Here are my results so far:

Inheritance

Inheritance, separated by module

Entity relationship

Entity relationship, with attributes

Entity relationship, separated by module

Some conclusions:

Boy, are there a lot of relations out there!

Personally, I didn't expect to see that many relations between the models. This is definitely going to influence the way I'll design things around, especially the web interface, in order to make it as easy as possible to configure relations in the seeded data.

That's about it for this week, I'll now start actually doing those design documents! :)

Cheers,

sttwister (Felix Kerekes)

GSoC 2010 : Week 1(Community Bonding)

2010-05-07T06:48:00.000-07:00

Hello Everyone!
My first week as a GSoC student for Melange has gone by :-). I had the first few meetings with my mentor (Madhusudan) and am right now feeling very excited about my project. Here is a short summary of the week and updates on my GSoC project:

I had a couple of meetings with my mentor where we discussed about my project design and other high-level details as well as some technical stuff. Here is a brief overview:

About my Project:
To develop and integrate a social features module into the current soc framework. The purpose of the module is to help users participating in the programs (such as GSoC) to get to interact and find out more about their fellow GSoC'ers and where they are from, what their project is about and anything else which they want to share with the rest of the community such as events they are organizing or would like to attend etc.

More Specifically:
As of now, there are two main features we have decided on at this stage:

Google Maps + Calendar
This feature allows users to find on a global map users, events, meets etc. based on location (hopefully reducing the quantity of "i-am-from" emails on gsoc-students-mailing list), organization, type of project etc. The Calendars will be used to show mainly details of events and meetings occurring within a certain date range for a certain organization etc.
User Pages
Once you locate users on the global map mentioned above, then more information can be found about them through their user pages. Users can use this page to share their personal/professional profiles, details and updates on their projects, their blog feeds.

The above is very brief and a more detailed version is available here.

Venturing into the Melange code-base...
To get started on the task of getting to know the code, my mentor suggested I could take up an issue and fix it. So I asked the team for the most suitable one and took up one (issue 890). After a good amount of trudging around in the code (and some tips from my mentor :-) ), I am on the verge of fixing it! And it has definitely helped me to get familiar with the architecture and code.

In the coming week:

Work on a more formal specification of the project design, features and work flow (to be added to wiki) and frameworks, APIs to be used.
Fix another bug (issue 884)
Start thinking about actual implementation details (idea of how the code might be structured etc.)
More meetings with my mentor :)