Tuesday, August 23, 2011

GSoC 2011 Integration with External APIs: GSoC ends with great experiences

Hi,

We came into the end of Summer of Code 2011. It was one of the best collaborative development environments i've ever been before. I learned lots of stuff especially about Google Data APIs, OAuth, Javascript, Rest structure . I improved my code writing skills for more beautiful Python code. I worked with very nice and smart people. I tried to look problems like from their perspectives which  led me better reasoning against problems. As a result, it was a long but short summer that I spent my most of my time in front of my computer, but that was really fun. I own special thanks to my mentor Mario and co-mentor Madhusudan for spending quite amount of their time with my problems, reviews and newbie questions.

Last week was a little rush for me. I worked on Carol's issue about shipment trackings. Program administrator wants to share some spreadsheets that holds shipment tracking numbers, notes, address for students. A task for this spreadsheet is going to run and data will be synced to Melange datastore so student will be able to see the shipments sent for them in the dashboard page. We hope it will be used for the final shipments this year.

That's a video i captured for its initial state:

Wednesday, August 17, 2011

GSoC 2011 GUI Overhaul: Generic Iconic Dashboard

Hi everyone,

This week I need to post remaining patches, especially the generic iconic dashboard. I just posted early so I can have much time iterating the review process and hopefully got the patches submitted this week. The generic dashboard class is now placed in app/soc/views/dashboard.py. It contains the underlying view base for iconic dashboard. Every nested-iconic-dashboard should be inherited from Dashboard class. You can take a loot at admin.py, dashboard.py and org_admin_dashboard.py views to get the idea how to use Dashboard class. Now all related base dashboard templates were placed in v2/soc/dashboard directory. It contains the base.html and _user_action.html. All stylesheet rules should be defined inside v2/gsoc/dashboard.css.

There were changes on the melange.list and melange.dependency JS after refactoring the dashboard. The changes were to hold the lists to be queued and executed when the lists were rendered inside the dashboard. After applying the changes, I run the test and it broke in the test_admin.py. I figure out the case immediately, should be the assertTemplate. Yeah, there were some templates removed and added. After fixing the test, I post all the patches and ping Daniel. I hope I can run fast for the review process ;)

Monday, August 15, 2011

GSoc 2011 : Melange Testing Project CP W12 Last Week

Hi,

Only a week is left for GSoC to end and I need to finish testing all the important views as soon as possible. I had been working on views.org_profile, profile_show, accepted_orgs, proposal_review, duplicates and proposal_review this week. Testing views is turning out to be a little more involved and complicated task as compared to testing the logic. I got stuck at some places and so could not submit them for review. I had thought that I would be able to finish the views much earlier and start functional testing but I was not fast enough and also I could not devote 10+ hours per day during the weekends as I had promised in the proposal due to heavy course work, classes and personal issues.

      I will be trying to commit as much tests as possible in this week and work on the remaining tests after GSoC. I aim to work on functional testing and buildbot after GSoC. Lets hope for the best :).

GSoC 2011 GUI Overhaul: What's Left

Hi everyone,

The GSoC 2011 program will end shortly. Last week I was working on the account management, assign host, adding evaluation group and dashboard. There's a problem with Django test client that I couldn't figure out. The handler always treat self.data.POST as regular dict for every requests (in my case via POST) made from Django test client. Since in validate method there's a line that expect QueryDict instance, by calling getlist method self.data.POST.getlist('assign_program'), the test always fail. Maybe I missed something here, so I read the client.py code from Django that comes along with Google App Engine SDK. It took me two days or more to trace this problem. I don't think we should remove getlist from the handler nor hacking the django test client. So, I left the test ATM and move to another to do list, that's it, the dashboard.

The dashboard will be moved to soc/views/helpers/dashboard.py and the code itself is self explanatory:
class Dashboard(Template):

"""Base dashboard for admin page
"""

def __init__(self, request, data, subpages=None):
"""Initializes the dashboard.

Args:
request: The HTTPRequest object
data: The RequestData object
subpages: Subpages of current dashboard
"""
self.request = request
self.data = data
self.subpages = subpages

def getSubpagesLink(self):
"""Returns the link to other dashboard that appears
on top of the dashboard.
"""
return self.subpages

def templatePath(self):
"""Returns the path to the template that should be used in render()
"""
return 'v2/modules/gsoc/dashboard/dashboard.html'

def _divideSubPages(self, subpages):
"""Returns the subpages divided into two columns.
"""
middle_ceil = int(math.ceil(float(len(subpages))/2))

return [
subpages[:middle_ceil],
subpages[middle_ceil:],
]

The templates is improved to be able to handle list components that we have seen from regular dashboard. I hope the patch will be posted for reviewing process lately this week and pushed before the program ends.

Monday, August 8, 2011

GSoC 2011 Melange Testing Project: CP W11

Hi,

I worked I tested gsoc.views.duplicates module. I have not submitted my tests which I wrote for org_profile, profile_show and student_forms for review as there is a need of some access checks to be defined new for the respective module in the codebase. I was rechecking my tests for org_profile and I felt that the timeline should also be checked if a program is active or not. I put this in the mailing list and Lennard suggested that isProgramActive be renamed to isProgramVisible and a new method isProgramRunning be introduced. Those pages which remain active after gsoc can include isProgramVisible in their access checks and which are current program specific can include isProgramRunning. I submitted the patch for the access checks and Lennard and Sverre reviewed them. I will be accommodating the changes soon.
    
There was also a problem with our coverage module which is outdated and so it does not work with the latest Nose 1.1.2. I reported the issue with Nose and they suggested that we use coverage > 3.0. Leo tried including it in the buildout dependencies so that the coverage module is always the latest but it too did not work. Just replacing the old coverage module with the latest coverage modules works fine.

GSoC 2011 Integration with External APIs: Proxy Url and Exporting Lists - W11

My final decision about cross-domain requests: They can be headache. Especially if your requests are bound on some headers. There are new recommendations for this like "Cross Origin Resource Sharing". But this is so new to apply because it makes your application browser dependent. There are still many users on internet using ancient browsers and we can't force them to use latest version of specific browsers.  Also server we're trying to access must support this. Because we actually asking server if it accepts some parameters, headers etc. Besides this, jsonp and script methods may be enough for lots of purposes. But to make oauth authenticated requests to gdata server, it wasn't. So i changed my approach. I saw oauth gadgets use  the same approach: providing a proxy url on same domain that serves as a gateway and removes all cross-domain concerns.








That proxy url doesn't care about what request does. It just takes method, gdata server, url, data, headers as parameters and makes a request to gdata server and returns response directly to the client.

After resolving cross-domain request problem, i started to develop "list exporting" feature. For now i'm able to create an empty spreadsheet when user clicks "export to spreadsheet" button. Here are some screenshots of current state:

Button for exporting:
 







Uploading may take time, so it's stressed in a dialog window.









After upload complete, user may click to see the document:











Document is exported with the name of the list:



Sunday, August 7, 2011

GSoC 2011 GUI Overhaul: More Handlers for PA

Hi everyone,

GSoC is near to the final evaluation and I in a run to catch all my TODO. Last week, I wrote some handlers for PA to manage account and assign a user to be a host. PA can assign programs to be hosted by particular user in edit user page. Here's how it looks:



The code is somewhat similar to assign_mentors.py and project_details.py. To set a user as a host, PA needs:

  1. Given an account to be set as a host, go to the Lookup page in the admin dashboard.
  2. If the account is found then click the "Edit user" link
  3. In the edit user form, there's a box at the bottom (see picture above) that contains program selection. To set multiple programs to be hosted by the user, click "Add new" link. If done with the selection, click the "SET AS A HOST" button

Actually there's some features that I've in mind such as add / edit sponsor, programs list that contains list of program name and its hosts as cols, page to view readonly program that in inactive state, and of course the page to create new program. I'll hold thus things for a moment because the dashboard is still being refactored and I promised to finish it by Aug 6th. I am late two days here.

Tuesday, August 2, 2011

GSoC 2011 Integration with External APIs: Same Origin Policy - W10

Browsers have some restrictions for cross-domain Javascript requests due to some security issues. For "List exporting" from client side, we need to make cross-domain requests to Google from user. Before this jsonp was only standart that i know as a solution. Jsonp allows loading a json from a remote web server. But i knew GData protocol supports XML format data exchange. I tried to find a solution to load a cross-domain XML source. No solution was neat enough to apply.

Script Approach
Then i thought GData should support JSON of course, and found related document :) We should just have to add a "alt=json" parameter to url GET params. Same page also was providing a method to access resources from a cross-domain with JavaScript. We should call a special script that calls our callback with related json as parameter when page is loaded. Here is an example script:

<script src="http://www.google.com/calendar/feeds/developer-calendar@google.com/public/full?alt=json-in-script&callback=listEvents"></script>

which returns a javascript response with a form in which json response is embed.

listEvents({"version":"1.0" ..... })

This provides another way for making cross-domain requests. In this solution server must support this method with a special effort to form a javascript response instead of just interpretting request as a regular json request.

Http Access Control
This approach was good but unfortunately not enough for our purpose because we need to set Authentication header with request. So i returned back to jsonp. I thought it should be possible to make a jsonp request and add a "Authentication" header with request like that:

jQuery.ajax({url: a_cross_domain_url+"&callback=?", data: data, headers: {"Authentication": auth_header}, success: a_callback)

This was almost working until i encountered with another cross-domain restriction. Sending headers with a cross-domain request was limited to another standard. I read this document about "Cross-Origin Resource Sharing". As far as i understand from document, For sending headers with request, this is defining a way for browser to first ask to server if it allows given headers, then makes actual request. This sounds good but because it's a new recommendation and some browsers may not support it (and perhaps GData server too) , i skipped this. Firefox supports this after 3.5 version.

At the end
I learned some stuff  but still was not happy as there wasn't a way sending OAuth header with request. GData documentation contains AuthSub authentication over Javascript but i couldn't find anything related OAuth with JavaScript in which we will provide OAuth header. Then, while i was hopelessly playing with OAuth playground, saw that choice in form:


Put oauth params in:  



This was just i was looking for. We were able to pass oauth params with url instead of header. This was something not documented in GData OAuth documentation. I was lucky to find this out.


At the and my approach was shaped: Make a jsonp request, put oauth params inside url.

Note: This didn't work either, to see my final solution to make Javascript requests see next post by me.

Monday, August 1, 2011

GSoC 2011 GUI Overhaul: Document List for Admin

Hi everyone,

Lets talk about document list. Carol asked this on issue 1257. To implement such list, I borrowed the view from accepted_orgs.py. Lets name the list "DocumentList" and the page "DocumentListPage". I only added two visible cols, the title and shortname, for DocumentList. When the row is clicked, it will go to the edit document page view. I also created the test for document list inside test_document.py. The test will make sure, the list will be shown to the host (based on checkAccess method). Okay here's the class for DocumentList and DocumentListPage:

class DocumentList(Template):
"""Template for list of documents.
"""

def __init__(self, request, data):
self.request = request
self.data = data
r = data.redirect

list_config = lists.ListConfiguration()
list_config.addColumn('title', 'Title',
lambda e, *args: e.title.strip())
list_config.addSimpleColumn('link_id', 'Link ID', hidden=True)
list_config.addColumn('short_name', 'Short Name',
lambda e, *args: e.short_name.strip())
list_config.setRowAction(
lambda e, *args: r.document(e).urlOf('edit_gsoc_document'))

list_config.setDefaultPagination(False)
list_config.setDefaultSort('title')

self._list_config = list_config

def context(self):
description = 'List of documents for %s' % (
self.data.program.name)

list = lists.ListConfigurationResponse(
self.data, self._list_config, 0, description)

return {
'lists': [list],
}

def getListData(self):
idx = lists.getListIndex(self.request)
if idx == 0:
fields = {'scope': self.data.program}
response_builder = lists.QueryContentResponseBuilder(
self.request, self._list_config, document_logic, fields)
return response_builder.build()
else:
return None

def templatePath(self):
return 'v2/modules/gsoc/document/_document_list.html'


class DocumentListPage(RequestHandler):
"""View for the list documents page.
"""

def templatePath(self):
return 'v2/modules/gsoc/document/document_list.html'

def djangoURLPatterns(self):
return [
url(r'documents/%s$' % url_patterns.PROGRAM, self,
name='list_gsoc_documents'),
]

def checkAccess(self):
self.check.isHost()

def jsonContext(self):
list_content = DocumentList(self.request, self.data).getListData()

if not list_content:
raise AccessViolation(
'You do not have access to this data')
return list_content.content()

def context(self):
return {
'page_name': "Documents for %s" % self.data.program.name,
'document_list': DocumentList(self.request, self.data),
'program_select': ProgramSelect(self.data, 'list_gsoc_documents'),
}

And here's the test for document list:
class ListDocumentTest(DjangoTestCase):
"""Test document list page.
"""

def setUp(self):
self.init()
self.data.createHost()

def testListDocument(self):
url = '/gsoc/documents/' + self.gsoc.key().name()
response = self.client.get(url)
self.assertGSoCTemplatesUsed(response)

response = self.getListResponse(url, 0)
self.assertIsJsonResponse(response)
data = response.context['data']['']
self.assertEqual(1, len(data))

Sunday, July 31, 2011

GSoC 2011 Integration with External APIs: Popup Blocking Policy - W10

This week, unfortunately i was not productive. I had quite problems mainly based on "same origin policy". Tonight, story of the week seems ending nice, because i finally resolved how to make cross-domain requests with JavaScript. I will write two blog posts this week, this one for another problem "popup blocking problem" and next one for "same origin policy". If you don't have extra time for reading two blog posts, i suggest you read next one, because it's the main story of the week for me.

Popup Blocker Policy
I want to mention this before "same origin policy" problem. We had to use a window popup in OAuth authentication. Popup was used to open immediately after user clicks an element whom click event is connected to an authentication required function. This authentication required function was either opening popup, or running its own logic if user was already authenticated. This was how it was working:
 
User (before page loads):
- User requests a page which uses melange.gdata.core.js on user side. This script provides gdata login methods and gdata API requests for JS.

Server:
- Page indicates melange.gdata.core.js dependecy in its template by using our dependecy *mechanism*. 
- Page also needs to init melange.gdata.core.js with some context values those tell script some initial values: gdata_is_logged_in, oauth_redirect_url.

User (after page loads):
- If a page specific Javascript function  has a logic that needs authentication to GData, for example for the proposal page:

In proposal/base.js:

function sync() {
    //code that nees authentication
}
authenticated_sync = melange.gdata.core.createAuthorizedFunction(sync);
button.click(authenticated_sync)


This is working in peace for proposal page which adds melange.gdata.core.js to dependencies in it's template and inits it with some context value which is produced by a context helper (A helper which creates gdata_is_logged_in and oauth_redirect_url values). But for the "List Exporting" this was quite redundant workforce to init melange.gdata.core.js in every template because a big per cent of Melange pages contain lists. We did not want any extra configuration for every list page. Indicating list dependecy should be enough. So we tried to skip initialization of melange.gdata.core.js until user expresses he wants to use an authorization required feature. We tried to make this initilaziation with a JSON request. For example when user clicks "Export" button, it asks server for initialization values and with respect to response, performs required authentication process if required. Every thing seemed fine until i realized tha, request time delay was causing popup window to be blocked by browser's popup blocker. If we want to open a popup it should be immediately after a click event occured, but we can't know if we need to open a popup without asking it to server (so without waiting).

So we needed to change this mechanism. Script melange.gdata.core.js must be aware of gdata_is_logged_in value before click event. We solved this by moving gdata initialization into Melange's general javascript config initialization which works for every page. So with little redundancy (because we don't need to calculate is_logged_in value for every page), we were able to init it without extra initialization code for every page that uses melange.gdata.core.js. If we can assume %90 of Melange pages contains lists, this is not a big CPU usage redundacy.

I don't know if this popup blocking policy is browser dependent or not. But from my experiences i had the conclusion that most guarantee way of opening a popup is doing immediately after a click event is occured.

Saturday, July 30, 2011

GSoC 2011 Melange Testing Project: CP W9 & W10

Hi,

I could not write a blog post for the work I did during last week because of the
the university work and some other factors. This post summarises the work I did in last two weeks.

  Last week, I fixed some tests which were failing due to the recent change done to the mentor property. The tests for soc.modules.gsoc.tasks.test_survey_conversion failed because the models were seeded(Model.put()) by explicitly defining the model properties. This method is definitely not a robust one since if we happen to add other required properties in future, the tests will fail again due to the missing properties while seeding. The solution is to use the data seeder which randomly seeds the properties not mentioned in the properties parameter. If there is no data provider for a property, we can patch the data provider to seed such properties and the tests would run fine. So, I edited the tests for gsoc.tasks.survey_conversion and made them to use data seeder to seed the models.

     I also fixed the profile_utils.createStudent() method which had been bugging me while testing gsoc.views.student_forms. My tests failed 1 out every 7-8 test runs. It was because of the failing access checks which resulted because of the number_of_projects property being seeded other than 0 for a student without a project. This problem bugged me for quite sometime and Madhu helped me to figure out that this was actually the problem. Also, I submitted some patches to remove the unused imports in some modules in gsoc.views.

   This week I tested a couple of views namely views.org_profile, views.profile_show and submitted them for review. Leo has reviewed them and I will be fixing them today. I also added createInactiveProfile() and createInactiveStudent() method to profile_utils which I needed while testing views.org_profile. I discovered a defect in DownloadForm request handler in student_forms and I filed a issue for that. The defect was that a student without a project should not be allowed to download the enrollment and tax forms and in the present case a 500 error is returned which should be 403 instead. As told by Sverre, I need to add an access check to fix this.
 
    Since I get very less time during the week days to work on the project, I will be working on Friday, Saturday and Sunday as I don't have classes on these days.

Thursday, July 28, 2011

GSoC 2011 GUI Overhaul: Uploadable Logo with Area Selection

Hi everyone,

When I discovered a problem with the logo url rendered in the homepage, I believe blobstore for the organization logo will be fine. Yeah, until I took pains with redirect response. So here's the specification to implement image area selection in blobstore, so you can crop the uploaded image:

  1. We need coordinate fields, x1 as a horizontally starting-point where the area selection begins, x2 as horizontally end-point, y1 as vertically starting-point where the area selection begins and y2 as vertically end-point. Since we're using blobstore, we need to set BlobReferenceProperty field.
  2. Create the form similar to the tax or enrollment form, but only set the BlobReferenceProperty field. To make it clear, the uploaded file will be handled by Blobstore API and handover to our written handler afterward. In our handler we can set initial values for coordinate fields and redirect again to upload form.
  3. When the blob key is available, we need another handler to request the blob.
  4. To render the uploaded image in the form, we can follow how Google Map is rendered in the profile form.
  5. Image area selection was implemented using jQuery imgAreaSelect. The documentation is pretty obvious.
  6. There should be coordinate fields update handler which is posted when we made selection over the image. There's onSelectEnd paramater that we can passed over when instantiate imgAreaSelect.
  7. To serve the image with area selection, we need to create a handler that reads the blob based on provided blob key. Resized and cropped image will be manipulated here by using Image API.
I can't give you live demo for uploadable logo. I don't have a dashboard in billing-mode. Here's the screenshot if you're wondering:



Monday, July 25, 2011

GSoC 2011 Integration with External APIs: Melange Lists and GData API Protocol - W9

Hi,
Last week, I started to work on "Melange list exporting to GDocs" that will be second deliverable of my project. I've been searcing through GData Protocol to use JavaScript for  Spreadsheets API. This API is used required to create a spreadsheet and manage it. Unfortunately there isn't a JS API for this provided by Google. But GData APIs are built on REST like protocols over HTTP. This gives possibilty to anyone to use them with any language as long as correct HTTP requests are provided.

Well, we want to use Spreadsheets API with JS, but i haven't tell the reason yet. Reason is we can not trust how big the data is. For example it's quite big enough for accepted stundents page. GAE has a limitatation, max 30 seconds per request. Melange loads some of the lists with multiple requests, but when we want to create and upload a list we can't make it with multiple request (or that would be really harder). So we need to export data on client side.

This also saves time preventing reprocessing of query. Another benefit is that, it allows uploading modified list on client-side. For example, user may filter list for a parameter, or sort it on client side. With this approach we present user a feature like, "what you see is what you export" that is exactly what user wants.

Tuesday, July 19, 2011

GSoC 2011 Integration with External APIs: Proposal Sync - W8

Sorry for late blog post. First evaluation results of GSoC 2011 was announced. I passed first evaluation and I'm very happy for it :) My first deliverable "proposal sync" is still in review process. But my guess is it's mainly complete and just needs code design fixes. I'm busy with fixing my patches for a few days. My patches are mostly contains JS code and I wasn't writing quality JavaScript code before but just ones that saved my days. Not suprisingly Mario pointed out lots of bad design choices in my code. I want to mention a few of i learned :

* Always use === and !== instead of == and !=, These operators also checks if operands are same type besides their values are equal.

* Do not rely on content of DOM elements. Do not use them as variables. Create global variables instead.

* Using if and else without braces if they have only one statement is tricky. This later may cause someone not to notice there are no braces, and put a statment outside of else scope, while thinking it's in the scope.

* Melange JavaScript Guide also has good tips, for readibilty and quality of source code.

From now on, i will also start to work on Melange lists. We basicly want all Melange lists to have upload option to Google Docs. Melange lists are built-on jqgrid library which allows easily create and manage interactive HTML tables. We need a generic button that will work for every list without extra configuration, so someone creating a new list will not take care about how it forms it's data and uploads itself to GDocs.

That's all about me. See you next week and cheers :)

Sunday, July 17, 2011

GSoC 2011 Melange Testing Project: CP W8 Mid-term

Hi,
I have completed the logic tests and the task for post mid-term period is to write the tests for all the gsoc.views modules. Leo and I had a discussion after the last conference call and we decided to prioritise which modules should be tested first. A survey of the modules which have tests and which needs to be tested can be found at [0]. Accordingly, I have to first test gsoc.views and gsoc.tasks and then go on to write the functional tests. I am presently working on gsoc.views.student_forms and stopped working on soc.views.helper.access_checker.
I have no experience with Django and I spent some time studying the basics and  then asked Madhu some doubts that I had got while studying the view's code. I will develop pace as I get experience with the views.
    This week was also assigned for mid-term evaluations and Leo has passed me :). My classes are starting from 20th July and I need to manage both gsoc and classes. The last two days went in travelling to the university and the next two days will go in registration and shifting to the new hostel room.

[0] https://spreadsheets.google.com/spreadsheet/ccc?key=0AuENNrqFbxWEdDJMclZySzFUalpsUTd6bzNJTjJGVlE&hl=en_US#gid=0

GSoC 2011 GUI Overhaul: Things that I've been working on last week

Hi everyone,

As we already knew, mid term evaluation already passed. I contacted my mentor, Daniel, about the result of my mid term evaluation. Daniel told me to focus on the remaining works that need to be done. If you're following my discussion on the mailing list, there are obviously requested features in accordance with my project scope. Here's the list of thins I've been working on from the last week :
  • Account management. Sverre posted on mailing list about the problem he encountered by manually modify accounts using stats.py. Sverre gave some possible use cases:
    1. People have their profile migrated, their email address stays the
      same but their user_id changes, the User.user_id property needs to be
      set to None.
    2. People have a @gmail.com or @googlemail.com address attached to
      their @some.domain.com address, and their User.account needs to be set
      to the new address.
    3. Lennie added his suggestion for ban user feature.
  • Regular dashboard using iconic information. Daniel said to me give a shot for iconic dashboard for regular users (students and mentors). This means there will be two options to vote later, toggleable list and iconic dashboard. Toggleable list has been reviewed by Mario and I've made some adjustments (not posted yet).
  • Leo gave some review on my test for withdraw projects view. I need to fix the test.
  • Lennie asked me to make the evaluation links 3 level deep. I've sent the patch, no response yet.
  • Participant locations statistic.
I will post the patch for the above features on Monday, or lately Tuesday. After that there will be one week to review all the patches before going to be pushed.

Monday, July 11, 2011

GCI 2011 GCI with New Architecture - W7

Hello all,

This week I started with the Public Task view which first showed all the static fields and then I have created a Form to publish comments for the task and to retrieve the work submissions on that view.In the meanwhile my redirects patch got reviewed and I refactored the GSoC Redirects module but due to my mistake, it broke the existing code, I sincerely apologize for that.I have again written the GCI Redirects module avoiding these mistakes.I then wrote the GCI Timeline Helper class and moved some common code to the core.This got reviewed and will be committing this after the previous patches gets approved.After this I wrote the views to create and edit a GCI Task and have learnt some new things in this process and I sent this one for review too and then as advised by Madhu I moved onto to the task workflow, as the first step I have written a form with a drop down that has the list of actions that are available to the Org Admin, Mentor or Student depending upon the task status.I will be sending this to review now in bit.This week I have spent much more time with my project and in communicating with Madhu which helped me clear many doubts and work better, I feel I have this week's progress is better than the previous ones and I'm happy about it.I will be increasing my time on the project this coming week and will try my best to cover the lag.

GSoC 2011 GUI Overhaul: Push the code and some tasks need to be done

Hi everyone,

Last week I've been working to fix my patches with Daniel. Today I've pushed it into repository. It took a time to convert my commits into chronological patch with Mq. When I sent my patches without Mq, I used the following command to output the patch :

hg export -r started_revision_of_admin_dashboard:tip > patch.diff

Meanwhile the upstream repo changes rapidly with Madhu's changes, so importing the patches that have different tip with my own tip may abort the import. Mq is totally helpful if you're working on many patches while there are rapidly changes in the upstream repo. All you need to do is push your patches into queue and refresh it while receiving feedback from review. And then do hg qfinish to convert the patch into permanent changeset. With the awesomeness of Mq and review/feedback from Daniel and Praveen I just pushed my code a couple hours ago.

The admin dashboard isn't finish yet, there are some functionalities need to be done for PA, such as statistic that Daniel has been working on. I also promised to create MapClusterer to display participant in a single map like the following one:


After finishing the MapClusterer and integrate the statistic module into admin dashboard, I'll working on uploadable org's logo with cropping feature. I've made silly app that prove how Image service from GAE was capable to implement this feature.

Sunday, July 10, 2011

GSoC 2011 Melange Testing Project: CP W7

Hi

The task for this week was to correct my previous tests and finish the remaining tests in soc.logic and soc.modules.gsoc.logic. I have corrected my previous tests and submitted them again for the review. I have tested all the logic functions which are used at least once in the code base. I have not tested the soc.logic.allocations, soc.logic.lists and soc.logic.helper.notifications because these are not in my priority list for now as I need more familiarity with the code base to test them correctly and efficiently. I will come back at some time in future to test them.

       I have started attempting to write the view tests. I have never written any view tests earlier and my experience with Django is 0.1, but sometimes we have to learn the hard way. I studied many of the new tests written by Sverre. Testing a view seems to be a complicated task and I did not understand many things which I had to ask Madhu. Just before writing this post, I had a discussion with Madhu and I asked him what 'Context' is, what tests_utils.StuboutHelper is for, what soc.views.helper.access_checker.Mutator is used for and a load of other topics. At the time of writing this blog post, I was testing the soc.views.helper.access_checker. It will consume more time to test because there are more than 1100 lines of code to test. After testing this, I will test the gsoc.views.helper.access_checker module.

      I have the target of completing all the view tests before the final evaluations. It will require a huge amount of effort which I am pouring in. Lets hope for the best. :)

GSoC 2011 Integration with External APIs: Proposal Sync - W7

Approaching to midterm, i partially finished my first deliverable and sent my patches for review. I'm really For now there is one way sync from GDocs to Melange. We are still discussing to allow students "sync back" their proposals to GDocs.

I was struggled with some problems that i want to mention during this blog post. First when using GData libraries on App Engine infrastructure, exporting documents is not directly possible. The reason is that App Engine uses a distrubuted system and this does not allow to write to filesystem. To export anything, GData restricts its API to save output into a file instead of returning exported content. You need little change on gdata library to make it possible:


diff -r 84a930b76e63 -r e65b82d9ad71 app/gdata/docs/service.py
--- a/app/gdata/docs/service.py Sat Jul 09 03:58:23 2011 +0530
+++ b/app/gdata/docs/service.py Sun Jul 10 11:27:08 2011 +0300
@@ -106,6 +106,11 @@
         server=server, additional_headers=additional_headers, **kwargs)
     self.ssl = True
+    # Variables used to hack-in Export function to use it
+    # with a file handler instead of file path name.
+    self.file_handler = None
+    self.use_file_handler = False
+
   def _MakeKindCategory(self, label):
     if label is None:
       return None
@@ -183,10 +188,17 @@
       raise gdata.service.RequestError, {'status': server_response.status,
                                          'reason': server_response.reason,
                                          'body': response_body}
-    f = open(file_path, 'wb')
-    f.write(response_body)
-    f.flush()
-    f.close()
+
+    def writeResponseToFile(f, close=True):
+      f.write(response_body)
+      f.flush()
+      if close:
+        f.close()
+
+    if self.use_file_handler:
+        writeResponseToFile(self.file_handler, close=False)
+    else:
+        writeResponseToFile(open(file_path, 'wb'))
   def MoveIntoFolder(self, source_entry, folder_entry):
     """Moves a document into a folder in the Document List Feed.
@@ -346,7 +358,7 @@
     self._DownloadFile(url, file_path)
-  def Export(self, entry_or_id_or_url, file_path, gid=None, extra_params=None):
+  def Export(self, entry_or_id_or_url, file_path, gid=None, extra_params=None, file_handler=None):
     """Downloads a document from the Document List in a different format.
     Args:
@@ -361,6 +373,13 @@
     Raises:
       RequestError if the service does not respond with success
     """
+
+    if file_handler:
+      self.file_handler = file_handler
+      self.use_file_handler = True
+    else:
+      self.use_file_handler = False
+
     ext = None
     match = self.__FILE_EXT_PATTERN.match(file_path)
     if match:


With this you can use a memory file (StringIO) with Export function to write content to file handler directly instead of giving a file name to function for opening.

Second problem was about OAuth authentication page. We were willing to put OAuth page inside a frame in a jQuery dialog instead of a popup window. For strange reasons, page did not show up in the frame. I started an issue to learn the reason behind that: http://code.google.com/p/gdata-issues/issues/detail?id=2651

You can see demo of proposal sync from here with username:testmelange@gmail.com , password:melange2011 and type to find "Test Document 1" that is in GDocs account of testmelange user.

Monday, July 4, 2011

GCI 2011 GCI with New Architecture

Hello all,

Apologies for the late blog post.This week I have started off with the url patterns for GCI and Lennard suggested that I move the common functions to the core and leave the GCI related constants in that module and I have done that.I then went onto the GCITask view but before that I had to add another method to the access_checker to check if the task is published or not but looks like, I still need to modify this method a bit to include other cases too.I still have to retrieve work submissions for the Task view and form to edit the task and to subscribe the task.And me and Madhu had a small discussion about the templates and we decided to use the gsoc templates for now and make modifications where needed and use gsoc's css and JS till we get the designs.Meanwhile my other patches have been reviewed.First I had to refactor the gsoc redirects module, as many of the redirects are moved to the core and some methods have to be overridden to set the _url_name.And I was asked to make a little change to the gsoc access_checker as it's DeveloperAccessChecker should also inherit from AccessChecker, and the same was done to the gci access_checker too.Made required changes to the other patches as mentioned.I haven't spent enough time last week, as I had to help a Professor with his coursework and I really could not avoid it, I discussed with him today and took permission till the end of August, So I will cover up this week and will spend extra time on my project.

GSoC 2011 Melange Testing Project: CP W6

Hi,
In the last conference call, I reported my problems to Sverre that most of my time is spent on reading the code itself and that I had problems in deciding about the use cases when writing tests. My productivity has been very low till now and I have written atmost two tests per week. This week I worked hard and I have written tests for all the modules in gsoc.logic and soc.logic except three modules. Though the tests are under review and they need drastic improvements, I have now become comfortable in deciding about the use cases and writing tests. I had a discussion with Leo and he told that I was unnecessarily using DjangoTestCase. I can use seeder_logic and inherit my test classes from unittest.TestCase and this will improve the performance and speed. I was using DjangoTestCase to avoid instantiating the GSoCOrganization, GSoCTimeline and other entities which I could use directly when inherited from DjangoTestCase. But seeder_logic is more useful and its also used in DjangoTestCase. So, I corrected the test for soc.logic.host and used seeder_logic to seed the entities and it was working fine.
     
I tested a total of 8 modules in this week and the coverage has increased by 4%. The present coverage of the tests in my local copy is 70%. The coverage will increase more when I write tests for the soc.views. 

My work for this week is to correct the previous tests, test the remaining three modules in gsoc.logic and soc.logic and then start the view tests.I hope to finish all the logic tests before the mid-term evaluations.

PS: Mercurial queues are AWESOME. Salute and gratitude to the guy who wrote that extension. :)

Sunday, July 3, 2011

GSoC 2011 GUI Overhaul: Typical PA Workflow and Admin Dashboard Usage

Hi everyone,

A week ago I asked Madhu about the typical workflow for PA (Program Admin). Madhu was so kind to explain the workflow for PA. I'll summarize the workflow and usage of admin dashboard by the PA. The followings are ordered things that should be done by PA to organize the GSoC program:
  1. Initialize the program settings such as program name, document page, or timeline of the program. These settings already provided by site.py and timeline.py views.
  2. Invite organizations to participate.
  3. Create the organization application survey. I believe this is related to what Madhu's did for survey with form builder. I can easily add the url inside the new admin dashboard to refer to the form builder.
  4. PA and her team discuss the organization selection internally.
  5. Accepted organization announced. I don't know yet if there's a view or task to accept organization.
  6. Student signup period. Students begin submitting proposals to organizations.
  7. PA can assign slot in this time or before the accepted projects announced. There's a view for slot assignment.
  8. PA can check the duplicate proposal. There's a view to trigger the duplicate proposals' task queue. This queue will iterate each proposals and filter the proposals by the same student that would be accepted by two or more organizations. The resolution was done manually by gathering a meeting.
  9. PA can check slot transfer by organization and accept/ignore the request of transfer by org admin. There's a view for this.
  10. After all organizations solve the duplication status and agree with the given slots. PA can trigger the accept_proposals' task queue. There's a view to trigger accept_proposals queue.
  11. PA creates survey for mentors and students. There's a survey builder committed by Madhu.
  12. PA freeze the program after the final term.
I don't know yet what the task needed by the PA after creating the survey. But the typical workflow should be like the list I mentioned above.

Sunday, June 26, 2011

GSoC 2011 Integration with External APIs: Learning new tools - W5

This week i was busy with fixing my patches. Madhusudan suggested me a tool to create mutable changesets : Mercurial Queues (aka MQ). It was  not possible to work on multiple commits without actually commiting them. This great extension makes this possible. You can create a patch, edit it, then create new one and return back to previous patch to edit it. (Patch is a mutable changeset) I benefit from this mostly in two ways : First as i am new to community my commits need to be reviewed. When i send a patch for review, i can start to work on another while first one is in review and mutating. Second reason which also will make me use MQ even after i'm good enough to send my commits without review is it helps divide and code! I can divide a job into parts which are slowly growing (each part corresponds to a commit). So i have a working job which is not complete but each parts are still growing and making working job better. Instead of finishing each part in order and trying to make them perfect as i dont have chance to go back, i'm working on every task at the same time synchronously. I also wrote on my Turkish blog about MQ :)

Another tool i was introduced was {{less}} which helps writing CSS in a hierarchical style, without requiring to write full selector path for each block. It also allows defining variables which prevents redundancy when using same values multiple times. It's clearly more programmers way to CSS.

While being little late for proposal syncing feautre. I still could find some time to think on it and send a mail to community to discuss on it. There is no harm to copy/paste my current idea here :)

  • When user goes to '.../proposal/submit...', represent a sub-page with two buttons : Edit HTML, Sync with GDocs
  • Go to a different page for each selection, each represents different form. (e.g. '/proposal/submit/gdocs', '/proposal/submit/default') One page requires to login GDocs and a gdocs flag will be True for proposal entity after saved. 
  • For updating part, 'edit proposal' link's destination will be changed with regards to proposal's gdocs flag (either go to '/proposal/update/gdocs/id' or '/proposal/update/default/id').
  • Widget 'GDocs Link' will have a 'sync now' button near it. When user pastes link to field and presses "sync now" button, an ajax request will be executed. Text area 'Content' (which is not editable) will be updated with result of that AJAX request. So actually, in server-side, we won't process document link again and treat form like its content edited by hand. 

GCI 2011 GCI with New Architecture

Hello all !!

This week I have worked on the views after completing the conversions tasks.I first started off with RedirectHelper for constructing redirects in GCI.Each of the methods in this helper returns the RedirectHelper object after setting certain parameters like _no_url, url_name, _url, args and kwargs.The clear method clears the current internal state and urlOf method returns the resolved url for name using current internal state for args and kwargs.At this stage certain field from GSoCProgram like about_page, events_page, events_frame_url, connect_with_us_page etc are required for GCI too and hence they are moved up to soc.models.program.Program.The RequestData patch for GCI got reviewed and I am about to commit it now after making a couple of changes.
I then proceeded to the Access Checker for GCI.For this not much is needed at the moment and more methods will be added as we move on with the views.For now, I have to written a couple of methods one is access_checker.Mutator.taskFromKwargs which sets the task entity in the RequestData object and the other is access_checker.AccessChecker.isTaskInURLValid to check if the task in the url exitst or not.After acess_checker I have moved onto site menus for GCI.I was told that I can use the same site menus as that of GSoC for now.So I went ahead and have written a smiliar site menus for GCI which contains siteMenuContext to generate URL links for the hard-coded GCI site menu items and Header, MainMenu and Footer templates.I have then finally moved onto RequestHandler for GCI.The render method of the soc.views.base.RequestHandler is overridden to extend the context object with header, mainmenu and footer values which are rendered Templates for the current RequestData object.I have now started with the Task view.

GSoC 2011 GUI Overhaul: The Admin View's Design

Hi everyone,

From my previous post I've talked about colorbox approach for admin dashboard. If you want to try, visit my demo instance at http://melange-new-dashboard.appspot.com/ using the following account:

password: melangetestdashboardpassword

There are some changes in the JS, CSS and template. Some of these were just new files. I'll talk about code that make a sense to review. Before that, I'll divide the topic into two sections so you can skip a topic that you don't want to read.

Changes in admin's view

If you've tried the admin dashboard from the link that I gave above. You should try to toggle the colorbox or click the "Program settings" link. If you click the "Program settings" link, the dashboard will change its content. The case is the same for "Manage organizations" link. If you inspect the page with firebug, you'll notice that it just switching between div elements. Let's call this a sub dashboard.


So, a link that have sub links at the bottom of its description will point to the div that hidden at that moment, that's a sub dashboard. When we clicked "Program settings", the div with id "program-settings-dashboard" will be shown, the "disabled" class was removed. And the "main-dashboard" will be hidden. I figured out what's the best solution to reorder the links or reorganizing sub dashboard without coupled too many files. So I came up with the idea that these links and sub dashboards should be managed inside soc.modules.gsoc.views.admin. I managed to create a new class named Dashboard that inherits Template class:

class Dashboard(Template):
"""Base dashboard for admin page
"""

def __init__(self, request, data, subpages=None):
"""Initializes the dashboard.

Args:
request: The HTTPRequest object
data: The RequestData object
subpages: Subpages of current dashboard
"""
self.request = request
self.data = data
self.subpages = subpages

def getSubpagesLink(self):
"""Returns the link to other dashboard that appears
on top of the dashboard.
"""
return self.subpages

def templatePath(self):
"""Returns the path to the template that should be used in render()
"""
return 'v2/modules/gsoc/admin/dashboard.html'

def _divideSubPages(self, subpages):
"""Returns the subpages divided into two columns.
"""
middle_ceil = int(math.ceil(float(len(subpages))/2))

return [
subpages[:middle_ceil],
subpages[middle_ceil:],
]

Currently there are three dashboards that inherits Dashboard class, they are: MainDashboard, ProgramSettigsDashboard and ManageOrganizationsDashboard. To get you the clear idea, here's the MainDashboard and ProgramSettigsDashboard classes:

class MainDashboard(Dashboard):
"""Dashboard for admin's main-dashboard
"""

def __init__(self, request, data):
"""Initializes the dashboard.

Args:
request: The HTTPRequest object
data: The RequestData object
"""
super(MainDashboard, self).__init__(request, data)

def context(self):
"""Returns the context of main dashboard.
"""
r = self.data.redirect
r.program()

manage_orgs = ManageOrganizationsDashboard(self.request, self.data)
program_settings = ProgramSettigsDashboard(self.request, self.data)

subpages = [
{
'name': 'lookup_profile',
'description': ugettext(
'Lookup profile of mentor or student from various program.'),
'title': 'Lookup profile',
'link': r.urlOf('lookup_gsoc_profile')
},
{
'name': 'allocate_slots',
'description': ugettext(
'Allocate slots (number of acceptable projects) per '
'organization'),
'title': 'Allocate slots',
'link': r.urlOf('gsoc_slots')
},
{
'name': 'slots_transfer',
'description': ugettext(
'Transfer slots for organizations'),
'title': 'Slots transfer',
'link': r.urlOf('gsoc_admin_slots_transfer')
},
{
'name': 'duplicates',
'description': ugettext(
'Calculate how many duplicate proposals'),
'title': 'Duplicates',
'link': r.urlOf('gsoc_view_duplicates')
},
{
'name': 'manage_organizations',
'description': ugettext(
'Manage organizations from active program. You can allocate '
'slots for organizations, manage invitations for '
'org admin/mentors, and withdraw/accept students/mentors '
'from various organizations'),
'title': 'Manage organizations',
'link': '',
'subpage_links': manage_orgs.getSubpagesLink(),
},
{
'name': 'reminder_emails',
'description': ugettext(
'Send reminder emails for evaluations.'),
'title': 'Send reminder emails for evaluations',
'link': r.urlOf('gsoc_survey_reminder_admin')
},
{
'name': 'program_settings',
'description': ugettext(
'Edit program settings and timeline'),
'title': 'Program settings',
'link': '',
'subpage_links': program_settings.getSubpagesLink(),
},
{
'name': 'participant_locations',
'description': ugettext(
'Show all participants (students and mentors) in a '
'clusterer map. You can also view a single profile map '
'after clicking the marker.'),
'title': 'Participant Locations',
'link': '#'
},
{
'name': 'report_statistic',
'description': ugettext(
'Reports and statistics of program'),
'title': 'Report/statistic',
'link': '#'
},
]

return {
'title': 'Admin Dashboard',
'name': 'main',
'subpages': self._divideSubPages(subpages),
'enabled': True
}


class ProgramSettigsDashboard(Dashboard):
"""Dashboard for admin's program-settings-dashboard
"""

def __init__(self, request, data):
"""Initializes the dashboard.

Args:
request: The HTTPRequest object
data: The RequestData object
"""
r = data.redirect
r.program()

subpages = [
{
'name': 'edit_program',
'description': ugettext(
'Edit your program settings such as information, slots, '
'documents, etc.'),
'title': 'Edit program',
'link': r.urlOf('edit_gsoc_program')
},
{
'name': 'edit_timeline',
'description': ugettext(
'Edit your program timeline such as program start/end date, '
'student signup start/end date, etc.'),
'title': 'Edit timeline',
'link': r.urlOf('edit_gsoc_timeline')
},
]

super(ProgramSettigsDashboard, self).__init__(request, data, subpages)

def context(self):
"""Returns the context of manage organizations dashboard.
"""
subpages = self._divideSubPages(self.subpages)

return {
'title': 'Program Settings',
'name': 'program_settings',
'backlink': {
'to': 'main',
'title': 'Admin dashboard'
},
'subpages': subpages
}

To set the links, we need to set it into subpages context. If you want a link to another dashboard inside MainDashboard, such as "Program Settings", just create another class that inherits Dashboard, supply the backlink to refer back to the parent's dashboard and set the subpages of its own dashboard inside the new class. In the parent dashboard's subpages context supply the subpage_links to indicate this is a dashboard link. How about the icon itself? This is where the CSS rules. The name you supply for each link inside the subpages content will be rendered as a class name of the link's container. Everything's that related to the style of admin dashboard is defined inside v2/gsoc/admin.css file. I got the icons from Tango Desktop Project which is licensed under GPL.

The next thing that you might need to know is the addition of colorbox base layout. A colorbox base layout is minimalistic template without sidebarmenu and header. The rest is the same. To make any page rendered in colorbox layout, you need to pass cbox query string. The RequestHandler will take care the base layout. Each page that extends the base.html template need to change the extends tag to base_layout, {% extends base_layout %}. The rest of changes inside soc.modules.gsoc.views.admin were new handlers with list.

What's done and what needs to be done or in-progress

I promised to finish the admin dashboard before the mid-term. This means the code it self should be reviewed lately in this week. The scope for the deadline before the mid-term was a fully functional admin dashboard that the mockup already given before. Okay, here are what's done in the admin dashboard:

  • New admin's view architecture that I've discussed in the previous section
  • Handling colorbox template
  • A few handlers inside soc.modules.gsos.views.admin

Here are what needs to be done and in-progress:
  • Submit my code for review, possibly after writing this post.
  • The test_slots was failed because I created new template and changed the list template for accepted organizations. I refactor the list for accepted organizations inside admin's view so other handlers can use it. I'll fix the test after my code being reviewed.
  • Adding test for soc.modules.gsoc.admin. By looking at other view tests, it just a test case to make sure the template is rendered with the right context.
  • A participant locations to mark all participant into a single map using MarkerClusterer. This is not a critical requirement and can be skipped.
  • Refined toggleable list patch as suggested by Mario.


Saturday, June 25, 2011

GSoC 2011 Melange Testing Project: CP W5

Hi,

This was quite a busy week. Learned a lot and worked a lot. I worked on writing the tests for soc.logic.tags and soc.logic.models.base. I had attempted to test soc.logic.tags during the pre-GSoC era but failed because I did not understand properly what 'scope' and 'fields' parameters were. I asked Madhu what a 'scope' is and he explained it to me that it was a solution adapted by Melange during the early days of development to define a relation such as a GSoC project belongs to a Student. So, a GSoC project entity was scoped under a Student profile. But, as the developers understood more about the Transactionality and other stuff in GAE, they switched to using 'parent' argument of an entity to define the scope. But, scoping will stay for tagging.

   Earlier during the week, after Leo reviewed my patch for soc.logic.accounts, I pushed the code without first updating my local repo. So, I first pushed the patch and then pulled. But, then I could not find the test I had written in the online repository. So, I pushed it again and as a result to this, I ended up creating two heads in the repository. Sverre asked me to take care of the extra head I had created. I did not know how to do that. So, I googled and asked some friends. Some suggested using the 'strip' extension, 'merge' and some 'mercurial queues'.
Ultimately, I asked Madhu how to revert what I had done. He looked at the case and asked me if I do not rebase my patches after pulling. rebase!!!???What is it??. I then asked him what rebase here refers to. He gave a very nice tutorial on the concepts behind rebasing and the two schools of thought which I will never forget, the 'Purists' and the 'Whatever-ists'. :). I like Madhu's way of explaining things. Things get so easily printed in the mind.

   What I learnt about the 'parent' argument from the discussion with Madhu, helped me to extend the old tests for soc.logic.models.base authored by Sverre and Leo. The tests reported errors when I ran because it could not find test_model module which I also thought does not exists and may be it has been removed. So, I defined my own dummy class TestModel extended from soc.models.base.modelWithAttributes, added some more tests to the module and submitted them for review. After this, when I was looking for other module to test, I saw that there was a test_model in old_app.soc.logic.models and this was the reason why it gave ImportError. So, lets see what Leo says :).

 I now plan to re-use the old tests which Leo had written last year. The goal of my project has been re-scoped to write the unit tests until mid-term evaluations. I had written in my proposal that I will start writing functional tests in June. But, there are modules which have to be unit tested before. So, the target until mid-terms is to add as much unit tests as I can.


Monday, June 20, 2011

GCI 2011 GCI with New Architecture

Hi all !

This week I have committed the GCIProfile and GCIStudentInfo models.And currently I'm working on the script to convert GCIStudent data to GCIStudentInfo and as I had some troubling understanding certain things I spent a couple of days in going through the Oreilly - Programming Google Apps Engine book.I've spoken to my mentors sometime back and got a better idea as to what to do.Instead of querying for the profile that each student entity points while creating the GCIStudentInfo entity, it can be created when the Profile for each entity is created,the parent-child relationship can be obtained easily like this.Besides this I have also written a basic RequestData object that will be created for each request in GCI and will add more properties and fields as we go ahead in the project.The patch is under review along with the script to convert roles and update references in the models.This week I haven't worked upto the mark.I will finish the conversion works and start with views at the earliest this week.

GSoC 2011 Integration with External APIs: OAuth Demo - Week 4

This week i'm finishing Open Auth integration and will start to work on proposal syncing with Google Documents next week.

A demo view that uses Google Docs Service is available here. I made a few modifications for OAuth workflow diagram that you can see here.


After login process is complete, Google documents of user are listed.
Currently, tokens are stored by memcache. Access tokens are long-lived. So i marked a TODO to store access tokens in datastore (with Sverre's suggestion). Memcache is not a convenient way to store  a token for such a long period. After talking to Mario (my mentor), we decided to put a 'remember me' checkbox with login button, so when user logs out access token will be preserved for next login ( that  is also current behaviour). These are changes that i can work asynchronously while i work on proposal syncing feature.

Proposal syncing feature is the first deliverable of my project. There are few potential problems awaiting:
  • Integrating a full-html document which also has seperate image, css files. 
  • To sync proposal without user signs to site, a task must be implemented to sync proposals when user is offline.

GSoC 2011 Melange Testing Project : CP W4

Hi!
I had submitted two patches last week: one for test of app.soc.logic.dicts and the other for app.soc.logic.accounts. There were lot of mistakes and issues pointed out in the review. I have corrected soc.logic.accounts and submitted it again for review today. I also missed some functions to test in soc.logic.dicts and will do it this week.
  I wrote the test for app.soc.logic.system this week and I had a long discussion with Madhu whether to use DjangoTestCase or unittest.DjangoTestCase. Madhu suggested using the DjangoTestCase and I insisted that I could test the functions with the help of unittest.TestCase itself. I had to test the soc.logic.system.getHostname(data=None) function and for this I needed a data object which had a 'site.hostname' attribute. After some grepping and exploration, I came to the conclusion that it should be a soc.views.helper.request_data.RequestData() object and the 'site' should be a soc.models.site.Site() entity which Sverre told me. But then I had a problem in instantiating the Site entity and the traceback reported that a link_id property is missing. So, after struggling for some time I told Madhu the problem and he referred me to test_utils.DjangoTestCase which had a site and data attribute which he said I use in my tests. He also said that RequestData entity should be populated and seeded to efficiently test it. My point was that all the entities of RequestData object are set to either False or None and also I do not need all of its attributes in testing. I only need the data.site and data.site.hostname attributes and this I could accomplish without using the DjangoTestCase and directly handling the RequestData and Site objects. Finally he asked me to go ahead and write the test with unittest.TestCase.
    85% of my time is spent on reading codes and there are multiple windows that remain open almost all the time. I have to hibernate my system to avoid opening so many windows again.
   I did not do much coding this week but hope to do more this week. Me and a friend of mine are planning to participate in ACM-ICPC regionals this year. So, we spent most of the time in studying for ICPC only.

Friday, June 17, 2011

GSoC 2011 GUI Overhaul: Redesigned Admin Dashboard

Hi everyone,

As I promised from last post, I'll talk about redesigned admin dashboard in this post. Currently, admin dashboard is just a page with list, without any information, to other pages. I've provided mockup for admin dashboard with iconic information. There will be also new functionalities such as manage organizations with another sub functionalities to manage org, proposal, mentors, and students, participant locations, etc. There are two approaches to render child page in admin dashboard. These approaches is discussed in the following section :

Admin Dashboard with AHAH for Child Page

From last conference, Sverre told that Carol needs page in scope of admin should be stay on same page with parent page, that's admin dashboard. He said, it could be achieved with AJAX request/response. I tried the AJAX approach by making a div container to hold the HTML response. It's better to call this AHAH rather than AJAX. So, this div container is a hidden one and will be displayed after the HTML response injected to the container. The container was placed below the iconic information. To make the request handled with a minimum HTML responded, I created another blank template, base_ahah.html which contains following code :

{% block main_content %}
{% if ds_write_disabled %}
<div id=user-message class=error>
<strong>ALERT: </strong>Google Appengine datastore write
capability has been disabled at the moment. You will
<strong>NOT</strong> be able to save your changes at this
time. We are sorry for the inconvenience caused. Please visit
<a href=http://code.google.com/status/appengine>
http://code.google.com/status/appengine</a> to check
Appengine datastore status.
</div>
{% endif %}
{% block page_content %}
{{ tmpl.render }}
{% endblock page_content %}
{% endblock main_content %}

Page requested via AHAH must extends this template. We also need to modify form_base.html to check whether it was requested via AHAH or regular request with full base layout. If a page requested via AHAH, the JS code will add another query string, ?ahah=1. In the AHAH base template, we don't request any CSS or JS asset. I don't know if there's a way for LABjs loads once for the same script, and skip injecting the head for subsequent script calls. To be safe, any assets should be provided first when admin dashboard page requested. The form submission should be changed with AJAX. This was easily done with jQuery.post function. Any response, either invalid or valid, will be injected again in the AHAH container. With this approach, any admin sub page that need to be added later must exclude the asset its needed to parent page, admin dashboard. This make any sub page coupled to its parent page, which is not good.

Admin Dashboard with Iframed Colorbox for Child Page

Colorbox is yet another thickbox-like jQuery plugin. Colorbox provides a way to hold the content of external page in iframed container. For iframed template we need to create another base layout again, say base_colorbox, with minimum markup and variabel template. In iframed container we could extend block stylesheet and add melange script dependencies. Here's what admin dashboard with colorbox looked alike :


Within iframe, we need to specify the width and height for colorbox. We can specify the width and height in URL query string. Colorbox approach seems the easy way to implement without coupling to the parent page.

In the next day I'll provide the live instance to experiment with the new dashboard, so stay tunned.

Tuesday, June 14, 2011

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 3

I've been developing generic methods to handle OAuth. When developing software it's important to create meaningful abstractions so a method can refer to these abstractions without need to understand low-level details what they do. For the OAuth part of my project a helper module handles this low-level details and caller that refers to these abstractions is an access_checker. It asks following questions:

  • Get access token for user.
  • Create a service for user.
  • Generate a OAuth authorization URL.
  • Process URL for OAuth verifications.
Low-level helper module does following operations for questions above:

Get access token for user:
  • Generate a unique key consists of user's key and current host (different consumer key,secret so different services and access tokens are used for each hosts). 
  • Ask memcache or a storage mechanism to fetch token corresponding to that key.
  • Return token.
  • If there isn't a token return None.
Create a service for user:
Service is a GDocsService object that is used to make API calls.
  • Create a service for the current host. 
  • Set service's input parematers to host's consumer_key and consumer_secret. 
  • Set encryption method to sha or rsa.
  • Return service.
Generate a OAuth authorization URL:
  • Fetch a request token. 
  • Save request token's secret to memcache for later use (when validating token).
  • Generate a authorization URL from request token.
  • Return URL.
Process URL for OAuth verifications:
  • If request URL contains a verification key as GET parameter, parse it.
  • Verify previously stored access token with parsed verification key.
  • Store verified access token to memcache or datastore.

Monday, June 13, 2011

GSoC 2011 GUI Overhaul: Toggleable List

Hi everyone,

Today I'll talk about toggleable list for regular dashboard. To make you clear, there are two dashboards in Melange, one is regular dashboard and the other one is admin dashboard. The regular dashboard is the page that we see if we clicking the "My Dashboard" link on the left sidebar menu, or if you visit the url at gsoc/dashboard/(host)/(program_name). The admin dashboard is available in the left sidebar menu as "Admin Dashboard" if you have a role as a host. If you're login with account that listed as an administrator of melange instance, you have access to that admin dashboard. Admin dashboard can be visited at gsoc/admin/(host)/(program_name)

Currently, regular dashboard is loading a bunch of list at one time. Each list, or called component in dashboard view, in regular dashboard is fetching a lot of entities from datastore in the background. Each role gets different components. If someone has a role both as a host and a mentor, Carol's case, then she gets much components. Too much component will reduce usability. User needs to scroll hardly if she/he looks for component at the bottom. Fetching all list at one time is inefficient too. I've provided three ideas with mockups to handle this problem, one is with tab, two is with iconic page (the same as admin dashboard that I'll talk in the next post) and the last is with toggleable button. Carol agreed with toggleable button. The reason for toggleable button is user can toggle which list/component she/he wants to show without leaving the dashboard page.


I've provided the patch to melange-soc-dev. The implementation is quite simple. To make a component toggleable we just set toggleable and collapse to True inside the context. The list template will check this variable. If we ignore toggleable and collapse then the list will be requested when loading the page. We can set a component toggleable but doesn't collapse at first load by set the toggleable to True and collapse to False. With this approach, other pages with list, such as accepted_orgs, wouldn't be affected. In the next post I'll talk about admin dashboard that being redesigned right now.

GSoC 2011 Melange Testing Project : CP W3

Hi,

For quite a long time I had been working on the online test runner and could not make it work on a live instance. As I had discussed it in my last blog post about my implementation that I intended to exploit some plugins of Nose to run the tests, it failed because of the reason that Nose uses some libraries which are restricted by GAE. So Leo asked me to take my other approach of discovering all the test modules in the tests directory and then make a suite of tests using the unittest.loadTestsFromName  method. I used the os.walk() built-in function to first list all the modules with their absolute paths and then changed the paths into a dotted notation so that these paths could be easily used by unittest.loadTestsFromName to load the tests. I have used a lot of code from the gaeunit source specifically which is related to the browser interface and the test runner. The test runner had to be modified so as to fit in with melange code. I also had to include the tests directory, gaetestbed, nose, webtest and  mox libraries in the app folder so that tests can easily use them when running on a live instance.
    The online test runner runs on my local development environment but it shows errors that are due to the bugs in the runner itself. Below is the video of the melange tests running on my local machine through the browser.

I have stopped working on the runner as of now. But in the end, I discovered that I have extended some of the features of gaeunit. It has been 2 years since gaeunit was last released, so I was planning to contact the gaeunit developers if I can develop for them.

     During the last week conference call, Sverre asked me to treat the online runner as a side project and concentrate on my testing project now. So, I have started working on my project and wrote the test for app.soc.logic.accounts and submitted it for review. At the time of writing this post, I have also almost completed the test for app.soc.logic.dicts with a few methods left to test.

Monday, June 6, 2011

GSoC 2011 Integration with External APIs: OAuth Roundtrip - Week 2

It's been two weeks since coding started, due to my heavy exams i haven't been working too much and had to skip first week's blog. I updated my timeline with target deliverables [0].

First thing to solve is how OAuth will be handled with a generic solution. As we wouldn't want to handle all jobs (e.g. token check, redirection, access token parsing) in every seperate view, there needs to be a generic handler (a 'token_required' decorator and a 'gdocs_service_object' middleware in Django's terminology) for views that needs to make GDocs api calls over OAuth.

I started to work on this diagram to discuss OAuth mechanism with community:


One of the particular problems when playing with gdata library was a not-self-defining error about unicodes. As GAE returns entity field values in unicode, a  method in built-in hmac library that expects 8-bit strings was crashing with this error message:
character mapping must return integer, None or unicode
Solution was simply encoding field value to utf-8: consumer_key.encode('utf-8'). I don't think same error would arise when using rsa instead of sha with authentication.

[0] -http://code.google.com/p/soc/wiki/GSoC2011IntegrationWithExternalAPIs#4._Project_Timeline

GSoC 2011 Melange Testing Project : CP W2

My task for this week was to implement an online test runner which enables us to run our unit tests online rather than on the local machine. I have written the necessary modules which provide a very basic functionality and features for now. More features can be added and improved with time. Leo had suggested me the gaeunit Open Source project to study and see how they do it.

I outline the basic process of how gaeunit does the magic :-
  • gaeunit defines two main request handler classes - MainTestPageHandler and JsonTestRunHandler. The request to run the tests is handled by the MainTestPageHandler class which first validates the URI arguments and renders an html /plain report according to the 'format' argument. Our interest lies in the generation of the html report.
  • When the request is to render an HTML report, gaeunit first creates a suite of all the tests found according to the 'name' parameter passed to it in the URIs. 'name' can be any package, module or class. This suite of tests is then converted into a JSON format with structure {"module_name": {
    "class_name": [list_of_methods]}} and passed with the HTML response to the browser.
  • As soon as the response is loaded in the browser, it triggers a javascript whose task is to iterate through the JSON object and pass the (module_name, class_name, method_name) tuple to the server by generating an asynchronous XMLHttpRequest which then waits for the data to be sent by the server.
  • The GAE server upon receiving the tuple runs the JsonTestRunner and the test result is collected in a unittest.TestResult object. This TestResult object is then again converted into a JSON object and returned to the browser. 
  • The browser then sends the data to the appropriate fields in the web page
My Implementation:
My implementation does not work presently and it needs lot of debugging. I moved the tests folder to melange/app/ so that GAE finds it easily. I wrote two modules online_run.py and online_tester.py. online_run is colleague of tests.run.py with lot of code borrowed from it to implement the test runner for online testing.
      online_tester implements classes to handle and manage all the tasks right from initiating the tests to printing the test reports. I thought of exploiting the 'testid' and 'collect' plugins of nose. Running the tests with these plugins only collects the tests without actually running the tests and assigns an integer id to all of them. We can run a specific test next time by just passing the integer ids. So I have defined three classes MainTestPageHandler, PrepareForTests and RunTests. MainTestPageHandler just renders a view where users can select which tests to run. The arguments are passed as POST arguments to the PrepareForTests handler which is supposed to collect the tests and assign IDs to them. The number of collected tests is then passed to the browser with HTML source which contains a javascript . This javascript passes the integer IDs to RunTests which makes use of the runner in online_run module to execute the tests and returns the HTML output to response.out stream.

This implementation is not working  and I asked Madhu for help. He said may be I have messed with the appengine paths. I have to try again keeping the tests in app/ and making the necessary changes in my scripts.

Also the paths like '/home/praveen/testmelange/app/tests' which I have included in the python paths according to my system will not work on GAE because we do not know about their file systems.

You can take a look at the error reported by my implementation at my instance[1]
Just choose 'Run all tests' and hit Run.

[1] http://melangetesting.appspot.com/runtest
[2] online_run.py    http://tinypaste.com/da4b5e
[3] online_tester.py http://tinypaste.com/1f3b88