GSoC 2011 GUI Overhaul: More Handlers for PA

Hi everyone,

GSoC is near to the final evaluation and I in a run to catch all my TODO. Last week, I wrote some handlers for PA to manage account and assign a user to be a host. PA can assign programs to be hosted by particular user in edit user page. Here's how it looks:



The code is somewhat similar to assign_mentors.py and project_details.py. To set a user as a host, PA needs:

1. Given an account to be set as a host, go to the Lookup page in the admin dashboard.
2. If the account is found then click the "Edit user" link
3. In the edit user form, there's a box at the bottom (see picture above) that contains program selection. To set multiple programs to be hosted by the user, click "Add new" link. If done with the selection, click the "SET AS A HOST" button

Actually there's some features that I've in mind such as add / edit sponsor, programs list that contains list of program name and its hosts as cols, page to view readonly program that in inactive state, and of course the page to create new program. I'll hold thus things for a moment because the dashboard is still being refactored and I promised to finish it by Aug 6th. I am late two days here.

GSoC 2011 Integration with External APIs: Same Origin Policy - W10

Browsers have some restrictions for cross-domain Javascript requests due to some security issues. For "List exporting" from client side, we need to make cross-domain requests to Google from user. Before this jsonp was only standart that i know as a solution. Jsonp allows loading a json from a remote web server. But i knew GData protocol supports XML format data exchange. I tried to find a solution to load a cross-domain XML source. No solution was neat enough to apply.

Script Approach
Then i thought GData should support JSON of course, and found related document :) We should just have to add a "alt=json" parameter to url GET params. Same page also was providing a method to access resources from a cross-domain with JavaScript. We should call a special script that calls our callback with related json as parameter when page is loaded. Here is an example script:

<script src="http://www.google.com/calendar/feeds/developer-calendar@google.com/public/full?alt=json-in-script&callback=listEvents"></script>

which returns a javascript response with a form in which json response is embed.

listEvents({"version":"1.0" ..... })

This provides another way for making cross-domain requests. In this solution server must support this method with a special effort to form a javascript response instead of just interpretting request as a regular json request.

Http Access Control
This approach was good but unfortunately not enough for our purpose because we need to set Authentication header with request. So i returned back to jsonp. I thought it should be possible to make a jsonp request and add a "Authentication" header with request like that:

jQuery.ajax({url: a_cross_domain_url+"&callback=?", data: data, headers: {"Authentication": auth_header}, success: a_callback)

This was almost working until i encountered with another cross-domain restriction. Sending headers with a cross-domain request was limited to another standard. I read this document about "Cross-Origin Resource Sharing". As far as i understand from document, For sending headers with request, this is defining a way for browser to first ask to server if it allows given headers, then makes actual request. This sounds good but because it's a new recommendation and some browsers may not support it (and perhaps GData server too) , i skipped this. Firefox supports this after 3.5 version.

At the end
I learned some stuff  but still was not happy as there wasn't a way sending OAuth header with request. GData documentation contains AuthSub authentication over Javascript but i couldn't find anything related OAuth with JavaScript in which we will provide OAuth header. Then, while i was hopelessly playing with OAuth playground, saw that choice in form:
Put oauth params in: Authorization header   URL as params

This was just i was looking for. We were able to pass oauth params with url instead of header. This was something not documented in GData OAuth documentation. I was lucky to find this out.


At the and my approach was shaped: Make a jsonp request, put oauth params inside url.

Note: This didn't work either, to see my final solution to make Javascript requests see next post by me.

GSoC 2011 GUI Overhaul: Document List for Admin

Hi everyone,

Lets talk about document list. Carol asked this on issue 1257. To implement such list, I borrowed the view from accepted_orgs.py. Lets name the list "DocumentList" and the page "DocumentListPage". I only added two visible cols, the title and shortname, for DocumentList. When the row is clicked, it will go to the edit document page view. I also created the test for document list inside test_document.py. The test will make sure, the list will be shown to the host (based on checkAccess method). Okay here's the class for DocumentList and DocumentListPage:

class DocumentList(Template):
 """Template for list of documents.
 """

 def __init__(self, request, data):
   self.request = request
   self.data = data
   r = data.redirect

   list_config = lists.ListConfiguration()
   list_config.addColumn('title', 'Title',
       lambda e, *args: e.title.strip())
   list_config.addSimpleColumn('link_id', 'Link ID', hidden=True)
   list_config.addColumn('short_name', 'Short Name',
       lambda e, *args: e.short_name.strip())
   list_config.setRowAction(
       lambda e, *args: r.document(e).urlOf('edit_gsoc_document'))

   list_config.setDefaultPagination(False)
   list_config.setDefaultSort('title')

   self._list_config = list_config

 def context(self):
   description = 'List of documents for %s' % (
           self.data.program.name)

   list = lists.ListConfigurationResponse(
       self.data, self._list_config, 0, description)

   return {
       'lists': [list],
   }

 def getListData(self):
   idx = lists.getListIndex(self.request)
   if idx == 0:
     fields = {'scope': self.data.program}
     response_builder = lists.QueryContentResponseBuilder(
         self.request, self._list_config, document_logic, fields)
     return response_builder.build()
   else:
     return None

 def templatePath(self):
   return 'v2/modules/gsoc/document/_document_list.html'


class DocumentListPage(RequestHandler):
 """View for the list documents page.
 """

 def templatePath(self):
   return 'v2/modules/gsoc/document/document_list.html'

 def djangoURLPatterns(self):
   return [
       url(r'documents/%s$' % url_patterns.PROGRAM, self,
           name='list_gsoc_documents'),
   ]

 def checkAccess(self):
   self.check.isHost()

 def jsonContext(self):
   list_content = DocumentList(self.request, self.data).getListData()

   if not list_content:
     raise AccessViolation(
         'You do not have access to this data')
   return list_content.content()

 def context(self):
   return {
       'page_name': "Documents for %s" % self.data.program.name,
       'document_list': DocumentList(self.request, self.data),
       'program_select': ProgramSelect(self.data, 'list_gsoc_documents'),
   }

And here's the test for document list:

class ListDocumentTest(DjangoTestCase):
 """Test document list page.
 """

 def setUp(self):
   self.init()
   self.data.createHost()

 def testListDocument(self):
   url = '/gsoc/documents/' + self.gsoc.key().name()
   response = self.client.get(url)
   self.assertGSoCTemplatesUsed(response)

   response = self.getListResponse(url, 0)
   self.assertIsJsonResponse(response)
   data = response.context['data']['']
   self.assertEqual(1, len(data))

GSoC 2011 Integration with External APIs: Popup Blocking Policy - W10

This week, unfortunately i was not productive. I had quite problems mainly based on "same origin policy". Tonight, story of the week seems ending nice, because i finally resolved how to make cross-domain requests with JavaScript. I will write two blog posts this week, this one for another problem "popup blocking problem" and next one for "same origin policy". If you don't have extra time for reading two blog posts, i suggest you read next one, because it's the main story of the week for me.

Popup Blocker Policy
I want to mention this before "same origin policy" problem. We had to use a window popup in OAuth authentication. Popup was used to open immediately after user clicks an element whom click event is connected to an authentication required function. This authentication required function was either opening popup, or running its own logic if user was already authenticated. This was how it was working:
 
User (before page loads):
- User requests a page which uses melange.gdata.core.js on user side. This script provides gdata login methods and gdata API requests for JS.

Server:
- Page indicates melange.gdata.core.js dependecy in its template by using our dependecy *mechanism*. 
- Page also needs to init melange.gdata.core.js with some context values those tell script some initial values: gdata_is_logged_in, oauth_redirect_url.

User (after page loads):
- If a page specific Javascript function  has a logic that needs authentication to GData, for example for the proposal page:

In proposal/base.js:

function sync() {
    //code that nees authentication
}
authenticated_sync = melange.gdata.core.createAuthorizedFunction(sync);
button.click(authenticated_sync)


This is working in peace for proposal page which adds melange.gdata.core.js to dependencies in it's template and inits it with some context value which is produced by a context helper (A helper which creates gdata_is_logged_in and oauth_redirect_url values). But for the "List Exporting" this was quite redundant workforce to init melange.gdata.core.js in every template because a big per cent of Melange pages contain lists. We did not want any extra configuration for every list page. Indicating list dependecy should be enough. So we tried to skip initialization of melange.gdata.core.js until user expresses he wants to use an authorization required feature. We tried to make this initilaziation with a JSON request. For example when user clicks "Export" button, it asks server for initialization values and with respect to response, performs required authentication process if required. Every thing seemed fine until i realized tha, request time delay was causing popup window to be blocked by browser's popup blocker. If we want to open a popup it should be immediately after a click event occured, but we can't know if we need to open a popup without asking it to server (so without waiting).

So we needed to change this mechanism. Script melange.gdata.core.js must be aware of gdata_is_logged_in value before click event. We solved this by moving gdata initialization into Melange's general javascript config initialization which works for every page. So with little redundancy (because we don't need to calculate is_logged_in value for every page), we were able to init it without extra initialization code for every page that uses melange.gdata.core.js. If we can assume %90 of Melange pages contains lists, this is not a big CPU usage redundacy.

I don't know if this popup blocking policy is browser dependent or not. But from my experiences i had the conclusion that most guarantee way of opening a popup is doing immediately after a click event is occured.

GSoC 2011 Melange Testing Project: CP W9 & W10

Hi,

I could not write a blog post for the work I did during last week because of the
the university work and some other factors. This post summarises the work I did in last two weeks.

  Last week, I fixed some tests which were failing due to the recent change done to the mentor property. The tests for soc.modules.gsoc.tasks.test_survey_conversion failed because the models were seeded(Model.put()) by explicitly defining the model properties. This method is definitely not a robust one since if we happen to add other required properties in future, the tests will fail again due to the missing properties while seeding. The solution is to use the data seeder which randomly seeds the properties not mentioned in the properties parameter. If there is no data provider for a property, we can patch the data provider to seed such properties and the tests would run fine. So, I edited the tests for gsoc.tasks.survey_conversion and made them to use data seeder to seed the models.

     I also fixed the profile_utils.createStudent() method which had been bugging me while testing gsoc.views.student_forms. My tests failed 1 out every 7-8 test runs. It was because of the failing access checks which resulted because of the number_of_projects property being seeded other than 0 for a student without a project. This problem bugged me for quite sometime and Madhu helped me to figure out that this was actually the problem. Also, I submitted some patches to remove the unused imports in some modules in gsoc.views.

   This week I tested a couple of views namely views.org_profile, views.profile_show and submitted them for review. Leo has reviewed them and I will be fixing them today. I also added createInactiveProfile() and createInactiveStudent() method to profile_utils which I needed while testing views.org_profile. I discovered a defect in DownloadForm request handler in student_forms and I filed a issue for that. The defect was that a student without a project should not be allowed to download the enrollment and tax forms and in the present case a 500 error is returned which should be 403 instead. As told by Sverre, I need to add an access check to fix this.
 
    Since I get very less time during the week days to work on the project, I will be working on Friday, Saturday and Sunday as I don't have classes on these days.

GSoC 2011 GUI Overhaul: Uploadable Logo with Area Selection

Hi everyone,

When I discovered a problem with the logo url rendered in the homepage, I believe blobstore for the organization logo will be fine. Yeah, until I took pains with redirect response. So here's the specification to implement image area selection in blobstore, so you can crop the uploaded image:

1. We need coordinate fields, x1 as a horizontally starting-point where the area selection begins, x2 as horizontally end-point, y1 as vertically starting-point where the area selection begins and y2 as vertically end-point. Since we're using blobstore, we need to set BlobReferenceProperty field.
2. Create the form similar to the tax or enrollment form, but only set the BlobReferenceProperty field. To make it clear, the uploaded file will be handled by Blobstore API and handover to our written handler afterward. In our handler we can set initial values for coordinate fields and redirect again to upload form.
3. When the blob key is available, we need another handler to request the blob.
4. To render the uploaded image in the form, we can follow how Google Map is rendered in the profile form.
5. Image area selection was implemented using jQuery imgAreaSelect. The documentation is pretty obvious.
6. There should be coordinate fields update handler which is posted when we made selection over the image. There's onSelectEnd paramater that we can passed over when instantiate imgAreaSelect.
7. To serve the image with area selection, we need to create a handler that reads the blob based on provided blob key. Resized and cropped image will be manipulated here by using Image API.

I can't give you live demo for uploadable logo. I don't have a dashboard in billing-mode. Here's the screenshot if you're wondering:

GSoC 2011 Integration with External APIs: Melange Lists and GData API Protocol - W9

Hi,
Last week, I started to work on "Melange list exporting to GDocs" that will be second deliverable of my project. I've been searcing through GData Protocol to use JavaScript for  Spreadsheets API. This API is used required to create a spreadsheet and manage it. Unfortunately there isn't a JS API for this provided by Google. But GData APIs are built on REST like protocols over HTTP. This gives possibilty to anyone to use them with any language as long as correct HTTP requests are provided.

Well, we want to use Spreadsheets API with JS, but i haven't tell the reason yet. Reason is we can not trust how big the data is. For example it's quite big enough for accepted stundents page. GAE has a limitatation, max 30 seconds per request. Melange loads some of the lists with multiple requests, but when we want to create and upload a list we can't make it with multiple request (or that would be really harder). So we need to export data on client side.

This also saves time preventing reprocessing of query. Another benefit is that, it allows uploading modified list on client-side. For example, user may filter list for a parameter, or sort it on client side. With this approach we present user a feature like, "what you see is what you export" that is exactly what user wants.