Tenth Week Newsfeed Development Update

At the end of last week, I submitted a major refactoring of the newsfeed component, which now includes a separate subscription component that manages e-mail subscriptions (which can contain more private information than the public ATOM feed).

After receiving feedback on this latest patch, the plan is to commit the Newsfeed. Before I commit, one very important change is to taskify the sendMail method of the mail dispatcher. Obviously, this doesn't just affect the Newsfeed component, but it's very important for newsfeed subscriptions as I discovered after deploying and testing on my jamstage.appspot.com staging instance. The e-mail notification processing is done from within a task, and when a task returns a 500 error the task is repeated. Because the task often sends out many e-mails at once, I hit a OverQuotaError from this task, which re-triggered the task, and it wasn't long before my e-mail inbox was flooded with e-mails.

This has been a good lesson about working with the Task Queue API. It's good that a task will retry if it fails, but this means that only atomic (and idempotent) functionality should be contained within each task, so that if it fails the retry behavior will not bring about undesired behavior.

Once the Newsfeed is commited, I'd like to finish the UpdateLogic which is necessary for automatically compiling subscriptions. This means that we need a list of users with read-access. I describe the problem further in the NewsFeed wiki page:

The subscriptions logic contains an UpdateLogic class that is meant to automatically compile a list of users with read-access to an arbitrary entity. Unfortunately, setting subscriptions manually is the only option right now, since the automatic subscription handler is not nearly where I hoped it would be by now. The idea is very simple - just subscribe users to the entities where they have read-access. But the reality has been much more difficult to do without creating an entirely new infrastructure, and I'm entertaining the idea of simply relying on a manual interface (subscribe-by-star). The takeaway from this is that perhaps the access component should be abstracted to a high-level API, because it appears to have gained dependencies and complexities that prevent it from being useful to the news feed logic.

Finally, I plan to implement a simple PubSub example implementation, and do more in-depth testing (especially of edge cases, etc.)

Sixth/Seventh Week Status Update

It appears as though there are now some diffs about which week we're on! Depends on whether we're indexing from 0 or 1?

This week I find myself working on two different tasks with a very similar theme.

The first task is in the News Feed module. When we generate a feed item, we're doing it on the logic side. the onCreate, onUpdate, and onDelete hooks that are used to create the feed item is in the model logic layer, and the feed item process itself (now being converted to a scheduled Task) is also in the logic layer.

However, the payload for a certain feed item requires access to view params in many cases so that model-specific payload templates can be re-used to render the payload for the activity item.

And you could say, "yes logic can't retrieve from view, but isn't the view retrieving from the logic anyways?" This is true, and this is why normally this is not a problem. But the problem arises when the view for one model type (site) wants to access view params for another model type (document, or survey).

This is also a challenge for the view for org admins to view the results from the manage a student project page. In this situation, the view model is for StudentProject, but the most obviously DRY approach would be to also instantiate a view class for survey view models so that we could render a list of ProjectSurveys or GradingProjectSurveys.

Short version: the view layer seems somewhat impenetrable from the logic layer, as it is tied up in the HTTP request. As a view should be. But what do we do about these edge cases?

I've been impressed with the orthognalness (is that a word?) of Melange, so it's entirely possible that I haven't yet figured out how to properly instantiate a view from the logic -- or why I don't want to do that in the first place.


On an unrelated note, another good learning experience has been in how I've been thinking about privacy in regard to how news feed activity can be consumed. While at first I was mostly alright with the idea of generating a hash key for hard-to-guess URLs. But the news feed activity is mostly HTML, content, and key words. In other words, if we're not careful, it could very easily end up crossing paths with a robot (edit: not if we set our robots.txt correctly!).

Newsfeed Status Update

Working on the news_feed feature has required work in all the different sections of the codebase (Model, logic, caching, view, template) and has required updates to about a dozen existing modules. As you can imagine, merging upstream is a fun exercise in getting aquainted with some of the more advanced features of version control.

The mechanics behind the news feed are very simple, at least to start with. Whenever an entity - a document, for instance - is created, updated, or deleted, it gathers a list of receivers. The implementation right now is limited to the scope of linkable models, but luckily that covers almost all of the use-cases where we'd want the news feed to be updated.

A new FeedItem entity is created for each receiver - since the receiver in the current implementation is the scope, the receiver is only one entity, but I'm working on a method to propagate news feed updates like an event bubbling up through a hierarchy.


A news feed is constructed for the receiver entities - such as Program, Organization, Club, etc. - and aggressively cached. The template for the news feed itself is very similar to how blogs are formatted in Melange.

One part of the template has been a big headache - the link redirecting to the sender entity.

Right now, this is what the link HTML template looks like -

href="/{{ item.sender.kind.lower }}/show/{{ item.sender.key.name }}"

Yes, it's hacky. But it works...most of the time. When the sender's kind name is "Document", then this translates to /document/, which is what we want. But when the sender's kind name is "StudentProject", this is rendered as /studentproject/, while it should be /student_project/. I'm still figuring out how to properly linkify an arbitrary entity...and I tried using the redirect views helper, but I was hoping to do something that wouldn't require any additional server-side code.

The next week has a lot of challenging tasks ahead. Now that the basic implementation is working, I'm going to work on automation and testing. I'm also going to code up an ATOM feed that can be used to export news feed updates. (This shouldn't be a security issue, since news feed updates only include very basic information and a link to the entity for further information)

Surveys also need to have their ACL component finished so they can be taken for midterms...I've struggled with this when attempting it before, but GSOC needs this feature working, so I'm going to have to figure out a way to do it.

Finally, I've also updated logic for surveys and documents so that based on the prefix of the document, it creates a scope property. This way, the scope doesn't have to be retrieved at runtime and the document model is more in sync with the rest of Linkable entities.