GSoC 2010 Data Seeder: Wrapping up for end of GSoC

So GSoC is coming to an end, it's time to wrap up the project to a pretty much fully functional and usable state, while there could still be much to do. So, my latest changes mostly include usability improvements like being able to test the configuration for a data provider and get some sample values, include provider and parameters descriptions, being able to validate, download or upload a configuration sheet.

Other improvements include some visual styling, although I personally find my work very ugly, I'm hoping someone with better visual appeal might help me with this part in the future.

As a final proof of concept, I will provide a number of configuration sheets that will demonstrate the full power of the module: being able to generate mass amounts of "real world" data very easily (although doing a first-time configuration might be a little time consuming).

I have also encountered some unexpected problems. It seems that GAE (or maybe the Mapper API, I'm not sure) limits the amount of data that a task can receive as a parameter. Configuration sheets can get pretty big (I won't be surprised to see some reach tens of KBs) so I needed to find a workaround. My solution involves storing the actual configuration sheet in a model, and then each task would read the configuration sheet from the datastore. While this solution speeds down the process by quite a bit, it might prove more scalable and also provide some historical value, should the configuration sheets remain in the datastore.

Enough about the technical stuff... It's been a real joy working with everyone around the Melange community and I'm sorry if I haven't been all that serious all the time during GSoC. Many thanks to my mentor Mario who has provided me with priceless information all the time and who has made all my work so far possible. Also, thanks to all the other developers (Lennard, Sverre, Madhusudan and Daniel), you are simply great and it's been a pleasure to work with you all. Cheers to all other GSoC students (Leo, Pavel and Savitha), I hope we'll see each other in the future as well. Having said that, it's time to continue working on the project, there's still a lot of things that can be improved.

Cheers!