Plat_Forms 2012 rev. 2

What has been going on? - part 3

Today we will conclude our little series of preliminary results. After the robustness data in part 1 and the activity data in part 2, today we will show you the completeness of our participants' solutions.

Completeness

For each requirement from the requirements document two randomly assigned judges evaluated a solution's implementation of that requirement and categorized it as missing, partially implemented, fully-implemented-but-flawed (e.g. failing in over 25% of the use cases or being buggy), fully implemented, and particularly-well-fully-implemented. Figure 1 shows all fully implemented requirements, i.e. omits missing and only partially implemented requirements.

Figure 1: Number of fully implemented requirements by requirement priority and team. The rightmost bar for comparison shows the possible maximum.

What has been going on? - part 2

As promised last week, today we share another bit of data with you.

Activity interviews

During the contest every 15 minutes we asked each team member what he is currently doing. We listened to all 2328 interviews thus collected and labeled the answers with codes corresponding to one of several activity types. Figure 1 shows the totals (per code) for each team.

Figure 1: Frequency of each type of activity performed at the time of our interviews.

What has been going on?

It has been a while since Plat_Forms 2012 and today we want to share with you what we have been doing in the meantime.

We have collected an enormous amount of data (interviews during the contest, code, version archives, running solutions, questionnaires, and more) and haven't finished our evaluation yet. In particular we sofar cannot draw any platform-specific conclusions. As you might remember, Plat_Forms 2012 wanted to find a new platform notion that is not tied to a particular programming language but based on characteristics of the technology stack. This turns out to be a difficult task that we are still working on.

In the following three weeks we will therefore simply show a few per-team charts without comment.

What we do have is data on the type of activity the teams were performing at the moment of our interviews (every 15 minutes), data on the completeness of the teams' solutions, and data on the robustness of the solutions.

What's missing and currently being worked on amongst others is data on the maintainability of the solutions, data on the static structure of the solutions (several notation-specific scanners still need to be written), and of course the attempt to explain the observations with the technology employed. So stay tuned for updates.

Robustness

For our robustness tests we stressed the solutions with different uncommon inputs -- some valid, others invalid. Figure 1 shows our observations.

Figure 1: Results of our robustness tests. Light red stands for broken, yellow for acceptable and green for OK. We submitted input with HTML tags ("</...>" column), veeeeeery long character strings ("long" column), Unicode characters ("int'l" column), 5 different types of invalid email addresses ("email" column), strings containing SQL quote characters ("SQL" column), and we disabled cookies before trying to login ("cookie" column).

The Task 2012

This page contains the Plat_Forms 2012 Task and additional files. The files are only available to the Plat_Forms contestants and will be made public after the contest on Oct 10 2012.

Plat_Forms 2012 Teams announced

We accepted the applications of 10 teams for Plat_Forms 2012.

The teams admitted to the contest are (in alphabetical order):

The Perl Foundation and Valiton unfortunately had to cancel their participation.

Application period extended, location details

Lutz Prechelt is going to give a keynote talk on Plat_Forms at the Developer Conference Hamburg on 7 Sep 2012. We decided to extend the application period until 14 Sep 2012 in order to give conference participants a chance to apply.

Also the venue details have been finalized now. Plat_Forms will take place at the Computer Science Faculty of Freie Universit├Ąt Berlin. The exact address is Takustr. 9, 14195 Berlin. See the map below for directions. We have a special rate for participants that want to stay at the Seminaris Campus Hotel next door. The number of rooms is limited though. Accepted participants will receive the details shortly after the application deadline.

Plat_Forms 2012 announcement is out

The Plat_Forms 2012 rev. 2 announcement and the Application Form are out. Applications for Plat_Forms 2012 will be accepted until Fri 2012-09-07.

We are looking forward to your applications!

Plat_Forms 2012 rev. 2

TLDR: After we had to cancel a first Plat_Forms 2012 effort in April this year, we are now going to try again and hold Plat_Forms 2012 rev. 2 on October 9-10 of 2012.

For the April 2012 execution we tried to put Plat_Forms under a cloud computing and scalable applications theme. There were two reasons for that: First we thought that having a popular technology such as cloud computing under scientific scrutinity would yield interesting results. Second we noticed that distinguishing a programming platform for web applications by the language only might not be accurate enough anymore since there are a number of other things, most importantly the web development framework, that play a role here. By moving the focus away from a typical HTML-producing web application to a scalable web service we hoped to get around that problem. The alternative would have been to systematically find out what constitutes a web framework, how to group "similar" frameworks, and what "similar" means in the context of web development frameworks which is something that hasn't been done before.

Although we received a lot of positive feedback during our planning phase and a lot of interest from potential participants, we failed to recruit enough teams. The main reason for that seemed to be lack of confidence in their own skills by potential teams. The argument we heard most often was that the teams feared they couldn't master the relatively complex cloud computing technology good enough to compete with other teams.

We therefore decided to take the extra effort and do the framework classification I mentioned earlier and return to Plat_Forms' roots. Plat_Forms 2012 rev. 2 will take place on October 9-10 2012 in Berlin on the premises of the Freie Universit├Ąt Berlin (exact location still to be determined). The contest will return to its origins and the focus will again be on "classic" web applications. We will however introduce some changes:

  • Platform grouping will not only be by language but by a mixture of language and technology stack used (exact details are still to be determined).
  • HTML prototypes will not only be provided this time but will also need to be strictly adhered to.
  • We will put some focus on good craftmanship. This means that the task will be relatively small (but not trivial) allowing for proper QA, deployment, etc. We don't want to see dirty hacks wink.

What you can do now

  • Assemble a team. We will open up registration for the contest shortly and will publish a detailed announcement.
  • In the meantime you can read up on the Plat_Forms 2011 announcement which will probably be quite similar to the one for Plat_Forms 2012 rev. 2 modulo the task-specific parts.
  • Subscribe to our announcements mailing list (low-volume: about one message per week) or use the RSS feed of our Blog, which will carry roughly the same content.
  • Become a sponsor of Plat_Forms 2012 rev. 2. (Make sure you also subscribe to the mailing list!)
  • Read about the history of Plat_Forms

What else has been going on

Apart from trying to find out how to progress with Plat_Forms, we were not inactive: