The Results 2011
On November 25th 2011 we presented the results of Plat_Forms 2011. The slides are available in our results presentation (PDF, 2.2 MB).
A detailed results report will follow.
A brief summary:
- Solutions of Java and in particular Ruby teams had a higher than average completeness (Figures 1 to 3).
Figure 1: number of fully implemented requirements, grouped by requirement priority
Figure 2: number of fully implemented requirements by functional area
Figure 3: number of partially and fully implemented requirements. Note team Java I that implemented almost as many requirements partially as completely, probably due to time management problems.
Compared to the 2007 results, the average level of completeness increased for Java and decreased for Perl and PHP (Figure 4).
Figure 4: 1.0 represents the mean number of fully implemented requirements over all platforms. Values above 1.0 thus mean more implemented requirements than the average, values below less than average.
- All Perl, PHP and Ruby solutions' SQL handling resisted our manipulation attempts. Only one solution, written in Java, showed signs of a possible SQL injection attack vector. See Figure 5.
Figure 5: Results of our robustness tests. Dark red means inacceptable, light red stands for broken, yellow for acceptable and green for OK. White squares indicate that we weren't able to perform that particular test on the solution. We tried to store HTML tags ("</...>" column), tried to input long character strings ("long" column), tried to store Unicode characters ("int'l" column), tried to enter 5 different types of invalid email addresses ("email" column), tried to input string containing SQL quote characters ("SQL" column) and disabled cookies before trying to login ("cookie" column).
- Perl was the only platform where all testable solutions correctly rejected the three types of invalid email addresses that are identifiable by simple syntax tests. On the other hand, all but one Java solution only rejected one type of invalid email addresses. Three of the four PHP solutions though not only rejected the syntactically invalid email addresses but also checked against a list of known top-level domains. For the fourth team we assume that this validation feature has been forgotten to be turned on. See Figure 5.
- The overall observed robustness has increased compared to the results of 2007: For example only 2 out of 16 (in 2007: 3 out of 9) solutions show signs of a possibility for cross site scripting attacks and only 1 out of 16 (4 out of 9) solutions seems to be vulnerable to SQL injection attacks (compare Figures 5 and 6).
Figure 6: results of the same robustness tests in 2007
- An average Ruby team participating in Plat_Forms posed questions to the on-site customer as often as an average Java, Perl and PHP team combined (see Figure 7).
Figure 7: average number of questions posed to the on-site customer. The wiskers denote the standard error of the mean.
- Our Ruby teams on average spend more time writing automated tests than Java, Perl and PHP teams combined. Interestingly, we observed the Java teams doing manual tests as often as we observed Ruby teams writing automated tests. See Figure 8.
Figure 8: average number of times we observed teams of a platform doing manual tests (red bar), writing automated tests (blue bar) or running an automated test (orange bar).
- The Ruby and Perl solutions tend to be more compact than the Java solutions. The PHP solutions don't show a consistent picture with some being more compact and some less. See Figure 9.
Figure 9: The x-axis shows the number of implemented requirements, the y-axis the lines of coded needed to implement these. The orange line denotes the average number of code lines needed to implement a certain number of requirements. Solutions lying below this line thus are more compact than the average, solutions lying above are less compact.
- We did not observe the uniform results among the PHP teams that we observed in 2007.
The teams' solutions are online now as well:
Winning Teams and Thanks
Our thanks go to the teams participating in Plat_Forms 2011. We hope to see you all again in March 2012 for Plat_Forms 2012.
Special congratulations go to our winning teams:
- for Java: Team A, Crealogix, in particular for the highest level of completeness among all Java solutions.
- for Perl: Team C, #austria.pm for the second-most complete Perl solution, the best robustness among all Perl solutions, and the most compact Perl solution
- for PHP: Team M, mindworks, for the most complete PHP solution
- for Ruby: Team P, Infopark, for the most complete Ruby solution
All winning teams look forward to eternal fame and a prize money of € 1.000 sponsored by the Open Source Business Foundation.