HackTheHearst Judging Procedures
Presentation judging procedures
Still being ironed out by the judges.
Technical judging procedures
A panel of four technical judges will assess each HackTheHearst submission against the following nine criteria:
- (20%) How good is the UX presented by the submission? (i.e. are the UI components laid out in a logical way and of the right sort, encouraging discovery and easy to understand and use; is the flow acceptable and the presentation of results clear; is it easy to recover from user errors?)
- (20%) How good is the overall design and software architecture (is the modularization and design ecumenical? or is it on the other hand eclectic and hard-to-understand? Is there room to expand and develop the application, adding new functionality?)
- (15%) How pretty / visually complete is the submission? (Is the css bright and clear? Is the overall graphic design appealing?)
- (15%) How good is the code quality (coding style, inline documentation, clarity of implementation, tests? Is the code concise and understandable?)?
- (10%) How clever or original is the submission from a software point of view? (e.g. Has this been done before? Does it present a novel use of existing patterns and techniques? Is the UX something that is likely to produce new, unforeseen, or previously unobtainable results?)
- (10%) How quick is the response time? How long does it take to get a useful result?
- (10%) How easy will it be to scale the design and implementation support more users, more data, and more features? That is, how efficient is the submission now, and can the required performance be sustained were more demands to be made?
- (yes/no—required to win) Does the submission save state in a way that would make it difficult to ensure the freshness of the results (given that the datastore is normally refreshed nightly)?
- (yes/no—required to win) Are steps necessary to deploy the interface and its dependencies, however complex or trivial, documented and cited so that a person with system administration skills but who is only marginally familiar with the technologies used can efficiently review and execute those steps?
The score assigned for these nine criteria will count towards 35% of the total project score.
Each criterion will be assigned a score between 0 and 5 by each technical judge according to the following guidelines:
5 - Excellent; far exceeds criteria/requirements/expectations
4 - Good; goes beyond criteria/requirements/expectations
3 - Meets criteria/requirements/expectations
2 - Weak; falls somewhat short of criteria/requirements/expectations
1 - Poor; fails to meaningfully address criteria/requirements/expectations
0 - Absent or missing
The results of judging will be recorded in a shared Google Docs spreadsheet that will be prepared prior to September 21st and made available to all judges. The spreadsheet will have a row for each team/submission, columns for scoring each criterion, and a place for technical notes. The spreadsheet will also contain (for each submission):
- the URL where judges can access the code and associated code documentation for each project;
- a URL where judges can review the submission in operation, with credentials if needed;
- the URL where judges can access the deployment documentation for each project;
- the dependencies (languages, libraries, frameworks, and/or platforms) used in developing each project, and whether any of these is not open source;
- the email address for each team's primary technical contact.
The presentation judges will use the same spreadsheet, and will mark which submissions made the cut into the top 16 (prize-eligible) submissions.
Technical judging can take place remotely, and can be done as a group or individually (as long as the judging criteria are applied evenly by the individual judges). If there is sufficient time, the organizer's preference is that each of the top 16 apps be assessed by each technical judge, with the final score on each criterion being the weighted mean of the judges' scores.
Technical judging can begin as early as 12:02 AM (Pacific Time) on Sunday, September 21st.
Only the top 16 apps need be judged for technical merit (these will be announced between 12:30 and 12:40 PM Pacific Time). The judges can choose to judge apps prior to the announcement of the 16 finalists, and/or can judge non-finalists, but the 16 finalist apps must be judged.
Technical judging must be concluded by 3:45 PM (Pacific Time), so that the Presentation Judges can decide the winning apps between 3:45 and 4:15 PM (Pacific Time).
The technical judges (or at least one of them) should be available in person or via Skype during the final judging (3:45–4:15 PM Pacific Time) to:
- answer any questions that might come up,
- allow the technical judges to express any opinions they may have that aren't sufficiently communicated by the spreadsheet,
- give the technical judges a say in the application of the judges' discretionary 15% score.