Most hackathon entries are judged by a system the team never sees. At DOGFOOD 2026, the teams built that system. Every entry was a submission and judging platform, and the people scoring it were judges who would be using exactly that kind of tool to score them. A weak normalisation model in any of them would decide who won.
That turned out to be a demanding mirror. Building a form that collects scores takes an afternoon. Building one whose ranking you can defend to the team that came fourth takes much longer, and the strongest submissions spent most of their 72 hours on that second problem.
The Challenge
DOGFOOD ran online from 18:00 UTC on 26 September to 18:00 UTC on 29 September 2026. The brief was a single product: a self-hosted portal that runs a hackathon end to end, from registration and team formation through submissions, judging, results and certificates. Every entry had to start with one docker compose up on a laptop and keep working with the network switched off at run time. Images could be pulled during the build; after that, the platform was on its own.
The scope came as a published four-tier ladder. Tier 1 covered the core event: accounts, teams, submissions and a deadline the server actually enforces. Tier 2 was judging: assignment, weighted rubrics, judges isolated from one another's scores, and cross-judge normalisation. Tier 3 added community voting with abuse controls, and Tier 4 the stretch features: an API, webhooks, signed certificates and embeddable galleries. An official acceptance checker verified the first two tiers against shared fixture data, and teams declared what they claimed in a manifest. Honest gap reporting was rewarded and inflated claims were penalised, which shaped the event as much as any feature list.
What Participants Built
194 projects were scored. The field divided less by technology stack than by how seriously each team took the question at the centre of the brief: when two judges disagree, whose number counts?
Correcting for the judge
The most consistent pattern among the top entries was an explicit model of judge leniency. Plain averaging rewards projects that happen to draw a generous judge, and several teams set out to prove it against the fixture data before fixing it.
Verdict, from team TBD, took first place with a final score of 4.611. Its pitch is "judging you can defend," and the engineering backs it. An additive leniency model with shrinkage evens out harsh and generous judges, and the team tested it in 300 simulated events built on the fixture's real judge-project pairs, where it recovered the true ranking better than raw averages or z-scores. Every project gets a 90% rank range rather than a single false-precision position. Verdict also adds a head-to-head mode, in which judges choose the stronger of two projects and a Bradley-Terry model produces a second ranking to check against the rubric.
RAPTOR DESK, from GSK VERIFIED, finished second at 4.583 with a similar model and a sharper sense of its own limits. The team showed that in the official fixtures one project finishes third only because it drew a generous judge, then built a desk that corrects for it and keeps doubting the result. Every project gets a probability of finishing in each prize place, close calls are flagged, and a "kingmaker" check names any single judge who alone decides a prize. Spare review capacity is sent to the judges whose extra opinion would actually settle something, and whatever is still close goes to a deliberation board where each decision carries a written reason.
EvenHand, from Len3, solved the same problem as a ridge regression of project quality plus judge leniency, and then refused to overstate what it had found. Scores come with their uncertainty and with tie groups, so a leaderboard that is really a near tie is shown as one. Dogfood Portal from LippInc took the same stance, estimating each judge's leniency from the event's own scores before anything is ranked and attaching the reviews, the arithmetic and a margin of error to every score, average and place.
BeyondBug, from the team of the same name, attacked a failure mode most entries handled with special cases: the judge who gives every project the same mark. Its additive fit makes that judge harmless by construction, and the team shipped a test-enforced planted-truth study showing that unregularised calibration does worse than plain averaging, a result most teams would rather not publish.
Refusing false precision
A second family went further and built the platform's honesty into the product itself.
Quorum, from Quorum Sensing, placed fifth at 4.402 on a simple principle: it refuses to print a podium the data cannot support. It keeps every project at its review target across an asynchronous judging window, spends spare judge time only where a prize is still in doubt, and settles ties with a head-to-head round whose rules are fixed in advance. Every team receives a scorecard with written feedback, and every judge a signed evaluation record. A local assistant answers organisers' questions from the event's own data and is never allowed to score.
ballotbench, from Cause Of Death, finished fourth at 4.418 and made calibration the core of the system. A reviewer model learns each judge's leniency, scale and noise. A judge who gives everyone the same score counts exactly as if absent, and shifting or stretching a judge's scores changes nothing, which the team tested to fifteen decimal places. After publication, each team can open a view that answers "why did we come fourth?" with the judges anonymised. That feature reads like a small courtesy and is in fact the whole brief in one screen.
Enforcing the rules below the interface
The third family treated the user interface as the least trustworthy layer and enforced everything again further down.
ballotbench is the clearest example on deadlines: the page, the API and a Postgres trigger running on the database's own clock all refuse a late edit, so no single bug can let one through. The same team proved judge isolation with 94 scripted requests that tried to read scores they should not see. Verdict took the same approach to permissions, with one integration test that calls every documented route as every role that should be refused, more than 600 calls in all, and a hash-chained audit log behind every change.
BroadSheet, from Binary Builders, took third place at 4.428 with the most complete reading of the ladder. From one Docker command it covers accounts, a server-enforced deadline and duplicate detection at Tier 1, then judge assignment, weighted rubrics, isolated scoring and normalisation at Tier 2, sealed community voting at Tier 3, and an API, signed webhooks and verifiable judge records at Tier 4. One of its design choices drew particular attention: duplicate submissions are kept out of the ranking but still used to calibrate the judges who scored them, so a nuisance becomes a measurement.
Built for a stranger to run
The last family optimised for the organiser who has never seen the code. Shipshape, from Cache Me, runs on one laptop with the network off and needs no cloud account, API key or sign-up. Participants form teams by invite link and edit their entries until a deadline the server enforces by its own clock, the community vote is quadratic with a cap per project so a group of friends cannot swing a result, and nothing is visible until an organiser publishes. Podium, from Team Carbon C, followed the same discipline in a single container, with weighted rubrics, judging that cannot leak, voting built to resist stuffing, and a straightforward way for an organiser to take the event's data back out.
The Write-ups
Teams could also submit an article about their build, and four of them shared the write-up award equally, ranked blind to the author. All four have the same quality: each author found the place where their own system was weaker than it looked and said so. Manjeet Singh, writing about Shipshape, traced a ranking flip to a change in how Python sums floating-point numbers between versions. GSK VERIFIED tested RAPTOR DESK's own 90% intervals and reported that they covered the truth far less often than claimed. Keshav Varshney showed that a headline metric in the podium build was true by construction and that the shared fixture data carried no recoverable winner at all. Daniel Amah's EvenHand article published how small the model's improvement over a plain mean actually was. In a hackathon about defensible judging, the best writing was about the limits of what each team could defend.
How the Projects Were Judged
Every project was scored from 0 to 5 on four weighted criteria: Tier Completion and Correctness (40%), Judging Integrity (25%), Adoptability and Operability (20%), and Code Quality and Innovation (15%). The panel combined 37 human judges and five AI judges, each counting as one equal vote. Judges who scored high or low across the board were adjusted for, projects with fewer reviews were pulled slightly toward the field average, and each result was published with a rank range showing how far it could move. The six projects submitted late were scored by the AI judges alone. As a final step, one model compared the claims made by the nine highest-ranked projects against their code; it gave no scores, and no score or rank changed.
The panel was headed by four senior engineers and product leaders. Narasimha Reddy Annapareddy, Software Engineering Manager leading NetSuite Business Systems at ChargePoint, judged through the lens of finance and supply-chain integration work, asking whether a platform would behave when connected to the rest of an organiser's tools, and scored an entry that had not built a hackathon platform at all exactly as it deserved. Rushikesh Hayatnagarkar, Senior Product Manager and AI technical builder, returned to a Hackathon Raptors panel after Zero Dependency and weighed architectural depth against the plain question of whether an organiser could adopt the product next month. Hari Krishna Bethanaboina, Lead Solutions Architect at Martin Marietta Materials and an IEEE Senior Member, probed the flows live and checked documentation against code, catching a normalisation method that was described in one project's docs but did not exist in its implementation. Rupesh Kumar Prasad, Principal Architect at Analog Devices and an IEEE Senior Member with 18 years of designing secure enterprise platforms on Azure and AWS, brought identity, security and API-design expertise to a brief in which isolation and access control carried a quarter of the score.
Engineering Challenges
Uncertainty is a product feature
The tension running through the strongest entries was that a better ranking model does not by itself give a more trustworthy ranking. Several teams discovered that their corrected scores were reproducible without being certain, and that a confident single number can be worse than an honest range. RAPTOR DESK's kingmaker check, Quorum's refusal to print an unsupported podium and EvenHand's tie groups are three answers to the same discovery: the platform has to tell the organiser when the data has run out.
Small panels break clean statistics
Many projects in a real event get only two or three reviews, and the textbook tools behave strangely at that size. Teams found that z-scores with two scores collapse to fixed values, that a judge with no spread breaks naive normalisation, and that resampling a project's few reviews understates how much a fourth judge might disagree. The entries that scored best on Judging Integrity met these cases directly, with shrinkage toward the field, explicit handling of flat judges and wider ranges where evidence was thin.
Offline as a design constraint
Running with the network off sounded like a deployment detail and became a design constraint. It ruled out hosted authentication, external email for team invites, and cloud services for anything from storage to signing. The platforms that handled it well, such as Shipshape and Dogfood Portal, treated the laptop as the whole world, with signing keys, verification and exports that work without ever calling home.
Looking Forward
The organisers set this brief because they intend to run the winner. That changes what "good" means. A platform that wins a hackathon has to survive real organisers, real judges who disagree, and real teams who want to know why they came fourth. The top of the DOGFOOD leaderboard suggests the field is ready for that: the leading entries did not just compute a ranking, they explained how sure they were and where a person still had to decide.
The more lasting lesson may be about judging itself. Every team that modelled judge leniency was, in effect, auditing the process that would score them, and several of them published what they found even when it cut against their own result. That is a reasonable standard for any panel, human or automated, and DOGFOOD gave 194 teams a reason to build tools that hold judges to it.
.jpg)