19 Hours, 16 Teams, and One Agent Named Aviad

Author: Asaf Goldstoff

At 4:00 AM, after 19 consecutive hours, the last group was still in the office. We started the previous morning, and as the light flooded the building's windows again, 16 teams had already finalized projects. More than 60 employees participated in the event, and the teams included not only engineers, but also HR and finance staff who came out with working end-to-end solutions.

This is not the first hackathon I have seen, but this time there was something different about it. I attribute the difference—and, in fact, the tremendous improvement in impact—to the preliminary planning. It is easy to organize a one-day hackathon so that it looks impressive, but it is much harder to organize it so that its products survive the morning after.

So how do you turn a hackathon from a day of activity into a place where real solutions are born that will continue to serve us long after?

1. Preparing the teams (long before sitting down to write code)

No one came to the hackathon without preparation. In the weeks before the event, we held basic workshops on AI tools and learned from real Use cases, so that even those who had never written a prompt knew where to start. At the same time, we asked each team to arrive with a list of manual and tedious processes from their daily routine—not an abstract idea, but a concrete problem they want to solve.

In our case, most of the participants were not developers, so we focused on training on Claude and Vibe Coding tools. Throughout the workshops, we presented Best Practices for working with the tools, and we built products, skills, and initial agents together—so that every participant would experience success even before the hackathon.

Two weeks before the event, each team submitted its idea for approval. The goal was to ensure it had feasibility and real value, so we wouldn't spend 24 hours building something that wouldn't be used in practice. Even before the day of the event, every team already knew exactly what it came to solve.

These are the three main parameters that stood behind every green light we gave to an idea:

Real feasibility for taking the solution to production.
Proven potential for saving at least one workday per month—in time or costs.
Correct implementation method of the solution (to ensure teams don't create good solutions in ways that are not efficient).

2. Building an infrastructure that centralizes the projects and preserves them

On the day after hackathons, most projects tend to remain in a folder on someone's computer and be neglected there. To prevent such a situation, our DevOps team prepared a dedicated architecture in GitLab in advance. New projects received a shared and pre-defined work environment, and projects based on existing systems were tagged under a uniform naming convention. All products were concentrated in one place, instead of being scattered among personal computers.

We prepared templates in advance that allowed even non-technical teams to set up their solutions in a simple and uniform way. In addition, a technical person was attached to each team as a mentor or as an active team member throughout the hackathon. From our perspective, this was an opportunity to expose non-technical people to challenges and ways of thinking about technical problems—and to get a broader view of the big picture.

In addition, we set a simple rule in advance: for a project to be considered successful, it must be mature enough to go to production. This changed the nature of the work during the day itself. Teams did not build a demo that works only on stage, but built something that needs to continue functioning in the week after. What makes a project mature enough? As part of the idea approval process, we accompanied and guided the teams to bring their solutions to full production within 24 hours. This includes:

Readiness of the solution for use by the rest of the users in the organization.
Information security and permission management.
Possibility to expand the solution later in a scalable way.

3. The question that must be asked already at registration

Already at the registration stage for the event, we asked each team to explain not only how it will build the tool, but how it plans to get its department to adopt it in daily life. Such a question at such an early stage forces people to think about actual use long before they write a single line of code.

Before the hackathon, as part of the idea approval process, we performed initial planning for the implementation of the solution in the department. An implementation process can include workshops, organized handover, and the definition of a departmental manager who will lead the adoption of the solution. A good example of this is the Onboarding team, which built an application that allows new customers to enter their data independently and see updates in the system in real time. This saves the team a lot of manual work, so the adoption of the solution was almost immediate. For the implementation, we opened a dedicated task board and divided the work among the members of the Onboarding team. Thus, every team member is a partner in the implementation of the solution and its adoption, and not just a user of it.

4. The twist: a non-human judge

Alongside the founders and managers who sat on the judging panel, we added Aviad—an agent we created especially for the event. After all the presentations of the hackathon participants, he went over their full transcript and ranked the teams according to the same criteria used by the human judges. His attention to detail surprised us, because he identified angles of analysis that did not always arise in the discussion of the judges themselves.

In the first stage, we introduced Aviad to all the ideas submitted, including what problem each idea was intended to solve, what solution the tool offers, and how it can be implemented. After that, we provided him with the five main parameters by which the judges planned to rank the teams. In the next stage, we ran a variety of tests on him, presented him with a detailed explanation of each solution, including a demonstration from the demo, and checked if he manages to evaluate it consistently. Along the way, we also fixed a variety of edge cases.

At the end of the presentations, we collected the transcript of each demo and passed it to Aviad, while ensuring he knew which team and which demo he was referring to. Aviad was asked to rank each demo according to the five parameters, with a score of 1 to 5 for each parameter, exactly as the judges were asked to do.

We decided to rank the groups by calculating the average of the scores of all the judges, including Aviad, so Aviad's score was given a weight identical to that of each of the other judges. Before announcing the winners, we checked that the scores given by Aviad were in reasonable correlation with the scores of the other judges, and indeed we saw that he evaluates the demos similarly to most of them. After that, we talked to him to understand the considerations behind his evaluations, and it seemed that he understood well the complexity and the value that the groups' solutions generate.

In one of the presentations, Aviad gave a significantly higher score than most of the judges, mainly in the parameter that examined the scope of users who could derive value from the solution. When we asked him to explain the decision, he showed us how the solution could serve two additional departments, and not just the department that the team presented, and even described several relevant use cases for them. This was interesting—both practically and because none of the human judges caught this possibility.

5. The day after

One of the goals of the process we built was to prevent the fading of enthusiasm around the projects. And it worked: weeks after the hackathon, we continue to accompany the group leaders in bi-weekly meetings, to ensure that the tools they built will not remain an experiment—but will reach full use in the field. Most organizations skip this stage, but in my opinion, it is what determines whether the hackathon was a real investment or a fun day with pizzas.

The goal of the bi-weekly meetings is to go over statuses, document open tasks that need to be solved to implement the ideas, and ensure that everything is progressing as planned. We divided the projects among several responsible persons who are in charge of pushing the solutions they were assigned to. In meetings with them, technical gaps sometimes arise that require support from R&D and DevOps, and for this, we have the CTO who accompanies us and takes care of providing the relevant resources. Operational problems in connecting to other products in the organization require the support of the RevOps department, and for gaps in the correct use of AI capabilities, the groups are assisted by my team, AI Operations.

Today, a few weeks after the hackathon, two solutions have already been fully implemented in their departments. We will be able to measure the value they generate in about three months.

Although relatively little time has passed, the projects that the teams managed to build in one night made me rethink the way we plan ahead and divide responsibility around AI in the organization. Technological cycles are so short today, and the ability to create solutions is distributed among all departments—not just among engineers. This requires different thinking about planning ahead.

If you are debating how to start, want to hold a similar event at your place, or are just curious about Aviad, you are welcome to write to me in the comments and I will be happy to share our experiences, mistakes, and successes.

The author is the Director of AI Operations at Agora.