When all students received 100 on exams, the university realized there was a problem
A new development at Bar-Ilan University places an AI examiner in front of computer science students: instead of fighting the use of artificial intelligence, the system asks them questions about the code they submitted to ensure they truly understand the material. Students initially resisted, but report that the method improved their knowledge – and now there are plans to expand the project to additional mathematics courses.
In the last two years, faculty members in the Department of Computer Science and Artificial Intelligence at Bar-Ilan University noticed a phenomenon recurring in almost every academic institution: almost all students passed the automatic checks of programming assignments perfectly, receiving high grades. However, this did not necessarily indicate that they understood what they submitted.
"This is a system we implemented in the operating systems course last semester. Until today, we checked these programming assignments automatically; we would inject tens of thousands of scenarios and check for the expected output. 'Surprisingly', in the last two years, everyone got 100. We realized that we were not checking the student's ability to understand, but how much access they have to artificial intelligence. This caused us to develop the system," Prof. David Sarna, a faculty member in the Department of Computer Science and Artificial Intelligence at Bar-Ilan University, tells mako.
It is important for Sarna to emphasize that he and his team are not trying to prevent students from using AI. On the contrary, they recognize these as essential modern skills, but want to ensure that students use artificial intelligence intelligently and truly understand their submissions.
The new artificial intelligence system they developed, which has already operated in a pilot program, is, as far as they know, the first of its kind in Israel and one of the first in the world. It analyzes the students' answers to provide an in-depth assessment of their understanding, rather than just verifying the final result.
How does it work?
After submitting a programming assignment, the student schedules a short exam and arrives at a designated room. They sit in front of a locked station that prevents switching to other programs or windows, and the system automatically pulls the submitted code.
After the student confirms the code is theirs, the screen splits: the code appears on one side, and a conversation with an AI-based examiner takes place on the other. During the 16-minute exam, the system usually asks three questions, highlighting relevant lines in the code each time.
"The system essentially engages the student in a 'conversation', asking various questions about the assignment to assess their level of control over the submitted code, their understanding of it, and their ability to modify its flow," explains Sarna. "The goal in an era where everyone uses artificial intelligence is to assess how much you really understand what you are doing, rather than just relying on AI."
The questions are not prepared in advance. The teaching assistant defines the topics, expected knowledge level, and focus areas, and the system creates dozens of possible templates. The beauty is that the template only becomes a personal question, dependent on the student's solution, when it meets their specific code.
The grade is based on several AI agents: one examines the code without seeing the exam, another reads the conversation without seeing the code, and a third weighs the findings. A quarter of the grade is based on code quality, and three-quarters on the degree of understanding demonstrated. At the end, the student receives an email with the grade, details of the answers, and topics for review.
Behind the development, alongside Sarna, are Shahar Sagi, the teaching assistant who led the practical development, and Dr. Yossi Ben Zion, an expert in learning technology, who oversaw the design and integration. The system was developed within three weeks using Claude Code.
Student feedback
About 260 students have already been examined using the system. As is common, many resisted the change. Dvir, a student who used the system throughout the semester, says he arrived at the first exam under pressure, but the joint preparation with friends forced them to review the code in depth. "In retrospect, this preparation itself caused us to understand the material much better," he says.
According to him, the exam felt less like a test and more like an explanation to a friend, and the system improved throughout the semester in how it evaluated answers. "Ultimately, it helped me understand the material much better and connect the assignments to the topics learned in the course," he says.
A student who does not succeed can schedule another date, with different questions appearing in each attempt. "If it didn't go well, you can schedule another exam, and as far as I'm concerned, you can do it as many times as you want. Once I can truly say you understand at the level of the grade I gave you, I don't care that everyone gets 100, because I know that real knowledge stands behind the grade," says Sarna.
Towards the next academic year, Bar-Ilan plans to expand the system's use to mathematics courses, including linear algebra, and Sarna's team is already in discussions with various academic bodies.
