"Set and forget": Why do you keep paying for AI tools that don't really help you?

The smoke detector on your office ceiling has never beeped. This is not proof that it works. This is exactly the same observation you would get if its battery had died a year ago. The difference between the two is not in the detector, it is in the question of whether and when someone checked it recently.

This is what measuring AI usage looks like today in most of the organizations I enter. There are tools, there are subscriptions, there are employees who use them, but there is no one who can say what has changed. When I ask managers how they know the implementation worked, the most common answer is that no one is complaining.

Here are five questions every manager should ask themselves to stop guessing:

1. What does fake success look like?

It looks calm. No malfunctions, no resistance, nothing to report upwards. The problem is that this is exactly what a tool that was quietly discontinued two months ago looks like, as well as a process that employees bypass because it is easier for them without it. Silence is not data, silence is the absence of data, and these are two completely different things. This is especially prominent in professions where the output is a document — accounting firms, lawyers, or finance departments. There it is hardest to see failure, because a document continues to be produced in any case. The question of whether it was produced faster or better is almost never asked.

1. So what is actually being measured?

Not general time savings, because no one is actually counting it. Define one narrow and clear metric: how much time does the task take today, how many times a week is the tool opened, or what happens when you stop using it for two days? If you can stop using the tool for two days and no one feels its absence, you have received a full answer about its value.

1. What is the employee supposed to do with it?

Not log hours and not fill out forms. The employee is the only one who knows if the tool saves them time or adds a step. The only question worth asking them is what they would ask to have back if the tool were taken away from them tomorrow. An immediate answer means the tool has entered the workflow. Hesitation means it is there, nothing more.

1. Where do we start tomorrow morning?

Start with a list: open a spreadsheet and add every AI tool the organization pays for, alongside the task it is supposed to serve and the person responsible for it. The cells that remained empty are the answer. In every business I have worked with in the last year, I found several tools that no one ever decided to close, they simply stopped being checked and were forgotten.

I say this about myself too. I have automatic checks that are supposed to alert my business when something breaks. When I went over them, I discovered that two of them reported that everything was "fine" for whole weeks on tools that had already been canceled. They didn't lie to me. Simply no one asked them anything, so they continued to return the last answer they knew.

1. When do we check again?

This is the question that is easiest to skip, because after one check it feels like we closed the subject. In practice, a tool that is checked once returns to the state from which we started: it works until it stops, without anyone knowing when. Set a date in the calendar, a quarter ahead, to go over the same list. A check without a deadline for the next check is not a check but a good feeling.

One check costs you half an hour. Avoiding it costs you a whole year of recurring charges and false confidence that everything is fine. The next time you look at an expense report containing your subscriptions, ask yourself: what of all these is really working, and what is simply silent?

Noam Cohen is the founder of CritiqeIL, an AI consulting and implementation company for organizations in Israel.