Good piece, and it lands the simpler problem plainly: NLP isn't new, and what most organisations learned early on was that you need someone interpreting what a person is actually asking for before you can address it. That interpretation layer hasn't gone away just because the bot got more capable — it still needs building in, and I've sat on calls where the technology is the thing making me angrier, not the problem I called about.
There's a sharper issue underneath that, though. Someone with a cognitive impairment, or in the middle of a mental health crisis, needs help at the point they're asking for it — not a bot that can't recognise the state they're in and routes them into another loop. That's not a general design flaw; it's a question of who an organisation is deploying a given technology to, and who it quietly can't support. One size doesn't fit anyone in a crisis.
That's the gap under your Test 1. The outcome metric gets chosen by the organisation, but the population who can't be served by a given deployment is rarely part of that choice — the vulnerable user the bot fails is invisible to a metric built around deflection or engagement, by design, not by accident.
I had one of the best experiences of this kind of technology years ago with an early Apple assistant — say one sentence, it understood you, it did the thing. Simple. Now the same category of tool is more complicated and works about half the time, and every call ends the same way: just put me through to a person.
Very valid remark, and it’s more than a person-in-crisis use case.
I speak French rather fluently but with a distinct accent.
You can imagine how many times a voice bot would simply not understand me (when a normal person actually would).
Sometimes I “make a scene” out of it, just for laughs. Playing anger voice , playing other accents.
The voice bot fails each time. By the time I get through to a human, I already got angry, released my steam, had a laugh. And now ready to speak to a human in a good mood.
You’re pointing at a new “disease”. Everyone wants to ship fast because AI is magic, right?
Nothing changed, with or without AI. You have to write acceptance tests, run automation, functional tests, etc.
My experience building my business on AI is that you need to go slow (1) and you should only automate stable and well-documented processes. Normally that automation should be there in Enterprise already and you’re just adding the AI layer.
Thank you Fabrice, you’ve define the right and only surface where AI should be applied in enterprise deployments. I would couple that with personal / nominated accountability list linking software, process and people to each outcome and result.
I also had some negative experience with AI voice agents in customer support. I would add: sometimes the data is not available to the agent. In my case the AI voice agent could not answer because the necessary information was not in the system. In your example, the already available data was not handed over to the AI.
And I would add: in your case there was also a UX problem. To make really a good solution, you also need skills in UI/UX and customer centricity as a philosophy.
The needed combination: AI, UI/UX, data, end-2-end outcome metrics, security.
You know how difficult it is for many companies to get so many things in parallel right and with high quality. This is why the team and leadership structure is crucial!
I bet that in these examples there have not been real cross-functional teams!
Yeah, this is a common pattern indeed. Let’s work on making this area a better place :) because we all will be finally happy customers (or unhappy victims) of it.
Good piece, and it lands the simpler problem plainly: NLP isn't new, and what most organisations learned early on was that you need someone interpreting what a person is actually asking for before you can address it. That interpretation layer hasn't gone away just because the bot got more capable — it still needs building in, and I've sat on calls where the technology is the thing making me angrier, not the problem I called about.
There's a sharper issue underneath that, though. Someone with a cognitive impairment, or in the middle of a mental health crisis, needs help at the point they're asking for it — not a bot that can't recognise the state they're in and routes them into another loop. That's not a general design flaw; it's a question of who an organisation is deploying a given technology to, and who it quietly can't support. One size doesn't fit anyone in a crisis.
That's the gap under your Test 1. The outcome metric gets chosen by the organisation, but the population who can't be served by a given deployment is rarely part of that choice — the vulnerable user the bot fails is invisible to a metric built around deflection or engagement, by design, not by accident.
I had one of the best experiences of this kind of technology years ago with an early Apple assistant — say one sentence, it understood you, it did the thing. Simple. Now the same category of tool is more complicated and works about half the time, and every call ends the same way: just put me through to a person.
Very valid remark, and it’s more than a person-in-crisis use case.
I speak French rather fluently but with a distinct accent.
You can imagine how many times a voice bot would simply not understand me (when a normal person actually would).
Sometimes I “make a scene” out of it, just for laughs. Playing anger voice , playing other accents.
The voice bot fails each time. By the time I get through to a human, I already got angry, released my steam, had a laugh. And now ready to speak to a human in a good mood.
Tout oui - j’ai bien rigolé 😂
You’re pointing at a new “disease”. Everyone wants to ship fast because AI is magic, right?
Nothing changed, with or without AI. You have to write acceptance tests, run automation, functional tests, etc.
My experience building my business on AI is that you need to go slow (1) and you should only automate stable and well-documented processes. Normally that automation should be there in Enterprise already and you’re just adding the AI layer.
Thank you Fabrice, you’ve define the right and only surface where AI should be applied in enterprise deployments. I would couple that with personal / nominated accountability list linking software, process and people to each outcome and result.
Probably as long as it’s not about finding a scapegoat. The quality of software and projects shipped tell a lot about the company culture IMHO.
I also had some negative experience with AI voice agents in customer support. I would add: sometimes the data is not available to the agent. In my case the AI voice agent could not answer because the necessary information was not in the system. In your example, the already available data was not handed over to the AI.
And I would add: in your case there was also a UX problem. To make really a good solution, you also need skills in UI/UX and customer centricity as a philosophy.
The needed combination: AI, UI/UX, data, end-2-end outcome metrics, security.
You know how difficult it is for many companies to get so many things in parallel right and with high quality. This is why the team and leadership structure is crucial!
I bet that in these examples there have not been real cross-functional teams!
Yeah, this is a common pattern indeed. Let’s work on making this area a better place :) because we all will be finally happy customers (or unhappy victims) of it.