Every HR platform on the market now describes itself as AI-powered. Because the label is applied to everything from a genuinely automated payroll check to a help widget that rephrases the employee handbook, it has stopped being useful as a way of telling products apart.
If you are evaluating options, the useful move is to ignore the word and ask what the system actually does when nobody is watching it. The questions below tend to separate the two quite quickly.
Does it act, or only answer?
This is the sharpest dividing line. A lot of what gets sold as AI in HR is a search layer: you ask a question, it finds the relevant policy or record, you still do the work.
That has value, but it is a different product from one that completes a task. Ask the vendor to show you something the system finishes on its own, end to end, without a person clicking through each step. If the demo is always a conversation rather than a completed action, you are looking at a search layer.
What does it read from?
An assistant is only as good as the data underneath it, and this is where many implementations quietly fail. If the AI reads from a copy of your employee data that refreshes overnight, then every answer it gives is a day old, and anything that changed today is invisible to it.
Ask directly whether it reads live records or a synced copy. Then ask what happens when a record changes while it is working. The answer tells you whether you are getting a genuine operational tool or a reporting layer with a friendlier interface.
Does it respect permissions?
This one is easy to overlook in a demo, where everyone is logged in as an administrator. In production it matters enormously.
If a line manager asks the system a question about their team, the answer should be scoped to what that manager is already allowed to see. If the same question from a manager and from the head of HR returns the same answer, the permission model is not being applied, and you have created a data exposure problem rather than solved a workload problem.
Can you see what it did?
Any system that takes action on employee records needs to write down what it did, when, and on whose authority. This is not only a compliance requirement. It is also the only practical way to build confidence internally.
When something looks wrong, the first question anyone asks is what happened. If the audit trail cannot answer that, your team will stop trusting the automation and go back to checking everything manually, which removes the benefit you bought it for.
Where are the limits drawn?
A serious vendor will tell you what their system does not decide. Termination, disciplinary outcomes, final hiring calls and pay decisions should sit with people, with the software preparing the ground rather than reaching the conclusion.
Be wary of a product that presents no boundaries at all. Either the boundaries exist and are not being discussed, or they have not been thought about, and neither is comfortable.
What happens to your data?
Ask whether your employee data is used to train models that other customers benefit from. Ask where it is processed and where it is stored. Get the answers in writing rather than in a conversation, because this is the question most likely to come back to you from legal or from your works council.
A practical way to run the evaluation
Rather than working through a feature matrix, pick three tasks your team genuinely dislikes. Chasing missing onboarding documents is a good candidate. So is checking a payroll run for gaps before submission, or answering the same leave balance question for the fifth time this week.
Ask each vendor to show those three tasks running on data that resembles yours. You will learn more in that half hour than from any amount of comparison material, because the tasks you chose are the ones where the difference between answering and acting actually shows up.