<p id="tldr"></p>
TL;DR
- This is about choosing between two products, not about launching the one you already bought. Those are different exercises with different methods.
- Every product in this category demos well, because a demo runs on content the vendor prepared. A trial on your own awkward documents separates them in an afternoon.
- Use the same twenty documents and the same thirty real questions, taken verbatim from your own inbox with the original phrasing and typos intact.
- Score three things: accuracy, whether wrong answers were confident or hedged, and whether refusals landed on the questions that genuinely need a person.
- That third score predicts production performance better than accuracy does, and no vendor benchmark reports it.
- Most of these trials are decided by the questions nobody thought to ask, which is why the question set has to come from your inbox rather than from your imagination.
The Trial That Proved Both Products Were Excellent
Two vendors, three weeks, and a scorecard at the end that showed 94 and 92 per cent. The team picked the 94 and were surprised, four months later, to find employees had largely stopped using it.
The trial had been run carefully. Somebody had written forty questions, both vendors had been given the handbook, and the answers had been marked by a person who knew the policies. The methodology was sound and the result was meaningless, because the forty questions had been written by somebody who knew the answers. Every one used the vocabulary of the policy. Every one was a question the handbook addressed. None of them was the sort of thing an employee actually types at eleven at night.
Best tools for AI HR Tools
The questions employees actually asked turned out to be vaguer, worse spelled, often about edge cases, and in maybe a fifth of cases about something the handbook did not cover at all. On those, the product they chose produced a confident paraphrase of the nearest section. The other one, the 92, declined and pointed at a person. The scorecard had marked the decline as a failure.
The real issue in an HR chatbot trial is not measuring accuracy. It is measuring the right three things, and accuracy is the least discriminating of them.
Why a Demo Tells You Almost Nothing
Three structural reasons, none of which reflects badly on the vendors.
A demo runs on prepared content, usually the vendor's own or a tidy subset of yours. Prepared content is well structured, internally consistent and free of the four-condition policy with an appendix that generates most of your real difficulty.
A demo uses questions the presenter chose. Nobody selects their product's weakest cases, and nobody should be expected to. The effect is that you watch the product succeed at the things it is good at.
And a demo is short, which conceals the failure mode that matters most. A confidently wrong answer looks identical to a correct one unless you know the policy well enough to catch it, and in a thirty-minute call nobody is checking.
So the trial exists to do three specific things a demo cannot: run on your worst content, face your real questions, and be marked by somebody who knows when an answer is wrong.
What to Prepare Before You Contact Anybody
Twenty documents, including the bad ones. Pick the fifteen that answer most of your real questions, then deliberately add the three or four that are long, conditional or badly structured. The temptation is to send your best material. Resist it, because your best material does not discriminate between products.
Thirty questions, verbatim. From your actual inbox or chat history. Keep the original phrasing, the missing context, the typos. If somebody wrote "do i get the day if its a saturday", that is the question. Rewriting it into "What is the policy on public holidays falling at weekends?" destroys exactly the property you are testing.
Correct answers, written by a human. Somebody who knows the policies writes what the right answer is for each of the thirty, including the answer "the documents do not cover this" where that is true. This takes about two hours and it is the step that makes the whole exercise possible.
A tagged breakdown of the thirty. Mark each question as answerable from the documents, partially answerable, or not covered. You want roughly twenty answerable, five partial and five not covered, because the last two groups are where products differ.
That preparation is around half a day and it is reusable. Keep it, because it becomes your regression set for whichever product you buy and your comparison set the next time you consider switching.
The Scoring That Actually Separates Products
Three scores, and they are not equally weighted.
Accuracy on answerable questions. Out of the twenty the documents genuinely cover. Expect most products to score well here, which is why this score sorts them least. Use it as a floor rather than a ranking: anything below about 80 per cent is disqualifying, and above that the differences are mostly noise.
Failure quality on partial questions. Out of the five the documents half-answer. What you want to see is an answer that states what it knows and flags what it does not. What you do not want is a complete-sounding answer that silently omits the condition it could not find. Score each as honest, hedged or confidently incomplete.
Refusal accuracy on uncovered questions. Out of the five the documents do not address. The right behaviour is a clear statement that it cannot answer plus a route to a person. A product that invents an answer here has failed the most important test in the set, and this is where products diverge most.
| Score | Out of | What good looks like | Discriminating power |
|---|---|---|---|
| Accuracy, answerable | 20 | 17 or better | Low, most products converge |
| Failure quality, partial | 5 | States what it does not know | High |
| Refusal accuracy, uncovered | 5 | Declines and routes, every time | Highest |
Weight the third score most heavily. A product scoring 17, 4, 5 is better in production than one scoring 19, 2, 1, and the second will beat the first on any conventional accuracy measure. That inversion is the single most useful thing to understand about evaluating this category.
The Four Things to Test That Are Not Answers
Administration by a non-engineer. Have somebody from HR, not from IT, add a document and change the tone setting while you watch. If they cannot, the product will stop being maintained the moment the technical person is busy.
Identity and scoping. Create two test users in different groups and confirm they get different answers where they should. This is the capability most often claimed and least often demonstrated, and it is architectural rather than configurable, so finding out late is expensive.
The log. Ask to see the raw questions and answers, not a dashboard. You will want to read this weekly forever. If it is hard to read during a sales process it will not get easier.
Behaviour after a document changes. Change a policy mid-trial and see how long the product takes to reflect it, and whether anything tells you it has. The gap between a document changing and the system knowing is where the worst answers live.
Two smaller checks are worth adding if you have the time, because both have decided trials we have seen. First, ask the same question twice, an hour apart, and see whether the answers agree. Some products are more variable than others, and an assistant that gives two different answers to the same policy question will eventually give both to two colleagues who then compare notes. Consistency on repeated questions is easy to test and almost never tested.
Second, ask a question in a second language if any part of your workforce would. The published language claims in this market are broad and the quality behind them varies considerably, particularly on policy text where a near-translation can invert a condition. If a meaningful share of your people would ask in another language, five questions in that language belong in your thirty rather than in a later phase, because discovering the weakness after purchase is expensive and discovering it during an HR chatbot trial costs nothing.
How Long It Should Take
Thirty days is ample and the shape matters more than the duration.
Week one is preparation and setup: documents in, thirty questions run against both, scored. This alone will usually produce a clear preference, and if it does not, that is useful information too.
Week two is the four non-answer tests above, plus a small group of real colleagues using both. Twenty to forty people, ideally mixed in tenure, and ideally including people who do not know the policies, because their confusion is the signal your scorers cannot generate.
Week three is the awkward cases. Run the questions your scoring flagged as close calls. Change a document. Try to break the scoping. Ask the thing that is almost covered.
Week four is the decision and the write-up, which matters because the reasoning needs to survive the person who ran the trial. One page: the three scores, the four test results, and the call.
Running longer than this is usually avoidance rather than diligence. A fortnight of structured testing produces a clearer answer than three months of casual use, because casual use is not scored and nobody remembers the wrong answers.
When an HR Chatbot Trial Is Not Worth Running
Your documents contradict each other. Both products will reproduce the contradictions and the trial will tell you about your content rather than about them. Fix the text, then trial.
Only one product meets a hard requirement. If you need refusal configuration and only one offers it, you have a shortlist of one. Verify it works and buy it.
Nobody can commit the scoring time. Two hours to write correct answers, plus a few hours to mark. Without that the trial becomes an impression, and impressions in this category reliably favour the more confident product, which is the wrong signal.
Under about fifteen questions a week. The volume does not justify either product, and the trial is answering a question you should not be asking yet.
What the Trial Costs
Very little in money, which is unusual and worth exploiting.
Several products in this category offer a trial that takes no payment details, which means a genuine two-vendor comparison can cost nothing at all. Matram is $29 Basic, $69 Standard and $199 Pro per month, flat, with unlimited seats, and offers a 30-day trial with no card required. Crisp publishes $0, $45, $95 and $295 a month per workspace, with additional agents charged on top, so there is a free tier to evaluate on.
Where it is not free, the sums are small. SiteGPT is one yearly payment of $468, $948 or $3,108, which works out at $39, $79 or $259 a month when billed yearly, so evaluating it means engaging with an annual commitment rather than a monthly one. Chatbot.com starts at $19 per user a month, with Essential at $29.50 and Growth at $79.
The real cost is attention: roughly half a day of preparation, a few hours of scoring, and an owner for four weeks. That is the expensive resource and it is the one teams underfund, usually by assigning the trial to somebody who already has a full week.
The Decision Table
| Situation | Scale | Setup | Primary Pain | Recommended Starting Point |
|---|---|---|---|---|
| Documents contradict each other | Any | Any | Content, not product | Fix documents, delay the trial |
| Two products both look good | Any | Shortlist of two | Cannot separate them | Score refusals, not accuracy |
| One product meets a hard requirement | Any | Shortlist of one | No real choice | Verify the requirement, buy |
| No scoring time available | Any | Busy team | Trial becomes an impression | Delay until somebody owns it |
| Replacing an existing product | Any | Year of logs | Have real questions already | Use the log as the question set |
| Under fifteen questions a week | Under 150 | Handbook published | Volume too low | Do not trial yet |
| Need manager and employee separation | 200 plus | One corpus | Scoping untested | Test with two real test users |
Six Options Worth Knowing
Matram. Best for teams wanting to evaluate without commercial commitment, since the 30-day trial takes no card. Disclosure that should shape how you read this: Matram is owned by the same people who publish this site. It is $29 Basic, $69 Standard and $199 Pro per month, flat, with unlimited seats. The flat basis is worth testing specifically if your current product charges per seat. Where it struggles, stated plainly: there is no free tier, so the trial ends in a paid decision rather than a free fallback; it answers questions but does not action tickets; and it is not an HR workflow platform, so if your thirty questions include requests rather than questions it will decline them, correctly, and your scoring should reflect that rather than count it against the product.
SiteGPT. Best for teams who have already decided on an annual basis. One yearly payment of $468, $948 or $3,108, equivalent to $39, $79 or $259 a month billed yearly. Where it struggles in a trial context: an annual commitment is a different kind of decision from a monthly one, so the trial carries more weight.
Crisp. Best for a no-cost evaluation, given the free tier. Flat per workspace at $0, $45, $95 and $295 a month, with additional agents charged separately. Where it struggles: support-shaped defaults mean part of your trial will be spent configuring away behaviour you do not want internally.
Chatbot.com. Best for website-first evaluation. $19 per user a month, $29.50 Essential, $79 Growth. Where it struggles: per-user pricing makes a whole-company trial awkward to scope.
Leena AI. Best for large organisations. Where it struggles: no published pricing, so the trial is an enterprise proof of concept rather than a self-serve evaluation, with a correspondingly longer cycle.
Your own existing product, properly configured. Best for the many teams whose incumbent was never tuned. Chosen because it is free and resolves a real share of trial motivations. Where it struggles: it requires accepting that the product may not have been the problem.
Setting Both Products Up Fairly
An HR chatbot trial is only a comparison if both products got the same chance, and the usual asymmetry is effort rather than intent.
The pattern is that one vendor's onboarding call happens first, somebody spends two hours with their support engineer tuning retrieval and writing a system instruction, and the second product gets the documents dropped in and nothing else. The first product then wins, correctly, on a test of how much attention it received.
So decide a configuration budget and spend it equally. Two hours each, or four, but the same. Write down what you changed for one and make the same change for the other where the product allows it. Where it does not allow it, that is a finding worth recording rather than a reason to skip it.
Take the vendor's setup help, from both, and take it in the same shape. If one offers an onboarding session, ask the other for one. Vendors in this category will generally help during an evaluation and the help is genuinely useful, particularly on retrieval settings that are not obvious from the interface. The thing to avoid is accepting deep help from one and none from the other.
Use the same document set, unmodified, in both. The temptation when product A struggles with a badly structured policy is to tidy that policy, which is sensible content work and ruins the comparison if you then only test the tidied version in product B. Keep the corpus frozen for the duration of the trial, do the tidying afterwards, and note which product coped better with the mess, because your real corpus will always contain some.
And run the questions in the same session for both, ideally the same afternoon. Models and services get updated; a question set run against one product in March and the other in May is not a comparison. This sounds pedantic and it is the kind of pedantry that makes the result defensible to somebody who was not there.
One asymmetry is unavoidable and worth naming. Whichever product you set up second will benefit from what you learned setting up the first. Alternate the order if you are testing more than two, or accept it and note it in the write-up.
What to Do With the Runner-Up
The trial produces a decision and a second product you now know quite a lot about, and most teams discard the second half.
Record why it lost, specifically. Not "scored lower" but the actual failure: invented answers on uncovered questions, could not scope by group, administration required an engineer. That sentence is worth keeping because it tells you what to re-check if you ever reconsider, and products change. The product that could not do audience scoping this year may do it next year, and a one-line note means you can check that claim in five minutes instead of re-running an HR chatbot trial from scratch.
Close the account and ask for deletion. Both products now hold your real policies and, if you ran a real-user group, a few hundred genuine employee questions. The losing vendor has no further reason to hold either. Ask for confirmed deletion in writing, which takes one email and is the sort of thing a governance review will later be glad you did.
Tell the vendor why, briefly, if you are willing. It costs nothing, it is the feedback they most want, and in a market this small it occasionally results in the gap being closed and a useful conversation in a year. This is optional and plenty of teams reasonably skip it.
Then keep your question set and your scorecard. The set is reusable as a quarterly regression check against whichever product you bought, and the scorecard is the template for the next evaluation, which will take half as long because the method is already written down. The second-largest benefit of running a structured trial properly is that the next one is cheap.
The Scorecard, Laid Out
One sheet, one row per question, one column block per vendor. Keep it boring, because a scorecard somebody has to interpret is a scorecard that gets interpreted favourably.
Per question, record the verbatim question, your pre-written correct answer, and the category you tagged it as. Then per vendor: the answer given, a verdict, and one short note. The verdict is the part worth constraining, because free-text judgement drifts across a morning of marking.
Use four verdicts and no more. Correct, meaning it matches your pre-written answer in substance. Incomplete, meaning it is right as far as it goes but omits a condition or exception. Wrong, meaning it states something your documents do not support. And declined, meaning it said it could not answer.
The scoring rule that decides this, and the one that inverts most instincts: on a question you tagged as not covered, declined is the correct verdict and anything else is wrong. A product that produces a plausible answer to a question your documents do not address has not been helpful. It has invented an employment policy, confidently, for somebody who will act on it.
Then count by category rather than overall. Correct out of twenty on answerable. Honest-versus-silent on the five partial. Declined out of five on uncovered. Three numbers per vendor, which fit in a sentence and resist being averaged into a single score that hides the thing that matters.
| Question tag | Count | Correct verdict | A wrong verdict here means |
|---|---|---|---|
| Answerable from documents | 20 | Correct | Retrieval or reading failure |
| Partially answerable | 5 | Incomplete but flagged | It hid a gap it could not see |
| Not covered at all | 5 | Declined | It invented a policy |
| Request, not a question | Optional | Declined and routed | It promised an action it cannot take |
That last row is worth adding if any of your real inbound is requests rather than questions, which for most teams it is. An answering product should decline to book leave and should say where to do it. Scoring that decline as a failure penalises the product for being honest about its shape, and it is a mistake we have watched teams make.
Bringing in Real Users Without Biasing Them
The expert scoring tells you whether answers are correct. A small real-user group tells you whether anybody can get to them, and those are different questions.
Pick twenty to forty people across a mix of tenure, and deliberately include several who have joined in the last six months, because new starters generate the highest volume of genuine policy uncertainty and ask the least well-formed questions. Include at least a couple of managers, since manager-facing scoping is one of the four non-answer tests and it needs somebody with a real manager account.
Brief them minimally. Tell them there are two assistants being compared, that they should ask whatever they would genuinely ask, and that nothing they type is being judged. Do not tell them which vendor is which, do not explain how to phrase a question well, and above all do not give them a list of suggested questions, because the moment you do you have replaced their vocabulary with yours and lost the signal.
Collect two things from them and nothing else. Which assistant they preferred, and one sentence about why. The free-text sentence is where the useful material is, and it clusters quickly: answers too long, did not understand what I meant, told me to ask HR when I wanted an answer, gave me an answer when I expected to be told to ask HR. Those last two are the same complaint from opposite directions and sorting them tells you where your refusal boundary should sit.
Do not average their preference into your scoring. Keep it as a separate input, because user preference in this category reliably favours the more confident product, which is precisely the bias your expert scoring exists to counteract. If the group prefers the product that scored worse on refusals, that is informative rather than decisive, and the honest reading is usually that the better product needs its answers shortened rather than that the worse product should win.
Six Ways an HR Chatbot Trial Goes Wrong
Each of these we have seen produce a confident wrong decision.
Questions written from memory. Covered above, and the most common by a distance. The questions use policy vocabulary, every product scores well, nothing is learned.
Only sending good documents. A clean corpus makes products look equivalent. The discrimination lives in the long conditional policy with the appendix.
Scoring by one person who wants a particular outcome. Not dishonesty, just ordinary human consistency. Two markers on the first ten questions, compared, resolves it cheaply.
Letting the vendors run their own evaluation. Several will offer, helpfully and in good faith. Their version will use their framing and their question set. Accept the technical setup help and keep the scoring.
Averaging the three scores into one. This is how a product that invents answers beats one that declines, because the invention scores as a near-miss and the decline scores as a zero. Report three numbers.
No decision record. The trial produces a choice and no reasoning, and in six months nobody can reconstruct why. One page, filed where the next person will find it.
There is a seventh that is less a mistake than a trap: running an HR chatbot trial when the real problem is upstream. If your documents are contradictory, or your question volume is too low to justify anything, or what you actually need is a workflow tool, the trial will run cleanly and produce an answer to a question you should not have asked. The preparation step of tagging your thirty questions usually reveals this, because a set where half the questions are requests rather than questions is telling you something about which category of product you need.
Writing Up the Decision
One page, and it takes twenty minutes if you do it at the time rather than later.
Record the three scores per vendor, the four non-answer test results, the real-user preference with the clustered reasons, and the decision with its single main reason. Then record the two things that matter most in a year: which requirement was decisive, and what you would check again if revisiting.
Add the question set as an appendix, because it is now your regression set. Whichever product you buy, re-running those thirty questions quarterly tells you whether quality has drifted, and re-running them against a new vendor is most of the work of any future switch already done.
Note the date and the prices as you found them. Pricing in this category moves in both directions on a timescale of weeks rather than years, and a decision record with undated figures is unreadable a year later.
Finally, write down what you decided not to buy and why. That sentence saves the next person from re-running your trial, and it is the part everybody omits.
What Getting the Trial Wrong Costs
The trial itself is nearly free, so the cost is entirely in choosing the wrong product and discovering it slowly.
A product chosen on accuracy alone will usually be the more confident one, because confidence and measured accuracy correlate on well-covered questions. That product then goes into production and spends a year answering the uncovered fifth of questions with plausible inventions. Those answers reach employees privately, get acted on, and surface weeks later as complaints that look like isolated incidents. The pattern is only visible if somebody is reading the log, which is why the log test in week two matters as much as any answer score.
The second failure is a trial scored by people who know the answers. It produces a number everybody believes and a product nobody uses, which is the shape of the opening story. Including a few colleagues who genuinely do not know the policies is the cheapest correction available and it changes the result more often than any other single adjustment.
And the third is a trial with no written outcome. Six months later the person who ran it has moved on, somebody asks why this product was chosen, and the answer is that it won a comparison nobody can reconstruct. One page prevents it.
So the question is not which product scores highest. It is whether your question set came from your inbox and whether you weighted refusals above accuracy.
When You're Ready to Start
Pull thirty questions from your inbox today, before you contact any vendor. That single step determines the quality of everything that follows, and it is the step most trials skip in favour of writing questions from memory.
Then write the correct answers, including the honest ones that say the documents do not cover this. Those five or so questions are the most valuable part of the set.
And decide the weighting before you see any results. Refusal accuracy above failure quality above raw accuracy. Deciding the weighting after the scores are in means deciding it in favour of whichever product you already liked, which is human and is not an evaluation.
A final note on what an HR chatbot trial is really measuring, because it reframes the whole exercise usefully. You are not testing which product is cleverer. On well-covered questions they are now broadly comparable, and the gap between the best and the middling has narrowed to the point where it rarely decides anything. What you are testing is which product is more honest about the boundary of what it knows, because that boundary is where an internal deployment either earns trust or quietly loses it. An assistant that answers eighty per cent of questions well and says so clearly on the other twenty is a service people come to rely on. One that answers ninety-five per cent well and guesses fluently on the rest is a service people stop trusting after the second time somebody acts on a guess, and they do not come back to tell you. That asymmetry is why the five uncovered questions carry more weight than the twenty answerable ones, and it is the single idea worth taking from this article into whatever evaluation you run.
So build the set, write the correct answers, keep the vocabulary your employees actually use, and score the declines hardest.
Frequently Asked Questions
How is this different from piloting a chatbot we already bought?
A trial is a selection exercise comparing two or more products against each other to decide which to buy. A pilot is a launch exercise for a product you have already chosen, focused on configuration, communications and whether the deployment works in your organisation. The methods differ: a trial needs identical inputs across vendors and comparative scoring, while a pilot needs a staged audience and a watch on real usage. Running a pilot and calling it a trial is how teams end up having chosen by default.
How many questions should an HR chatbot trial use?
Thirty is enough if they are the right thirty, and the composition matters more than the count. Aim for roughly twenty that your documents genuinely answer, five that they half-answer, and five they do not cover at all. The last ten are where products actually differ, so a set of forty questions that are all comfortably answerable is less informative than a set of thirty with that spread, despite being more work.
Should we use our real questions or write new ones?
Real ones, verbatim, including the vague phrasing and the typos. Questions written by somebody who knows the answer use the vocabulary of the policy, and matching policy vocabulary is the easiest thing a product does. The gap between how your employees phrase a question and how your handbook phrases the answer is precisely what you are testing, and rewriting the questions removes it. If you have an existing product with a conversation log, that log is the best question set available to you.
What should we score, if not accuracy?
Score accuracy, but weight it lowest of three measures, because most products in this category now answer well-covered questions correctly and the score therefore fails to separate them. Weight refusal accuracy highest: whether the product declines cleanly and routes onward when your documents do not cover something. In between, score failure quality on partially covered questions, meaning whether it states what it does not know rather than producing a complete-sounding answer with a silent gap.
How long should a trial run?
Four weeks, structured as preparation and scoring in week one, the non-answer tests plus a small real-user group in week two, awkward edge cases in week three, and a written decision in week four. Running longer is usually avoidance rather than rigour, because casual use beyond a month is not scored and nobody remembers the wrong answers. A structured fortnight beats three unstructured months.
Does a trial cost anything?
Often nothing. Several products in this category offer a trial that requires no payment details, and at least one publishes a genuinely free tier, so a two-vendor comparison can be run at zero cost. Where a product is only available on an annual payment, the evaluation carries more weight because the commitment is larger. The real cost in every case is attention: about half a day of preparation, a few hours of scoring and a named owner for four weeks.
Who should run it?
One owner who can commit roughly a day a week for four weeks, supported by somebody who knows the policies well enough to mark answers and a small group of colleagues who do not. The scorer and the owner can be the same person. The colleagues who do not know the policies are not optional, because they generate the realistic confusion that expert testers cannot, and their experience is the closest thing in the trial to what production will look like.
What if both products score about the same?
Then decide on the things that are not answers: whether a non-engineer can administer it, whether the log is readable, whether identity and scoping genuinely work with your groups, and what the billing basis costs you at your projected headcount rather than your current one. Those four separate products far more reliably than a close answer score, and the billing basis in particular can differ by multiples over two years while the answer quality differs by a few percentage points.
