Your Policy Changed. Your Bot Did Not Notice.

TL;DR

• An HR chatbot answers from a corpus, and a policy change takes effect on a date. The corpus behind your assistant updates whenever somebody remembers, and the gap between those two moments is where wrong answers live.
• Three separate lags stack up: nobody told the person who owns the corpus, the document was updated but not re-ingested, and the old version is still in there competing with the new one.
• The third is the worst because it produces inconsistency rather than staleness. The same question returns different answers depending on which passage matched, and no error is logged.
• The commonest question after any change is which rule applies to something already submitted. A document that silently overwrites itself cannot answer it.
• The fix is three habits, not a feature: a change notification that fires where the change happens, a re-ingestion check, and a dated change note left in the document.
• Verify on the rendered answer, not the document. A corpus that contains the right text can still be answering from the wrong passage.

Three Weeks of the Old Notice Period

A company changed its notice period for internal transfers, from four weeks to two, effective the first of the month. The change was approved properly, the policy document was updated the same week, and the HR team announced it in a company-wide message.

For the following three weeks the assistant told everybody who asked that the notice period was four weeks.

The document had been updated. That was not the problem. The updated document lived in a shared drive folder that the assistant ingested on a schedule nobody had checked, and the schedule had been set to monthly during setup because the content was described at the time as rarely changing. So the new text sat in the right place, correct and invisible, for three weeks.

Recommended Tools

Best tools for HR Operations

B BambooHR HR management system R Rippling HR & IT automation W Workday Enterprise HRIS

View all recommended tools

Two people planned around four weeks. One of them delayed a conversation with their manager by a fortnight for no reason, which is a small harm. The other had already agreed a start date with the receiving team on the basis of the answer, and unpicking that took three conversations and left everybody slightly annoyed with HR, which is the kind of cost that never gets attributed to a re-ingestion schedule.

And when it was found, the fix took four minutes. That is the shape of this entire problem: trivial to correct, invisible until somebody is affected, and guaranteed to recur without a habit attached.

The Three Lags, Which Are Different Problems

People talk about the corpus being out of date as one thing. It is three, they have different causes, and only one of them is about software.

Lag one: nobody told the owner. The policy changed and the person responsible for the answers did not know. This is the largest of the three and it is entirely organisational. Policy changes go through approval, communications and sometimes legal, and none of those steps has a line saying tell whoever owns the assistant, because that role did not exist when the process was written.

Lag two: the document changed and the corpus did not. Ingestion is scheduled, or manual, or triggered by something nobody remembers configuring. A document updated in place does not announce itself. This is the lag in the story above and it is the easiest to eliminate, because it is a setting and a check rather than a behaviour change.

Lag three: both versions are now in the corpus. The new document was added and the old one was never removed, so two passages answer the same question differently. This is the worst of the three because it does not look like staleness. It looks like the system being unreliable, and it is the one most likely to be diagnosed as a product fault.

Every row takes minutes to fix and months to notice. That asymmetry is the whole topic.

Why the Duplicate Is the Dangerous One

A stale answer is wrong and consistent. Ask twice, get the same wrong answer, somebody eventually notices and reports it. It is a bug with a clear shape.

A duplicate is wrong intermittently. Two passages exist, both plausible, and which one gets retrieved depends on the exact phrasing of the question. Ask about the notice period and get two weeks. Ask how much notice you need to give and get four. Both answers cite a real document, both documents are real, and the system is behaving exactly as designed.

This produces three bad outcomes at once. Employees conclude the assistant makes things up, which is the worst possible reputation for an answering system and is here undeserved. The person investigating cannot reproduce the fault reliably, so it gets logged as intermittent and deprioritised. And a spot check will usually pass, because whoever checks asks the question in the phrasing they have in their head, gets the right answer, and closes it.

So the specific habit worth building is this: when a document is replaced, confirm the old one is gone, not just that the new one is present. Those are two different checks and only the second one is ever done.

The version of this that catches people out is the archive folder. Somebody moves the superseded policy to an archive directory, which feels like removal and is not, because the archive is inside the ingested path. The corpus now contains a document explicitly labelled as old and will happily answer from it.

The Question That Only a Change Note Answers

After any policy change, the single most common question is not what the new rule is. It is whether the new rule applies to something already in flight.

I submitted my request last week, under the old notice period, does the new one apply to me. I booked that leave in March, before the carry-over rule changed. My probation started under the previous policy. These are reasonable questions, they have answers, and a document that silently replaced its own text cannot produce them, because the information needed is the relationship between two versions and only one version exists.

The fix is one line in the document and it is the single highest-return habit in this article.

State what changed, and from when. A short note at the top of the section: that the notice period changed from four weeks to two with effect from a date, and that requests submitted before that date follow the previous arrangement. Whether that last part is true depends on your own policy and circumstances, which is exactly why somebody has to decide it and write it down rather than leaving the assistant to infer it.

Keep the note for a defined period. Six months is a reasonable default, long enough to cover anything in flight and short enough that documents do not accumulate a changelog nobody reads. The owner removes it on the monthly cycle.

Write it as a fact, not as an announcement. The note is content that has to answer a question months later, not a communication with a tone. Avoid phrasings tied to the moment such as recently or as of now, because they stop meaning anything the week after and the assistant will reproduce them verbatim.

And where the honest answer is that it depends on the individual circumstances, say which document governs and name a route to a person. That is a useful answer. A confident guess about which rule applies to somebody's already-submitted request is the worst available outcome.

There is a second reason the note earns its place, which has nothing to do with the assistant. It gives the HR team a single place to point when the same question arrives by email, and it means the answer given in March matches the answer given in July, because both come from the same sentence rather than from two people's recollection of a decision made at the time. Most companies discover this benefit after writing their first one and then keep doing it for reasons unrelated to the software.

What People Get Wrong About This

Why did nobody notice for three weeks?

Because wrong answers from a confident system do not look like faults. An employee who is told four weeks has no reason to doubt it, and the one person who would have spotted it, whoever changed the policy, is not the one asking. Staleness is only ever discovered by somebody who already knows the right answer, and those people do not use the assistant.

Should ingestion just run continuously?

More often is better and it does not solve lag one or lag three. Real-time ingestion of a stale document set gives you the old answers faster. The scheduling question is the easiest of the three to fix and the least important, which is the opposite of where most of the attention goes.

Can the assistant flag that a document is old?

Yes, if the document carries a last-reviewed date, and this is worth doing. An answer that notes the policy was last reviewed fourteen months ago gives the reader the information they need to decide whether to check. It does not help when the document was updated last week and the corpus has the version from last year, because the date it reports is the date in the text it holds.

Who should be told when a policy changes?

Whoever owns the answers, as one line in the checklist of whoever approves the change. Putting the trigger at the point of approval is what makes it reliable, because that is the only place in the organisation where the change is definitely known. A monthly question from the corpus owner asking what changed works and depends on memory.

Does the announcement cover it?

No, and assuming it does is a common failure. A company-wide message reaches people who read it on the day and not the person who asks the question in August. The announcement and the corpus are different channels with different audiences, and the assistant is now the channel most people will actually use.

The Change Routine, Which Is Four Checks

Attach this to the policy change process rather than to the assistant, because that is where it will actually run.

One: tell the owner, at approval. One line in the approval checklist naming the person who owns the answers. Not a copy of the announcement, which arrives alongside everything else. A direct note that a specific policy changed and from when.

Two: confirm the corpus has the new text. Not that the document was updated. That the corpus has it, which means either waiting for the scheduled ingestion and checking afterwards, or triggering it manually. Open the admin view and look.

Three: confirm the old version is gone. Search the corpus for a distinctive phrase from the superseded text. If it comes back, the old document is still in there, including in any archive folder inside the ingested path. This is the check almost nobody runs and the one that prevents the inconsistency case.

Four: ask the question three ways and read the answers. Not one way. Three phrasings, including the formal policy language and the way an employee would actually ask. This is the only check that verifies behaviour rather than content, and it is the step that catches a duplicate that the phrase search missed because the wording differed.

Four checks, about ten minutes, triggered by a policy change rather than by a calendar. The tenth minute is the one that matters: a corpus containing the right text can still be answering from the wrong passage, and only the rendered answer shows that.

Two notes on making the routine stick. Put it on the change, not on a person. A routine attached to a named individual stops when they are on leave during the week a policy changes, which is exactly when it will happen. Written into the approval template, it survives absence and role changes.

And do the checks in that order, because each one is only meaningful if the previous passed. Verifying the answer before confirming the corpus has the new text tells you nothing you can act on, since a wrong answer at that point has at least four possible causes. Running them in sequence means a failure at any step names its own cause, which turns a ten-minute routine into a diagnosis rather than a test.

The Changes Most Likely to Slip Through

Not all changes carry the same risk of going unnoticed, and knowing which ones slip is more useful than a general instruction to be diligent.

Changes made by somebody outside HR. A new benefits provider, a payroll calendar shift, a finance rule about expense approval limits. These are real policy changes affecting answers employees will ask HR about, and they do not go through the HR change process at all, so no notification step catches them. This is the largest category of missed change and the hardest to fix, because the trigger has to sit in three or four different functions.

Changes to a figure rather than to a rule. The structure stays identical and one number moves: an allowance, a threshold, a mileage rate, a contribution percentage. Nobody thinks of it as a policy change because the policy did not change, and a figure is exactly what an employee will act on directly without checking. A wrong rule produces confusion. A wrong number produces a wrong decision.

Changes that apply to one population. A new arrangement for a single office, a contract type, or a country. The document for everybody else stays correct, so a spot check passes, and the only people getting a wrong answer are the ones the change was for.

Changes communicated verbally or in a meeting. Somebody announces a new approach in a town hall and the practice shifts immediately while the document does not. The corpus is now behind the organisation rather than behind a document, which no ingestion schedule can catch.

Changes made to the document by somebody who did not tell anybody. A policy owner tidies up wording, and in doing so alters meaning or restructures the text into a form that retrieves poorly. The change log shows an edit, nobody flagged it, and answer quality moves without an identifiable cause.

So the monthly question is not only what policies changed. It is what changed that affects what people ask us, which catches the finance rule and the town hall announcement that a narrower question misses.

The Options, and How Each Handles a Change

Matram, named below, is owned by the same people who publish HROpsLab. Its limitations are stated as plainly as everybody else's.

Matram

A document-trained answering agent, flat at $29, $69 and $199 a month with unlimited seats. Strict mode declines rather than guessing when the corpus does not cover a question.

On this topic: strict mode does not help with staleness, because a stale document is coverage. It helps with the opposite case, where a policy is new and undocumented, by declining rather than answering from the old one. What to verify in your own instance: how re-ingestion is triggered, whether replacing a source removes the previous version or adds alongside it, and whether you can search the corpus for a phrase to confirm removal. Its cons stand: no free tier, it answers rather than actioning tickets, and it is not an HR or ITSM workflow platform.

Freshservice

Service management with Freddy AI over a knowledge base, at $19, $49 and $99 per agent per month plus a $29 per agent per month AI add-on.

On this topic: the knowledge base is inside the product, which is a real advantage here. Editing an article updates the thing being answered from, with no ingestion step and no duplicate risk, because there is one copy. The constraint is that your policies then live in a service desk rather than wherever HR keeps documents, and maintaining both is how divergence starts.

Zendesk

Suite Team at $55 per agent per month and Copilot at $50 per agent per month, so $105 per agent per month together. At twenty agents that is $2,100 a month.

On this topic: the same structural advantage, with better article versioning and publishing controls than anything else on this list, including scheduled publishing and visible revision history. Built for support content, so using it as the system of record for HR policy is a decision with consequences beyond this article.

Leena AI and Moveworks

Enterprise platforms ingesting your existing sources. Neither publishes a price.

On this topic: both handle source management as part of an implementation, which usually means somebody has thought about re-ingestion properly. The risk shifts to the notification problem, because a well-configured pipeline over documents nobody told anybody about is still three weeks of the old notice period.

SiteGPT

A document and site-trained agent at $59 and $129 a month billed monthly, or $468 and $948 per year billed yearly. The annual plans work out near $39 and $79 a month billed yearly and should not be quoted bare.

On this topic: it is built around crawling, so if your policies live as pages that get edited in place, the update path is the cleanest of any option here and the duplicate problem largely disappears. If they live as uploaded files, you own the version management manually.

Crisp

A messaging platform with a lighter knowledge layer, per workspace at a free tier and then $45, $95 and $295 a month.

On this topic: less content to manage means fewer places for a stale version to hide, which is a genuine if accidental advantage. The knowledge layer is thinner, so more of the answer quality rests on documents you maintain elsewhere.

The Comparison

The last column every evaluation reads. The third is the one that produces inconsistent answers nobody can reproduce.

The Decision Table

Catching It Without Waiting for a Complaint

The four checks run at a change. Something also has to catch the changes nobody told you about, and there are three cheap ways that do not require anybody to be diligent.

Watch the gap log for a new topic appearing suddenly. A cluster of questions about a subject the corpus has never covered usually means something happened: a new provider, a new process, an announcement. The log is the earliest available signal that the organisation moved and the corpus did not, and it arrives days before anybody files a complaint.

Watch for a rise in the same question being asked twice in one session. When an answer has gone stale, people do not always report it. They rephrase, because they assume they asked badly. A rising repeat rate on one topic is the fingerprint of an answer that has started being wrong, and it is visible in aggregate with no identifiers attached.

Keep a list of the documents with the earliest last-reviewed dates, and look at the top five monthly. Not an audit. Five documents, sorted by age. The oldest things in the corpus are where staleness concentrates, and reviewing the top of that list monthly means every document gets attention eventually without anybody scheduling a full review.

None of those is monitoring in the proper sense, because this failure cannot be monitored: there is no signal that a document has become untrue. They are three proxies that fire a few days earlier than a complaint would, which at a cost of about fifteen minutes a month is the best available deal.

One more, if you have the relationship for it. Ask three people who deal with employees directly whether anything has changed that HR would not know about. A finance manager, an office lead, a team lead. They know about the expense threshold and the parking arrangement and the new provider, and nobody has ever asked them, because the assumption is that policy changes come from HR.

Verifying on the Answer, Not the Document

This deserves its own section because it is the step teams skip and the reason fixes fail quietly.

Checking the document tells you what the corpus should contain. Checking the answer tells you what it does. Those diverge for at least four reasons: the ingestion has not run, it ran and failed silently, an old version is competing, or the new text is structured in a way that retrieves poorly even though it is present and correct.

The last of those is the subtle one. A policy rewritten into a single dense paragraph during an update is now harder to retrieve from than the version it replaced, so the answer gets worse despite the content being newer and more accurate. A document-level check passes. The answer is still wrong.

So the verification is always: ask the question, in three phrasings, and read what comes back. Compare it to what you intended. It takes two minutes and it is the only check that tests the thing employees experience.

Then record the three phrasings in the test set, with the new correct answer. The change you just made becomes a permanent regression check, which costs nothing and means the next person who touches the corpus cannot silently undo it.

Over a year this is how the test set stops being a launch artefact and becomes the real record of what the system is supposed to say. Thirty questions written at launch tell you whether the original build still works. Thirty plus every change since tells you whether the current policy set is being answered correctly, which is a materially different and more useful thing, and it grows by three questions per change rather than by a scheduled review nobody books.

The Period Where Both Rules Are Live

A change with an effective date creates a window in which two answers are both correct, for different people, and this is the hardest thing in the article to get right because it is not a maintenance failure. It is a genuine conditional that the corpus has to carry.

Three shapes come up, and they want different handling.

A clean cutover. The new rule applies to everybody from a date, with nothing in flight to worry about. This is the easy case and the dated change note covers it: state the old rule, the new rule, and the date. The assistant can answer both the current question and the historical one, and the note comes out six months later.

A grandfathered population. The new rule applies to new joiners or new requests, and an existing group continues under the previous arrangement, sometimes indefinitely. This is not a change note, it is a permanent second rule, and treating it as a note means the information gets deleted in six months while the population still exists. Write it as a scoped statement of two arrangements with the boundary defined, and give it its own heading, because people will ask which group they are in and that is a different question from what the rules are.

An in-flight distinction. Requests submitted before a date follow the old rule, those after follow the new one. Whether that is how your policy works is a decision somebody has to make and write down, and it is the detail most often left unstated because at the time of the change it seems obvious to the people making it.

The thing to avoid in all three is letting the assistant infer the answer. A model given a document that says the rule is now two weeks, and another saying it was four weeks until a date, will produce something plausible about which applies to a request from last Tuesday. It will be confident and it will not be grounded in a decision anybody made.

So the rule is: if the transition has a conditional, write the conditional. If nobody has decided it, that is the finding, and it is better surfaced by somebody writing the document than by an employee receiving a confident guess.

What Getting This Wrong Costs

The direct cost is people acting on the old rule. Somebody gives the wrong notice, misses a window, or plans around a figure that changed last month. Each instance is small and attributable to nothing in particular, which is why it never gets counted.

The compounding cost is the reputation of the system. Inconsistent answers are the fastest way to lose an audience, because an employee who gets two different answers to one question has learnt something more damaging than if they had got one wrong answer: they have learnt that they cannot tell when it is right. People do not come back from that, and usage declines without a single complaint being filed.

And there is the misdiagnosis. A team that sees inconsistency concludes the model is unreliable, and the conversation turns to switching products or to whether this technology is mature enough. The actual cause is a superseded document in an archive folder, which nobody will find while the discussion is about model quality. That is a quarter spent on the wrong question.

Ask it this way at the next policy change. Who will tell the corpus, who will check the old version is gone, and who will ask the question three ways afterwards? If any of those is nobody, the three weeks in the opening of this article are already scheduled.

When You're Ready to Make This Structural

Two steps beyond the four checks, once the habit is holding.

Put the notification in the approval workflow rather than in a person's memory. Whatever tool approves policy changes, add the corpus owner as a required notification. The checks are reliable and the trigger is the weak link, because it depends on somebody remembering during the busiest part of a change. Moving it into the approval step is the one structural change worth making.

Keep a changed-policies list with dates, reviewed monthly. Not a changelog per document, which nobody maintains. One list: policy name, what changed, effective date, date the corpus was verified. It takes a line per change and it answers the three questions that otherwise require investigation: what has changed recently, was the corpus updated, and which change notes are now old enough to remove.

A third step, once both of those are routine and only then. Make the effective date visible in the answer itself. Where a document carries a date from which its content applies, the assistant can state it, which turns every answer about a recently changed rule into a self-dating one. Employees then have what they need to notice a discrepancy themselves, which is the only form of monitoring that scales to the whole workforce.

That list is also the artefact to hand somebody when they ask whether the assistant is current. A dated list of verified changes is an answer. An assurance that somebody keeps it up to date is not, and after the first three-week gap nobody will accept it as one.

Frequently Asked Questions

HROpsLab takes no vendor money and publishes no paid placements.