TL;DR
- Most pilots fail as evidence rather than as operations. They run smoothly, everybody feels positive, and nobody can say afterwards what was actually proven.
- Pick the hard markets deliberately. A pilot in your easiest country tests the vendor's best case and tells you almost nothing you did not already assume.
- Sixty days is the minimum that includes both a delivery cycle and at least one real retrieval. Thirty days tests half the product.
- Write the pass criteria before you start, in terms a spreadsheet could settle. Criteria written afterwards get written to match the result.
- Measure four things from day one: order to hand by country, collection completion without chasing, exception count, and what the vendor did when something went wrong.
- Decide in advance what a failure means. A pilot with no defined exit is a procurement step wearing a pilot's clothes.
The pilot that went perfectly and proved nothing
A 350-person company ran a sixty-day pilot with an equipment vendor. Twelve devices, all to employees in the company's two largest markets, all standard specification, all new joiners with four weeks of notice before their start dates.
Everything arrived on time. The vendor's account team was responsive. The portal was clean. At the end of the pilot everybody involved felt good about it and the recommendation to proceed was unanimous.
Eleven months later the arrangement was in trouble, and the reasons were all things the pilot had not touched. Deliveries into the company's three smaller markets took far longer than the two tested ones. Retrieval was slow and required chasing, which the pilot had never exercised because no pilot participant had left. The exception process, when a device arrived damaged, involved a conversation that took two weeks, and nothing had gone wrong during the pilot so nobody knew.
Best tools for HR Operations
The pilot was not badly run. It was badly designed. It tested the part of the service that works well for every vendor in this category, over a period short enough that the return journey never happened, in markets where nothing is hard. It produced a positive feeling and no evidence.
When a pilot is not worth running
The commitment is small and reversible. If you can sign a short arrangement with no minimum and leave after a quarter, that is a better test than a pilot, because it is the real thing with a real exit.
You only have one market and it is a straightforward one. There is nothing to vary. Order a few devices, see how it goes, and treat the first month as the pilot without calling it one.
You have no capacity to run it properly. A pilot staffed by whoever has a spare afternoon produces a result nobody trusts. If you genuinely cannot allocate somebody, a short paid trial of the real service is more honest than a pilot that nobody has time to observe.
The decision is already made. Some pilots exist to justify a choice that has been taken. That is not inherently wrong, but be honest internally about what it is, because designing it as evidence and using it as justification wastes the effort of everybody involved.
Design the pilot around what you are afraid of
The instinct is to start easy, build confidence and expand. For a pilot that is backwards, because the purpose is not to build confidence but to find out whether the thing works where it is hard.
Write down the three things you are most worried about before choosing anything else. Typical answers: a specific market where customs has caught you out, the speed of retrieval, whether anybody actually answers when something goes wrong, or whether the asset data will be usable.
Then design the pilot so each of those is exercised. If you are worried about a market, that market is in the pilot. If you are worried about retrieval, the pilot must contain at least one real collection. If you are worried about the exception path, you need something to go wrong, which is the one thing you cannot schedule and the reason for the deliberate-failure section below.
A pilot that exercises none of your actual worries is a demonstration. Demonstrations are fine and vendors run them for free. What you are paying for here is a test.
Picking the cohort
Three to five markets, twelve to twenty devices, and a deliberate mix rather than a convenient one.
Include your hardest market. The one with the customs history, the long lead times, or the single employee. This is non-negotiable and it is the most common thing missing from pilots.
Include one easy market as a control. You need to know whether a problem is the vendor or the market, and without a baseline every result is ambiguous.
Include a single-person market if you have one. These behave differently from markets with volume and most vendors handle them through a partner, which is exactly the arrangement you want to see working.
Include at least one non-standard case. A different device specification, an accessibility requirement, a contractor rather than an employee. Pilots built entirely from the standard case tell you about the standard case, and the standard case is not where vendors differ. If an adjustment or a specialist configuration is in scope, handle it through whoever normally owns that rather than as a logistics variation, since the point is to test whether the vendor can accommodate it without the person waiting longer than everybody else.
And include real people with real start dates. A pilot shipping to an office for somebody to unbox proves the logistics and skips the experience. The employee's view of what arrived and when is part of what you are testing, and it is the part that determines whether the arrangement feels good to the rest of the company once it is live. A pilot run entirely into a storeroom tests a courier relationship rather than an onboarding service.
| Cohort element | Why it is in the pilot | What it tests |
|---|---|---|
| Hardest market | Where the risk actually is | Depth, customs handling, escalation |
| Easy market as control | Disambiguates vendor from geography | Baseline performance |
| Single-person market | Usually partner-served | The partner arrangement |
| Non-standard specification | Pilots skew to the standard case | Flexibility and lead time on exceptions |
| At least one retrieval | The half most pilots never test | The return journey end to end |
| Real employees, real dates | Logistics is not the whole product | The experience and the communication |
Why sixty days and not thirty
Thirty days tests delivery. Sixty tests the service.
The reason is the return journey. Equipment goes out continuously and comes back only when somebody leaves or swaps a device, and in a cohort of fifteen people over thirty days the probability of a real retrieval happening naturally is low. Without one you have tested half the product and the half that every vendor does well.
Sixty days also gives you a second ordering cycle, which matters because the first one gets the vendor's full attention during onboarding and the second is closer to how the relationship will actually feel.
If you genuinely cannot run sixty days, the compromise is to manufacture a retrieval rather than shorten the pilot. Find a device that needs collecting anyway, an old machine sitting with somebody who no longer uses it, and put it through the full process. That is not as good as an organic retrieval and it is considerably better than none.
Ninety days is better than sixty if you have the patience, mostly because it increases the chance that something goes wrong naturally, and how a vendor handles a genuine problem is the most informative thing in the whole exercise.
Write the pass criteria before you start
This is the step that separates a pilot from an extended demo, and it is the one most often skipped because it feels bureaucratic at the start and obvious at the end.
Criteria written afterwards get written to match what happened. That is not dishonesty, it is how people work: you observe an outcome, you reason about whether it was acceptable, and the reasoning accommodates the result. The only protection is writing the standard down while you still do not know what it will be.
Good criteria are things a spreadsheet could settle. Order to hand within a stated number of working days in each pilot market, with the number set per market rather than globally, because a uniform target is either unrealistic somewhere or trivial everywhere. At least one collection completed without your team chasing it. An asset export you can open and understand without the vendor explaining it. No more than a stated number of exceptions requiring a conversation.
Bad criteria are things requiring judgement. The vendor was responsive. The portal was intuitive. The team liked it. Those may all be true and none of them survives a disagreement six months later about whether the pilot actually passed.
Agree the criteria with whoever will make the decision, not just with the person running the pilot. A criterion the decision-maker has not seen is a criterion they can decline to recognise.
The four measurements
Record these as they happen. None can be reconstructed afterwards and all four are routinely lost because nobody set up anywhere to put them.
Order to hand, by country. Working days from your request to the person having a usable device. Not from dispatch, which is the number vendors report naturally and which excludes exactly the part you cannot control.
Collection completion without chasing. Of the retrievals initiated, how many completed without somebody on your side following up. This is the single most predictive measure of what the relationship will feel like in year two.
Exception count. How many of your shipments needed a human conversation rather than running through the normal path. A high number during a pilot, when the vendor is paying maximum attention, is a strong signal.
What happened when something went wrong. Not a number, a narrative. Write down the incident, who you contacted, how long it took and what the resolution was. This is the part everybody remembers vaguely and nobody records, and it is the most useful page in the final report.
Let something go wrong, or at least be ready when it does
You cannot schedule a failure, but you can refuse to smooth one over when it happens.
The natural instinct during a pilot is to help. A shipment is late, somebody on your team phones the vendor's account manager directly, it gets fixed, and everybody moves on. That is a reasonable thing to do operationally and it has destroyed your evidence, because what you tested was the escalation path available to a prospect rather than the one available to a customer.
So when something goes wrong, use the normal route. Raise it the way an ordinary user would, through whatever channel the vendor has given you, and let it take the time it takes. Note the time.
Two things you can do deliberately without sabotaging anything. Send one request through the standard channel rather than to your account contact, and see what happens. Ask a question you know the answer to, through support, and see how long the response takes and whether it is right. Both are honest tests and both tell you about the service rather than about the sales process.
Who runs it, and the staffing nobody budgets
A pilot needs an owner with time, and the most common reason pilots produce weak evidence is that the person running it was doing it alongside their actual job.
The work is not large but it is continuous. Somebody has to place the orders, record the four measurements as they happen, notice when something goes wrong and resist fixing it informally, chase the retrieval, collect feedback from the people who received devices and write it up at the end. Across sixty days that is perhaps three or four hours a week, concentrated unevenly.
What happens when nobody owns it is predictable. The measurements get reconstructed from memory at the end. The retrieval never happens because initiating one requires somebody to decide to. The exception gets escalated through a personal relationship because that was faster. And the final review is a conversation about how everybody felt, which is where you started.
Name the person before the pilot begins and confirm they have the hours. Then name a second person who is not involved in running it to chair the day-sixty review, because the person who ran a pilot has an investment in it having gone well that is entirely human and not conducive to a clear verdict.
| Role | Who | What they do |
|---|---|---|
| Pilot owner | IT operations usually | Orders, measurements, exception handling via normal routes |
| Decision-maker | Whoever signs | Agrees criteria in week one, chairs the review |
| Participants | Real employees in scope | Receive devices, give feedback on the experience |
| Vendor contact | Named on their side | Deliberately not used for routine escalation during the pilot |
| Finance | Light touch | Confirms what the pilot costs and what a rollout would |
What the pilot cannot tell you
Worth knowing the limits, because a pilot oversold internally creates expectations that the live arrangement then fails.
It cannot tell you about scale. Fifteen devices over sixty days exercises the service, not its capacity. A vendor comfortable with your pilot volume may handle two hundred devices in a quarter differently, and nothing in a pilot surfaces that.
It cannot tell you about year two. Onboarding attention is real and it fades. The account team assigned to a new customer is not necessarily the one you will have in eighteen months, and the responsiveness during a pilot is close to the best you will see.
It cannot tell you much about the contract. Pilots run on goodwill. How a vendor behaves when something goes wrong and there is money attached to the answer is a different question, and the only proxy available is how clearly they explain their own terms.
And it cannot test markets you did not include. This sounds obvious and it is the specific failure in the story at the top of this article, where three markets were outside the pilot and all three turned out to be the problem.
The honest framing at the day-sixty review is that the pilot tested specific things in specific markets and those things worked. Resisting the generalisation is what keeps the exercise useful.
Running it alongside the existing process
The pilot does not replace your current arrangement, it runs beside it, and the overlap needs managing or the pilot becomes a source of confusion rather than evidence.
Decide which system is authoritative for pilot devices, and make it the pilot. Devices in two registers will diverge, and the divergence will be attributed to the vendor.
Tell the people receiving pilot devices that they are in a pilot. Not to lower expectations, but because their feedback is part of what you are gathering and they will give it more usefully if they know it is wanted. Say explicitly that reporting a problem is helpful rather than unhelpful, since new joiners in particular tend to absorb a poor experience silently rather than complain in their first fortnight.
Keep the old route available for anything urgent. A pilot that blocks a genuinely urgent request creates pressure to declare it successful early, and somebody waiting for a laptop is not a good reason to accept a vendor. Record it when you use the fallback, though, because a pilot where three urgent cases quietly went the old way has tested less than its device count suggests.
And set a clear boundary on scope creep. Pilots expand. Somebody asks whether the vendor can also handle a market that was not in scope, the vendor obliges, and by week six you are running a half-migration that nobody decided on and cannot cleanly assess. The vendor is not at fault here; saying yes to a customer request is the correct commercial instinct, and the discipline has to come from your side. Agree in week one that anything outside the cohort goes through the old route, and hold that line even when the vendor could obviously handle it faster.
Week by week, roughly
A shape rather than a schedule, because pilots vary. What matters is that the retrieval and the deliberate support test happen early enough to leave room for a second attempt.
Week one. Criteria written and agreed with the decision-maker. Cohort confirmed including the hard market. Measurement sheet created, with somebody owning it. Vendor onboarding completed and the asset baseline recorded.
Weeks two and three. First orders placed across all pilot markets simultaneously rather than sequentially, because staggering them hides market-level differences behind timing. Measurements recorded as deliveries land.
Week four. First participant feedback, while the experience is recent. Initiate a retrieval if none has occurred naturally, using a device that genuinely needs collecting.
Weeks five and six. Second ordering cycle, which is the one that shows you the relationship rather than the onboarding. Run the deliberate support test here: one request through the standard channel, one question to support whose answer you already know.
Week seven. Retrieval should be complete or visibly late, and either is informative. Request the asset export and try to use it without help. Collect remaining participant feedback.
Week eight. Write it up against the criteria from week one. Hold the review on the date set at the start, chaired by somebody who did not run the pilot.
The two things most often left too late are the retrieval and the export request. Both have a habit of taking longer than expected, and discovering that in week eight leaves no time to find out whether it was a one-off.
The decision at day sixty
Hold the review on a date set at the start. Bring the four measurements and the criteria you wrote in week one.
If it passed, write down what good looked like. The numbers from the pilot become your baseline for the relationship, and having them written down means the first quarterly review has something to compare against. Companies that skip this lose the only clean measurement they will ever take, because every later measurement is of a live arrangement with no control.
If it passed in some markets and not others, that is the most common result and it is a specific conversation rather than a verdict. The useful question is whether the failing markets are fixable, whether the vendor knew they were weak there, and whether their explanation matches what you observed.
If it failed, say so and use the evidence. A pilot that produces a no has done its job and saved you considerably more than it cost. The reason to write criteria in advance is precisely so this conversation is about the numbers rather than about whose impression was right. Share the findings with the vendor too, specifically and without softening them. Some will have a genuine answer, particularly where a market was weak for a reason they are already addressing, and a vendor who responds well to a failed pilot is giving you better information about the relationship than one who passed easily.
And if it was inconclusive, extend rather than decide. An inconclusive pilot usually means the cohort was too easy or too small. Another thirty days with the hard market added is cheaper than a three-year commitment made on insufficient evidence.
Asking the participants, which is half the evidence
The four measurements capture the operation. They do not capture what it was like to be on the receiving end, and that is a meaningful part of what you are buying.
Ask each participant four short questions at the end rather than one open-ended one, because open-ended questions about equipment produce either nothing or a complaint about something unrelated.
Did you know when it was arriving? Communication during transit is where vendors differ enormously and where the employee experience is actually made. A device that arrives on day nine having been silent since day one feels worse than one that arrives on day eleven with two updates.
Was anything missing or unexpected? Cables, adapters, the dock, the regional power supply. These are the small failures that generate a second shipment and a bad first week, and they rarely reach whoever is running the pilot.
Did you have to contact anybody, and what happened? Any contact at all is a signal, since in the ideal case the employee never needs to.
Was it ready to use, or did you have work to do? A machine that arrives needing ninety minutes of setup is a different product from one that is ready, and the difference is invisible from the ordering side.
Write the answers down verbatim rather than summarising them. Two or three sentences from somebody who received a laptop in your hardest market is more persuasive at the day-sixty review than any number you will produce, and it is the part that stops the discussion becoming abstract.
The costs of the pilot itself
Small, and worth knowing so nobody is surprised halfway through.
The devices are real. You are buying or leasing machines that people will keep using, so this is not incremental spend in most cases. What can be incremental is a premium on small volume, since pilot quantities will not attract whatever pricing a full arrangement would.
Shipping at pilot volumes is worse per unit. Fifteen shipments do not get volume rates, and your hardest market will be the most expensive of the fifteen. Expect the per-device cost during a pilot to overstate what a rollout would cost, and say so when presenting the numbers, because somebody will otherwise extrapolate it.
Internal time is the real cost. Three to four hours a week for the owner across sixty days, plus the review. That is the number to put in front of whoever approves it, rather than pretending the pilot is free because the devices were going to be bought anyway.
And there may be a setup or onboarding fee, which some vendors waive for pilots and some do not. Ask, and ask specifically whether it is waived or deferred, since a deferred fee that appears on the first invoice after a successful pilot is a surprise nobody enjoys.
Where the vendors fit
A free trial offered by the vendor. Common and worth taking, with the caveat that a vendor-designed trial will naturally exercise what the vendor does well. Take the free trial and impose your own cohort and criteria on it.
A paid short-term arrangement. More honest in some ways, because you are a customer rather than a prospect and the service you receive is closer to the real thing. Costs more and is the better test.
Running two vendors in parallel. Expensive in attention and the best evidence available, because the same cohort through two vendors removes every ambiguity about whether a problem is the vendor or your situation. Most companies cannot staff it; those that can learn more than from any other approach.
RemoAsset is our own product and that is the disclosure rather than a footnote. It covers procurement and retrieval across 80+ countries and publishes a Subscription tier from $1,499 a year, alongside a pay-as-you-go tier which is relevant to pilots specifically because it lets you run one without committing to a subscription first. Three limits, stated because a pilot designed on a wrong assumption wastes sixty days: it enrols devices but is not an MDM in its own right, so do not design a pilot to test endpoint management; it wipes devices but is not a certified ITAD, so a pilot testing formal disposal evidence will need a separate provider in scope; and coverage and lead times vary by market, which is exactly why the hard-market-first rule above matters and why you should put our weakest market in your pilot rather than our strongest.
The decision table
| Situation | Scale | Setup | Primary Pain | Recommended Starting Point |
|---|---|---|---|---|
| First equipment vendor | 50 to 200 | Couriers and a spreadsheet | No baseline to compare against | Measure current performance for a month first |
| Replacing an incumbent | Any | Existing vendor, known problems | Pilot tests the new one in isolation | Run the same cohort through both |
| Small commitment available | Any | Vendor offers month to month | Pilot overhead exceeds the risk | Skip the pilot, buy the short arrangement |
| Regulated or audit in scope | Any | Evidence requirements | Pilots rarely test disposal | Put disposal evidence in the criteria |
| Many markets, uneven coverage | 200 plus | Mixed performance today | Averages hide market-level failure | Criteria set per market, not globally |
| No internal capacity | Any | Nobody has time to observe | Result nobody trusts | Shorter paid trial, or defer until staffed |
Testing the data, which almost nobody does
Every pilot tests whether devices arrive. Very few test whether the record of them is any good, and that record is what you will live with for years.
Ask for a full export in week seven and try to use it unaided. Not a demo of the reporting screen, the actual file. Open it, and see whether you can answer three questions from it without help: what do we have, who has it, and when did they get it. If you cannot, that is a finding, because the export is what you will rely on during an audit, a migration or an exit.
Check whether it contains serial numbers for everything. A record without serials is extremely difficult to reconcile later, and pilots are exactly when gaps in this are easiest to spot because the volume is small enough to check every row.
Check what happens to the record when a device comes back. Does the retrieval close the assignment, does it move the device to a stock status, does it leave it assigned to somebody who no longer works for you. This is where asset registers rot and it is visible in a pilot if you look.
And check it against reality. You have fifteen devices and you know where they are. Compare the export against what you know. A pilot is the only time you will ever have a dataset small enough to verify completely, and a discrepancy at fifteen devices is a much louder signal than the same discrepancy rate at fifteen hundred.
This is the least glamorous part of a pilot and the one with the longest tail, because a vendor whose logistics are excellent and whose data is poor will cost you a migration eventually.
What getting this wrong actually costs
The visible cost is sixty days, which is real but recoverable.
The larger cost is a commitment made on evidence that was never evidence. A pilot that tested the easy path produces a confident decision, and confident decisions are harder to revisit than uncertain ones. The company in the story at the top of this article did not just sign a vendor that was wrong for three of its markets; it signed with a conviction that made the problem take eleven months to surface, because everybody remembered the pilot going well.
The cost nobody anticipates is the second pilot. Having run one that proved nothing, a company that discovers problems later has to either live with them or start the whole process again, and the appetite for a second sixty-day exercise after an expensive mistake is very low. In practice most companies live with it.
So the question is not whether the pilot went well. It is whether anything in it would have gone badly if the vendor were wrong for you, because a test that cannot fail has not tested anything.
When a pilot is the wrong instrument entirely
For most equipment vendor decisions a well-designed pilot is the right tool. There are two situations where it is not.
The first is when you have no baseline. A pilot measures a vendor against criteria, but if you have no idea what your current performance is, the criteria are guesses. Spending a month measuring your existing process before piloting anything is unglamorous and it makes every subsequent comparison meaningful. Order to hand, collection completion and exception count are all measurable on what you do today.
The second is when the decision is really about model rather than vendor. If the open question is whether to use a platform at all, rather than which one, a pilot of one vendor answers the wrong question. That decision is made by comparing your current costs and capability against the category, which is analysis rather than trial.
If you recognise either situation, the sixty days are better spent on the thing that is actually uncertain. A pilot run to answer a question it cannot answer is a well-executed waste of a quarter.
Frequently asked questions
How long should an equipment platform pilot run?
Sixty days as a minimum, because that is roughly what it takes to include both a delivery cycle and at least one genuine retrieval. Thirty days tests delivery only, which is the half of the service every vendor in this category performs well. Ninety is better if you have the patience, mainly because it raises the chance that something goes wrong naturally, and how a vendor handles a real problem is the most informative thing the exercise can produce. If you cannot run sixty, manufacture a retrieval rather than shortening the pilot.
Which markets should be in the pilot?
Your hardest one, without exception, plus an easy market as a control so you can tell whether a problem is the vendor or the geography. Add a single-person market if you have one, since those are usually partner-served and that arrangement is what you most want to see working. The most common design error is piloting in the two largest, most straightforward markets, which tests the vendor's best case and produces a positive result that predicts nothing about the markets where you actually need help.
What should the pass criteria be?
Things a spreadsheet could settle rather than things requiring judgement. Order to hand within a stated number of working days per market, with the number set per market rather than globally. At least one collection completed without your team chasing. An asset export you can open and understand unaided. A cap on exceptions needing a human conversation. Write them before the pilot starts and agree them with whoever makes the final decision, because criteria written afterwards get written to match the result.
What should we measure during the pilot?
Four things, recorded as they happen since none can be reconstructed later. Order to hand in working days by country, measured to the person having a usable device rather than to dispatch. Collection completion without chasing, which is the most predictive measure of how year two will feel. Exception count, meaning shipments needing a conversation rather than running through. And a written narrative of what happened when something went wrong, which is the page everybody remembers vaguely and nobody records.
Should we help the vendor when something goes wrong during the pilot?
Use the normal route rather than your account contact, and note how long it takes. The instinct to phone somebody directly and get it fixed is operationally reasonable and it destroys the evidence, because you have then tested the escalation path available to a prospect rather than the one available to a customer. Two honest deliberate tests: send one request through the standard channel instead of your account manager, and ask support a question you already know the answer to.
What if the pilot result is inconclusive?
Extend rather than decide, because inconclusive almost always means the cohort was too small or too easy. Another thirty days with your hardest market added costs very little against a multi-year commitment made on insufficient evidence. Resist the pressure to call it successful because the time is up; a pilot that has not exercised the things you were worried about has not finished, regardless of what the calendar says. The one thing not to do is proceed while describing the result as positive.
Should we run two vendors at once?
It is the best evidence available and most companies cannot staff it. Putting the same cohort through two vendors removes every ambiguity about whether a problem is the vendor or your own situation, which is the question that makes single-vendor pilots hard to interpret. The cost is attention rather than money: two onboardings, two sets of measurements and two relationships to manage over the same sixty days. If you have the capacity it is worth more than any other approach described here.
What do we do with the results if the pilot passes?
Write down what good looked like, in numbers, before the arrangement goes live. The pilot measurements become your baseline, and without them every later review compares a live arrangement against an impression. This is the only clean measurement you will ever take, because afterwards there is no control. Companies that skip this step find that a year later nobody can say whether performance has drifted, since there was never a recorded starting point to drift from.
Where this leaves you
A pilot is worth running when it can fail. Most cannot, because they are designed around the easy path over a period too short to include the return journey.
Pick the markets you are afraid of. Run sixty days so a real retrieval happens. Write criteria a spreadsheet could settle, before you start, agreed with whoever decides. Measure four things as they occur. Use the normal support route when something breaks, and write down what happened.
Then hold the review on the date you set in week one, and be willing to let the answer be no.
Two habits make the difference between a pilot and an extended demonstration, and neither costs anything. The first is writing the pass criteria down before you know what the results will be, because criteria written afterwards accommodate whatever happened and everybody involved will genuinely believe they were the standard all along. The second is refusing to smooth over the first thing that goes wrong, which is harder than it sounds because fixing it quickly is both the helpful instinct and the thing that deletes your most useful data point.
Everything else in this article follows from those two. Pick hard markets so something can go wrong. Run long enough that the return journey happens. Measure as you go so the numbers exist. Let a stranger chair the review so the person who invested sixty days is not the one grading it.
A pilot that could not have failed has told you nothing, and it will have told you that nothing very convincingly.
