Legal Tech and AI

Running a Technology Pilot at a Law Firm

A pilot method that ends in a clear yes or no: charter, baseline, pilot group, scorecard, decision memo and the mistakes that sink legal tech trials.

AssistCall research teamPublished Reviewed by Faruque Amin, VP of Sales and Marketing6 min read

On this page
  1. Why pilots drift
  2. The pilot charter
  3. Measure the baseline first
  4. Choose the pilot group
  5. Do the security review before day one
  6. Run it in phases
  7. Score it honestly
  8. Debrief interviews
  9. Example: a pilot that ended in no
  10. Write the decision memo
  11. Mistakes that sink pilots
  12. Using the method across the firm
  13. Frequently asked questions
  14. References
Legal Tech and AI: Running a Technology Pilot at a Law FirmArticle coverLEGAL TECH AND AIRunning a TechnologyPilot at a Law FirmAva for LawyersInsightsLegal Tech and AI: Running a Technology Pilot at a Law FirmArticle coverLEGAL TECH AND AIRunning a TechnologyPilot at a Law FirmAva for Lawyers

Key takeaways

  • Write down the decision you want to make before the pilot starts, with thresholds for yes, adjust and no.
  • Measure a baseline first; only 20% of legal professionals measure the return on investment of generative AI[1].
  • Choose a pilot group that does the real work, includes a skeptic and has protected time to give feedback.
  • End with a one-page decision memo, even when the answer is no.

Why pilots drift

Most legal tech trials end without a real decision. The free period runs out, a few people liked the tool, nobody measured anything, and the firm either renews by default or lets it lapse. Months later someone suggests the same tool again, and nobody can say why it was dropped.

The pace of adoption makes disciplined pilots more important. In Thomson Reuters Institute's 2025 report, 26% of legal professionals said they were already using generative AI, up from 14% in the 2024 report, and 28% of law firms had adopted it at the organization level[1]. Across the wider economy, US business AI use hovered between 17% and 20% from December 2025 to May 2026[2], while a Federal Reserve analysis put adoption in professional, scientific and technical services, the sector that includes law firms, at about 33%[3]. Vendors know this, and the pressure to try something new will keep growing.

Legal professionals already using generative AIThomson Reuters Institute survey
Legal professionals already using generative AIColumn chart. 2024 report: 14%; 2025 report: 26%.0%10%20%30%2024 report: 14%14%2024 report2025 report: 26%26%2025 reportLegal professionals already using generative AIColumn chart. 2024 report: 14%; 2025 report: 26%.0%10%20%30%2024 report: 14%14%2024 report2025 report: 26%26%2025 report

Source: Thomson Reuters Institute, 2025[1]

Show the data
ItemValue
2024 report14%
2025 report26%

The pilot charter

Write a one-page charter before anyone logs in. It forces the firm to agree what problem the tool should solve and what result would justify paying for it. Example charter for a fictional document automation pilot:

Pilot charter fields with an example
Charter fieldWhat to writeExample
ProblemThe specific workflow and what it costs todayEngagement letters take about forty minutes each to assemble
HypothesisWhat should change if the tool worksAssembly time falls below fifteen minutes with no increase in errors
ScopeWho, which matters and how longTwo paralegals, new estate planning matters, six weeks
MeasuresTwo to four numbers, each with a baselineMinutes per letter, errors found at review, staff rating
ThresholdsWhat counts as yes, adjust and noYes below fifteen minutes; adjust from fifteen to twenty-five; no above that or with more errors
OwnerOne person who runs the pilot and writes the memoOffice manager
Data and security checkWhat data the tool sees and on what termsVendor confirms in writing that firm data is not used for training

Have the supervising partner sign the charter. A signature turns a casual trial into a commitment to decide, and it makes it harder to move the goalposts later.

Measure the baseline first

Without a baseline, people compare the pilot with their memory of the old way, and memory is generous to whatever they are used to. Spend two to four weeks timing the current process on the same kinds of matters. Thomson Reuters found that only 20% of legal professionals were measuring the return on investment of generative AI[1]; a stopwatch and a shared spreadsheet are enough to do better than that.

  • Time: minutes per task from start to finished draft.
  • Quality: errors found at review, recorded by type.
  • Volume: tasks completed per week.
  • Experience: a one-question staff rating after each task, on a scale of one to five.

Measure the same things in the same way during the pilot. If the measurement method changes, the comparison is meaningless.

Choose the pilot group

  • People who do the work every day, not only the partner who found the tool.
  • At least one skeptic, whose objections you want to hear early rather than after rollout.
  • Enough volume to produce useful data within the pilot window.
  • Protected time for setup and feedback, written into their week.

Training matters more than enthusiasm. In a 2025 Thomson Reuters survey, 64% of professionals had received no training on using generative AI for their work[4]. Budget a proper session at the start and a check-in at the end of week two, and make sure the pilot group knows the firm's rules on what data may go into the tool.

Do the security review before day one

Many pilots are delayed or abandoned halfway because someone asks a data question that should have been asked at the start. Settle it before anyone loads a real document. Ask the vendor, in writing, where data is stored, who at the vendor can access it, whether it is used to train models, how long it is kept after you stop, and how you export it. Have the supervising attorney decide what categories of client information may be used during the pilot, and use anonymized or closed matters if the answers are not yet good enough.

Record the answers in the charter. If the tool goes firm-wide, the same answers become part of your vendor file and your client communications.

Run it in phases

  1. Setup, week one

    Configure the tool, load templates, confirm the security settings and train the group.

  2. Controlled use, weeks two and three

    Use the tool on a defined set of matters, and log every problem and workaround.

  3. Midpoint check, end of week three

    Review the log and early numbers. Fix configuration problems, but do not change the measures.

  4. Full use, weeks four and five

    Run at normal volume so the data reflects real working conditions.

  5. Measure and debrief, week six

    Compare results with the baseline, interview each user and collect the vendor's renewal terms in writing.

Score it honestly

Example pilot scorecard for a fictional estate planning team
MeasureBaselinePilotThreshold met?
Minutes per engagement letter4017Adjust band
Errors found at review per ten letters32Yes
Staff rating, one to five2.84.1Yes
Setup hoursNot applicable12Within plan

Example: the fictional team is faster and happier but lands in the adjust band on time. The memo recommends a two-week extension to fix two template problems that caused most of the delay, using the same measures, followed by a final decision. That is a good outcome. A pilot is supposed to surface exactly this kind of detail.

Debrief interviews

Numbers show what happened. Short interviews explain why. Spend fifteen minutes with each person in the pilot group, separately, and ask the same questions each time:

  • Which tasks got easier, and which got harder?
  • Where did you work around the tool rather than use it?
  • What did you check by hand because you did not trust the output?
  • If we stopped using it tomorrow, what would you miss?
  • What would need to change before the rest of the firm uses it?

Listen hardest to the skeptic. If the skeptic says the tool saved time, that is strong evidence. If the enthusiast says it did not, take it seriously too.

Example: a pilot that ended in no

Example: a fictional immigration firm pilots an AI tool for summarizing client documents. Summaries arrive quickly, but paralegals spend so long checking dates and names against the originals that total time per file barely changes, and two summaries mix up family members. The memo recommends no, records the specific failure, and notes that the firm would reconsider a tool that links every summary line to its source page. The firm saves an annual subscription and has a clear standard for the next vendor that calls.

Write the decision memo

  1. Decision: yes, adjust and extend once, or no.
  2. Evidence: the scorecard, with baseline and pilot figures side by side.
  3. Costs: subscription, setup, training and integration over two years.
  4. Risks: data handling, vendor stability, lock-in and how you would leave.
  5. Rollout plan if yes: who next, by when and who owns training.
  6. Lessons if no: the problem may still need solving another way.

File the memo where the next person considering a similar tool will find it. The most expensive pilot is the one the firm runs twice because nobody kept the first result.

Mistakes that sink pilots

  • Changing the success measures halfway through.
  • Piloting on unusual matters that do not represent normal work.
  • Letting the vendor run the measurement and the debrief.
  • Signing an annual contract to get pilot pricing before the decision is made.
  • Giving the pilot group no time, so the tool is used only when it is convenient.
  • Ending without a memo, so the next pilot repeats the same mistakes.

Using the method across the firm

A yes is the start of a rollout, not the end of the work. Expand one team at a time, keep measuring the same numbers for the first quarter, and have the pilot users train the next group. Results often dip briefly when less enthusiastic users join, so judge the rollout on the trend rather than the first week.

The same charter, baseline, scorecard and memo work for almost any tool, from shared inboxes and ticketing to call analytics such as searching call transcripts for intake insights. Fund pilots through a dedicated line in your technology budget, so testing is planned rather than squeezed in. If the pilot is for a new practice management system, plan the data migration before you commit, because migration effort often decides whether a switch is worth it.

Frequently asked questions

How long should a legal tech pilot last?

Four to eight weeks for most tools, with two to four weeks of baseline measurement before it starts. Long enough for normal volume, short enough to keep attention.

How many people should be in the pilot group?

Enough to produce real volume, usually two to five people in a small firm, including at least one skeptic.

Should we let the vendor run the pilot?

Let the vendor handle setup and training, but keep measurement and the decision inside the firm.

What if the results are mixed?

Use the adjust band in your charter: fix the configuration, extend once with the same measures, then decide.

References

  1. Thomson Reuters Institute (2025). 2025 GenAI report: Executive summary for legal professionals (opens in a new tab) โ†‘

  2. U.S. Census Bureau (2026). Large Firms With at Least 20 Employees Biggest AI Users (opens in a new tab) โ†‘

  3. Board of Governors of the Federal Reserve System (2026). Monitoring AI Adoption in the U.S. Economy (FEDS Notes) (opens in a new tab) โ†‘

  4. Thomson Reuters (2025). Generative AI Adoption Nearly Doubles as Professional Services Reach Crossroads (opens in a new tab) โ†‘

Book a demo

See Ava handle your calls.

Book a short demo. Weโ€™ll walk through your calls and workflow, and agree what Ava should take care of.

Book a demo