PowerPoint slide production was over seven minutes faster on average but required corrections due to inferior quality
A recent three-month trial of Microsoft’s M365 Copilot within a UK government department has found no definitive evidence of improved productivity, despite some promising results for specific tasks.
The Department for Business and Trade conducted the pilot from October to December 2024, distributing 1,000 licences primarily to volunteers, with 30% randomly selected participants.
About 300 participants consented to data analysis, enabling the department to assess the tool’s impact on work efficiency, output quality, and user satisfaction.
While M365 Copilot showed potential in speeding up certain activities – most notably writing emails, summarising meetings and creating report summaries – the quality and accuracy of outputs varied significantly depending on the task.
Users completed email writing and report summaries faster and with higher quality compared to non-users, although the time saved for email writing was reported as minimal.
Conversely, tasks such as Excel data analysis and PowerPoint slide creation were completed slower or produced lower-quality results with M365 Copilot.
Specifically, PowerPoint slide production was over seven minutes faster on average but required corrections due to inferior quality. Data analysis in Excel also took longer and was less accurate among Copilot users, contradicting diary study data that suggested possible time savings.
No clear link between time savings and productivity gains
While some time savings were identified, the assessment [pdf] found no conclusive evidence that these led to measurable productivity improvement. The department acknowledged this limitation, citing that productivity impact was not the main focus of the evaluation and that further data would be needed to verify such claims.
Usage statistics revealed relatively low engagement with Copilot features. On average, each user performed just over one Copilot action per working day (1.14 actions) out of 63 working days in the trial.
Applications like Word, Teams and Outlook saw the most use, while Loop and OneNote usage remained below 3% daily. PowerPoint and Excel usage peaked with about 7% of license holders utilising Copilot features on any given day.
Positive user satisfaction
Despite modest usage rates, user satisfaction was generally positive. Overall, 72% of participants expressed satisfaction or high satisfaction with M365 Copilot, and many were disappointed when the trial ended.
The trial yielded a Net Promoter Score (NPS) of 31, considered a good rating, indicating that many users appreciated the tool for routine administrative activities and felt it allowed them to redirect time toward more strategic or valuable work, such as training or breaks.
However, the trial highlights concerns about cost-effectiveness. With commercial M365 Copilot licenses ranging between £4.90 and £18.10 per user per month depending on the business plan, the expense for extensive government deployment could become substantial considering the relatively limited use observed.
The Department for Business and Trade is continuing to analyse environmental impacts and return on investment to better understand the tool’s overall value.
“The evaluation findings will feed into a series of internal recommendations and next steps. Further evaluation is needed to assess the value for money and environmental impact of M365 Copilot usage in the department,” the report says.