National Disability Insurance Agency staff are using machine learning to help create draft plans for NDIS participants, documents obtained by Guardian Australia reveal.<\/p>\n
Documents related to NDIA\u2019s use of AI released under freedom of information laws showed 300 staff participated in a six-month trial of Microsoft\u2019s Copilot AI from January last year.<\/p>\n
The agency said Copilot uses generative AI, which was only utilised for the NDIA\u2019s emails, meetings and other non-client-facing tasks \u2013 not for participant plans.<\/p>\n
But the documents reveal before the Copilot trial began, the NDIA was already using a form of AI \u2013 machine learning \u2013 to prepare draft budget plans for participants.<\/p>\n
Machine learning was defined as \u201ca subset of AI that involves the use of algorithms to learn from data and make predictions or decisions without being explicitly programmed\u201d.<\/p>\n
The NDIA stated that NDIS staff made all final decisions on plans and its AI policy document from April 2024 stated that \u201cAI tools must not access participant records\u201d unless otherwise expressly authorised by the chief information officer and authorised under the NDIS Act.<\/p>\n
The briefing document, which was prepared for Senate estimates 2023-24, read: \u201cWhile machine learning is utilised within draft budgets (or Typical Support Package) for first plans based on key information from a participant\u2019s profiles.<\/p>\n
\u201cThe algorithm is only ever used to make recommendations, with decisions made by actual delegates.\u201d<\/p>\n
The documents continued that \u201cthe machine learning recommendations are used to assist delegates by speeding up the initial analysis to provide quicker resolutions for participants and improved service\u201d.<\/p>\n
Sign up: AU Breaking News email<\/a><\/p>\n In a report in June 2024, it stated staff had experienced improved productivity in preparing documents and emails by \u201cinterpreting NDIA policies and generating a summary of the purpose\u201d.<\/p>\n NDIA staff overall reported a 20% reduction in task completion times during the Copilot trial and a 90% satisfaction rating \u2013 including hearing-impaired staff who reported positively of the use of live-transcription during meetings.<\/p>\n The report noted difficulties facing the trial included staff concern about the findings of the robodebt royal commission on automated decision-making, and concerns about AI being used to reduce staff numbers.<\/p>\n The end-of-trial report notes that one of the risks of using Copilot was accidental data exposure, but the agency said it would have robust access controls, regular audits, and training for employees.<\/p>\n Dr Georgia Van Toorn, from the University of New South Wales, who has written about the impact of algorithmic decision-making in the public sector, said machine-learning and data-driven approaches often fail \u201cin dealing with complexity and nuance\u201d.<\/p>\n \u201cI don\u2019t think it\u2019s necessarily a bad thing, especially for cases that are relatively straightforward, but \u2026 we can\u2019t expect a machine-learning approach to be able to predict the types of support someone will need if that person doesn\u2019t fit neatly into a box. And that\u2019s most people, right?\u201d<\/p>\n Van Toorn added that machine learning also had a \u201cblack box\u201d problem \u2013 in which it is hard for humans to know which data points the machine is using, what weight it is giving to them and what biases it is assuming \u2013 as it learns and makes decisions.<\/p>\n \u201cI think there\u2019s an assumption that because it\u2019s data-driven, it\u2019s accurate and personalised \u2026 But in this case, I think that the crucial part is the human in the loop needs to understand the limitations [of the technology] \u2026 and exercise their discretion and judgment.<\/p>\n skip past newsletter promotion<\/a><\/p>\n Sign up to Breaking News Australia<\/p>\n Get the most important news as it breaks<\/p>\n Privacy Notice: Newsletters may contain information about charities, online ads, and content funded by outside parties. If you do not have an account, we will create a guest account for you on theguardian.com<\/a> to send you this newsletter. You can complete full registration at any time. For more information about how we use your data see our Privacy Policy<\/a>. We use Google reCaptcha to protect our website and the Google Privacy Policy<\/a> and Terms of Service<\/a> apply.<\/p>\n after newsletter promotion<\/p>\n \u201cAnd they need to be properly trained and supported to do so at the right moment.\u201d<\/p>\n Van Toorn said the fact the NDIA documents explicitly state that decisions on support plans are made by humans was important.<\/p>\n However, she cautioned that there is a lot of evidence for what researchers call \u201cautomation bias\u201d \u2013 where people are influenced by AI recommendations when making decisions.<\/p>\n \u201cThere might be time constraints or pressures on planners to get through a certain amount of plans to meet KPIs, or there might be pressures on the NDIA generally to be reducing the number or cost of plans,\u201d she said.<\/p>\n \u201cThe risk is that if it makes their job quicker or easier, a planner might be more likely to go along with the recommended plan, leaning on the algorithm instead of using their own judgment or really listening to NDIS participants.\u201d<\/p>\n \u2018These things are so serious and so significant\u2019 \u2026 disability advocate Stevie Lang Howson<\/p>\n Dr Stevie Lang Howson, an NDIS participant and disability advocate, said his \u201cbiggest concern\u201d was whether staff were trained, equipped and given time to \u201cmeaningfully make our plans suit our needs as individuals\u201d.<\/p>\n \u201cThese are actually people\u2019s lives. They\u2019re how many times people are able to get to the bathroom \u2026 it\u2019s how often you\u2019re able to leave our house, it\u2019s whether the wheelchair that you\u2019re sitting in is too small and causes you pain \u2026 These things are so serious and so significant they need to be made with care and transparency and in a way that reflects people\u2019s individual needs.\u201d<\/p>\n A spokesperson for the NDIA said AI was not used in systems \u201cthat interact directly with participants or providers or for any decisions on NDIS funding or eligibility\u201d.<\/p>\n \u201cDelegates make decisions on a participant\u2019s NDIS funding using information and evidence provided by participants in accordance with the NDIS Act,\u201d the spokesperson said.<\/p>\n