An attempt by the Australian Bureau of Statistics to cut costs and corners led to the “shocking” and last-minute cancellation of a key survey that would have cast light on how households have navigated the cost-of-living crisis.

Newly released ABS briefing documents to Andrew Leigh, the assistant minister for Treasury, reveal how the Canberra-based agency scrambled but ultimately failed to salvage its long-awaited survey on income and housing for 2023-24, which had been due for release in July.

The survey of about 30,000 people provides the only comprehensive statistics on household income and wealth and housing occupancy and costs.

Thanks to the pandemic, the survey was last completed for the 2019-20 financial year. Now the ABS is now looking to release the data for 2025-26 in mid-2027 – a full five years after the last report.

Sign up: AU Breaking News email

Ben Phillips, an associate professor in economics and public policy at ANU, said the failure to release the survey data had eroded trust in the agency.

“Everyone is a bit shocked. The ABS have not done a good job on this at all.”

“We are missing a lot of good data on two of the biggest issues facing the country,” in the housing and cost-of-living crises, he said.

“We have aggregate-level data, but we don’t know how it is impacting a lot of important populations, like low-income households and single parents. We don’t really know what’s happening with income inequality. We are not increasing jobseeker, and this survey would be useful whether that is causing problems or not.”

The briefing document sent to Leigh on 15 July was published on Monday after the independent senator Fatima Payman moved for an order of production of documents in late August, and comes ahead of a review the ABS intends to publish on Friday.

Statistical bodies around the world are finding it harder and more costly to survey their citizens.

The ABS for the 2023-24 survey “implemented several innovations … to improve response rates and reduce the cost of data collection”.

“We also took steps to reduce the survey content to make it easier and faster for households to complete the survey,” the briefing document says.

These steps included asking Australians to do the survey online, rather than sending somebody out in person.

skip past newsletter promotion

Sign up to Breaking News Australia

Get the most important news as it breaks

Privacy Notice: Newsletters may contain information about charities, online ads, and content funded by outside parties. If you do not have an account, we will create a guest account for you on theguardian.com to send you this newsletter. You can complete full registration at any time. For more information about how we use your data see our Privacy Policy. We use Google reCaptcha to protect our website and the Google Privacy Policy and Terms of Service apply.

The ABS also believed it could streamline the surveys by replacing questions with administrative data, such as on superannuation balances.

The whole survey design process was rushed, the ABS said, which meant forms were dispatched six months after the survey was “stood up” in December 2022, rather than the more typical 18 months, which would have allowed for more testing.

In particular, there wasn’t time to test the “extraction” process for the administrative data the agency hoped would replace the survey questions – which proved more complicated and was only solved at the end of last year.

The more “streamlined” survey also meant respondents were able to skip important questions, including about their mortgages.

In early 2025 they then began the “weighting” process to make the survey sample of about 30,000 respondents nationally representative.

It became clear there were issues, and that the sample remained biased towards poorer households.

The results were not “credible”, the ABS concluded, as the data suggested household incomes were lower in 2023-24 than they were in 2019-20: a conclusion which failed to match up with every other piece of data.