After her wedding service in 2003, Mandy Sayer stopped traffic. Or at least the musicians leading her and her new husband, Louis Nowra, through the streets of Kings Cross did. “Saxophones wailing, tambourines jingling, drums booming, even managing to pick up one or two rough sleepers along the way,” she writes in her latest memoir, No Dancing In The Lift.

Three years earlier, in 2000, the same scenario had played out at the funeral of her jazz musician father, Gerry. The congregation followed the hearse down Darlinghurst Road, “all playing percussion instruments to the saxophonist’s fast blues,” she tells Guardian Australia.

Important occasions require a band of musicians.

Mandy Sayer’s wedding day in 2003 in Kings Cross, Sydney. Photograph: Roslyn Sharp

Gerry believed he and his daughter were so artistically alike because she was conceived after he had swallowed a block of hash. “He always fondly referred to me as his hash baby,” Sayers says. He couldn’t read or write: “He could write his name in shaky capital letters, but that was about it. But he was a brilliant storyteller.”

Gerry pursued his music with evangelical fervour, practising for hours every day, sacrificing family and “relinquishing the bright and shiny chance at uncomplicated unhappiness for the longer odds of creative contentment,” as Sayer writes in her new memoir, which is addressed directly to Gerry. As a little girl, Sayer would sit between his legs as he played Mingus or Coltrane, “lightly tapping against my skull, literally drumming your rhythms into my head. In the beginning, to me, music and language were one. I heard stories in saxophones, and syncopation in sentences.”

“I think any artist has to make a choice,” Sayer says now. “How much are you willing to risk? How important is it to you? What are you willing to not have in your life?”

Mandy and Gerry Sayer performing in Chinatown, Sydney, 1980. Photograph: Supplied by Mandy Sayer

Sayer has managed to achieve an artistic career that creates great beauty from hard times. Her parents separated by the time she was eight, after Gerry realised during an LSD trip that he had married the wrong woman. Although she still saw her father occasionally, he was largely absent until she sought him out as a teenager: “He was so wonderful and warm, I kept returning to him.”

Sayer, her brother, Gene, and sister, Lisa, were left with their mother, Betty, who was descending into chronic alcoholism and entered relationships with dangerous men; one of them was so violent the family had to go on the run. Sayer’s childhood was spent on the move, living in rooms in pubs and dosshouses.

“Betty’s inability to express her rage directly resulted in an extended performance of passive-aggressive behaviour,” she writes. “By day, she worked as a live-in housekeeper and nanny; by night, she got drunk and capriciously threw herself into the arms of a bookie, then a sports journalist, then an undertaker.”

“She was so beautiful and sad,” Sayer says of her mother. “I used to write poetry daily to cheer her up, and it worked, even if only for a few hours.”

No Dancing in the Lift is the fourth in a quartet of memoirs; there was also Velocity (2005), which covered Sayer’s childhood, and The Poets Wife (2014), which spanned the 10 years she spent with African American poet Yusef Komunyakaa, as he grew in fame and won a Pulitzer.

But her first, Dreamtime Alice (1998), which won the National Biography award, covered the three itinerant years in the early 80s she and Gerry spent busking in New York, New Orleans and Colorado. She was 21 and he was 62 when they started; she would tap dance to the beat of his drums. Father and daughter “almost turned breaking the law into a performance art: busking in New York City without a permit and dodging debt collectors by frequently moving hotels.”

‘I know how to hustle on the streets of New York City’ … Mandy Sayer today. Photograph: Supplied by Mandy Sayer

This period of her life “certainly made me the person I am today,” she says now. “Whenever someone in the industry is trying to mess with me, I’m thinking, ‘Look, you’ve got no chance. I know how to hustle on the streets of New York City.’ You cannot grift a grifter.”

For decades, Kings Cross was Sayer’s habitat; she was backlit, even in the daytime, by the garish fluorescent neon signs on every storefront, showgirls, strippers, eccentrics, grifters walking in the pulsating streets. “It was theatre,” she says, “and a diverse, accepting community.” Sayer was an ethereal figure wandering among all this with her long blonde hair, her flowing vintage dresses, and large picture hats. The now defunct Bourbon and Beefsteak bar “was my living room because it was just across the street”.

Gone now, all of it gone. Even Sayers has defected “over the border” to Darlinghurst.

Mandy and Gerry Sayer performing during Mardi Gras, on Bourbon Street, 1982. Photograph: Supplied by Mandy Sayer

For years, Gerry lived in single rooms above pubs in The Rocks, with his beers, weed and flutters on the horses. When he was diagnosed with inoperable lung cancer, Sayer became his devoted carer. He moved between the hospice and her home; she would carefully roll his joints the way she had been instructed from an early age – trying to recreate, she writes, “the many beer gardens of my childhood”. In sickness, he was sometimes angry and belligerent, and sometimes his hilarious, warm, affectionate self. After a Lear-like outburst at a nurse, Sayer told him that being terminally ill did not give him the right “to act like an arsehole”.

He was a flamboyant sight at the hospice, wearing his daughter’s bright pyjamas and her Chinese satin embroidered robe, which made him look like a “wise old sage”.

“He enjoyed shocking people out of their inertia,” she says.

He had his opinions on her suitors, too. His face lit up when she told him she was going to dinner with Nowra, a writer, playwright, screenwriter and librettist. Nowra was married to someone else at the time; nevertheless, Sayer overheard Gerry admonishing Nowra during a visit to the hospice: “What’s wrong with my daughter? Why haven’t you made a pass at her?”

Sayer and Nowra had been introduced to produce an anthology of Kings Cross. They were drinking wine one afternoon, with the photocopied excerpts laid out in rows on her living room floor, when suddenly they were kissing. “His head was lying against an excerpt of George Johnson’s Clean Straw For Nothing, my arm elbowed an extract of Aunts Up The Cross by Robin Dalton,” she writes. “Soon we were lying on Patrick White, Betty Roland and Kenneth Slessor, paper sheets beneath us instead of cotton ones.”

Her father “knew Louis would be a good person to be with me after his death. He knew that Louis was a good man and he wouldn’t hurt me,” Sayers says. Later, she texts me: “All [Gerry] really wanted me to do is to keep writing and being myself. He was concerned that some arsehole would come along and interfere with that process.”

The cover of No Dancing in the Lift by Mandy Sayer. Photograph: Transit Lounge Publishing

Sayer was swimming in the sea when the voice of the book came to her. “I could hear the rhythms of the prose, and I could hear the point of view, which was me talking to him. I was following his voice, a bit like the Pied Piper.” Then she found the memoirist’s mother lode when she was clearing out a storage unit: “I’d forgotten that I’d stored every scrap of paper, every coaster and bar bill I had written on and put in this box. My diaries, medical bills, appointment books. It was this great gift.”

Sayer realises now, “one way or another”, that Gerry is threaded through all of her work, memoir and fiction. Though he couldn’t read or write, his “sublime” storytelling had “inspired my imagination”. He had unwittingly influenced her work with his “flair for improvisation, both in music and in life”.

No Dancing in the Lift is not a grief memoir, as such. There is enough distance for absurdities and honesty to shine through. “He was a very funny man,” Sayer says. “Sometimes when I was writing, it was making me laugh so much that I scared my chihuahua. I almost felt euphoric. And I realised, it was because I was spending every day with my father.”