One morning a few years ago a broker showed me into a nice apartment building in midtown Manhattan. She opened the door to a flat and we stepped past a narrow slot of a kitchen, into a single room that was dominated by a giant pink bed. It was just about within our budget.

“It has a good-sized bathroom,” said the broker.

I made some appreciative noises about this. Then I said: “So, um, do you think we could get our children in here?”

“Your children?” the broker said.

Brokers are an optimistic bunch. They have to be. They only need one person to sign on the dotted line to make a quick five or ten thousand dollars, depending on their fees. There must be a natural inclination to look at each new punter as single, financially stable and sane.

The broker looked at me, aghast. “How many children do you have?” she asked.

“Two,” I said. I could see she was furious at me. “They’re quite small,” I added, trying to be helpful.

This did not help one bit. She shook her head at me and hustled me out of the flat and thanked me for wasting her morning. I think of her sometimes as I step over the detritus left by the rampages of our now three children.

You are allowed to have children in New York City. But having three usually feels like an extravagant and reckless enterprise: like heli-skiing or crypto-trading or bicycling in Midtown. Other New Yorkers say: “This is quite the circus.” Or “Well, you’ve got your hands full.” Then they step carefully out of range.

Will Pavia offering his three sons ice cream at Anita La Mamma de Gelato in New York.

The Pavias live in “one of the last relatively affordable parts of the Upper East Side”

TIMOTHY FADEK FOR THE TIMES

It is a niche pursuit. Proof of this comes in an analysis by the Center for an Urban Future, which shows the city steadily losing larger families over the past decade. The number of one-child families has grown slightly, the pool of two-child families is about the same and the three-childers have dropped by 17 per cent.

The centre suspected it had something to do with the relatively small number of places in New York with three or more bedrooms, the astonishing sums it costs to rent one of them and the fact you are up against young, single professionals house-sharing and splitting the rent.

A report about this study went ricocheting through the neighbourhood where I live, one of the last relatively affordable parts of the Upper East Side, where quite a lot of families have three kids. People sent it to us, to commiserate.

The truth is you have to ignore all this “three or more bedrooms” business if you want to have three children in New York. The median rent for three bedrooms or more in 2015 was $3,500. It is now $4,800 city-wide, according to Streeteasy; in Manhattan it’s $7,200.

Meet Zohran Mamdani, the man who promises to make NYC affordable

So you find a smaller place, and then have a very awkward conversation about whether your children might fit in there. Eventually, an agent with a refreshingly can-do attitude told us about a building of railroad apartments on the Upper East Side where the rent was about $2,700. Railroads are long, narrow flats where the rooms are strung out in a line like train carriages.

“I think you could all get in there,” she said.

“What if we have a third child?” I asked a lawyer, before we committed.

“They can’t throw you out because you have more children,” he said.

Once you are in, you are free to make your life as complicated as possible. There were a few raised eyebrows from the neighbours, certainly. But the guy upstairs was very supportive. He grew up in the building when it was a tenement filled with Irish and Italian families. He was one of four children, in a one-bedroom flat. “It was fine,” he said.

My baby is destroying my home. I can’t decide if I should stop him

The other people who were very supportive were these families we met through one of my children’s soccer teams. “They’re amazing,” I said to my wife. “This one family has four children! In Manhattan! I was so impressed. But the mum explained that she grew up with five siblings, and her husband was one of ten…”

My wife nodded. “They are Mormon,” she said.

“Oh!”

The Mormons were really a breath of fresh air. I used to think it might look a bit odd to people that one of our children slept inside a cupboard.

I gave Mormon ‘dirty sodas’ to my children — what have I done?

One of our Mormon friends, Holly Greenburg, 44, said this was perfectly normal. They did this with her eldest, when she was pregnant with number two. The cupboard “was right by the front door”, she said. “He would wake up and hear you and, like, push the door open.”

But all this stuff — floor plans, bunk beds, children in cupboards, the queue for the bathroom — is not so different from the gripes of any other New Yorker, with no children underfoot. New Yorkers talk about real estate the way Londoners discuss Chelsea and Arsenal: who is up, who is down, who has more cupboards. You always want more of them, or at least more than the people next door.

Will Pavia and his three sons at an ice cream shop.

The expensive ice cream place near Pavia’s home, where a large portion can feed all three of his boys

TIMOTHY FADEK FOR THE TIMES

I worry, of course, that none of my children have any space to themselves, except in their bunk beds. One of them used to keep all of his stuff in his bed: when he posted himself into it at night, matchbox cars, stuffed animals, Nerf-gun bullets and plastic dinosaurs would shift and then settle around him. It was like watching a badger climb into a bin bag.

But the larger battles, with three, are the same in New York as they are everywhere else: getting your kids into a good school, do they have good friends, and most inescapably, what on earth are we going to do with them this afternoon?

Sometimes the Mormons seemed better prepared for it. They took turns to look after each other’s children, sometimes for days on end. We felt that this was quite impossible, with our three. They are all boys. You have to spend a lot of time breaking up fights. A hardened grandmother can manage it for four hours at the outside. Our own family would baulk at the idea of having them for more than an afternoon. We have no one we could leave them with, my wife said. One of the mums looked at her, quite seriously, and said: “You do now.”

Times writers on raising children: ‘I wish their childhoods had lasted ten times as long’

I asked Greenburg for some tips on how to get by in New York with three kids. There are the big things, obviously: childcare, schools, groceries (her husband drives weekly, with another dad, to a supermarket in New Jersey). But did we always equate children with spending money? Didn’t we used to think they would make us richer, somehow, in the best of all worlds?

“We do a lot of free stuff,” Greenburg said. They do children’s ballet afternoons; they do the free children’s races staged by the New York Road Runners club.

We have a couple of these too. For instance, at the Lego store, you can buy the right to build three Lego figures for $10. If you have three children, that does you quite nicely on a rainy Sunday. Here’s another. There’s an expensive ice cream place near us where the queue on summer nights stretches down the street. If you buy a large ice cream from there, it is $12. But it comes with three scoops. They can each choose a flavour.

Then for about ten minutes it is like Karl Marx’s utopia, at our table, “to each according to his needs”. Provided the smallest can get it inside him fast enough, before the other spoons start circling.