San Francisco’s AI economy is mostly being defined by the companies spending the most. Foundation model labs raise billions, run training clusters that draw the power of mid-sized data centers, and operate at significant losses for the foreseeable future. The standard playbook assumes that the path to building something foundational runs through years of heavy capital burn before any revenue catches up. A quieter story is taking shape underneath that one. Some of the city’s AI companies aren’t building the models. They’re building the layer beneath the models, the infrastructure that determines whether AI systems can actually use the real world.
According to Cloudflare’s Q1 2026 data, AI crawlers now generate 22% of all bot traffic on the web, the fastest-growing category Cloudflare tracks. Dedicated AI training crawlers crossed the 50% threshold of all AI bot traffic in the first quarter of 2026, a full quarter ahead of forecast. Behind those numbers are automated agents trying to read websites that were built for humans, and a meaningful share of those attempts come back with incomplete or unusable data.
The bottleneck nobody noticed
The problem is straightforward but easy to underestimate until you hit it. Modern websites are rendered by JavaScript that has to execute before any content shows up. Information lives behind cookie banners, dropdown menus, paywalls, and login walls. Anti-bot systems designed to stop denial-of-service attacks treat legitimate AI agents the same way they treat malicious ones. Layouts change frequently and silently break the extraction pipelines AI products depend on.
For developers building agents, meaning the autonomous AI systems that book flights, monitor competitor pricing, answer customer support questions, and pull live data into applications, this is the operational reality. The model itself is rarely the limiting factor. Getting the model the right information from the live web, at scale, in a form it can actually use, is.
One of the San Francisco companies emerging as a central piece of this layer is Firecrawl. The platform handles the work of getting AI agents access to the live web, exposing search, scraping, and interaction capabilities through a single API call. Adoption has been rapid. Almost a million developers have signed up since the company launched, and the open-source repository has crossed 120,000 stars on GitHub, making it the most-starred project in its category. Apple, Canva, and Lovable run Firecrawl in production. The company closed a $14.5 million Series A in August 2025, led by Nexus Venture Partners, with Shopify CEO Tobi Lütke participating after first becoming a customer of the platform. The core utility is that an agent built on Firecrawl doesn’t need engineers writing custom extraction code for every site it touches. The platform absorbs the rendering, the navigation, and the interaction, and returns structured content the model can immediately use.
“Every AI company needed clean web data and nobody was solving it well,” says Eric Ciarla, one of Firecrawl’s cofounders. “So we built Firecrawl.”
The pattern that built the company
The pattern that became Firecrawl showed up first at the cofounders’ previous company. Mendable was the AI search product they ran before Firecrawl, with Snapchat, Coinbase, DoorDash, and MongoDB as enterprise customers. The search experience worked. The thing that didn’t work was everything underneath it. Each new customer integration meant standing up custom extraction code for that customer’s website, and the code broke the next time the customer redesigned the site. The team spent more time maintaining scrapers than building the product they actually wanted to ship.
That experience surfaced a recurring industry pattern. Every AI company integrating with web data, and increasingly every AI company that wanted to be useful, was rebuilding the same brittle tools in-house, badly, over and over. The bottleneck wasn’t model quality. It was data access.
Firecrawl emerged from that realization, designed around three capabilities the team had needed at Mendable and never been able to find. The first is search: finding relevant information across the live web in a form an agent can use, rather than getting back a list of links to chase down. The second is scraping: taking a page that was built for human eyes and turning it into structured data a model can read cleanly. The third is interaction: clicking buttons, submitting forms, navigating multi-step flows, and reaching information that lives behind a single static URL.
That third capability is what separates a basic scraping tool from infrastructure, and it’s where most of the useful information on the modern web actually lives. An insurance company deploying an AI chatbot to answer customer questions needs the bot to know what its policies actually say right now, not what they said when the underlying model was last trained. Firecrawl is the layer that goes out, finds the policy documents on the company’s own site, reads them the way a person would, and hands the information back to the AI in a form it can use to answer the customer. The same pattern repeats across most AI products moving into production: a model plus a system around it that gets it the information the user actually cares about.
How AI is changing who finds your business
Engineering teams aren’t the only ones affected. The way customers find businesses online is in the middle of a structural shift. For two decades, almost every digital path from “a customer is looking for something” to “a customer finds your business” went through Google. Sites were optimized for search rankings, content was built for human readers, and the rest of the flow was familiar. That model is fracturing. AI assistants from OpenAI, Anthropic, and others increasingly handle the front end of that journey, doing the searching and the page-reading on the human’s behalf and returning a synthesized answer. If the assistant couldn’t reach a particular site, the human never learns the business exists.
This reshuffles how businesses should think about AI crawlers, and Ciarla is direct about it. “Behind every AI agent is a human trying to find something,” he says. The standard industry posture toward AI crawlers still treats them the way nuisance bots were treated a decade ago: as drains on server resources, as automated traffic with no human attached. The framing made sense when search engine bots were the only things reading sites at scale and the resulting human traffic came through links the bots produced. It makes much less sense when the AI agent reading the site is the customer’s path to discovering the business.
Ciarla compares the current pattern to blocking Google in 2005. The defensive instinct cuts businesses off from a channel their customers are already migrating to, often without anyone in the building noticing.
The structural difference in Firecrawl’s approach to this shift is that businesses don’t have to participate in it. Most other approaches to AI visibility ask the site owner to do work: implement structured data, expose new endpoints, restructure pages, take on a new optimization discipline on top of the existing SEO one. Firecrawl operates from the opposite side of the transaction. The platform sits between the agent and the live site and converts the human-oriented presentation into machine-readable output in real time. The site owner doesn’t have to know that AI agents are reading the page. The agents read it correctly anyway.
The ecosystem question
The Cloudflare data that captures the explosive growth of AI crawler traffic captures something else underneath it: the leading AI crawlers today pull thousands of pages for every single human visitor they send back to a publisher, with some operators measured at ratios in the tens of thousands. A model where AI extracts value from web content at that ratio, without anything flowing back to the people who created it, isn’t durable. The pushback is already underway, through lawsuits from major publishers and access blocks from sites that have decided the cost of being crawled exceeds the benefit of being indexed.
A small but telling counter-example landed in March 2026, when Firecrawl restructured its entire relationship with Wikipedia. Instead of continuing to scrape the public site directly, the company signed an enterprise agreement with Wikimedia and now routes the 2 to 3 million requests its agents make to Wikipedia each month through Wikimedia’s paid commercial APIs. The arrangement is structured to support the volunteer community maintaining Wikipedia rather than draining the foundation’s infrastructure to feed AI systems.
“The community members who write and edit these articles hold immense power in the age of AI,” Ciarla said when the partnership was announced. “They are providing the essential service of defining what is true. We want to ensure our infrastructure supports their work rather than just consuming it.”
The Wikimedia arrangement is the most public example yet of Firecrawl pursuing this kind of structured relationship, and more are likely to follow. The companies building the infrastructure that AI agents rely on are also, by default, helping shape the economics of an AI-mediated web: what gets crawled, what gets paid for, and which sources of human-created information survive long enough to keep training the next generation of models.
What it means that this got built in San Francisco
San Francisco has plenty of AI companies. The interesting thing about Firecrawl is what kind of AI company it is. The city’s AI story has been told mostly through the foundation model labs: the headline capital raises, the billion-dollar valuations, the training runs that consume their own power grids. Underneath those companies, a different kind of AI infrastructure is being built on much smaller budgets by teams that often have to figure out what the foundation model labs actually need before those labs know themselves.
The bottleneck Firecrawl spotted in 2022, when it was an internal frustration at a Y Combinator startup most people had never heard of, turned out to be the same bottleneck a wave of AI companies integrating with the live web would hit a year or two later. The foundation model labs and the infrastructure layer are now in a strange kind of mutual dependency. The models keep getting better, but the advantage from that improvement increasingly accrues to whoever can give the model the right context at the moment it needs it. Cloudflare can measure the bot traffic. The labs can measure model performance. The gap between the two is the layer that decides what an AI system can actually reach on the live web. That’s the company Firecrawl is building, and it’s being built here.
*The San Francisco Weekly newsroom and editorial were not involved in the creation of this content.