SAN DIEGO \u2014 For the past week, academics, startup founders and researchers representing industrial titans from around the globe descended on sunny San Diego for the top gathering in the field of artificial intelligence.<\/p>\n
The Neural Information Processing Systems<\/a>, or NeurIPS, conference has been held for 39 years, but it drew a record-breaking 26,000 attendees this year, twice as many as just six years ago.<\/p>\n Since its founding in 1987<\/a>, NeurIPS has been devoted to researching neural networks and the interplay among computation, neurobiology and physics. While neural networks, computational structures inspired by human and animal cognitive systems, were once an esoteric academic fixation, their role underpinning AI systems has transformed NeurIPS from a niche meeting in a Colorado hotel to an event filling the entire San Diego Convention Center \u2014 also home to the world-famous Comic-Con.<\/p>\n But even as the gathering boomed along with the AI industry and sessions on hyperspecific topics like AI-created music<\/a> proliferated, one of the buzziest points of discussion was basic and foundational to the field of AI: the mystery around how frontier systems actually work.<\/p>\n Most \u2014 if not all \u2014 leading AI researchers<\/a> and CEOs readily admit<\/a> that they do not understand how today\u2019s leading AI systems function. The pursuit of understanding models\u2019 internal structure is called interpretability, given the desire to \u201cinterpret\u201d how the models function.<\/p>\n Shriyash Upadhyay, an AI researcher and co-founder of an interpretability-focused company called Martian<\/a>, said the interpretability field is still in its infancy: \u201cPeople don\u2019t really understand fully what the field is about. There\u2019s a lot of ferment in ideas, and people have different agendas.\u201d <\/p>\n \u201cIn traditional incremental science, for example, where the ideas are mostly settled, scientists might attempt to add an additional decimal point of accuracy of measurement to particular properties of an electron,\u201d Upadhyay said. <\/p>\n \u201cWith interpretability, we\u2019re in the phase of asking: \u2018What are electrons? Do electrons exist? Are they measurable?\u2019 It\u2019s the same question with interpretability: We\u2019re asking, \u2018What does it mean to have an interpretable AI system?\u2019\u201d Upadhyay and Martian used the NeurIPS occasion to launch a $1 million prize <\/a>to boost interpretability efforts.<\/p>\n As the conference unfolded, leading AI companies\u2019 interpretability teams signaled new and diverging approaches to understanding how their increasingly advanced systems work. <\/p>\n Early last week, Google\u2019s team announced a significant pivot<\/a> to shift away from attempts to understand every part of a model to more practical methods focused on real-world impact.<\/p>\n Neel Nanda, one of Google\u2019s interpretability leaders, wrote in a statement that \u201cgrand goals like near-complete reverse-engineering still feel far out of reach\u201d given that \u201cwe want our work to pay off within ~10 years.\u201d Nanda highlighted AI\u2019s rapid progress and lackluster advancements on the teams\u2019 previous, more \u201cambitious reverse-engineering\u201d approach as reasons for the switch.<\/p>\n On the other hand, OpenAI\u2019s head of interpretability, Leo Gao, announced Friday<\/a> and discussed at NeurIPS that he was doubling down on a deeper, more ambitious form of interpretability \u201cto fully understand how neural networks work.\u201d<\/p>\n Adam Gleave, an AI researcher and co-founder of the FAR.AI<\/a> research and education nonprofit organization, said he was skeptical of the ability to fully understand models\u2019 behavior: \u201cI suspect deep-learning models don\u2019t have a simple explanation \u2014 so it\u2019s simply not possible to fully reverse engineer a large-scale neural network in a way that is comprehensible to a person.\u201d<\/p>\n Despite the barrier to complete understanding of the complex systems, Gleave said he was hopeful that researchers would still make meaningful progress in understanding how models behave on many levels, which would help researchers and companies create more reliable and trustworthy systems. <\/p>\n \u201cI\u2019m excited by the growing interest in issues of safety and alignment in the machine-learning research community,\u201d Gleave told NBC News, though he noted that NeurIPS meetings dedicated to increasing AI capabilities were so large that they \u201ctook place in rooms that could double as aircraft hangars.\u201d<\/p>\n In addition to uncertainties in how models behave, most researchers are unimpressed with current methods for evaluating and measuring AI systems\u2019 current capabilities.<\/p>\n \u201cWe don\u2019t have the measurement tools to measure more complicated concepts and bigger questions about models\u2019 general behavior, things like intelligence and reasoning,\u201d said Sanmi Koyejo, a professor of computer science and leader of the Trustworthy AI Research Lab at Stanford University. <\/p>\n \u201cLots of evaluations and benchmarks were built for a different time when researchers were measuring specific downstream tasks,\u201d Koyejo said, emphasizing the need for more resources and attention to create new, reliable and meaningful tests for AI systems.<\/p>\n The same questions about what aspects of AI systems should be measured and how to measure them apply to AI models being used for specific scientific domains.<\/p>\n Ziv Bar-Joseph, a professor at Carnegie Mellon University, the founder of GenBio AI<\/a> and an expert in advanced AI models for biology, said evaluations of biology-specific AI systems are also in their infancy. <\/p>\n \u201cIt\u2019s extremely, extremely early stages for biology evaluations. Extremely early stages,\u201d Bar-Joseph told NBC News. \u201cI think we are still working out what should be the way we evaluate things, let alone what we should even be studying for evaluation.\u201d<\/p>\n Despite incremental and halting progress in understanding how cutting-edge AI systems work and how to actually measure their progress, researchers nonetheless see rapid advancements in AI systems\u2019 ability to enhance scientific research itself. <\/p>\n \u201cPeople built bridges before Isaac Newton figured out physics,\u201d said Upadhyay, of Martian, pointing out that complete understanding of AI systems is not necessary to unleash significant real-world change.<\/p>\n For the fourth year in a row, researchers organized an offshoot of the main NeurIPS conference to focus on the latest AI methods to boost scientific discovery<\/a>. One of the event\u2019s organizers, Ada Fang, a Ph.D. student studying the intersection of AI and chemistry at Harvard, said this year\u2019s NeurIPS edition was \u201ca great success.\u201d<\/p>\n \u201cFrontier research in AI for science is happening separately in areas of biology to materials, chemistry and physics, yet the underlying challenges and ideas are deeply shared,\u201d Fang told NBC News. \u201cOur goal was to create a space where researchers could discuss not only the breakthroughs, but also the reach and limits of AI for science.\u201d<\/p>\n