Word of the Year and Math Discoveries of LLM

In the New Year’s Eve issue of the #InfocusAI digest, we’ll talk about a synaptic transistor that doesn’t require cryogenic temperatures, and a development from Google DeepMind that enabled the first mathematical discovery using LLMs. You will also learn about Gramota.ru’s word of the year, Stanford University’s predictions for 2024, and Apple’s open source LLM.

Issue 32, December 14-28, 2023

Scientists at Northwestern University, Boston College and MIT have developed a new synaptic transistor with simultaneous processing and storage of information, much like in the human brain. This is more energy efficient than moving data back and forth between microprocessor and memory as is done in conventional computing systems. The idea of neuromorphic computing devices is not exactly new. However, scientists say that previous designs couldn’t function outside of cryogenic temperatures. Their invention, on the other hand, works stably at room temperature with faster computing speeds, less power consumption and retaining stored information even when power is taken off. The latest advances in the physics of moiré patterns have helped to achieve this. These are a type of geometrical design where two patterns overlap. “When two-dimensional materials are stacked, new properties emerge that do not exist in one layer alone. And when those layers are twisted to form a moiré pattern, unprecedented tunability of electronic properties becomes possible,” as explained in the Northwestern University publication. The combination of bilayer graphene and hexagonal boron nitride allowed the scientists to achieve the required properties for efficient operation of the transistor at room temperature. The invention is described scientifically in Nature, and in simpler terms — on the Northwestern University website.

Researchers at Google DeepMind have developed a method to apply LLMs for making discoveries in maths and computer sciences, according to the developer’s website. The fact is that large language models are prone to confabulations, or hallucinations, which can cause them to make plausible but fundamentally incorrect statements. This is a significant limitation to their application in science. The FunSearch method proposed by Google DeepMind avoids this limitation by combining LLMs with an automated evaluator. First the pre-trained language model generates new solutions for certain tasks and presents them as computer code, then the evaluator filters out any solutions with a hint of confabulations. Alternating between these two components allows scientists to get new knowledge from initial solutions. The researchers applied FunSearch to the cap set problem and immediately discovered new solutions to this long-standing combinatorics problem. Essentially, this is the first discovery in mathematics made by LLMs. Here’s an article in Nature (not yet edited) with scientific details. 

For the first time, Gramota.ru held its own Word of the Year campaign and selected “нейросеть” (neural network) as the main word of the passing 2023 in Russian. The choice was made based on data analysis, query frequency and expert judgement. Its closest competitors were “джипити” (GPT), “база” (base), and “возвращенцы” (returnees). Gramota.ru notes that recently the word “нейросеть” has become widespread and has even acquired diminutive forms and other derivatives, but has not yet entered the normative dictionaries. At the same time, in the everyday sense, it began to be used to name the human-facing side of AI, as a neural network is now perceived as a living being that can be taught and with which you can communicate. Read more interesting facts about the word of the year here. By the way, the words of the year in English were also chosen from the artificial intelligence field. Collins Dictionary has chosen “AI”, and Cambridge Dictionary — “hallucinate” (when used in relation to AI, it means to produce false/incorrect information). 

At the end of the year, Stanford HAI faculty members and scholars have published their expectations for 2024 regarding the development and impact of artificial intelligence. In particular, they predict the emergence of new multimodal models for video generation and, as a consequence, the proliferation of deepfakes, especially during election campaigns. The scientists also foresee a shortage of GPUs, which may spur the development of innovative hardware solutions that are cheaper and easier to manufacture. And of course, the conversations about AI regulation will be around for a long time to come. All predictions are summarised here. 

And to finish, the news we missed. Or, rather, not only us…  In October, Apple together with Columbia University quietly released an open source multimodal language model called Ferret, VentureBeat reports. For now, the model is open for researchers only. In other words, a company famous for the closed nature of its developments has joined an open source AI landscape. 

VentureBeat notes that many members in the AI community celebrated Apple’s unexpected entry into the open source LLM sphere, albeit belatedly. Specifically, they’re citing the post made by a tech blogger Ben Dickson, where he wrote that this was the least expected development of the year for him. Although, the expert then explains that for Apple, entering the LLM market with an open source model makes sense. “To compete with models like ChatGPT, you either need to be a hyperscaler or be in partnership with one. Apple might have a lot of resources, but its infrastructure is not built to serve LLMs at scale. The alternative would be to become dependent on a cloud provider like Microsoft or Google (both bitter enemies) or start releasing its own open source models, a la Meta*,” VentureBeat quotes him. Speaking of which, you might be interested in reading another article on this portal — about Apple’s latest breakthrough AI developments.  

*Meta is recognised as extremist and banned in the Russian Federation.