In 2023, investment institutions and shareholders are asking listed companies in the field of AI chips three deadly questions:
Have you made a storage and calculation integrated chip yet?
(Source of investors asking questions to Montage Technology: Tonghuashun Finance)
How is the progress of the memory-computing integrated chip?
(Investors ask Dongxin shares the source of the question: Tonghuashun Finance)
What is the prospect of integrated storage and computing chips?
(Source of investors’ questions to Hengshuo shares: Tonghuashun Finance)
The above series of questions means that the integration of storage and calculation no longer "stays" in the academic and industrial circles, and it seems to be popular in the secondary market: stockholders and brokerage institutions are working overtime to make up for the integration of storage and calculation, so traditional chip manufacturers from Since the beginning of 2023, the layout of the integration of storage and computing has been "tortured".
A natural question is why the integration of deposit and calculation has suddenly become so popular?
Huawei, which released the new storage product OceanStor A310 on 7.14, gave its own answer. Zhou Yuefeng, president of Huawei's data storage product line, said that there are many data problems in the training and reasoning of general-purpose large models and industry large models: slow cross-domain collection of massive data, low data interaction efficiency in preprocessing and training, and difficult data flow security.
The large AI model brings new challenges to storage. For example, the stability is worse than that of traditional AI. At the same time, there is a large amount of data preprocessing and preparation work, which requires near-storage computing (a type of storage and calculation) to systematically deal with this problem.
So, how much is the "large amount of data" here? According to the calculations of SI Rui Insight, ChatGPT currently has 100 million daily users. If 5% of people ask questions every second at the same time every day, it will eat up the power generation of the Three Gorges Dam for nearly a year.
In other words, large models are frantically demanding computing power, and emerging solutions represented by the integration of storage and computing can solve many challenges brought about by large models. In fact, in addition to placing strict requirements on computing power, the large model also gives AI chip manufacturers "a lot of sugar to eat".
This article attempts to explore, under the background of the large model, what benefits are given to AI chip manufacturers? What is the future pattern of storage and computing integrated manufacturers that have been "fired" by ChatGPT?
PART-01 "Fortune and misfortune depend on each other" AI chip manufacturers
The wind of the mockup is still blowing, and the discussion about the mockup is still in full swing.
At the beginning of August, at the sub-forum of the China Computer Federation (CCF) Artificial Intelligence Conference - "Seeing the Frontier of Kunlun AI Large-scale Model Computing Power", experts and scholars agreed that in the era of large-scale models, intelligent computing power has become a "scarce asset". , How to make good use of computing resources and maximize their benefits has become a challenge that every participant must face.
As for the amount of computing power demanded by GPT-3 alone, under conservative circumstances, the Three Gorges Dam needs to "work" all night to meet:
Assuming that there are 100 million people online at the same time, 5% of them ask questions every second at the same time every day, each occupying 30 tokens, and 15.3 million H100 chips are needed to provide its inference computing power; and the power consumption of each H100 is 750W About, if it runs for 10 hours a day, the annual power consumption of the data center where these H100 chips are located will reach 84 billion kWh, while the annual power generation of the Three Gorges Dam is 88.2 billion kWh.
What makes AI chip manufacturers even more anxious is that this is only the power consumption of a large ChatGPT model in the inference stage.
As a result, a series of emerging solutions have been unearthed by manufacturers: integrated storage and computing, chiplets, HBM, and so on. Among them, because the integration of storage and computing overthrew the storage wall under the traditional von Neumann architecture, it has actually achieved cost reduction and efficiency enhancement, becoming this year's "Ziweixing".
(Photo source: "AI Big Computing Power Chip Industry Report-Hundreds of Battleships, Innovators First" by Cirui Insights)
In addition to the emergence of emerging solutions that have given AI chip manufacturers a breath, the big model also brings good news to chip manufacturers, especially start-up chip manufacturers: the importance of software ecology is declining.
When the technology was not mature enough in the early days, researchers could only start by solving a specific problem, and small models with less than one million parameters were born. For example, DeepMind, an AI company owned by Google, allows AlphaGO to conduct special "learning" on the chess-playing steps of millions of human professional players.
After there are more small models, the problem of hardware such as chip adaptation is imminent. Therefore, when Nvidia launched the unified ecological CUDA, GPU+CUDA quickly won the recognition of the computer science community and became the standard configuration for artificial intelligence development.
The large models that are emerging one after another today have multi-modal capabilities, can handle text, pictures, programming and other issues, and can also cover multiple vertical fields such as office, education, and medical care. This also means that adapting to the mainstream ecology is not the only option: when the demand for chips for large models is skyrocketing, chip manufacturers may be able to complete orders for multiple small models in the past by only adapting to 1-2 large models.
In other words, the emergence of ChatGPT provides start-up chip manufacturers with the opportunity to overtake on corners. This means that the AI chip market structure will undergo tremendous changes: it will no longer be a one-man show of individual manufacturers, but a group play of multiple innovators.
When computing power becomes a scarce commodity and adapting to the mainstream ecology is no longer a must, the integration of storage and computing can hardly hide its light. At this time, whether to invest and how much to invest has become the second problem facing AI chip manufacturers.
In this regard, the answer given by the "experienced" chip giant NVIDIA for many years is to be brave in innovation and invest heavily:
Every R&D manufacturer of an emerging technology will undoubtedly face problems at various levels such as technical exploration obstacles and downstream manufacturers' disagreement. In the early stage, whoever predicts the future development trend first, takes the courage to take the steps of exploration, and lays down reasonable resources to try will seize the opportunity.
When the wave of data centers has not yet overwhelmingly hit, and artificial intelligence training is still a niche field, Nvidia has invested heavily in the development of general-purpose computing GPUs and unified programming software CUDA, and it is a good job for Nvidia-computing platforms.
At that time, making the GPU programmable was "useless and loss-making": I don't know if its performance can be doubled, but product development will be doubled. For this reason, no customer is willing to pay for it. However, Nvidia, who predicted that a single-function graphics processor is not a long-term solution, decided to apply CUDA to all product lines.
In an interview between Xindongxi and Dr. Lai Junjie, Senior Director of Engineering and Solutions of Nvidia China, Lai Junjie said: "For the vision of the computing platform, Huang Renxun quickly mobilized a lot of resources from Nvidia up and down in the early days."
Foresight + heavy investment, in 2012, Nvidia won the innovator's reward: In 2012, the computing performance of the deep learning algorithm caused a sensation in the academic circle. As a high-computing, more versatile and easy-to-use productivity tool, GPU+CUDA quickly became popular The computer science community has become the "standard configuration" for the development of artificial intelligence.
On the journey of AI chips with large computing power, integrated storage and computing chips have ushered in their own "golden period", and the investment in advance is the right answer.
PART-02 Non-technical, financially rich, do not enter
Seeing the various benefits of the integration of storage and computing, at this stage, the camp of players integrating storage and computing chips is growing.
(Photo source: "AI Big Computing Power Chip Industry Report-Hundreds of Battleships, Innovators First" by Cirui Insights)
According to the incomplete statistics of CIRUI Insight, since 2019, most of the new AI chip manufacturers have integrated storage and computing: there are 20 new AI chip manufacturers in 2019-2021, and 10 of them Choose the deposit and calculation integration route.
This all shows that the integration of storage and computing will become a rising star after GPGPU, ASIC and other architectures. And this new star, not everyone can pick it.
Under the circumstance that academia, industry, and capital are unanimously optimistic about the integration of storage and computing, strong technical strength, solid talent reserves, and precise control over the acceptance of migration costs are the key to maintaining competitiveness in the industry for start-up companies. Three barriers for new players.
Strong technical strength is always the highest mountain in the chip field.
The first is the integration of storage and computing, which involves the whole process of chip manufacturing: from the lowest device, to circuit design, architecture design, tool chain, and then to the research and development of the software layer; Whole body”: While making corresponding changes in each layer, the degree of adaptation between each layer should also be considered.
Let's look at it layer by layer, what kind of technical problems are there when a memory-computing integrated chip is manufactured.
First of all, in terms of device selection, manufacturers are "walking on thin ice": the memory design determines the yield rate of the chip, and once the direction is wrong, the chip may not be mass-produced.
The second is the circuit design level. After the device is available at the circuit level, it needs to be used for the circuit design of the storage array. At present, in circuit design, there is no EDA tool guidance for in-memory calculation, and it needs to be done manually, which undoubtedly greatly increases the difficulty of operation.
Immediately afterwards, after there are circuits at the architectural level, it is necessary to design the architectural layer. Each circuit is a basic computing module, and the entire architecture is composed of different modules. The design of the memory-computing integrated module determines the energy efficiency ratio of the chip. Analog circuits will be disturbed by noise, and chips will encounter many problems when they are affected by noise.
In this case, chip architects are required to fully understand the process characteristics of analog in-memory computing, and at the same time design the architecture according to these characteristics. On this basis, the adaptability between the architecture and software development must also be considered. After the software-level architecture design is completed, the corresponding tool chain needs to be developed.
(Photo source: "AI Big Computing Power Chip Industry Report-Hundreds of Battleships, Innovators First" by Cirui Insights)
Since the original model of storage-computing integration is different from the model under the traditional architecture, the compiler needs to adapt to a completely different storage-computing integration architecture to ensure that all computing units can be mapped to hardware and run smoothly.
A complete technical chain will test the ability of each link of device, circuit design, architecture design, tool chain, and software layer development, and coordinate the adaptation ability of each link. It is a protracted battle that consumes time, effort and money.
At the same time, according to the operation process of the above links, it can be seen that the memory-computing integrated chip urgently needs experienced circuit designers and chip architects.
In view of the particularity of the integration of storage and calculation, companies that can integrate storage and calculation need to have the following two characteristics in terms of personnel reserves:
The leader needs to have enough courage. There must be a clear idea in the choice of device selection (RRAM, SRAM, etc.) and computing mode (traditional von Neumann, integrated storage and calculation, etc.). This is because, as a subversive and innovative technology, the integration of storage and calculation has no one to lead, and the cost of trial and error is extremely high. The founders of companies that can achieve commercialization often have rich experience in the industry, large manufacturers, and academic backgrounds, and can lead the team to quickly complete product iterations.
In the core team, it is necessary to equip experienced talents in all levels of technology. For example, the architect, which is the core of the team. Architects need to have a deep understanding and cognition of the underlying hardware and software tools, and be able to realize the storage and computing architecture in the concept through technology, and finally achieve product landing;
In addition, according to the qubit report, there is a lack of high-end talents in circuit design in China, especially in the field of hybrid circuits. In-memory computing involves a large number of analog circuit designs. Compared with digital circuit design that emphasizes teamwork, analog circuit design requires individual designers who are extremely familiar with process, design, layout, model pdk, and packaging.
And this series of talents and technologies must take the landing effect as the ultimate goal-the landing is the primary productive force. At the time of delivery, what customers consider is not just the storage-computing integrated technology, but whether the performance indicators of the storage-computing integrated SoC, such as energy efficiency ratio, area efficiency ratio, and usability, have been improved enough compared with previous products. , and more importantly, whether the migration costs are affordable.
If choosing a new chip to improve the performance of the algorithm requires relearning a programming system, and the labor cost of model migration is higher than the cost of purchasing a new GPU, then customers will most likely not choose to use a new chip.
Therefore, whether the integration of storage and computing can minimize the migration cost during the landing process is a key factor for customers when choosing products.
In the context of large-scale models, the memory-computing integrated chip is becoming a rising star in the chip track by virtue of its low power consumption but high energy efficiency ratio. Nowadays, the integrated deposit and calculation market is still in the stage of "Xiaohe is just emerging".
But we cannot deny that the integrated storage and calculation players have already built three high walls, and those with strong technical strength and solid talent reserves should not enter.
View Original
This page may contain third-party content, which is provided for information purposes only (not representations/warranties) and should not be considered as an endorsement of its views by Gate, nor as financial or professional advice. See Disclaimer for details.
In 2023, AI chip companies are being asked three times
In 2023, investment institutions and shareholders are asking listed companies in the field of AI chips three deadly questions:
Have you made a storage and calculation integrated chip yet?
(Source of investors asking questions to Montage Technology: Tonghuashun Finance)
How is the progress of the memory-computing integrated chip?
(Investors ask Dongxin shares the source of the question: Tonghuashun Finance)
What is the prospect of integrated storage and computing chips?
(Source of investors’ questions to Hengshuo shares: Tonghuashun Finance)
The above series of questions means that the integration of storage and calculation no longer "stays" in the academic and industrial circles, and it seems to be popular in the secondary market: stockholders and brokerage institutions are working overtime to make up for the integration of storage and calculation, so traditional chip manufacturers from Since the beginning of 2023, the layout of the integration of storage and computing has been "tortured".
A natural question is why the integration of deposit and calculation has suddenly become so popular?
Huawei, which released the new storage product OceanStor A310 on 7.14, gave its own answer. Zhou Yuefeng, president of Huawei's data storage product line, said that there are many data problems in the training and reasoning of general-purpose large models and industry large models: slow cross-domain collection of massive data, low data interaction efficiency in preprocessing and training, and difficult data flow security.
The large AI model brings new challenges to storage. For example, the stability is worse than that of traditional AI. At the same time, there is a large amount of data preprocessing and preparation work, which requires near-storage computing (a type of storage and calculation) to systematically deal with this problem.
So, how much is the "large amount of data" here? According to the calculations of SI Rui Insight, ChatGPT currently has 100 million daily users. If 5% of people ask questions every second at the same time every day, it will eat up the power generation of the Three Gorges Dam for nearly a year.
In other words, large models are frantically demanding computing power, and emerging solutions represented by the integration of storage and computing can solve many challenges brought about by large models. In fact, in addition to placing strict requirements on computing power, the large model also gives AI chip manufacturers "a lot of sugar to eat".
This article attempts to explore, under the background of the large model, what benefits are given to AI chip manufacturers? What is the future pattern of storage and computing integrated manufacturers that have been "fired" by ChatGPT?
PART-01 "Fortune and misfortune depend on each other" AI chip manufacturers
The wind of the mockup is still blowing, and the discussion about the mockup is still in full swing.
At the beginning of August, at the sub-forum of the China Computer Federation (CCF) Artificial Intelligence Conference - "Seeing the Frontier of Kunlun AI Large-scale Model Computing Power", experts and scholars agreed that in the era of large-scale models, intelligent computing power has become a "scarce asset". , How to make good use of computing resources and maximize their benefits has become a challenge that every participant must face.
As for the amount of computing power demanded by GPT-3 alone, under conservative circumstances, the Three Gorges Dam needs to "work" all night to meet:
Assuming that there are 100 million people online at the same time, 5% of them ask questions every second at the same time every day, each occupying 30 tokens, and 15.3 million H100 chips are needed to provide its inference computing power; and the power consumption of each H100 is 750W About, if it runs for 10 hours a day, the annual power consumption of the data center where these H100 chips are located will reach 84 billion kWh, while the annual power generation of the Three Gorges Dam is 88.2 billion kWh.
What makes AI chip manufacturers even more anxious is that this is only the power consumption of a large ChatGPT model in the inference stage.
As a result, a series of emerging solutions have been unearthed by manufacturers: integrated storage and computing, chiplets, HBM, and so on. Among them, because the integration of storage and computing overthrew the storage wall under the traditional von Neumann architecture, it has actually achieved cost reduction and efficiency enhancement, becoming this year's "Ziweixing".
(Photo source: "AI Big Computing Power Chip Industry Report-Hundreds of Battleships, Innovators First" by Cirui Insights)
In addition to the emergence of emerging solutions that have given AI chip manufacturers a breath, the big model also brings good news to chip manufacturers, especially start-up chip manufacturers: the importance of software ecology is declining.
When the technology was not mature enough in the early days, researchers could only start by solving a specific problem, and small models with less than one million parameters were born. For example, DeepMind, an AI company owned by Google, allows AlphaGO to conduct special "learning" on the chess-playing steps of millions of human professional players.
After there are more small models, the problem of hardware such as chip adaptation is imminent. Therefore, when Nvidia launched the unified ecological CUDA, GPU+CUDA quickly won the recognition of the computer science community and became the standard configuration for artificial intelligence development.
The large models that are emerging one after another today have multi-modal capabilities, can handle text, pictures, programming and other issues, and can also cover multiple vertical fields such as office, education, and medical care. This also means that adapting to the mainstream ecology is not the only option: when the demand for chips for large models is skyrocketing, chip manufacturers may be able to complete orders for multiple small models in the past by only adapting to 1-2 large models.
In other words, the emergence of ChatGPT provides start-up chip manufacturers with the opportunity to overtake on corners. This means that the AI chip market structure will undergo tremendous changes: it will no longer be a one-man show of individual manufacturers, but a group play of multiple innovators.
When computing power becomes a scarce commodity and adapting to the mainstream ecology is no longer a must, the integration of storage and computing can hardly hide its light. At this time, whether to invest and how much to invest has become the second problem facing AI chip manufacturers.
In this regard, the answer given by the "experienced" chip giant NVIDIA for many years is to be brave in innovation and invest heavily:
Every R&D manufacturer of an emerging technology will undoubtedly face problems at various levels such as technical exploration obstacles and downstream manufacturers' disagreement. In the early stage, whoever predicts the future development trend first, takes the courage to take the steps of exploration, and lays down reasonable resources to try will seize the opportunity.
When the wave of data centers has not yet overwhelmingly hit, and artificial intelligence training is still a niche field, Nvidia has invested heavily in the development of general-purpose computing GPUs and unified programming software CUDA, and it is a good job for Nvidia-computing platforms.
At that time, making the GPU programmable was "useless and loss-making": I don't know if its performance can be doubled, but product development will be doubled. For this reason, no customer is willing to pay for it. However, Nvidia, who predicted that a single-function graphics processor is not a long-term solution, decided to apply CUDA to all product lines.
In an interview between Xindongxi and Dr. Lai Junjie, Senior Director of Engineering and Solutions of Nvidia China, Lai Junjie said: "For the vision of the computing platform, Huang Renxun quickly mobilized a lot of resources from Nvidia up and down in the early days."
Foresight + heavy investment, in 2012, Nvidia won the innovator's reward: In 2012, the computing performance of the deep learning algorithm caused a sensation in the academic circle. As a high-computing, more versatile and easy-to-use productivity tool, GPU+CUDA quickly became popular The computer science community has become the "standard configuration" for the development of artificial intelligence.
On the journey of AI chips with large computing power, integrated storage and computing chips have ushered in their own "golden period", and the investment in advance is the right answer.
PART-02 Non-technical, financially rich, do not enter
Seeing the various benefits of the integration of storage and computing, at this stage, the camp of players integrating storage and computing chips is growing.
(Photo source: "AI Big Computing Power Chip Industry Report-Hundreds of Battleships, Innovators First" by Cirui Insights)
According to the incomplete statistics of CIRUI Insight, since 2019, most of the new AI chip manufacturers have integrated storage and computing: there are 20 new AI chip manufacturers in 2019-2021, and 10 of them Choose the deposit and calculation integration route.
This all shows that the integration of storage and computing will become a rising star after GPGPU, ASIC and other architectures. And this new star, not everyone can pick it.
Under the circumstance that academia, industry, and capital are unanimously optimistic about the integration of storage and computing, strong technical strength, solid talent reserves, and precise control over the acceptance of migration costs are the key to maintaining competitiveness in the industry for start-up companies. Three barriers for new players.
Strong technical strength is always the highest mountain in the chip field.
The first is the integration of storage and computing, which involves the whole process of chip manufacturing: from the lowest device, to circuit design, architecture design, tool chain, and then to the research and development of the software layer; Whole body”: While making corresponding changes in each layer, the degree of adaptation between each layer should also be considered.
Let's look at it layer by layer, what kind of technical problems are there when a memory-computing integrated chip is manufactured.
First of all, in terms of device selection, manufacturers are "walking on thin ice": the memory design determines the yield rate of the chip, and once the direction is wrong, the chip may not be mass-produced.
The second is the circuit design level. After the device is available at the circuit level, it needs to be used for the circuit design of the storage array. At present, in circuit design, there is no EDA tool guidance for in-memory calculation, and it needs to be done manually, which undoubtedly greatly increases the difficulty of operation.
Immediately afterwards, after there are circuits at the architectural level, it is necessary to design the architectural layer. Each circuit is a basic computing module, and the entire architecture is composed of different modules. The design of the memory-computing integrated module determines the energy efficiency ratio of the chip. Analog circuits will be disturbed by noise, and chips will encounter many problems when they are affected by noise.
In this case, chip architects are required to fully understand the process characteristics of analog in-memory computing, and at the same time design the architecture according to these characteristics. On this basis, the adaptability between the architecture and software development must also be considered. After the software-level architecture design is completed, the corresponding tool chain needs to be developed.
(Photo source: "AI Big Computing Power Chip Industry Report-Hundreds of Battleships, Innovators First" by Cirui Insights)
Since the original model of storage-computing integration is different from the model under the traditional architecture, the compiler needs to adapt to a completely different storage-computing integration architecture to ensure that all computing units can be mapped to hardware and run smoothly.
A complete technical chain will test the ability of each link of device, circuit design, architecture design, tool chain, and software layer development, and coordinate the adaptation ability of each link. It is a protracted battle that consumes time, effort and money.
At the same time, according to the operation process of the above links, it can be seen that the memory-computing integrated chip urgently needs experienced circuit designers and chip architects.
In view of the particularity of the integration of storage and calculation, companies that can integrate storage and calculation need to have the following two characteristics in terms of personnel reserves:
The leader needs to have enough courage. There must be a clear idea in the choice of device selection (RRAM, SRAM, etc.) and computing mode (traditional von Neumann, integrated storage and calculation, etc.). This is because, as a subversive and innovative technology, the integration of storage and calculation has no one to lead, and the cost of trial and error is extremely high. The founders of companies that can achieve commercialization often have rich experience in the industry, large manufacturers, and academic backgrounds, and can lead the team to quickly complete product iterations.
In the core team, it is necessary to equip experienced talents in all levels of technology. For example, the architect, which is the core of the team. Architects need to have a deep understanding and cognition of the underlying hardware and software tools, and be able to realize the storage and computing architecture in the concept through technology, and finally achieve product landing;
In addition, according to the qubit report, there is a lack of high-end talents in circuit design in China, especially in the field of hybrid circuits. In-memory computing involves a large number of analog circuit designs. Compared with digital circuit design that emphasizes teamwork, analog circuit design requires individual designers who are extremely familiar with process, design, layout, model pdk, and packaging.
And this series of talents and technologies must take the landing effect as the ultimate goal-the landing is the primary productive force. At the time of delivery, what customers consider is not just the storage-computing integrated technology, but whether the performance indicators of the storage-computing integrated SoC, such as energy efficiency ratio, area efficiency ratio, and usability, have been improved enough compared with previous products. , and more importantly, whether the migration costs are affordable.
If choosing a new chip to improve the performance of the algorithm requires relearning a programming system, and the labor cost of model migration is higher than the cost of purchasing a new GPU, then customers will most likely not choose to use a new chip.
Therefore, whether the integration of storage and computing can minimize the migration cost during the landing process is a key factor for customers when choosing products.
In the context of large-scale models, the memory-computing integrated chip is becoming a rising star in the chip track by virtue of its low power consumption but high energy efficiency ratio. Nowadays, the integrated deposit and calculation market is still in the stage of "Xiaohe is just emerging".
But we cannot deny that the integrated storage and calculation players have already built three high walls, and those with strong technical strength and solid talent reserves should not enter.