AIGC, the wave of content generation has come

Original: Fudan Business Knowledge

Image source: Generated by Unbounded AI‌

Human civilization can be thought of as the sum total of records of the imprints of all human life. So, if there is a super brain that learns all the knowledge recorded by human beings, can it help us create a more brilliant and rich civilization?

In December 2022, AIGC, which was selected as one of the top ten breakthroughs in science by “Science” magazine, will bring this dream into reality. The full name of AIGC is AI-Generated Content, that is, generative AI, which uses artificial intelligence technology to automatically produce content. For the AIGC, 2022 is considered a year of incredible growth. **AIGC can be said to be the most popular and fantasy-filled development direction in the field of AI today. The development of AIGC has given birth to explosive applications such as writing assistants, AI painting, dialogue robots, digital humans, and office software assistants, which are formed through human-computer interaction. new paradigms of recording, learning and recreating. **A compelling question then arises: How will AIGC boost the new wave of artificial intelligence?

Record, learn and recreate

Heroes don’t necessarily start at the bottom. The story of OpenAI begins with a group of upstart entrepreneurs who are full of fear about the future of artificial intelligence. The GPT series is like a super brain cultivated by various feeds carefully prepared by OpenAI.

The GPT-2 model, which was created nearly three and a half years after its establishment, is OpenAI’s first truly representative work. GPT-2 contains 1.5 billion parameters, feeds on 8 million Reddit forum posts, and a total of 40GB of text, showing the ability to continue writing text. For example, if you enter a sentence in “The Lord of the Rings”, it will generate a continuation that makes it impossible to distinguish between true and false, and the plot is different from the original, but it seems logical.

OpenAI frantically wanted to know what kind of capabilities this super brain would have if it could eat more corpus, so GPT-3 with 175 billion parameters was born. GPT-3 training alone costs tens of millions of dollars. Experts feed the model hundreds of billions of English words, including news reports, posts, full-text books, and various web pages collected from 60 million domain names in the past 12 years. . This time, GPT-3 not only has a more powerful language generation ability, but also has excellent context learning ability and a lot of world knowledge. It is proficient in writing poetry, writing news reports, answering questions, and writing code. The latest GPT-4 has stronger data processing and understanding capabilities. It can receive and generate 25,000-word text, which is 8 times that of the previous ChatGPT.

In addition, its logical thinking ability and image understanding ability have also made great leaps. OpenAI is perhaps the most steadfast practitioner of the paradigm of AI recording, learning, and recreating. According to the speculation of the University of Edinburgh and the Allen Institute of Artificial Intelligence, from GPT-3 to GPT-3.5 to GPT-4, OpenAI has iterated multiple versions internally. **ChatGPT performs instruction fine-tuning based on human feedback reinforcement learning, and improves the four abilities of detailed response, fair response, rejection of inappropriate questions, and rejection of questions beyond its knowledge scope by greatly reducing the context learning ability. **

AIGC along the way

In addition to the language generation technology represented by ChatGPT, AIGC also includes image generation, video generation, audio generation, etc. The long development process of AIGC, according to the “AIGC White Paper” issued by China Academy of Information and Communications Technology, can be roughly divided into the following three stages:

**Early embryonic stage (1950s-1990s): **Limited by the level of technology, AIGC is limited to small-scale experiments, and the generated content is not very realistic. In 1957, Lejaren Hiller and Leonard Isaacson completed the first computer-generated musical composition in history by changing the control variables in the computer program into musical notes— - String Quartet “Ilyac Suite”. In 1966, Joseph Weizenbaum (Joseph Weizenbaum) and Kenneth Colby (Kenneth Colby) jointly developed the world’s first robot “Eliza” (Eliza), which uses keyword scanning and recombination to Complete interactive missions. In the mid-1980s, IBM created the voice-controlled typewriter “Tangora” based on the hidden Markov chain model, which was able to process 20,000 words. At this stage, AIGC is only generated by learning the rules written by experts, and its generalization ability is extremely limited, just like a marionette played by experts.

**Sediment accumulation stage (1990s-2010s): **AIGC gradually changed from experimental to practical. Major breakthroughs have been made in deep learning algorithms, graphics processing units (GPUs), tensor processors (TPUs), and training data scale. In 2007, the artificial intelligence system assembled by New York University artificial intelligence researcher Ross Goodwin (Ross Goodwin) wrote the world’s first completely artificial intelligence creation by recording and perceiving what he saw and heard during the road trip. Fiction - 1 The Road. In 2012, Microsoft publicly demonstrated a fully automatic simultaneous interpretation system. Through deep neural network (DNN), the content of English speakers can be automatically generated into Chinese speech through speech recognition, language translation, speech synthesis and other technologies. At this stage, AIGC began to automatically learn a small amount of data recorded by humans, and mastered a certain generalization ability, but limited by the bottleneck of the algorithm, the generation effect needs to be improved. At this time, the AIGC is like a parrot good at imitating, which seems to be decent but knows nothing.

**Rapid development stage (2010s to present):**Since 2014, with the introduction of generative deep learning algorithms and the rapid expansion of training data scale, the effect of AIGC-generated content has gradually become so realistic that it is difficult for humans to distinguish. In 2017, Microsoft’s artificial intelligence girl “Xiaobing” launched the world’s first poetry collection “Sunshine Lost the Glass Window” created entirely by artificial intelligence. In 2018, the StyleGAN model released by Nvidia can automatically generate pictures, and the high-resolution pictures generated by it can hardly be distinguished by the human eye. In 2021, OpenAI launched DALL-E and launched an upgraded version DALL-E-2 a year later. Users only need to enter a short descriptive text, and DALL-E-2 can create corresponding extremely high-quality cartoons, Realistic, abstract and other styles of painting. In July 2022, the open source AI painting tool Stable Diffusion was released, enabling ordinary people to create professional painter-level works. In August of the same year, an art work called “Space Opera House” won the first prize at the Colorado State Fair in the United States. This work was made by AI. Models such as Make-A-Video, Imagen Video, and Phenaki released later can generate videos with text descriptions. ** On November 30, 2022, OpenAI released the chat robot ChatGPT. So far, the era of AIGC has fully opened, and the generated content is flourishing. **

Data Fuel

After nearly 70 years of technological precipitation, AIGC has become an important form of the artificial intelligence industry. In 2022, researchers at Google published a paper entitled “Emerging Ability of Large Language Models”, and found that when the language model is too large to exceed a certain critical value, capabilities that smaller models do not have will emerge. **In recent years, the remarkable achievements of large-scale model technologies represented by GPT-4 and ChatGPT have shown that increasing the scale of models and data is an effective way to break through the bottleneck of existing technologies.

AI models are getting larger in size, essentially to accommodate more data, yet the high-quality data recorded by humans may be exhausted in the near future. Epoch, an artificial intelligence research and forecasting organization, predicted in a non-peer-reviewed paper that high-quality text data, low-quality text data, and image data will be artificially processed in 2023-2027, 2030-2050, and 2030-2070, respectively. Intelligence drained.

At that time, AIGC-based data synthesis will become a new fuel for artificial intelligence. At present, the data generated by artificial intelligence accounts for less than 1% of all data.**According to the forecast of the consulting firm Gartner (Gartner), by 2025, the data generated by artificial intelligence will account for 10% of all data. ** Therefore, establishing a complete AIGC industrial ecology as soon as possible, allowing users to actively interact with AIGC to generate data, thereby forming a data flywheel, will continue to promote the advancement of artificial intelligence technology.

Looking into the future, AIGC for science may become a deep-water area and a new main battlefield for the application of artificial intelligence technology, that is, “artificial intelligence opens the future of scientific research”. **In the past, the data dividends of Internet companies have been exhausted, but a large amount of experimental data has been accumulated in the scientific field. After feeding 280 million amino acid sequences, a start-up company in Berkeley, California, let the model learn the language of proteins, realizing the synthesis of new proteins from zero for the first time. The innovation AIGC brings to science is in full swing. **It is predicted that by 2025, more than 30% of drugs and materials will be discovered with the help of AIGC. **

In the future, human beings will be linked with AIGC to form a symbiosis of content creation and knowledge discovery, but all the deep-seated things of human beings will not be changed by AI. OpenAI CEO Sam Altman (Sam Altman) once envisioned: **As human beings, we still pay attention to the interaction between people, the reward mechanism of the human brain has not changed, we still pursue happiness and have the desire to create And the desire for competition, the desire to form a family… What humans cared about 50,000 years ago, humans will care about a hundred years later. **

Sam Altman also said that the popularity of ChatGPT makes everyone feel that AGI (General Artificial Intelligence) seems to be closer to us, but in fact, a large language model similar to ChatGPT is still very far away from AGI, and we still have a long way to go in the future. way to go. Between change and invariance, the wave set off by AIGC has arrived.