LeCun's Observations#
Continued reviewing the conversation between LeCun and Lex this morning. LeCun is a staunch critic of autoregressive models, believing that such models lack sufficient intelligence. The most insightful observation I heard today came from computational power - LLM provides the same computational power for any problem, which is a manifestation of a lack of planning. When faced with difficult or simple problems, humans allocate different levels of mental effort based on the difficulty.
However, for autoregressive models, the amount of computational power consumed for answering any question is the same for each word. This reveals that the intelligence of the existing LLM is a more reflective, primitive intelligence that does not have the ability to deeply understand the problem.
So why does the current LLM appear to be so good at answering questions? My observation on this issue is that because LLM is trained on a large-scale text, it is very easy for it to appear to understand the problem, giving people an illusion of understanding. However, human understanding often involves more hidden information that is difficult to describe in language. I may discuss this problem slowly later.
One observation that verifies this point is the long tail effect mentioned by LeCun in LLM applications. In testing and research, LLM may have been fine-tuned for 90% of frequently asked questions, providing appropriate answers. But there are still 10% of questions that were not anticipated during training, and this small tail is actually very long, creating many difficulties in LLM applications due to out-of-distribution problems.
Of course, I don't think this problem will exist for too long. Many people are doing research in the direction of reasoning and planning, and I believe that this field will make significant progress in a few years.
Recharge and Questions#
After being banned for several months, Ma Qianzu started looking for new opportunities and opened his own knowledge planet. I have always felt that China's self-media and new media lack the foundation of a subscription system, and there is no stable method like The New York Times to obtain economic resources from readers, relying on indirect advertising to maintain cash flow, making them more constrained. Not to mention the current cultural management environment.
So I supported Ma Qianzu's knowledge planet, which operates on a subscription basis, as soon as possible. Of course, recharging is not just about spending money, but also getting the opportunity to ask questions every month. Let me share Ma Qianzu's evaluation of the German economy.
Exactly ten years ago, a friend traveled to Europe and told me about his experiences... In Germany, there are a number of prosperous small cities, and their prosperity is based on occupying a key link in the automotive industry chain. The living atmosphere in these small cities is like the living quarters of state-owned enterprises in China in the 1980s, peaceful and harmonious. If young people fail to enter a decent university, they can look forward to learning their ancestors' craftsmanship and become skilled workers in the local area through a "similar" succession method.
At that time, I answered based on my own life experience, saying that in places where everything is like the living quarters of old state-owned enterprises, good times will only last for one generation. Because there are too many hardworking young people in the world who want to enjoy the peaceful years of others and use them as stepping stones to climb the industrial ladder. If it's like the living quarters of state-owned enterprises in the 1980s, it means that they are only about 15 years away from massive layoffs.
This judgment is very similar to mine. In terms of the future of Germany, I believe it cannot avoid competition with late-developing countries like China. If the peaceful years last too long, there may not be much future. There are other parts to the complete answer, but I think they are a bit nonsense, and I don't want to type too much of his response. Interested friends can join his knowledge planet themselves.
Miscellaneous#
- Outside the office, a tree that has been bare all winter is starting to sprout green buds. Upon closer inspection, they are not green buds, but small yellow-green flowers, with bees flying around them, which looks very pleasing.
- From Global Science (the Chinese operation of Scientific American) news summary, a participant in Musk's brain-machine interface said they played Civilization 6 until 6 a.m., and the biggest drawback is the need for charging. This is somewhat funny. However, Brainlink still cannot control the body to perform complex movements, so satisfying the craving for games is at least a legitimate demand. I would want that too.
- In the same summary, it is mentioned that rinsing the nose with tap water may lead to infection with Acanthamoeba, with a survival rate of less than 7%. Although this incident happened in the United States, it also sounded an alarm for me. It is indeed a bit scary. I will buy a new water filter cartridge for my water pitcher tomorrow.
Also, a little pig is very unhappy that I didn't mention her in the daily insight. Otherwise, she claims to be the best in the world. But I still think it is inappropriate to talk about privacy in the daily insight, so I will just add this note here to show that I haven't forgotten about her.