Preface#
This is a new series, and if I persist, I will separate it into its own category. This series was inspired by Ye Yining, who updates LLM-related paper insights on his blog daily. He said that his inspiration for writing insights came from Academician Tan's blog, but Academician Tan's style is too concise, and I don't want to imitate it.
For me, papers are not the most important part of my life right now. So my insights are more about facing life and understanding myself in these directions.
During my time writing blogs on xlog, I had a series of publicly summarized weekly summaries. Later, my weekly summaries turned into self monologues, which means writing letters to myself for the previous and next day. This kind of content is not suitable for publication, but I have intermittently persisted in this habit for about half a year, which is a very good practice.
I record many detailed details in such conversations because there is no pressure in talking to myself, and it doesn't make me refine what I want to say the most. The length of the content also reduces the frequency of reviewing in the future, which is contrary to my original intention of recording life—for the purpose of enriching my life, not for the sake of recording itself.
Therefore, I am now moving my own summaries back to the public space, summarizing the parts that are suitable for sharing every day, and organizing them into daily content about interesting things I read and my own insights. I hope to use the pressure of publicity to extract some memorable content worth remembering every day.
Insights on Debugging#
Recently, the core work has been reworking the new experiments for the thesis. Sometimes, you really encounter bugs that seem to have no clue at all, such as the model and tensor both being on cuda:0 but still getting an error saying they are not on the same device.
Eugenio previously mentioned a summary of insights on deep learning in his repo, one of which is to be with the data, to see the problem from the perspective of the data flowing through your pipeline, and to see it from the perspective of a participant.
Recently, I have relied on this mindset for many debugging tasks. Many times, when I really feel that I have no clue at all, the only solution in the end is to follow your data one step forward, step into a function, and then continue to move forward. Walk step by step from the perspective of the data until you reach the point where the error occurs.
Low-level Errors in Meta#
Some time ago, I was looking at v-jepa's repo. Everything was fine when using things inside the library, but there were problems when trying to call it from outside. It was strange that even though I had confirmed that the library was installed, I couldn't call it using the library name "jepa" as defined in setup.py.
Finally, I found that the author did not pay attention to the format of the repo when implementing it. For a Python project, there are generally two ways to organize the directories: one is to use the library name as the folder name for the source files, and the other is to use the format src/library_name.
# Method 1
example_python_project
├── example_python_project
│ ├── __init__.py
│ └── something.py
└── setup.py
# Method 2
example_python_project
├── setup.py
└── src
└── example_python_project
├── __init__.py
└── something.py
Both of these methods can be recognized and the source code will be registered under example_python_project. But it's funny that this library produced by Meta chose something in between - it used src but didn't have a subfolder with the same name as the project. This caused all of his source files to be registered under src.
And the author made the mistake of using from src.xxx.xxx import xxx throughout the entire repo. This is actually probably a misunderstanding of a relative import, starting from the root directory of the project. But because most commands are run in the root directory, he didn't notice the problem.
But this is actually a very unreasonable operation, and I didn't realize this problem while looking at the source code until I discovered that I could reference the libraries inside jepa anywhere using the format src.xxx.xxx. Then I realized that even people who work at FAIR can make such low-level mistakes in their own projects.
What I Read#
Lei Luo's article "My Wife Was Diagnosed with Lung Cancer, I Hope to Get Your Help" details the process of his wife being diagnosed with lung cancer. Reading it made me feel very sad and empathetic, and I am also very afraid of similar things happening to my family.
Because of the above article, I discovered his blog and read his description of gradually disconnecting from the Chinese internet in the past two years here. I feel that my mentality is very similar. Except for some irrelevant social media and certain rogue social apps that I can't completely get rid of, I have almost completely disappeared from the Chinese internet. Through self-hosting (mainly) and alternative services abroad, I can ensure that my privacy truly belongs to me and is not a tool that anyone can profit from, nor can it be freely accessed by small-minded people. I am very grateful for the existence of the OSS movement, which allows me to have such choices.
Miscellaneous#
- The OST of Civilization VI is very beautiful, and you can hear the ancient style of Scarborough Fair, but it is obviously just projecting their own era onto the ancient times. It is actually creating a bias that people associate with a nation as an attribute that has existed "since ancient times."
- Just realized that nn.Module does not necessarily have a device attribute.
- I live in a village near Freiburg, and every night when I ride my bike home, I can look up and see the starry sky. I can't tell if the feeling in my heart is melancholy or delight.