14 Mar 2023
It’s been a minute since I’ve updated this thing. Today I woke up with a pretty fun dream. I found a secret part of my high school filled with California Scrub-Jays (my favorite bird :D) and rushed to take photographs of them. On that note, why not write an article on my observations and techniques for photographing birds ¯\_(ツ)_/¯
Common Reason for Failed Photographs
Leaves. Branches. More leaves. Half the time these things become the area of focus for my camera lens instead of the fast-moving birds I am going to capture. Therefore, always try to find a clearing or area where no leaves/branches are in front of the bird (in the background as scenery is not a problem).
Another common reason for a bird photograph is cause the bird flies off. Here’s how to avoid that.
How To Prevent The Bird From Flying Off
To some extent, the motions of a bird are out of control. However, simple things like not talking near the bird or not making too loud steps are controllable. Only approach a still bird further after taking a photograph of it. It doesn’t make sense to give up your guaranteed picture in hopes of a more close-up picture.
Avoid shaking the branches of the bird either as that tends to make the birds go away. Of course, this stuff all sounds obvious but I’m not lying when I say there is a spidey sense to know when a bird will disappear a split-second before it happens.
What makes a good bird photograph?
I’m not the best in photographing these creatures but the main thing I’ve noticed is that having the face (and both eyes) is extremely crucial. We look at bird photographs somewhat in the same way as humans. Having as much focus on the bird’s face - and the emotion created by the bird’s hopeful glance as it turns its head - is extremely important.
Why Bird Photography is Enjoyable
Bird photography (at least for an amateur like me) is a little bit like monopoly. There’s undeniable skill, but undeniable luck as well. Sometimes I get multiple of my best photographs ever from one trip (Alaska 👀) and sometimes return borderline empty-handed. But there are things to improve upon - better aim of the camera, the ability to recognize birds faster, being able to shift lens focus in a split-second, knowing which birds show when - there is a reason we have photographs like these.
Furthermore observing birds as they hop around free while you stress about APLAC is oddly annoying and reassuring at the same time.
Conclusion
It’s not as boring as it sounds. Also peep the amateur field guide I made for Bay Area birds.
22 Feb 2023
Having ~1.5 years of experience with napping, I think I have am pro enough to have some insights. In fact, I just took a 10min nap at 12:00 (pm!) so that I could get myself to study electric potential & voltage. Naps are typically taken wrong but are also pretty dangerous if you don’t take them right. Here’s to everything I know about naps on 2/23/2023 at 2:14am.
Benefits
Energy Saver
The rationale for taking a nap is the idea of admitting that you will not be productive if you do not take a nap. For example, after arriving from school I am pretty certain I will not immediately do homework and instead just browse YouTube. If on average I estimate I will waste 20 minutes of watching YouTube, which not only will drain my energy by being on a scren but also lose motivation to get through homework without watching YouTube - it makes no sense to avoid YouTube altogether. Instead, taking a 20-30 minute nap is ideal for productivity. By accepting that you need a break as an inevitable, taking a nap means not only that you get that break but that break actually will lead to increased focus/motivation/patience for BS after you wakeup. It’s sort of like making a contract like I’m going to kill X minutes but will immediately be productive after. In reality, it’s worth sacrificing 20-30 minutes for 30minutes of work at 95% productivity compared to 15 minutes of YouTube at 50-75% productivity levels.
Day Breaker
Naps, especially those of 20-30min taken in the afternoon after something intensive like school, allow for the brain to shut down. This essentially means that after waking up from the nap, memory of what happened before the nap will suffer. Thus, what you just did an hour ago in school will seem distant and somewhat feel like a completely different day. Also, if you are a “morning person” (inceased productivity after waking up) - ig waking up more in a day can only be useful then?? Or instead you like sleeping/dreaming (or the feeling of normal force supporting your side), naps can give you a way to sleep more with the pretense of “productivity”. I’ve had a good amount of times when while napping I feel like I am awake with eyes open - it’s always pretty fun.
Escape from Reality
Naps are a great way to escape your problems.
Drawbacks
Naps can really screw you up. For one, if you lack self-control and extend your nap to more than 1 hour, your basically screwing up the time when your body will want you to sleep. Second, I guess people feel unproductive taking naps or something? I guess you just have to really believe that 20min of dead time outweighs switching back-and-forth tabs with distractions for 30min of work.
01 Feb 2023
Keywords: Random
This article was made at 2am. I will update as I become more aware of my own observations and discover more.
It’s been a while since I’ve posted and I figure I should. I’ve made a series of observations in my life which are undoubtedly true (I may be biased ;) but generally seem to prove themselves true time and time again. Whether that is a self-fulfilling prophecy of me believing in my own BS, I do not know.
Experiences Can Not Be Repeated Twice
I remember when I was younger, visiting a beach in Europe. It was pretty fun to be in the waves and all day. The next day, I wanted to have exactly what I had yesterday - a nice day on the sand - but the tides were going crazy and the weather was cloudy and gloomy. From that experience in itself around ~7 years ago, I’ve come to realize that any memory cannot be repeated more than once. Time is a monotonically increasing function - you cannot go back - to the first time you listented to your favorite song, read your favorite book, or had a genuinely good time.
What this implies is the idea of living in the moment as being necessary to experience life better. If we are having a fun, spontaneous time, we have to be smart enough to realize that 1) we are having fun and that 2) this time will never come again in its pure, natural form and thus the best thing that can be done is to make it the best time of it. While in the moment we are just living - just like we always are - and thus we don’t see the moment’s importance (e.g. importance of spontantenously going out or looking around), it’s important to see every second of this moment will be valuable in the future. Memories appreciate in value over time, so make the best investments with what time you have now.
If this seems too idealistic for you, here’s a simpler one - just next time you find a song that hits your ears in a nice way just try to listen to that song only because the pleasure you get from its novelties, unexpected beats, etc. will die with each listen.
Useful vs. Useless Arguments
Today, I remember seeing an argument about who was the most oppressed/privileged/etc. demographic throughout history. This may seem obvious, but regardless of how won - there would be no real winner because they are really just fighting for approval over an opinion not over policy. Useless arguments argue over opinions, useful arguments argue over policy. Try to aim for useful arguments.
Implicit Order of the World
For this mysterious thing called prestige, there seems to be an implicit order of the world. The world is full of diverse people yet at the same time follows predictable, statistically-significant patterns en masse: students offered a selection of universities generally choose some over others, certain countries are seen as just “better” (e.g. America) despite growing bodies of evidence saying anything but. Rankings for companies change yearly and yet the same implicit order of which companies are more prestigious stay - there is a general inertia for learnt rankings to change when they are founded on unstated social assumptions and not facts.
Things are more deterministic than we realize
My desk right now is an absolute mess. It is completely disorganized and there are random pens everywhere. I can’t even tell you how I got to this state. But it is deterministc. When I search for something, like a pen, that I will inevitably lose in this mess, I can guarantee I will try to search for multiple different locations. This is fine, but it’s better for me to realize there’s a reason why it’s the place it is beyond the fact that I “lost” it.
Decisions are not made in a vaccuum; they are not random. There is a reason why the U.S. sides with Israel or why India today may be reluctant to condemn Russia in the Ukraine-Russia conflict. The current standing of things - be it where my history notes went or a country’s foreign relations - can logically be retraced to their initial states.
The principle that things are the way they are for a reason (deterministic in machine learning) is an interesting one.
Your Music Taste Is Not (Really) Unique
(Might just be me.) I am always hungry about songs. It is always super funny to see how an unfamiliar song with a slightly weird name but a lot of streams slowly goes from something I completely don’t resonate or feel is part of my taste to my playlist. Time and time again, I have seen that whatever my initial reaction to the song - I am not unique - I probably will find myself liking it. This means that I should try to explore different genres and not judge.
You Are Always Closer to the End
Basically, I have noticed more than likely there is always some way to frame your current timeframe in which you are at the end of the proces. During the summer, I will be near the end of high school. Right now, I am one day away from the last day of school. At the start of second semester in senior year, I am towards the end of the year (and the end of high-school.) At the start of first semester in senior year, I am towards the end of the college application process (and nearing the end of the year.) There is always an end in sight!
11 Jan 2023
Keywords: Random
Basically, at some point or another we’ve probably heard of the saying count your stars or be happy with what you have. I recently learned about hedonistic treadmills and made me make a (probably not unique) connection to this saying. For those, unaware the hedonistic treadmill refers to the stability of one’s happiness regardless of events of positive and unhappy occurences. Essentially, the principle of the hedonistic treadmill says that after a positive or a negative event, we will feel some slight delta in happiness only to return back to where we were before. This is not to say that our happiness is the same regardless of our lives - improvements in living conditions of course improve happiness (generally) - but is more so to describe the fact that we are generally pretty stable.
Before we get to the adage in the title, it’s worth noting that the hedonistic treadmill (at least how I see it) is great at proving another adage about the journey and not the destination. By the hedonistic treadmill, if we wanted to maximize our happiness (like this is some sort of differentiable function idk?) working in jobs and progressing careers would be pretty irrelevant. However, many people find satisfaction from achieving their goals, even if they return to the same happiness before: this is pretty good proof that the journey leads to happiness more than the destination. Perhaps this is a higher-form of happiness like meaning or some other self-help buzzword. Ironically, this means that the hedonistic treadmill measuring happiness missed the fact that just traveling on this treadmill makes us happy. Who else is thinking that this type of satisfaction is an integral of the hedonistic treadmill’s sine waves??
I digress, but the returning of a baseline stands. We really can’t change this baseline happiness other than preventing negative events from taking hold and finding more positive events to genuinely celebrate. To do this, I guess it’s best to just be happy with have. I am still curious about its function though.
This article may have been kinda cringe 🙃 :D
01 Jan 2023
Keywords: Backpropgation, Gradient Descent, Theoretical ML
Hope you are enjoying the new year 2023! It’s been a while since I’ve uploaded a theoretical algorithm so here we are :)
I’ve made a few posts and challenges about typical first-order gradient descent that I think it’s time I move on to other aspects of ML. But before I do, here’s one last gradient-descent-ish article on a variant of standard optimization algorithms and my theory on its potential usefulness in generative models.
Double backpropagation was first created by Le Cun and Drucker in 1992 and since then has been widely dismissed (maybe for computational reasons).Introductory ML courses and books don’t even cover it, so perhaps this article will help explain some things.
Double Backpropagation Algorithm
The name for double backpropagation is a little weird because it’s not actually backpropagation (basically gradient descent) applied twice but just an addition of another term in the objective function. For an objective function \(J\) to optimize parameters \(\theta\) on a training set \(X\) , the double backpropagation function \(J'\) is given by:
\[J'(\theta) = J(\theta) + \lambda \lVert \frac{\partial J(\theta)}{\partial X} \rVert^{2}\]
It’s actually pretty simple. The new objective function is just the regular objective functive plus the differentiable L2-norm of the gradient of the regular objective function with respect to the input scaled by some hyperparameter (you choose the value) \(\lambda\).
Why?
Double Backpropagation is motivated by forcing the parameters to be as generalizable as possible.
More concretely, the value of the adjusted function \(J'\) does not care just about the performance of the model with the parameters \(\theta\) but also how stable they are. If the input \(X\) is changed slightly, the model’s performance (objective function \(J\)) should ideally not change much. This is similar to augmentations where we want the predictions (e.g. a cat image predicted as a cat) to be the same regardless of small changes in the input - both double backpropagation + augmentations try to force the model to be generalizable to differing input. This is of course helpful because when a model is deployed, the input the model will encounter may not be similar to the data gathered use for training.
This is why the (partial) derivative in the equation measures the model’s performance (objective function value) sensitivity to small changes in the input \(X\) is added. The algorithm is known as double backpropagation because a gradient of a gradient is utilized.
Double-Backpropagation for Smooth Latent Spaces (?)
Just some background here. Generative models (for our purposes) are any type of model which takes in random vector input and generates some novel output. For example, a generative model \(G\) is a function that takes in a random generated vector v and returns an image \(Y\).
Let’s say this generative model is trained through some black-box witchcraft (or a GAN training procedure) to generate images of people through these vectors. In this case, differing vectors will produce different people (e.g. different genders, races, body type, etc.)
A latent space is the full extent of values the vector can take, where similar generated images will come from vectors which are near each other in this space.
A short aside: one of the most underrated parts about machine learning is its ability to frame an infinite space (e.g. vector input values in a generative model) in a way that makes sense: similar output images are generated with vectors that are close to each other. More impressive, these computers don’t really know the meaning of words or images (they just see numbers) but can formulate a good understanding of the world.
Ideally in these models, the mapping of the input vector x to the image \(Y\) learned by \(G\) is smooth - small changes in x shouldn’t lead to drastic changes in \(Y\). Do you see where I am going with this?
