Fooled by the Randomness: Book Notes

This post contains notes from my reading of the book: Fooled by the Randomness. It’s a excellent read on how randomness affects our life, in more ways we know.

Intro:

• Authors goal in life is to tease those who take themselves and their quality of their knowledge too seriously
• Book is about how a large part of success for top successful people is attributed to luck and not skill
• Skill can make you wealthy but to become wealthiest you need luck
• Book is about luck perceived as non luck. Randomness perceived as non random (deterministic)

Chapter 1:

• Nero is a risk averse trader who makes decent money, is afraid of loosing to avoid the alternate life style.
• John, his neighbour is relatively agressive trader who makes 10x more than Nero and looks down on him. He recently went bust (lost all his money on trading), is less hardworking and knowledgeable than Nero

Chapter 2:

• People use result to measure the performance in any field. Instead use the cost of alternative. (Expectation)
• Game of Russian roulette. 1 in 6 chance, you get shot, otherwise you get a million dollars.
• People who play and win forget the risk in playing the game. If a person is playing this game every year from 30 to 50 year of age. Highly unlikely that he will be alive at 50
• If 10K people play this game for 10 years, the few surviving will be very rich and will mistake luck for skill. On the other side, there will be morgue full of dead bodies.
• To complete PhD, one need to be high motivated to solve a narrow problem these days. This often creates boredom. In contrast, people going into finance get to apply their scientific rigor in various aspects of the world. From stock, derivatives, commodities; everything has a market. Human psychology, etc
• Heros are heros not because they win or loose. Because they are heroic in nature. History judges man not by its outcome but also with the role of randomness it played in their battles.
• Probability in real life = alternate realities

Chapter 3

• 30 percent profit with 10% variance annually leads to 50.02% of success if you are trading every second. Or rather checking your portfolio every second for profit and loss
• At close to 50 pecent profit and loss, the changes you observe at second level is mostly noise. Rather observing at monthly or quarterly or yearly is better
• Assuming a full time trader, you will have slightly more positive feeling than negative feelings but since toll of negative feelings is much higher than positive, a trader doing this will be in an emotional deficit
• Emotional deficit on long term basis, leads to various health , physical and mental problem. More stress, reduced neural plasticity over time and high blood pressure, etc
• It’s better to not look at the daily and second level signals because they are mostly random and doesnt reflect any information of company fundamental
• Create good sources of information such that only signals are provided and not noise

Chapter 4

• If you have to be fooled by randomness, it should be good kind. Like poetry, music, philosophy.
• Not the bad kind like someone recommending stock. Buying lottery

Chapter 5:

• Biases and story changes when one is fooled by randomness:
  • Overestimation of accuracy in some measure
  • Tendency to get married to a position (sunken cost fallacy)
  • Tendency to change their story
  • No planning for such events because it never occurred in last
  • Lack of critical thinking during black swan event
  • Denial
• One can make money in financial market totally out of randomness
• How does randomness fool evolution? Because evolution that only fit will survive, then how come less competent people survive in the business of trading?
• First, evolution doesn’t say about overall fitness of an individual… it talks about reproductive fitness
• Next, for it to take its course there needs enough time. Time aggregation eliminates effects of randomness. Enough regime change that can eliminate bad apple during hardships. But if reproduction happens before that then genes are passed to next gen
• Evolution says that on average the population will be fit for the environment. Not everyone of them and not at all times.

Chapter 6:

• Thinking terms of distribution helps over just looking at mean or median
• Translating this to markets, rather than thinking bullish or bearish, think next step on how much gain or loss will occur in bullish or bearish event.
• This also includes bet sizing. When you are bullish or bearish, the question is how much
• People tend to be sensitive to the presence of stimulus rather than magnitude of it.
• Stats cannot detect rare event because the absence of rare events leads to less data, recency bias.
• Also modelling of markets is hard because of the non stationary nature of the market. i.e. the underlying distribution of market is always changing

Chapter 7:

• Problem of induction: induction is idea of building theories from instances or data. i.e. building theories from specific instances to generic.
• Are probabilistic in nature. As things that didnot happen in past may happen in future
• The other way is deduction where we go from general to specific. For example: all human are mortal. Hari is a human. Thus, hari is mortal.
• In financial markets, relying solely on historical data can lead to disasters because of the induction problem.
• Event with very low probability might not have happened. Like Russian roulette.
• If it happens and you are not able to sustain the loss, you are out of game.
• Take historical surprises for example: they were the black swan events
• Certainties are only provided by falsification not by induction
• For example: no amount of white swan can provide there are no black swan. But a single white swan, can provide that not all swans are black.
• Induction cannot provide certainties
• Popper’s solution to induction. There are two types of theories:
  • Theories that have been falsified i.e. have been proven wrong
  • Theory that have not been falsified i.e. but is expected to proven wrong
• Popper said “there are men with bold ideas, but highly critical of their own ideas; they try to find whether their ideas are right by trying first to find whether they are not perhaps wrong. They work with bold conjectures and severe attempts at refuting their own conjectures”
• Induction is very natural to human beings because it helps reduce the information to specific to generic, reducing the amount of information required in the brain.
• Pascal’s wager: optimal strategy for humans is to believe in god. If God exists, they are rewarded and if God doesn’t exists, they have nothing to loose
• Similarly, use statistics and induction where the cost of being wrong is bearable.
• Author says he uses it to generate trades but not in risk management

Part II: Monkeys on Typewriters

• Infinite number of monkeys pressing keys infinitely on type writer, one of the them will write a beautiful poem
• What are the chances of that monkey writing another good poem? Very low
• Another explanation for : past does predict future
• But sample size matters: if among 5 monkeys, one generate beautiful poem that’s skill. If there are one billion monkey’s trying to write a poem and one writes a good poem, that’s luck
• Same things happens in business: larger the number of businessman, larger the chance that one of them performed stellar just by luck.
• In real life, it’s also harder to get sense of sample counts i.e. total number of monkeys attempting something
• Moreover we only see the winners, while the losers vanishes completely.
• One only sees the survivors, which skews the assessment of correct odds for success
• Author’s comments on the book “millionaire next door”: the book suggest that most millionaire are people who defer consumption for investment. I.e the accumulate more than their immediate consumption
  • Survivership bias 1: the book only considered people who accumulated things that made them wealthy later. If someone would accumulated something which they expected to grow in future like stocks or land or valuable assets but their expectations didnot land, their accumulation would not have lead to wealth. Most millionares accumulate but everyone who accumulated doesn’t become millionares

Chapter 9:

• Imagine group of 10K investment manager, they get +10% return or -10% return every year with equal probability.
  • You will have 313 managers in 5th year who can say that they never loose in market.
• Even lets say you have good to bad year ratio of 45:55, making the loss in the longer run. You will still have 184 manager boasting about winning streak and mistaking skill for luck
• Expectation of “maximum of track record” is dependent more on size of initial sample than odds of winning.
• Eventually the winners revert to the mean.
• Large number of prediction even with noise can make someone prophet and create all sort of scams
• Adverse selection : investment that comes to you required more checking compared to investment seeked by you.
• Reverse Survivor: there are also people who are skilled but had the bad odds and are out of market due to that
• In a group of 23 individuals, there is 50% chance of two people having same birthdays.
• Survivorship bias also occurs when trying to build trading strategy. We keep the winning strategies while filtering out the losers. The more we try to fit the data by adding more rules and filters, the more the chances that winning strategy is just due to luck.
• Survivorship bias also occurs on stock, universe and market selection. We only consider things like the returns of market X in last 50 years. But what about all the other markets what went bust.
• Randomness is compounded when calculating relative performance. Imagine scenarios: skill-skill, skill-luck, luck-luck, luck-skill. Error due to luck is much more in the comparison.
• Reference case problem: there is not true attainable randomness in practice, only in theory

Chapter 10:

• There exists non linearities in life i.e. simple linear or incremental changes leads to non linear output.
• Sandpile effect: the last sand grain put on sand castle can sink the whole castle
• Network effects: lot of time the option chosen by individual is not an due to better performance but due to known features. As with case with QWERTY keyboard or windows software. People use this because people are familiar with it.
• Most formulation of real world in mathematics makes assumption like iid (independent and identically distributed) which doesn’t hold strongly in real world
• Complex and chaos theories provides a better way to capture these dependent and non linear variables. When they fail, fallback is always numerical solutions or monte carlo simulations
• There exists a tipping point in network effect, which suggests that after the tipping point, the network grows exponentially (non linearly)
• Our brain is not build to grasp non linearity. Brain looks for linear causality (it’s easy to correlate). This it hard to understand all sorts of non linear events like delayed reward, butterfly effect
• There are non random paths to success, but very few people have mental stamina to follow them.
• Process >> Event

Chapter 11:

• Our mind fails to understand probability in real life.
• Need for system 1 (fast brain) in real life because it allows us to take actions instantly; where one doesn’t have time to think
• Herbert simon proposed “satisficing” i.e. our brain stops when it gets to a near satisfactory solution
• Daniel kahnman pointed that there are more stronger heuristics which human mind takes which causes biases in results
• Some heuristics from trader POV:
  • Anchoring bias: “i am as good as my last trade” i.e. people need to continuously perform. To perform they focus on immediate actions or do local optimization rather than global optimization. Goodhart’s law
  • Corollary: People react more to relative events rather than absolute events. Winning or loosing a lottery will make your networth: 100K or 99K. Doesn’t seem much a difference but thinking in terms of relative, win or loss of 1K dollars is more exciting
  • People doesn’t react to the wealth they have rather than “change in their wealth”.
  • Availability heuristic: estimating frequency of event according to the ease with which instances of the event can be recalled. Death due to terrorist activity deemed more likely than death due to any reason
  • Representatives heuristic: gauging if person belongs to a group based on how similar the characteristics are to the typical group member.
  • Simulating heuristic: the ease of mentally performing an alternative scenarios make it highly likely
  • Affect heuristic: the emotion elicited by events determine their probability in your mind. You believe good outcome will happen even when you know that most probable outcome is bad.
• Formally defined as system 1 and system 2.
  • System 1 is effortless, automatic, associative, rapid, parallel process, opaque, emotional, concrete, specific, social and personalized
  • System 2 is effortful, controlled, deductive, slow, serial, self aware, neural abstract, sets, asocial and depersonalized
• Evolutionary psychology: “our brains are made for fitness, not for truth”.
• Agreement between sociobiologist and the kahnman-tversky school of thought:
  • We do not think when making choices but use heuristics
  • We make serious probabistic mistakes in today’s world — whatever the true reason
• The environment for which we have built our endowment is not the one that prevails today
• Probabilistic thinking was not required in early humans life because life was mostly deterministic. We have evolved faster than our genes
• Most of developments in probability theory happened during the growth of gambling industry
• Gerd gigerenzer suggests evolution produces a unique type of rationality called “ecological rationality”
  • Our brain is designed to solve probabilistic problems like stock selection if information is presented correctly.
  • Our brain doesn’t react well to probabilities but well to frequencies
  • This is due to evolution which caused us to have “domain specific adaptions” and “domain general adaptions”
• Descartes’ error: a purely rational human being (without any emotion) cannot do anything. He will continue to weight decisions until time is over
• Mathematics also suggests the same. If one were to perform optimization operations on the large collection of variables, even with the large brains ours, it takes a long time
• There is a need of quick heuristics for decision making. Emotions provide that. “Lubricant of reason”
• Joseph ledoux’s theory: emotion affect one’s thinking.
  • Connections from emotional systems to cognitive system are more compared to vice versa
  • We feel emotions first (limbic brain) then find an explanation (neocortex)
• Corallery: most of our risk assessment and opinions and thoughts may be simple result of emotions.
• Lack of understanding of probabilities occurs everywhere from law, courtroom, doctors to trading systems.
• Options are generally priced as expectation of the possible price at the end time. People confuse it with the worst case scenario prices
• another reason people like options is that it provide “flow” i.e. feeling of gaining or loosing substantial money daily. (This is important for full time involvement like business)
• Option sellers and option buys. Sellers make steady money but can often loose all at once. Buyers experiences vice versa
• Most people are sellers because loosing one day out of 100 is less painful even if at the end you loose it all.
• Journalists gete confused between absence of evidence and evidence of absence
• Index math: the maximum of average of random variables is less volatile than average maximum
• Conditional probability should be taken in calculating morality rates. For example average person lives: 70 years. This doesnt mean the a person who is 65 years old will live 5 years. Conditional expectation needs to be calculated.
• Statistical significance is important in claiming something. Moreover, it’s is not enough because one need show cause and effect, if the claim is a consequence. For example: Nifty is up 1.03 on lower interest rates
• Change in probability and percentage in finance are due to non linear effects and cause. A 5% change in index is not relevant as 5x of 1% change. It’s rather 100x more important
• In financial world, filtering methods are often used to reduce the noise and increase the signals in the data
• Confidence intervals: “is the not the estimate of the forecast matters so much as the degree of confidence with the opinion”

Part III: wax in my ears

• Put wax in your ears when dealing with noise in the information age
• “Unless you have confidence in ruler’s reliability, if you use a ruler to measure a table, you may also be using the table to measure the ruler” — the more frequently you use such ruler, you make know more about the ruler and less about the measured object
• This part presents few tricks to provide insulation from randomness

Chapter 12: gamblers ticks and pigeons in a box

• When luck favours you, you often try to correlate your success which some noise. This is often called “gambler’s tick”
• Our mind is designed to derive causual link between two events, even if there is none. This leads to superstitions and hypothesis. These hypothesis helps us in taking actions.
• Scientists suggests that its emotionally harder to reject your hypothesis than to accept it
• Optimize your environment such that, it prevents you to become fooled by randomness
  • Add restrictions to information which are noise. For example: a strategy is only successful if it hits a predetermined threshold.
• Most of us know pretty much how we should behave. It’s the execution that is the problem, not the absence of probem

Chapter 13, Cardrades Comes to Rome.

• To believe in probability is to believe that there are alternative truth. No knowledge is certain.
• Thinking in terms of probability allows us to become self-critical, not stubbornly believe to an opinion. It allows us to contradict ourselves
• Self contradiction is important because are your grow and encounter new information your understanding becomes better and thus your opinion changes.
• People like Soros are devoid of path dependence. Every day is a clean slate for actions. Your future actions should not be changed based on your past action. It should only be to maximize future performance.
• Endowment effect: if you don’t want to buy something at current price, why keep it at that price? It recommends that you are married to your position
• Attachment to ideas is common in humans due to evolutionary traits. People get attached to things they put time and energy on. Lack of emotional attachment means the subject is literally a psychopath.
• Attribution bias: we attribute success to skill and failures to randomness. Attributing failures to “10 sigma” rare event, thinking they were right but luck played against him.
• Why? It’s is a human heuristic to make us actually believe so in order not to kill over self-esteem and keep us going against adversity. Evolutionary traits that keep us going.
• Author start meeting with convincing everyone that they are idiots prone to making mistakes but are endowed with the privilege of knowing it
• Attribution bias has another side effects i.e. it gives people that illusion of being better at what they do. This explains the findings that 80-90% people think they are above average

Chapter 14

• When often defeated by randomness, we are only left with dignity. It is defined as the execution of a protocol of behaviour that doesn’t depend on the immediate circumstance.
• Doing the right thing. This is what stoicism means
• This stoic is a person who combines the qualities of wisdom, upright dealing and courage. The stoic will be this be immune from some of life’s dirty tricks.
• The only article Lady Fortuna has no control over is you behaviour or action

Postscript:

• Inverse skill problem: in company, people on the bottom are judged on process and result while people on the top are judged only on result. This make the growth highly dependent on randomness (this excludes entrepreneurs).
• We continue to worship those who won battles and despise those who lost, no matter the process
• Research shows those to live under self-imposed pressure to be optimal in their enjoyment of things suffer a measure of distress.
• Satisficer vs maximizer
• Maximizer always seek a better deal keeping them always occupied and working. This makes someone unhappy because they have to always keep looking for better deals and predictable outcomes.
• Randomness acts as cure to this behaviour.
• Suppose you work hard towards optimzing your investment portfolio making the best money. One bad trade and you returns are as good as index. Or you buy a house with it, and that turns out to be bad purchase due to bad construction quality. You are left with all that loss.
• Randomness in life should make you humble that anything can happen anytime that might impact your life good or bad
• Human are not designed for schedules and thus randomness also helps in breaking these often.
• Randomness or unpredictability helps when competing with opponent. You need to be random enough that your opponents cannot predict you moves and plan.