X

Machine Learning is Like Floating-Point Arithmetic on the Real World

2024-12-26T00:00:00+00:00

A trained model can be treated just like a regular computer program.

I still think that view is correct. But now, with the benefit of additional knowledge, I am finetuning how I reason about machine learning models; I’ll now think:

A trained model can be treated like a function in a regular computer program.

And it’s not just making predictions using a trained model. Training a model using a computer is also done using a function in a computer program. What do the training and prediction (and possibly also the hyperparameter tuning) functions share? Some global variable(s) of course. The variables are the parameters/weights of the model in the computer memory. The training function updates those weights, and the prediction/inference function uses those weights to make predictions.

So I got to thinking about how the weights are stored in memory, and what the weights in memory actually represent. The weights represent the model/function’s understanding of the real world. Weights are usually tensors (multidimensional arrays) containing floats (real numbers). Imagine that you are training a deep neural network on some data from the real world. It is possible that, because you are using physics data, the network might learn about Pi, the gravitational constant, and Planck’s constant. And so after training there might be three values residing somewhere in your model weights with values 3.14159..., 0.0000000000667430..., and 0.000000000000000000000000000000000662607015... for each of those physical constants respectively.

But the thing is that when computers store real numbers, like the three examples above, they are commonly stored as a floating-point approximate representation (aka floats). This approximation is necessary because there are infinitely many real numbers; if you wanted to store all real numbers accurately, you will need infinite memory (and this is true even if you limit it to say real numbers between 0 and 1). Anyway, how a real number is stored in computer memory depends on the floating-point format used by the computer. The physical constants I have used as an example might be represented as ‘pairs’ of numbers where the first element is the signficand and the second is the exponent: [3.14159, 0], [6.67430, -11], and [6.62607, -34] respectively. This is assuming a base of 10 and a precision of 6. (But of course, computers use a base of 2; there are also lots of other things to talk about regarding the actual floating-point formats used in practice, like the sign of the number. See What Every Computer Scientist Should Know About Floating-Point Arithmetic for further reading).

So we can see that this approximation is already one source of noise/error for the model, especially for a regression model where the predictions are real numbers/continuous.

Now, what if the weights learned by the model are more complex than that? What if, for instance, the model learned some floating-point representation for Pi, gravitational constant and Planck’s constant? So instead of trying to store those three constants directly in three parameters, the model has assigned 6 parameters to store significand and exponent for each of the three constants. There might also be additional learned parameters that represent the ‘floating-point format’ that the model has learned. And this is not just about floating-point representations, this could be some approximation of some law/truth/function that the model has learned. In this case, in addition to the noise introduced by the way the actual weights are stored, there is also the noise introduced by the approximation that the model is doing with the weights.

To put it another way: Computers have to approximate real numbers because real numbers are unlimited and the computer memory is limited; machine learning models have to approximate the real world because machine learning models can only be so large but real-world functions that they predict are usually ‘infinite’ (like the decimals of Pi, or the sine function having an infinite number of terms when you do a Taylor series expansion).

Many computer programmers already know to be careful of calculations with floats because they ‘hallucinate’. A common example is that this piece of code (in Kotlin)

fun main() {
    val a: Double = 0.1
    val b: Double = 0.2
    println(a + b)
}

prints 0.30000000000000004 (or 0.30000001192092896 for Kotlin/Wasm) instead of 0.3 because of how the Double (which is a float with high precision and high exponent range) is represented in memory. Maybe, in the same way, we just have to live with machine learning models occasionally being wrong.

When it comes to modelling language rules, like in Large Language Models (LLMs), this is the perfect tool for it because the grammar rules of any human language are finite, and so can be modelled exhaustively. Also, at least in the English language, there are usually multiple acceptable ways to say the same thing, so there is flexibility. The LLM can output consistently good grammar. But when it tries to explain about real world stuff based on an understanding modelled with floats, then mistakes are still a risk, because the space of ideas is infinite.

Book Review: The Dream Machine

2024-06-16T00:00:00+00:00

The full title of this 2001 book by M. Mitchell Waldrop is The Dream Machine: J.C.R. Licklider and the Revolution That Made Computing Personal. I enjoyed reading this book a lot, especially because I read a hard cover version of it, a change from the mostly digital versions of books and papers I have been reading lately.

I learnt quite a lot about computers from this book. I will split the learnings into two categories: First there is all the interesting stories, tidbits, trivia about the history of computing that are exciting to know. But also, this book relates the human aspects of computing that are often overlooked by computer science practitioners. I will start with the cool stories, and then conclude with the deep lessons about computing because I actually came away from this book with a lot more appreciation for what computer science really is.

When I started learning how to program, I took it for granted that I could write code and save it in the computer and then run that code to get some desired behavior from the computer. It was not always like that. The idea of stored-program, which is that the program a computer runs could be stored in memory was an important breakthrough. Before that, machines were designed to do only one thing, and to program a machine to do a different task, you had to change wires or some other mechanical thing. For instance, the ENIAC was programmed by changing wires.

This idea and several others that have brought computing to where it is now came from many different sources. Written in an engaging style with lots of analogies to make concepts more understandable, this book shows you how those inventions were not made in a vacuum by giving some context to their development. CPU time-sharing, for instance, was invented because MIT had only one IBM 704 computer and when multiple people needed to work on it, people (like John McCarthy) had to wait to run jobs and were getting impatient. Time-sharing was a way to make the CPU do multiple tasks ‘at the same time’ by switching between the different tasks so quickly it appears to a human that the tasks are being done at the same time.

And oh, I learnt about why the internet protocols (and some other proposals people make today) are called RFCs (Request for Comments) (see pp 280, 292). How networking was not a sexy side of computing right from the start; people like John McCarthy and Steve Jobs did not like networking at first. But of course it is very important, connecting people to each other, and it is what the Internet is built on, and just imagine what using a computer without the Internet would be like.

I also learnt how artificial intelligence had been an important part of the history of computing right from the start. You could call it the Holy Grail of computer science. AI research goes back way before even the invention of personal computers. We have always wanted to model and create intelligence like ourselves, it appears. Computation is not just about writing commands for a computer to execute, it is about modelling behaviour. It is the language to model human behavior just like mathematics is the language to model the physical world around us. So in this way, it seems to me that computer science is related to cognitive sciences/psychology just like mathematics is related to physics. Like the human mind, the computer is an information processor, and Information Theory, I learnt from this book, appeared to apply to the human mind just like it did to physical communication channels used in communications and computers. There is a lot of information in this paragraph.

The Dartmouth 1956 summer research conference on Artificial Intelligence was when the term Artificial Intelligence started gaining prominence. At that time, there was a paradigm of studying human behavior which was starting to die off. This was behaviorism, and its premise is that the human is a black box, and so proponents of behaviorism experimented by studying human ‘output’ (behavior) given certain inputs. The revolution that was going on then was that psychologists recognized that “there were rules that could generate behavior, and that behavior wasn’t just the accumulation of reinforced responses” (pg. 144). And so they strongly rejected behaviorism. So at this conference, there were approaches to artificial intelligence presented like the heuristic approach (Logic Theorist by Newell and Simon) and the logical-deduction approach (Advice Taker by McCarthy). These approaches were trying to create intelligent systems based on well-defined rules.

Well, this is not part of the book, but reinforcement learning has become the most prominent and successful way that natural language processing is being done today. OpenAI uses reinforcement learning with human feedback (RLHF) to train large language models, and that has made them wildly successful. Most of the models that are the most successful today for AI are like black boxes, and I thought that was exactly what a behaviorist would do, isn’t it? So when I read the part of the book about the rejection of behaviorism and developing rules-based AI, I was like ah, have we come full circle! I immediately checked online to see what Noam Chomsky (who practically led the demolition of the behaviorist movement in psychology) thought about deep learning and reinforcement learning, and it was entertaining (he doesn’t like it, doesn’t think it’s science). Interesting read here, this podcast, and this response by Peter Norvig.

The main thread running through the book is the vision of ‘human-machine symbiosis’ that J.C.R. Licklider (Lick) and others had, and how Lick, while working at ARPA, directed funding to research labs across the US to achieve that vision.

That was what people like Vannevar Bush, J. C. R Licklider, Wes Clark, and Doug Engelbart had always perceived so well, he thought. The real significance of computing was to be found not in this gadget or that gadget, but in how the technology was woven into the fabric of human life - how computers could change the way people thought, the way they created, the way they communicated, the way they worked together, the way they organized themselves, even the way they apportioned power and responsibility. That was what resonated so deeply in Taylor’s mind. … if Pake could be made to understand what computing was really about, then there could be a tremendous opportunity here to make the dreams into something real. (pp. 329)

I saw a parallel to this in a more primitive, first-principles way in the movie “Godzilla x Kong: The New Empire” when the scientists augmented Kong’s right hand with a new metal arm, and Kong was now able to fight more effectively with the final boss ape (Skar King) who had a chain with a hook at the end that would slash if caught with an arm of flesh. Making gadgets are all well and good, but what can it help Kong do? What human process can it augment and make better?

The biggest lesson from this book for me is the nature of changing the world. It is all about people. We all want to change the world, but I learnt that how to go about it is to change how people do things. Less about making things, and more about impacting people. Change how people pay for goods and services, how they access information, how they interact with each other, how they think about something like weather, the universe, governments, make it easier for folks to move from place to place, etc. It could be a really simple idea, it often is. But once it gets accepted and implemented on a large scale, you have just changed the world. Sounds very obvious (what is the world without people?) but I admittedly never thought of it that way.

Book Review: Brown Girl in the Ring

2023-08-27T00:00:00+00:00

It had been a long time since I read fiction when I found this novel by Nalo Hopkinson titled Brown Girl in the Ring. It was very different from the kinds of fiction I have read because first I don’t recall that I have ever read any Caribbean literature (I am definitely putting some Derek Walcott in my to-read list). Secondly, it combines this cultural background (like the names and dialect of the characters) with fantastic elements of magic, and is set in a very different kind of Toronto, Canada. And again I have also not read a lot of books in the magic realism genre.

So to me, the story was quite strange and gripping. Duppy, and spirits and visions and magic and all. And the book even features some of the Yoruba deities I have heard of like Ogun, Oshun, and Sango. I particularly liked the part about the ritual that Mami Gros-Jeanne carried out. The author did not stint, but described it in detail, and then the effect of the magic afterwards. Some of the descriptions (Rudy’s rituals) were so vivid and visceral I had to skim the paragraphs so I won’t feel sick.

The book was also touching. The way the author described the relationships and dynamics between Gros-Jeanne and Ti-Jeanne, and later Mi-Jeanne, and the relationship between Ti-Jeanne, Tony and Gros-Jeanne. I think this is something that makes spending time reading a book different from watching a movie: you get a fuller depth of the feeling between the characters.

I read a large part of it while waiting for a plane in the airport, and time went by really fast! It was a good story overall.

The Education of a British-Protected Child: A Review

2022-11-02T00:00:00+00:00

I had almost forgotten how unique a writer Chinua Achebe is. This work is like a body of water that appears listlessly quiet on the surface but is quite deep, swift flowing and full of fish as you would immediately realize if you went ahead and stepped into it. This might be among the least popular works of the author but it dragged emotions out of me that the other works of his have never done. It was not a comfortable book for me to read.

A key overall takeaway from the book, one that is addressed, or touched on briefly, in several of the essays (Spelling Our Proper Name, Africa’s Tarnished Image, Martin Luther King and Africa, Africa is People, My Daughters, etc.) is “the image burden that Africa bears today” and “how that image has molded contemporary attitudes, including perhaps our own, to that continent.” It is part of the author’s goal to identify, to put it in the way he would, where the rain that is currently beating Africa started.

This book was published in 2009. To think that, around then, I was still getting well entertained by The Gods Must Be Crazy a movie that is filled with the kind of narrative that Achebe was fighting against in this book, and which movie he indeed alluded to in one of the essays. I have to mention how the brunt of his tirade against incorrect narratives about Africa was directed at the works of Joseph Conrad. He did it so well it is amusing to think how hard @realCaptainJConrad would be dragged and cancelled had this book been published in an age of social media like we have today. But I think it would be hard to project the full depth of feeling in this work onto social media.

The book contains essays that explore other topics: Racism (Traveling White), Biafra (Stanley Diamond), issues in African Literature (Politics and Polititians of Language in African Literature, African Literature as Restoration of Celebration), and personal topics (My Daughters, What is Nigeria to Me?) among others.

There are plenty of quotations, references to other relevant works, and of course, proverbs. The author frequently used stories to make his points. And stories he had in abundance. I particularly liked how he carefully told the stories referencing their contexts in time and in geography to drive home his points. This is kind of what scientists do with facts and empirical data, but what Achebe had was stories (many of them true and provable), and he had them in abundance. I might even go ahead to say that stories in the way he use them are more effective: first, it puts things in perspective (in a not very usual perspective outside of Africa, and hence a much needed perspective) in ways plain reporting cannot; and second, like the book also makes clear, many of the official records were prejudiced against the African to promote the slave trade.

In all, you find that the author takes pain to maintain balance while navigating various topics. In fact, he outrightly champions the middle ground quite early in the first and title essay of the book. That first essay, if you start with it as I did, could come across as underwhelming. What is the man doing going on about middle ground and sweet remisciences and all? But then if you do read on, you get a feeling that it was part of the design so that the surface of the river remains calm and undisturbed. You will find the torrent in the middle essays!

Even though I did not know it when I started reading, I needed this book. And even though it has not answered all the questions that I have, it is a work carefully, and with courage too, put together to set matters in proper perspective.

Have you ever been confused in your dream?

2022-09-21T00:00:00+00:00

Have you ever been confused in your dream? I have. Earlier this year, I woke from a dream that felt unusual so I wrote it down in my diary as I usually do with vivid waking dreams. I did not think anything of it again until last month when I read the transcripts of this Quanta podcast about dreams¹. I went back to my diary entry and I will reproduce it here. This is the entry for that day as I recorded it, except that I changed the names of people mentioned. Mary was a family member, John a classmate in University, and Lola a neighbour we had in my childhood.

Monday May 2, 2022

I had a peculiar kind of dream today: At one point I was confused in the dream. I was with Mary and we were looking to buy coke. We were in a shop owned by John and I asked him for some coke and I really meant the soft drink coke. But in this dream, we had spent quite some time looking for coke that while describing it to John, I remember I did say “hard coke”. John handed me a sachet containing a dark brown content that looked like dirty wood chippings. He gave it to us then left to attend to someone else. (He went into a compund where some moments ago we had sighted Lola sliding from a handrail on the first floor of the building down to the ground.) We (Mary & I) were confused by the package John handed us. We kept looking at it and turning it over until it struck me that this would be a hard drug (weed). We then took it to John and said we need coke, coca-cola, the soft drink, why was he giving us this. He was like “ohhh okay”.

The point I am making is that I was genuinely confused by that package. But I guess my brain was not. I find it interesting. It makes me think about dreams as a conversation between me and my brain. I have control over what I do but not over the other characters in the dream.

School starts today. […]

I cannot remember this dream itself (I don’t think anyone does) but I do remember having the dream and recording it. I don’t believe that I have ever seen hard drugs apart from in movies, and in the movies they certainly don’t look brown.

What is striking to me is that my brain knew something that I did not know. And it knew it in a kind of oh-you-did-not-know-that way.

It was as if the actual me was only a part of my brain while the rest of my brain operated independently. I find that this apparent nature of the brain has been well-documented in neuroscience and psychology literature ².

Further material:

A Case for Analog

2022-01-25T00:00:00+00:00

Analog vs Digital is like Image vs Text. Images are more difficult to replicate but they are richer! Text could be more formal.

I have heard my dad self-deprecatingly use the adjective “analog” to show how out of touch he is with modern technology, specifically modern digital computers. ‘Analog’ to him and to almost everybody today has the connotation of being all those old-fashioned tech like the abacus (the abacus by the way is a digital computer), while digital are the shiny new ones like the Macbook Pro. However, recently I have seen a few projects¹ that explore analog computing as a more energy-efficient alternative to digital computers. And there are trends too.

It seems that beginning from the 1970’s, the technology world picked up the digital computing paradigm and ran very fast with it, growing at the rate predicted by Moore’s law until now when people are predicting that physical limitations will bring about the end of the law. They are also predictions that the growth in computing capacity will the driven by other areas that include AI (machine learning) and quantum computing. These two fields are anything but digital. Quantum computing can be seen as a whole different computing paradigm where the other two paradigms are analog and digital.

Machine learning on the other hand seems to me like a field that would really benefit from analog computing. The term analog itself comes from a Greek word that could be translated to mean “same ratio”. In effect, anolog systems try to create a measurable model of the real world. The analog clock is a model of the earth rotation, the thermometer is a model of a body’s temperature. The most advanced analog model of reality is found in the human brain, which has gradually evolved over millions of years to become a complex and intriguing piece of nature. Machine learning researchers frequently look to the human brain for inspiration while developing artificial intelligence. For instance, these days, neural networks often look for patterns in large amounts of data to enable it make predictions. This is similar to how the brain uses heuristics (aka biases) that have been acquired over several years of evolution to make many decisions. These decisions may not be logical (which is the area where digital computers are king) or rational or even correct, but they are a good reflection of how the individual percieves the world. It is a model of the imperfect world.

As a computing tool for financial transactions or for some security systems, precise calculations are needed and digital computers could remain the defacto tools for it. But when we require tools that can make predictions, recommendations, analog may be the most energy-efficient way to go.

Further material:

https://jnd.org/being_analog/
We’re building computers wrong

https://the-analog-thing.org/ ↩

Recommended movies

2021-07-11T00:00:00+00:00

Just saying..

The Princess and the Frog (2009)
The Lion King (1994)
Thappad (2020)
Black Panther (2018)
Avatar (2009)
Roma (2018)
Helvetica (2007)
The Terminal (2004)
Fiddler on the Roof (1971)
Soul (2020)
A River Runs Through It (1992)
The Love Punch (2013)
Yakari, A Spectacular Journey (2020)
Kung Fu Panda 3 (2016)
Teenage Mutant Ninja Turtles: Mutant Mayhem (2023)
Moscow Does Not Believe in Tears (1980)

The Senior Software Engineer Speaks of Beans

2021-07-10T00:00:00+00:00

A Poem¹

I’ve known beans²:

I’ve known beans that kept many a poor dev[il] awake in the dead of the night reflecting on their life choices.

Beans is the foundation of my worst nightmares.

My granduncle cooked one of the first major beans at Apple in the late 1970s.

Seven years ago I wrestled with one beans that have cost two million dollars for Microsoft since back in the 90s.

Two famous beans in Russia are named after me.

I spent three years cooking beans at OPay - I wish I had 10 bucks for every beans I cooked at OPay.

Then I went to SpaceX in 2017 and was part of the team that cooked the one that crashed a starship prototype.

Forget,

I’ve known beans:

Sneaky, little beans.

Beans is the foundation of my worst nightmares.

inspired by the poem written by Langston Hughes ↩
beans: another name for a software bug ↩

Yoba Chukwu - Lyrics & Translation

2021-06-10T00:00:00+00:00

Igbo	English
Ka-anyi yobanu Chukwu anyi a na-ekpe ekpere ka-anyi di ndu, anyi aka nka, anyi acha isi awo ka-anyi we rie ife anyi luta n’uwa nu	Let us plead with the Almighty and also continue in prayers that we may live, and grow old, and develop grey hair that we may enjoy the fruits of our worldly labor

Okongwu chalum n’uzo, chalum n’uzo kam ga. Ife I na-alu ka mu na-alu nu Ngi bu boyoyo! Yoba Chukwu kpebe ekpere nu Ka I di ndu, ka nka, cha isi awo, ka I bulu Okongwu. Na Okongwu aburo cha-cha fecha-fecha. Na ngi bu boyoyo mecha na I ga-bukwa Okongwu I ga-enye kwa ndi ozo efe nu Ma luo olu gi ofuma nu Ka o dabaalu gi nu Ka I we ka nka, rie ife I lutaa Na emesia na I ga-abu Okongwu. Nkwa!	Okongwu, give me space, give me space let me pass, We are of the same occupation But you there boyoyo! Plead with the Almighty, and pray That you may live, grow old, develop grey hair and become an Okongwu To be an Okongwu is by no means trivial You boyoyo will later become an Okongwu In turn you will give way to others Do your work exceptionally So that it will work out for you That you will grow old and enjoy the fruits of your work And finally you will become an Okongwu Dance!

Chorus (x3): Yoba Chukwu, kpebe ekpere Ka anyi di ndu, ka anyi ka nka, cha isi awo, rie ife anyi n’uwa	Plead with the Almighty, pray That we may live, and grow old, develop grey hair and enjoy the fruits of our worldly labor.

Biko yoba nu Chukwu n’elu uwa gi Biko kpebe nu ekpere nu Ka I di ndu n’elu uwaghi Ka I wee kazie nu nka nu! I chazie isi awo n’uwaaaa I wee rie ife I luta n’uwa nuu o Asim Okongwu puba nu Ife I na alu ka Okongwu na-alu nu Ngi bu boyoyo! Biko jisie ike kpebenu ekpere nu Ka I luo ife Okongwu lu I ga-amuta ife na imeye Emesia na I ga-abu Okongwu oo Na oburo ngi bu Chukwu Onye putaru uwa na ayo Chukwu Ka ya di ndu, cha isi awo, ka o ka nka Ka ofu ife o luta nu, ka ona eri ya Boyoyo! I ga-ato na boyoyo we nwuo? Emesia na I ga-abia kwa I zabakwa Okongwu Onye agbachinalu onye chi ya agbachilo Maka ife I na-alu ka mu na-alu Anochirim gi uzo? Yoba Chi, kpebe ekpere Tinye uchu n’olu gi nu, ka I puta Nkwa!	Please plead with the Almighty in your life Please continue in prayers That you may live your life That you may grow old And develop grey hair in your life That you may enjoy the fruits of your labor on earth I said Okongwu give way We are of the same occupation But you there boyoyo! Please keep on in prayers That you may do the exploits of an Okongwu You will learn from it You will become an Okongwu Because you are not the Almighty Everyone born into the world asks the Almighty That they may live, develop grey hair, grow old And see the fruit of their labor, and enjoy it Boyoyo! Will you remain a boyoyo till death? Will you come later and answer the name of Okongwu? Let no man block who his chi has not blocked Because we have the same occupation Am I in your way? Plead with the Almighty, pray Apply diligence to your work, and emerge Dance!

Chorus (outro): Yoba Chukwu, kpebe ekpere Ka anyi di ndu ka anyi ka nka, cha isi awo rie ife anyi n’uwa	Plead with the Almighty, pray That we may live, and grow old, develop grey hair and enjoy the fruits of our worldly labor.

Design Lessons: It’s for Real People!

2021-04-10T00:00:00+00:00

Have you ever used a product for the first time and it just feels intuitive? You did not have to try and try and try again to find out how to do something with it. I have had that experience with several Google products. When I first used Google Docs for instance, I had been using Microsoft Word for a long time. Google Docs was just like a cleaner version that still had every thing I needed where I could easily find them: The same keyboard shortcuts that I was used to, easy to download my final document in whatever format I wished, etc. It was smooth, white, clean, simple, not so many widgets. This was in contrast to two other word processors I also tried: LibreOffice and WPS Office. With both of these two guys, I was mentally scarred. They were clunky, as though made by aliens. LibreOffice was like a bad imitation of Word, while WPS kept making me feel bad that I was using a free version. I was sure the premium version would not be any better.

This lesson is on Design Research. Design research is the process of gathering information that will be useful to design the product that best meets the needs of your customers. This information includes the problems of the customers, how they use or are going to use your product, their context/situation which may be different from yours in ways you did not imagine.

Design research must be user focused. It requires empathy. To execute it properly, you have to be ready to rethink your assumptions, biases and most of the things you already take for granted, and put yourself in your customer’s shoes.

The best way to carry out design research is going on field trips to understand firsthand how users interact with your product. That experience can be very valuable. Then there are surveys, interviews, and secondary sources of data. Secondary sources are when you get information from third parties like data aggregators, or from your existing competitors in the field.

Your research goals are to answer: Who are the target customers? Why do they or will they use your product? What are their motivations? What do they expect from your product? How do they currently solve the problems you are targeting to solve? And any questions you think are relevant to you and your product development. Ask open-ended questions of your interviewees. You’re trying to gain insights from them, to get them to open up to you, and potentially share ideas you didn’t know to look for.

A quote from Luke Wroblewski: “Stop looking at what other companies are doing. Start spending more time with existing/potential customers.”