Matt Ritter's Blog

With words!

Nimble Machine Learning with Data Pipelines

It went extremely well. Because my wife is so visible as a vocal soloist in our community, people often ask what my artistic talent is. I’ve long been at a loss, but I (totally seriously!) think that it might be semi-technical communication. The moment I realized that was when I was tossing and turning at night, overcome by ideas for little tweaks that I could make to the presentation. In the same way that my wife obsesses over the details of her music (most of which I cannot even conceptually understand) I cannot help but obsess over the conceptual flow, metaphors and energy throughout my presentations.

When people think of bureaucracy they often imagine the DMV. And it often deserves its reputation. I think there’s a 1% chance that all of the offices have been infiltrated by foreign operatives with the long term goal of undermining American faith in government. Our own Central Intelligence Agency distributed a guide for accomplishing exactly that (see page 28).

But, in my role as slight-apologist-for-everything, I think our cultural version to it throws some baby out with the bathwater. This was my (weirdly abstract) take away from a pair of books that I have read recently which seem extremely disconnected: Killers of the Flower Moon and Bad Blood. The first is about the cold blooded murders of Osage Native Americans for the purpose of stealing their reservation’s oil riches. Some of victims were essentially killed by their own white spouses – a betrayal shocking even in the context of a horrific history of mistreatment. The second is about a blood testing company Theranos, which lied to investors, regulators and ultimately patients about their ability to detect disease.

Both stories are written chronologically, giving the reader a growing dread as suspicions slowly solidify into horrifying reality. In both cases, the eventual scale of the harm overwhelms even those who initially started looking into it. But there are more subtle connections between the two books, even though the Osage murders took place nearly 100 years earlier than the Theranos you-can’t-quite-prove-that-there-was-manslaughter.

Then, as now, most people are not actively evil. However, choices are rarely made with full information. And people’s curiosity can vary quite a bit. As Upton Sinclair said “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

For the Osage, it was the local law enforcement, and townspeople who shrugged their shoulders when no progress was made on the case. For Theranos, it was the many executives who hindered and overruled their own due diligence processes.

Processes – that’s where the bureaucrats start to look awfully attractive. They really, truly, don’t care about you. They don’t care if you’re an oppressed minority, or a charismatic Silicon Valley founder. There’s no upside to treating anyone differently, no glory in being the one who approved or denied some specific application. Just a stack of papers and a vague fear of being fired for breaking the rules.

Of course, they’re not all perfect. Some are susceptible to bribes, or become petty tyrants. But every system is made up of components. And a system that rewards by-the-book conservatism 99% of the time is going to be made up of people who are, as individuals and as a culture, pretty resilient in the face of fishy behavior. So let’s definitely have some elements of the Wild West in our research labs and movie studios. Even in our companies! But when there’s pressure to ignore age-old processes of justice, regulation, policy, or even democracy “just this once”, it’s worth understanding why our society found it worthwhile to build them up in the first place.

As a child, the word “science“ evoked image of making volcanoes out of vinegar and baking soda. Of course, I enjoyed making the ensuing mess as much as any kid. But I think it is a tragedy that I (and maybe you too!) finished school without realizing what science really means.

Making volcanoes is not experiment

We followed the volcano worksheet,  and got exactly the result we were promised. It was like a cooking recipe, with a paragraph about chemistry at the end. We never got to say “I wonder what would happen if I tried it that way?“ Nor did we ask about the principles at play. At the root of it, that’s all scientists are doing: Looking for universal patterns and principles.

Does everything move towards the center of the earth?

The specific techniques vary based on what is being measured, but all scientific disciplines share the basic cycle of brainstorming a theory, designing an experiment and analyzing their data.

If you want personally relevant insights, you might need to discover them yourself

As much as astrophysics is fascinating, I don’t have the money for a radio telescope. As much as cancer research is important, MIT wisely prohibits amateurs from messing around in their labs. But smaller things matter, too. Maybe they’re not life threatening, or only impact a tiny percentage of the population. But I think that we’re all “a tiny percentage of the population” in one way or another. If you’re frustrated that nobody’s figured out the universal solution to your problem yet: Good news! You and your sample size of 1 are uniquely well positioned to do some SCIENCE.

SCIENCE

And, as anyone with a mosquito bite can tell you, nothing is so motivating as your own itch.

So, Quantified Self

The British Royal Society’s 358-year-old motto is Nullius in verba, “Take nobody’s word for it”. Quantified Self says that you do not need to rely exclusively on an expert or even a book to make your decisions. You can brainstorm a hypothesis, design an experiment, and analyze the data – all by yourself. If you don’t get the outcome you want, just run another! It doesn’t have to be complex, and most experiments that I’ve seen people do are virtually risk free.

Nor does it invalidate expert advice. In fact, one of the benefits of experimentation is that you come into conversations with experts with much more knowledge (and confidence!). Many of my most successful experiments have been in collaboration with doctors and other medical professionals.

Whether it’s managing an ache, improving my personal productivity or even deciding on my retirement plan, the scientific mindset has enabled you to systematically develop understanding and make progress in life every day, and I hope it can help you too!

My wife has been encouraging me to find ways to engage with the FUUSN community, as we’re relatively new to checking it out. Most of the things which appear to compel longtime worshipers (regardless of specific denomination/religion) have never resonated with me, but Samuel Foster’s rookie sermon had just provoked a bunch of thoughts. And since writing a sermon review can’t be considered weirder than putting Barney in your TensorFlow installation guide, I figured this was as good a first step towards experimenting with building a relationship with the church.

Theme 1: Coming To Agreement

Sam started with a recurring dream about finding his sister in a big city. She was also looking for him, but they had no way of communicating, so they each needed to guess where the other was likely to guess they would be likely to go. I almost yelled “Aha, a classic Schelling Point!” but I’m still not clear whether this is one of those congregations where people yell about spirits or game theoretic concepts.

The sermon deftly wove in the morning’s childrens’ story, about the blind men coming to different conclusions based on feeling different parts of an elephant (summary here). This had a twist which I hadn’t heard before: They argued angrily until the Prince asserted his comprehensive understanding of its nature, and gave them a pleasant elephant ride home. This segued into the next theme:

Theme 2: Objective Reality as Power

The moment I decided I loved the sermon is when Sam pulled apart the layers of the children’s story and questioned the implied superiority of the Prince’s viewpoint, even pointing out the political power dynamics both within the story and around its creative goal: “After all, storytellers must also be paid. ” Fortunately, he was respectful enough to do so after the children left so that they did not become upset or postmodernists.

This is a good illustration of his third theme, which I’m going to summarize with a word that I don’t think he actually used:

Theme 3: Empathy

The final major touchpoint was a poem which had been read, about a gas station and its visitor’s transition from judgement to humanization of the dirty grease-covered owners. That transition, based on no new objective information, started with the recognition of details that suggested home. In particular, a home that somebody loved and cared for, just as much as the visitor presumably cared about her own.

Throughout, there were references to the church community. Like the elephant, it’s bigger than any one of us can touch. Like the urban search, the ‘right answer’ depends entirely on what everyone agrees on. And like the gas station, we’re going to have different initial reactions to the same space (physical, social, political, etc), but we need to at least consider each others’ viewpoints when passing judgement.

In fact, particularly in the elephant segment, Sam celebrates the diversity of truths that the blind men initially came to, and suggests that their abandonment in favor of the Prince’s was a loss. I’d argue that the acceptance and rejection of divergent truths needs a more nuanced treatment

My Thoughts

In the elephant story, it’s not clear what the blind men plan to do with their understanding of the elephant. Are they trying to decide whether to be scared of it? Because then the one who felt the trunk, and declared it like a snake, would sound awfully alarmist to the one who felt its rope-like tail. If they planned on cooling off with it, the one who felt its ears could never coordinate with the one who push on its side. In fact, the very existence of differences caused them to get into a loud argument that attracted the prince. It was only when he aligned them on a completely different model of elephants (as things you could comfortably ride), that they got any benefit from the whole exercise.

Many situations are like this. There’s no moral reason to drive on the left or right side of the road, but it’s very important that we all agree. The book Sapiens argues that the primary advantage that humans had over the apes was our ability to create ‘myths’, which started with specific variants of “God will kill you if you are not pro-social” and have evolved into “You can trust that this dollar bill will have value tomorrow” and “voting is a thing you should probably do”. They allow us to coordinate at a level that would be unthinkable in an I-only-trust-my-family-and-friends culture. And if you think that organization based strictly on mutually held ideas is unstable, try changing one.

There are certainly situations where a diversity of opinions is valuable. The average tourist’s obsession with the Mona Lisa is probably more of a headache than benefit to the Louvre. Science is driven forward when one person thinks “Maybe things work a little differently” and proves it. But what’s critical is how they act in the meantime.

Specifically, respecting other people’s dignity. It’s easy to see how telling someone that one of their beliefs is wrong could disrespect their dignity. Worse still is not telling them, and simply disregarding both their opinions (which might be invalid, if based on flawed assumptions) and their needs (which are quite likely still valid!).

There are actually plenty of examples of the opposite, where everyone has a similar mental model, but some are being massively victimized anyway. The Stanford Prison Experiment is case in point. The participants knew that their assignment into prisoner and guard were totally random, and the prisoners knew that they were getting the short end of the stick. They certainly disagreed on things, but largely did it within the shared model of how prisons worked. There didn’t need to be a breakdown in either shared facts or beliefs for one group to be systematically deprived of dignity.

So I guess what I’m saying is that, in general, we should be striving to converge our understanding of reality. We should also acknowledge that we’re far from it at the moment, and need to have meta agreements for managing our disagreements in the meantime (voting is a common one, though not without drawbacks). But, critically, it should all be in the context of trying to lift the common denominator, and for the love of god prevent our children from becoming postmodernists.

I’ve never been so jubilant to see custard apple (score = 0.00147) in my terminal window. It meant that I had finally classified an image using TensorFlow on my brand new GPU. Despite my confidence as I sat down with the visually appealing official guide, I found the process to be time consuming and frustrating. Based on the number and diversity of issues I saw others having as I Googled (actually DDGed) around, it looks like I’m not alone. As the beneficiary of their hard won experience, I wanted to contribute some of the things that I learned in the process.

I’m going to experiment a bit with the structure, alternating between abstract and specific thoughts. The value of specific thoughts is intuitive, but worth illuminating: None of this article has any value if it doesn’t help you, the reader, do something differently. Not “change your viewpoint” or “deepen your understanding”, but literally tap a different sequence of keys on your keyboard than you would have otherwise. Directly saying “Type this, not that” is the shortest path to this goal, and shorter paths are less likely to be waylaid.

Unfortunately, as Barney the Purple dinosaur tried to warn us, we’re all unique in our own way . This is mostly a good thing, but it can make it difficult to share advice. If nothing else, simply copying my .bash_history would start to fail as soon as you got to paths starting with `/home/mritter/`. You’re smart enough to trivially take that, abstract it up to “He means his home directory” and granularize it back to `/home/jsmith/` or whatever. You’re smart, you could do this, but there’s no reason I should make all of my readers perform that same first step, particularly for less obvious situations.

Specific: Ensure the right graphics driver is being used by blacklisting the default

Even after going through the installation steps, my Samsung Odyssey laptop wasn’t recognizing the existance of my GPU. The final step to fixing this was editing my /etc/modprobe.d/blacklist-nouveau.conf file to contain:

blacklist nouveau
options nouveau modeset=0

then running sudo update-initramfs -u

and restarting. I could then confirm the recognition of the GPU with `lshw -C video`. I tried other things beforehand (the probably-relevant parts of which will be detailed below), but I can’t know whether they were critical to this final bug or totally separate.

Abstract: The state space with positive outcomes was much smaller than I expected

Because the guides that I was reading were a few months old (which is years in internet time, and centuries in Deep Learning time), I assumed that I should just use the latest version of each suggested library or driver. This assumption has served me well for dozens of previous installation processes, but it failed this time. Perhaps I should have been more suspicious because of the unusual cross-corporate nature of the situation, or maybe you just win some and lose some. I won’t get into all of the other instances where a minor deviation from the advice cause cascading issues, but it was an important reminder that “extremely similar” configurations are not always good enough.

Specific: Be careful with CUDA 9.1!

The first major issue that I identified after trying to follow this comprehensive guide is that I’d installed CUDA 9.1 instead of 9.0 I assumed that since it wasn’t a major version number, it would be fully backwards compatible. To its credit the official documentation mentions the correct version number, but some of the commands it suggests default to the more recent version of various libraries, which have presumably changed since it was published. This short video  does a good job of outlining the small changes you need to make for it to work.

Note that you can get away with 9.1 if you build TensorFlow from source. But that sounded like opening up a shipping container of boxes of cans of worms, so I didn’t go down that route.

General: This stuff is still bleeding edge

I’ve always had a romantic notion of what it would have been like to work with steam engines during the Victorian age, or airplanes when they were new. New records being set every day! Limitless opportunity! …And frustrating setbacks caused by obscure parts!

The Wright Brothers, for example, had attempted a flight before the one which went down in history. It took two whole days to repair the ‘minor’ damage that the machine suffered, so that they could make their successful attempt. Their inspiration, a world famous glider pilot named Lilienthal, had (over the course of his 5 years in the spotlight) spent just 5 hours in the air. About half a workday actually doing the thing he was world famous for, the rest of the time handling logistics.

Good user experience fades into the background, and it’s easy to forget how hard and complex things are. When you’re at the bleeding edge, there’s nobody in change of making your experience pleasant, or even guaranteeing that what you want to do is even possible. When you’re lucky enough to find a guide, it usually assumes that you have considerable experience, which will let you fill in the gaps. For example, when was the last time someone digressed from their Stack Overflow answer to clarify “sudo means that you have to type your admin password”? That’s just a common denominator on that website, as are hundreds of other little bits of knowledge. Somehow our computing culture has come together on some de facto curriculum that lets most people understand each other, most of the time. But on the bleeding edge, when you’re talking about graphics drivers and rapidly updating libraries, those gaps can become impossible to bridge.

Specific: These commands are your friend

sudo dpkg/apt-get --purge <package> # Completely remove an installed system package, including drivers
apt list --installed | grep <package> # Search through installed packages (make sure they're all the right version!)
sudo dpkg -l | grep "cuda" # Search through installed packages (make sure they're all the right version!)
lshw -C video # See whether the GPU is visible to the machine
lsmod | grep nvidia # See list of relevant drivers (Make sure none are of the wrong version)
cat /proc/driver/nvidia/version # See Driver information
/usr/lib/nvidia-384/bin/nvidia-smi # See GPU details

The hardest part of the project was not doing things, but UNdoing them. Followed closely by knowing whether I had to undo them in the first place.

General: Learn to quickly Create, Read, Update and Delete in the system you’re debugging

Because I was largely operating in a space that I’m unfamiliar with, I didn’t know how to verify that I was on track until the end of the installation process. That would not have been as bad if the errors I got there had been more specific, but I was left with a diagnosis that boiled down to “One (or more!) of the 10 steps that you took is interacting with one (or more!) of your unknowable number of system configurations incorrectly” That, combined with my lack of fluency with the basic CRUD operations around drivers, made debugging by elimination extremely slow.

Working through it with a friend who both had this background, plus a running system to validate against, was critical for getting mine set up. THANK YOU STAN!!!

Specific: CUDNN doesn’t usually seem to be the root of issues, and CUDA versions often are

The download page is mercifully specific about which CUDA version each CUDNN option requires. I didn’t have to re-install it after moving a bunch of other things around – it really is just a few files, which you can see with the ls /usr/local/cuda-9.0/lib64/libcudnn* command.

Make sure that you’ve got the right CUDA version (denoted by the 3 digit number) on your PATH (and its parent directory on your LD_LIBRARY_PATH) /usr/lib/nvidia-384/bin

That’s what I’ve got for you. If you’re trying to get TensorFlow set up, I wish you the best of luck – it’s definitely possible (as long as you actually have a GPU!), and actually doesn’t take too long if you’re lucky enough to find a guide that aligns with your needs perfectly 🙂 If you run into issues, I definitely recommend finding someone who’s been through it before recently. In this and so many things, there’s a lot to be said for good friends! (I’ll take this opportunity to thank Stan again – I couldn’t have done it without the 150+ chat messages that we shared while debugging everything.)