📖 Read chapter one of Weapons of Math Destruction by Cathy O’Neil

📖 Read chapter one of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil

I don’t think she’s used the specific words in the book yet, but O’Neil is fundamentally writing about social justice and transparency. To a great extent both governments and increasingly large corporations are using these Weapons of Math Destruction inappropriately. Often it may be the case that the algorithms are so opaque as to be incomprehensible by their creators/users, but, as I suspect in many cases, they’re being used to actively create social injustice by benefiting some classes and decimating others. The evolving case of Facebook’s involvement in potentially shifting the outcome of the 2016 Presidential election especially via “dark posts” is an interesting case in point with regard to these examples.

In some sense these algorithms are like viruses running rampant in a large population without the availability of antibiotics to tamp down or modify their effects. Without feedback mechanisms and the ability to see what is going on as it happens the scale issue she touches on can quickly cause even greater harm over short periods of time.

I like that one of the first examples she uses for modeling is that of preparing food for a family. It’s simple, accessible, and generic enough that the majority of people can relate directly to it. It has lots of transparency (even more than her sabermetrics example from baseball). Sadly, however, there is a large swath of the American population that is poor, uneducated, and living in horrific food deserts that they may not grasp the subtleties of even this simple model. As I was reading, it occurred to me that there is a reasonable political football that gets pushed around from time to time in many countries that relates to food and food subsidies. In the United States it’s known as the Supplemental Nutrition Assistance Program (aka SNAP) and it’s regularly changing, though fortunately for many it has some nutritionists who help to provide a feedback mechanism for it. I suspect it would make a great example of the type of Weapon of Mass Destruction she’s discussing in this book. Those who are interested in a quick overview of it and some of the consequences can find a short audio introduction to it via the Eat This Podcast episode How much does a nutritious diet cost? Depends what you mean by “nutritious” or Crime and nourishment Some costs and consequences of the Supplemental Nutrition Assistance Program which discusses an interesting crime related sub-consequence of something as simple as when SNAP benefits are distributed.

I suspect that O’Neil won’t go as far as to bring religion into her thesis, so I’ll do it for her, but I’ll do so from a more general moral philosophical standpoint which underpins much of the Judeo-Christian heritage so prevalent in our society. One of my pet peeves of moralizing (often Republican) conservatives (who often both wear their religion on their sleeves as well as beat others with it–here’s a good recent case in point) is that they never seem to follow the Golden Rule which is stated in multiple ways in the Bible including:

He will reply, ‘Truly I tell you, whatever you did not do for one of the least of these, you did not do for me.

Matthew 25:45

In a country that (says it) values meritocracy, much of the establishment doesn’t seem to put much, if any value, into these basic principles as they would like to indicate that they do.

I’ve previously highlighted the application of mathematical game theory before briefly in relation to the Golden Rule, but from a meritocracy perspective, why can’t it operate at all levels? By this I’ll make tangential reference to Cesar Hidalgo‘s thesis in his book Why Information Grows in which he looks not at just individuals (person-bytes), but larger structures like firms/companies (firmbytes), governments, and even nations. Why can’t these larger structures have their own meritocracy? When America “competes” against other countries, why shouldn’t it be doing so in a meritocracy of nations? To do this requires that we as individuals (as well as corporations, city, state, and even national governments) need to help each other out to do what we can’t do alone. One often hears the aphorism that “a chain is only as strong as it’s weakest link”, why then would we actively go out of our way to create weak links within our own society, particularly as many in government decry the cultures and actions of other nations which we view as trying to defeat us? To me the statistical mechanics of the situation require that we help each other to advance the status quo of humanity. Evolution and the Red Queeen Hypothesis dictates that humanity won’t regress back to the mean, it may be regressing itself toward extinction otherwise.

Highlights, Quotes, & Marginalia

Chapter One – Bomb Parts: What is a Model

You can often see troubles when grandparents visit a grandchild they haven’t seen for a while.

Highlight (yellow) page 22 | Location 409-410
Added on Thursday, October 12, 2017 11:19:23 PM

Upon meeting her a year later, they can suffer a few awkward hours because their models are out of date.

Highlight (yellow) page 22 | Location 411-412
Added on Thursday, October 12, 2017 11:19:41 PM

Racism, at the individual level, can be seen as a predictive model whirring away in billions of human minds around the world. It is built from faulty, incomplete, or generalized data. Whether it comes from experience or hearsay, the data indicates that certain types of people have behaved badly. That generates a binary prediction that all people of that race will behave that same way.

Highlight (yellow) page 22 | Location 416-420
Added on Thursday, October 12, 2017 11:20:34 PM

Needless to say, racists don’t spend a lot of time hunting down reliable data to train their twisted models.

Highlight (yellow) page 23 | Location 420-421
Added on Thursday, October 12, 2017 11:20:52 PM

the workings of a recidivism model are tucked away in algorithms, intelligible only to a tiny elite.

Highlight (yellow) page 25 | Location 454-455
Added on Thursday, October 12, 2017 11:24:46 PM

A 2013 study by the New York Civil Liberties Union found that while black and Latino males between the ages of fourteen and twenty-four made up only 4.7 percent of the city’s population, they accounted for 40.6 percent of the stop-and-frisk checks by police.

Highlight (yellow) page 25 | Location 462-463
Added on Thursday, October 12, 2017 11:25:50 PM

So if early “involvement” with the police signals recidivism, poor people and racial minorities look far riskier.

Highlight (yellow) page 26 | Location 465-466
Added on Thursday, October 12, 2017 11:26:15 PM

The questionnaire does avoid asking about race, which is illegal. But with the wealth of detail each prisoner provides, that single illegal question is almost superfluous.

Highlight (yellow) page 26 | Location 468-469
Added on Friday, October 13, 2017 6:01:28 PM

judge would sustain it. This is the basis of our legal system. We are judged by what we do, not by who we are.

Highlight (yellow) page 26 | Location 478-478
Added on Friday, October 13, 2017 6:02:53 PM

(And they’ll be free to create them when they start buying their own food.) I should add that my model is highly unlikely to scale. I don’t see Walmart or the US Agriculture Department or any other titan embracing my app and imposing it on hundreds of millions of people, like some of the WMDs we’ll be discussing.

You have to love the obligatory parental aphorism about making your own rules when you have your own house.
Yet the US SNAP program does just this. It could be an interesting example of this type of WMD.
Highlight (yellow) page 28 | Location 497-499
Added on Friday, October 13, 2017 6:06:04 PM

three kinds of models.

namely: baseball, food, recidivism
Highlight (yellow) page 27 | Location 489-489
Added on Friday, October 13, 2017 6:08:26 PM

The first question: Even if the participant is aware of being modeled, or what the model is used for, is the model opaque, or even invisible?

Highlight (yellow) page 28 | Location 502-503
Added on Friday, October 13, 2017 6:08:59 PM

many companies go out of their way to hide the results of their models or even their existence. One common justification is that the algorithm constitutes a “secret sauce” crucial to their business. It’s intellectual property, and it must be defended,

Highlight (yellow) page 29 | Location 513-514
Added on Friday, October 13, 2017 6:11:03 PM

the second question: Does the model work against the subject’s interest? In short, is it unfair? Does it damage or destroy lives?

Highlight (yellow) page 29 | Location 516-518
Added on Friday, October 13, 2017 6:11:22 PM

While many may benefit from it, it leads to suffering for others.

Highlight (yellow) page 29 | Location 521-522
Added on Friday, October 13, 2017 6:12:19 PM

The third question is whether a model has the capacity to grow exponentially. As a statistician would put it, can it scale?

Highlight (yellow) page 29 | Location 524-525
Added on Friday, October 13, 2017 6:13:00 PM

scale is what turns WMDs from local nuisances into tsunami forces, ones that define and delimit our lives.

Highlight (yellow) page 30 | Location 526-527
Added on Friday, October 13, 2017 6:13:20 PM

So to sum up, these are the three elements of a WMD: Opacity, Scale, and Damage. All of them will be present, to one degree or another, in the examples we’ll be covering

Think about this for a bit. Are there other potential characteristics?
Highlight (yellow) page 31 | Location 540-542
Added on Friday, October 13, 2017 6:18:52 PM

You could argue, for example, that the recidivism scores are not totally opaque, since they spit out scores that prisoners, in some cases, can see. Yet they’re brimming with mystery, since the prisoners cannot see how their answers produce their score. The scoring algorithm is hidden.

This is similar to anti-class action laws and arbitration clauses that prevent classes from realizing they’re being discriminated against in the workplace or within healthcare. On behalf of insurance companies primarily, many lawmakers work to cap awards from litigation as well as to prevent class action suits which show much larger inequities that corporations would prefer to keep quiet. Some of the recent incidences like the cases of Ellen Pao, Susan J. Fowler, or even Harvey Weinstein are helping to remedy these types of things despite individuals being pressured to stay quiet so as not to bring others to the forefront and show a broader pattern of bad actions on the part of companies or individuals. (This topic could be an extended article or even book of its own.)
Highlight (yellow) page 31 | Location 542-544
Added on Friday, October 13, 2017 6:20:59 PM

the point is not whether some people benefit. It’s that so many suffer.

Highlight (yellow) page 31 | Location 547-547
Added on Friday, October 13, 2017 6:23:35 PM

And here’s one more thing about algorithms: they can leap from one field to the next, and they often do. Research in epidemiology can hold insights for box office predictions; spam filters are being retooled to identify the AIDS virus. This is true of WMDs as well. So if mathematical models in prisons appear to succeed at their job—which really boils down to efficient management of people—they could spread into the rest of the economy along with the other WMDs, leaving us as collateral damage.

Highlight (yellow) page 31 | Location 549-552
Added on Friday, October 13, 2017 6:24:09 PM

Guide to highlight colors

Yellow–general highlights and highlights which don’t fit under another category below
Orange–Vocabulary word; interesting and/or rare word
Green–Reference to read
Blue–Interesting Quote
Gray–Typography Problem
Red–Example to work through

I’m reading this as part of Bryan Alexander’s online book club.

Syndicated copies to:

As part of Bryan Alexander’s online book club, I’ve recalled that GoodReads.com allows users with linked Amazon accounts to make their Kindle highlights and notes publicly available. Though I expect that I’ll post most/all of them here on my site over time, I thought I’d still add a link to my highlights and annotations for Weapons for Math Destruction here and use this as a reminder to others in the group who might want to take advantage of this functionality as well.

Details on the functionality can be found at Share Your Kindle Notes and Highlights with Your Friends (Beta).

Syndicated copies to:

📖 Read pages 112-121 of Abstract Algebra: An Introduction by Thomas W. Hungerford

📖 Read pages 112-121 of Abstract Algebra: An Introduction (First Edition) by Thomas W. Hungerford
Chapter 5: Congruence in F[x] and Congruence-Class arithmetic, Sections 1 and 2

Reviewing over some algebra for my algebraic geometry class tonight. I always did love the pedagogic design of this textbook. The way he builds up algebraic structures is really lovely.

Abstract Algebra: An Introduction

Syndicated copies to:

📗 Started reading Weapons of Math Destruction by Cathy O’Neil

📖 Read introduction of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil

Based on the opening, I’m expecting some great examples many which are going to be as heavily biased as things like redlining seen in lending practices in the last century. They’ll come about as the result of missing data, missing assumptions, and even incorrect assumptions.

I’m aware that one of the biggest problems in so-called Big Data is that one needs to spend an inordinate amount of time cleaning up the data (often by hand) to get something even remotely usable. Even with this done I’ve heard about people not testing out their data and then relying on the results only to later find ridiculous error rates (sometimes over 100%!)

Of course there is some space here for the intelligent mathematician, scientist, or quant to create alternate models to take advantage of overlays in such areas, and particularly markets. By overlay here, I mean the gambling definition of the word in which the odds of a particular wager are higher than they should be, thus tending to favor an individual player (who typically has more knowledge or information about the game) rather than the house, which usually relies on a statistically biased game or by taking a rake off of the top of a parimutuel financial structure, or the bulk of other players who aren’t aware of the inequity. The mathematical models based on big data (aka Weapons of Math Destruction or WMDs) described here, particularly in financial markets, are going to often create such large inequities that users of alternate means can take tremendous advantage of the differences for their own benefits. Perhaps it’s the evolutionary competition that will more actively drive these differences to zero? If this is the case, it’s likely that it’s going to be a long time before they equilibrate based on current usage, especially when these algorithms are so opaque.

I suspect that some of this book will highlight uses of statistical errors and logical fallacies like cherry picking data, but which are hidden behind much more opaque mathematical algorithms thereby making them even harder to detect than simple policy decisions which use the simpler form. It’s this type of opacity that has caused major market shifts like the 2008 economic crash, which is still heavily unregulated to protect the masses.

I suspect that folks within Bryan Alexander’s book club will find that the example of Sarah Wysocki to be very compelling and damning evidence of how these big data algorithms work (or don’t work, as the case may be.) In this particular example, there are so many signals which are not only difficult to measure, if at all, that the thing they’re attempting to measure is so swamped with noise as to be unusable. Equally interesting, but not presented here, would be the alternate case of someone tremendously incompetent (perhaps who is cheating as indicated in the example) who actually scored tremendously high on the scale who was kept in their job.

Highlights, Quotes, & Marginalia


Do you see the paradox? An algorithm processes a slew of statistics and comes up with a probability that a certain person might be a bad hire, a risky borrower, a terrorist, or a miserable teacher. That probability is distilled into a score, which can turn someone’s life upside down. And yet when the person fights back, “suggestive” countervailing evidence simply won’t cut it. The case must be ironclad. The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.

Highlight (yellow) – Introduction > Location xxxx
Added on Sunday, October 9, 2017

[WMDs are] opaque, unquestioned, and unaccountable, and they operate at a scale to sort, target or “optimize” millions of people. By confusing their findings with on-the-ground reality, most of them create pernicious WMD feedback loops.

Highlight (yellow) – Introduction > Location xxxx
Added on Sunday, October 9, 2017

The software is doing it’s job. The trouble is that profits end up serving as a stand-in, or proxy, for truth. We’ll see this dangerous confusion crop up again and again.

Highlight (yellow) – Introduction > Location xxxx
Added on Sunday, October 9, 2017

I’m reading this as part of Bryan Alexander’s online book club.

Syndicated copies to:

I don’t often weigh myself, but a few people in the past week have said I look like I’ve lost weight. Since I recently unpacked the scale, I thought I’d change out the batteries and see what I’m up to. I was shocked to discover that I’m down to 169.3 pounds! I think this may be the first time in over 20 years that I’m sub-170. For most of the past year I think I’d been hovering around 185 with a high somewhere around 192.

I’m hoping that the changes I’ve made recently in the move will continue on. Perhaps I can get down to the 155-160ish that I ought to be for my height?

I’ve spent part of the day cleaning up some of my checkin data on my website. As I was going through I realized a quirky error had duplicated many and was somehow stripping out additional photos.

There was also another code error which was mixing many of the checkins into my longer form articles. I’ve got a temporary fix, but need to create a filter to fix things longer term. While fixing it, I couldn’t help hearing the haunting words of Richard MacManus who recently said “…I certainly don’t want a bunch of other peoples’ checkins clogging up my feed reader.” Though I’ve spent some time trying to split out content types, I can’t help but think he was referring specifically to me. Sorry Richard!

Doing a demo of POSSE and Webmentions at the Pasadena WordPress Developer Meetup.

Here are some resources for those attending:


Syndicated copies to: