Lecture 6: Probability Weighting
"Probability weighting” describes how people tend to convert objective information about probability into a subjective sense of what may happen—which can lead to bias and error. Observe how this applies to real-life situations such as buying life or travel insurance, and learn two tools to change how you deal with probabilities.
Different quantities that go by the name of entropy are used in variational principles to infer probability distributions from limited data. Shore and Johnson showed that maximizing the Boltzmann-Gibbs form of the entropy ensures that probability distributions inferred satisfy the multiplication rule of probability for independent events in the absence of data coupling such events. Other types of entropies that violate the Shore and Johnson axioms, including nonadditive entropies such as the Tsallis entropy, violate this basic consistency requirement. Here we use the axiomatic framework of Shore and Johnson to show how such nonadditive entropy functions generate biases in probability distributions that are not warranted by the underlying data.
It was almost New Year's Eve and I wanted to do something special on Twitter. I had 69,800 followers and because I admittedly am an imperfect and superficial human addicted to vanity metrics, I wanted to get to 70,000 followers before midnight and it becoming 2020. To celebrate
My friend Marc again to the rescue. He suggested that since there was 10,000+ people RT’ing and following, I could just pick a random follower from my current total follower list (78,000 at this point), then go to their profile to check if they RT’d it and see. If they didn’t, get another random follower and repeat, until you find someone. With 78,000 followers this should take about ~8 tries. ❧
Technically he said it would be random among those who retweeted, but he’s chosen a much smaller subset of people who are BOTH following him and who retweeted it. Oops!
Annotated on January 13, 2020 at 01:10PM
So, based on your write up it sounds like you’re saying that if one retweeted, but wasn’t following you, one had no chance of winning. This means a few thousand people still got lost in the shuffle. Keep in mind that some states have laws regarding lotteries, giveaways, games like this. Hopefully they don’t apply to you or your jurisdiction.
John D. Cook (johndcook.com)One of the shortcomings of the Poisson distribution is that its variance exactly equals its mean. It is common in practice for the variance of count data to be larger than the mean, so it’s natural to look for a distribution like the Poisson but with larger variance. We start with a Poisson random variable X with mean λ, but then we make λ itself random and suppose that λ comes from a gamma(α, β) distribution. Then the marginal distribution on X is a negative binomial distribution with parameters r = α and p = 1/(β + 1).
The previous post said that the negative binomial is useful because it has more variance than the Poisson. The derivation above explains why the negative binomial should have more variance than the Poisson.
Steph Stammel (Twitter)At some point, you have to decide how to raise the children. https://t.co/sy8sCiAUlA
— Steph Stammel (@StephStammel) November 18, 2019I'm sorry, what? pic.twitter.com/h28TBv0SiB
— Yonathan Arbel (@ProfArbel) November 17, 2019
Python For Data Analysis
Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow is by far the best book to get started with machine learning.
Introduction to Statistical Learning.
Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to hyped claims and the dismissal of possibly crucial effects.
Over the years, Lennon and McCartney have revealed who really wrote what, but some songs are still up for debate. The two even debate between themselves — their memories seem to differ when it comes to who wrote the music for 1965's "In My Life."
Mathematics professor Jason Brown spent 10 years working with statistics to solve the magical mystery. Brown's the findings were presented on Aug. 1 at the Joint Statistical Meeting in a presentation called "Assessing Authorship of Beatles Songs from Musical Content: Bayesian Classification Modeling from Bags-Of-Words Representations."
Jerome Jacobson and his network of mobsters, psychics, strip-club owners, and drug traffickers won almost every prize for 12 years, until the FBI launched Operation ‘Final Answer.’
Why you should use percentages, not words, to express probabilities.
Phil Tetlock, a professor of psychology at the University of Pennsylvania, who has studied forecasting in depth, suggests that “vague verbiage gives you political safety.” ❧
This result is consistent with analysis by the data science team at Quora, a site where users ask and answer questions. That team found that women use uncertain words and phrases more often than men do, even when they are just as confident. ❧
A large literature shows that we tend to be overconfident in our judgments. ❧
The best forecasters make lots of precise forecasts and keep track of their performance with a metric such as a Brier score. ❧
The competition, whose finals play out tonight, is as famed for its politics as its cheesy
Three new books on the challenge of drawing confident conclusions from an uncertain world.
This has some nice overview material for the general public on probability theory and science, but given the state of research, I’d even recommend this and some of the references to working scientists.
I remember bookmarking one of the texts back in November. This is a good reminder to circle back and read it.
What happens when several thousand distinguished physicists, researchers, and students descend on the nation’s gambling capital for a conference? The answer is "a bad week for the casino"—but you'd never guess why. The year was 1986, and the American Physical Society’s annual April meeting was slated to be held in San Diego. But when scheduling conflicts caused the hotel arrangements to fall through just a few months before, the conference's organizers were left scrambling to find an alternative destination that could accommodate the crowd—and ended up settling on Las Vegas's MGM grand.
Brian Wansink won fame, funding, and influence for his science-backed advice on healthy eating. Now, emails show how the Cornell professor and his colleagues have hacked and massaged low-quality data into headline-friendly studies to “go virally big time.”
We really need people to begin publishing their negative results and doing a better job on understanding and practicing statistics. Science is already not “believed” by far too many in the United States, we really don’t need bad actors like this eroding the solid foundations we’ve otherwise built.
Observational data about human behavior is often heterogeneous, i.e., generated by subgroups within the population under study that vary in size and behavior. Heterogeneity predisposes analysis to Simpson's paradox, whereby the trends observed in data that has been aggregated over the entire population may be substantially different from those of the underlying subgroups. I illustrate Simpson's paradox with several examples coming from studies of online behavior and show that aggregate response leads to wrong conclusions about the underlying individual behavior. I then present a simple method to test whether Simpson's paradox is affecting results of analysis. The presence of Simpson's paradox in social data suggests that important behavioral differences exist within the population, and failure to take these differences into account can distort the studies' findings.
