Bookmarked Information Theory and Statistical Mechanics by E. T. Jaynes (Physical Review, 106, 620 – Published 15 May 1957)

Information theory provides a constructive criterion for setting up probability distributions on the basis of partial knowledge, and leads to a type of statistical inference which is called the maximum-entropy estimate. It is the least biased estimate possible on the given information; i.e., it is maximally noncommittal with regard to missing information. If one considers statistical mechanics as a form of statistical inference rather than as a physical theory, it is found that the usual computational rules, starting with the determination of the partition function, are an immediate consequence of the maximum-entropy principle. In the resulting "subjective statistical mechanics," the usual rules are thus justified independently of any physical argument, and in particular independently of experimental verification; whether or not the results agree with experiment, they still represent the best estimates that could have been made on the basis of the information available.

It is concluded that statistical mechanics need not be regarded as a physical theory dependent for its validity on the truth of additional assumptions not contained in the laws of mechanics (such as ergodicity, metric transitivity, equal a priori probabilities, etc.). Furthermore, it is possible to maintain a sharp distinction between its physical and statistical aspects. The former consists only of the correct enumeration of the states of a system and their properties; the latter is a straightforward example of statistical inference.


On the Fallacy of Diminishing Returns

Nominated for quote of the week, which I encountered while reading Matt Ridley’s The Rational Optimist:

Thomas Jefferson (), American Founding Father and the principal author of the Declaration of Independence (1776)
in a letter to Isaac McPherson


Darwin Library, Now Online, Reveals Mind of 19th-Century Naturalist

Charles Darwin’s Library from the Biodiversity Heritage Library

A portion of Charles Darwin’s vast scientific library—including handwritten notes that the 19-century English naturalist scribbled in the margins of his books—has been digitized and is available online. Readers can now get a firsthand look into the mind of the man behind the theory of evolution.

The project to digitize Darwin’s extensive library, which includes 1,480 scientific books, was a joint effort with the University of Cambridge, the Darwin Manuscripts Project at the American Museum of Natural History, the Natural History Museum in Britain, and the Biodiversity Heritage Library.

The digital library, which includes 330 of the most heavily annotated books in the collection, is fully indexed—allowing readers to search through transcriptions of the naturalist’s handwritten notes that were compiled by the Darwin scholars Mario A. Di Gregorio and Nick Gill in 1990.

The Chronicle of Higher Education
in Darwin Library, Now Online, Reveals Mind of 19th-Century Naturalist


📅 18th International C. elegans Meeting, 22nd-26th June 2011

RSVPed Attending 18th International C. elegans Meeting
The Organizing Committee invites you to attend the 18th International C. elegans Meeting, sponsored by the Genetics Society of America. The meeting will be held June 22 – 26, 2011 at the University of California, Los Angeles campus. The meeting will begin on Wednesday evening, June 22 at 7:00 pm and will end on Sunday, June 26 at 12:00 noon. On Friday, June 24 at 5:00 pm there will be a Keynote Address by Joseph Culotti, Samuel Lunenfeld Research Institute, Toronto, Canada

Entropy Is Universal Rule of Language | Wired Science

Reposted Entropy Is Universal Rule of Language (Wired)
The amount of information carried in the arrangement of words is the same across all languages, even languages that aren't related to each other. This consistency could hint at a single common ancestral language, or universal features of how human brains process speech. "It doesn't matter what language or style you take," said systems biologist…

The research this article is based on is quite interesting for those doing language research.

The amount of information carried in the arrangement of words is the same across all languages, even languages that aren’t related to each other. This consistency could hint at a single common ancestral language, or universal features of how human brains process speech.

“It doesn’t matter what language or style you take,” said systems biologist Marcelo Montemurro of England’s University of Manchester, lead author of a study May 13 in PLoS ONE. “In languages as diverse as Chinese, English and Sumerian, a measure of the linguistic order, in the way words are arranged, is something that seems to be a universal of languages.”

Language carries meaning both in the words we choose, and the order we put them in. Some languages, like Finnish, carry most of their meaning in tags on the words themselves, and are fairly free-form in how words are arranged. Others, like English, are more strict “John loves Mary” means something different from “Mary loves John.”

Montemurro realized that he could quantify the amount of information encoded in word order by computing a text’s “entropy,” or a measure of how evenly distributed the words are. Drawing on methods from information theory, Montemurro co-author Dami??n Zanette of the National Atomic Energy Commission in Argentina calculated the entropy of thousands of texts in eight different languages: English, French, German, Finnish, Tagalog, Sumerian, Old Egyptian and Chinese.

Then the researchers randomly rearranged all the words in the texts, which ranged from the complete works of Shakespeare to The Origin of Species to prayers written on Sumerian tablets.

“If we destroy the original text by scrambling all the words, we are preserving the vocabulary,” Montemurro said. “What we are destroying is the linguistic order, the patterns that we use to encode information.”

The researchers found that the original texts spanned a variety of entropy values in different languages, reflecting differences in grammar and structure.

But strangely, the difference in entropy between the original, ordered text and the randomly scrambled text was constant across languages. This difference is a way to measure the amount of information encoded in word order, Montemurro says. The amount of information lost when they scrambled the text was about 3.5 bits per word.

“We found, very interestingly, that for all languages we got almost exactly the same value,” he said. “For some reason these languages evolved to be constrained in this framework, in these patterns of word ordering.”

This consistency could reflect some cognitive constraints that all human brains run up against, or give insight into the evolution of language, Montemurro suggests.

Cognitive scientists are still debating whether languages have universal features. Some pioneering linguists suggested that languages should evolve according to a limited set of rules, which would produce similar features of grammar and structure. But a study published last month that looked at the structure and syntax of thousands of languages found no such rules.

It may be that universal properties of language show up only at a higher level of organization, suggests linguist Kenny Smith of the University of Edinburgh.

“Maybe these broad-brushed features get down to what’s really essential” about language, he said. “Having words, and having rules for how the words are ordered, maybe those are the things that help you do the really basic functions of language. And the places where linguists traditionally look to see universals are not where the fundamentals of language are.”

Image: James Morrison/Flickr.

Citation:”Universal Entropy of Word Ordering Across Linguistic Families.” Marcelo A. Montemurro and Damián H. Zanette. PLoS ONE, Vol. 6, Issue 5, May 13, 2011. DOI: 10.1371/journal.pone.0019875.



Barnes & Noble Board Would Face Tough Choices in a Buyout Vote | Dealbook

Read Barnes & Noble Faces Tough Choices in a Buyout Vote by Steven Davidoff Solomon (DealBook)
If Leonard Riggio, Barnes & Noble's chairman, joins Liberty Media's proposed buyout of his company, the board needs to decide how to handle his 30 percent stake before shareholders vote on the deal.

This story from the New York Times’ Dealbook is a good quick read on some of the details and machinations of the Barnes & Noble buyout. Perhaps additional analysis on it from a game theoretical viewpoint would yield new insight?