Information Theory is Something Like the Logarithm of Probability Theory

Dr. Daniel Polani, reader in Artificial Life, University of Hertfordshire
in “Research Questions”


Not only a great quote, but an interesting way to view the subjects.

Syndicated copies to:

Tool Review: Timbuk2 Command Messenger Bag

Overview: Simply the best messenger bag out there

5out of 5

Pros: Delightful To Use, Roomy , Excellent Design and Engineering,  Napoleon Pocket, Lightweight , Great Strap Length, High Quality, Attractive, Great Craftsmanship, Comfortable , Durable

Best Uses: Commuting, School, Office, Day Trips, Computer Laptop, Airplane travel

Timbuk2 Command Messenger Bag

I spent quite a while (months/years) looking at almost every messenger bag in existence (and even contemplated designing my own) for my mobile office and for weekly trips to study abstract mathematics at UCLA. I’ve been through dozens of bags (including one that could only charitably have been called a “murse”) and had problems with all of them – particularly shoulder and neck problems from carrying around so much weight. The Timbuk2 Command messenger bag ($139, medium, black) seems to have remedied all of that. My neck and shoulder pains have gone away because this bag is simply so comfortable it feels like I’m carrying half the load that I used to.

The two length adjustment mechanisms on the shoulder strap of this bag should be a requirement for every bag on the planet. I simply don’t know how I managed without them for all this time and now I can never go back. One makes it dead simple to take the bag on and off and the other allows for additional easy length change – the combination makes the bag wear incredibly comfortably.

As an engineer I can readily appreciate some of the very subtle design and manufacturing elements that truly make this bag a wonder. It’s not only functional and sturdy, but it’s both beautiful inside and out. Timbuk2 has certainly put some serious thought into how to make a bag. In particular the velcro strips at the top of the the flap to keep water out, the Napoleon pocket (so one doesn’t have to completely open the bag to retrieve frequently used items), the plastic strips sewn into the lining to provide additional internal structural support, and an ingenious custom pocket at the bottom of the bag for my cables and computer power brick are simply genius.

My only minor caveats about the bag are:

Although there are quite a number of great and useful pockets, I could do with maybe half a dozen more for daily use to keep either small items (I carry my desk in my bag) or organizing additional papers from floating around inside the bag. One can’t really fault Timbuk2 for this as it’s personal preference on my part and I haven’t seen any other bags on the market with a better designed grouping of pocket spaces for such things.

Having a slightly larger water bottle carrier on the outside of the bag would also be excellent, but it’s nearly perfect for my Zojirushi stainless steel thermos and most small (<16 ounce) plastic water bottles. I’ll mention that one of Timbuk2’s line of Classic messenger bags includes internal water bottle pockets for those that desire something like this — or who need full waterproof interiors. The Command bag is roomy enough that I’ve also contemplated using a Camelback-type of fluid reservoir and drinking tube inside for longer day-trips.

As a comparison, the next closest high quality bag I’ve seen in my research is Tumi’s Alpha Bravo Benning Deluxe Messenger Bag ($275). It rates incredibly high for design and beauty, but slightly lower on the functionality scale (which still makes it one of the top 0.1% of bags on the market in my mind, so if you’ve got the money, it’s definitely worth it).  It’s almost twice the price and though it’s got equivalent design touches and is made from some equivalently excellent materials with fantastic craftsmanship, it is missing some of the more interesting engineering touches including the engineering work on the shoulder strap.

Timbuk2 Command Messenger Bag Open

Syndicated copies to:

Tool Review: Zojirushi Stainless Steel Mug

Designer/Artist William Morris once said, “Have nothing in your house that you do not know to be useful, or believe to be beautiful.” My Zojirushi stainless steel mug is one of the few things I’ve ever owned that I feel truly meets both of these criteria.

The design, materials, manufacturing and workmanship of the mug are nothing short of outstanding; the aesthetics and heft in the hand are truly fantastic. I really could not want for more out of such a product. I love looking at it, I love holding it, and I love using it.

I hope one day to come back and write a review worthy of how truly great this travel mug is, but for now, suffice it to say that I’m in love. I spent a LOT of time reading reviews on Amazon and elsewhere, and searching stores and vendors to find the best thermos/mug on the planet and settled on this one. Not only is it easy and intuitive to take completely apart and wash thoroughly (too many I’ve come across are impossible to take apart and clean properly, if at all), but it seals completely and doesn’t spill.

Zojirushi Stainless Steel Mug

Even better it keeps my beverages piping hot or cold for far longer than I wish it would. There have been days that I’ve filled it with hot coffee or tea and come back several times to drink it hoping that it had cooled a bit only to find it still too hot to consume. After several rounds with this over an eight hour span, I finally opened it up and put in some ice so I could finally drink my coffee. Now I often just leave the cap open (or off) to let it cool a bit more quickly, although even this is a fairly slow process. Now I try to put my beverages in at the temperature I want to drink them knowing that that’s generally the temperature they’ll be when I get around to drinking them.

I love the fact that the cap is designed with a two stage opening mechanism (which probably won’t be noticed by most users because it’s so subtle). One pushes the button and the top opens just a few millimeters. Then letting go of the button allows the top to spring back and click neatly into place so that it doesn’t fall forward and bonk one on the nose when attempting to take a drink.

When I first came across it, I will admit I was a bit reticent at it’s relatively high price (particularly in comparison with cheaper mugs on the market, many of which I’ve tried and been highly disappointed with), but the Zojirushi is certainly worth ever penny; I would not hesitate for a moment to buy more of these.

As a small aside, I will mention that due to physics and the design of the mug that it can occasionally leak a bit when filled with carbonated beverages and then shaken. Doing this creates additional interior pressure that pushes up the internal seal mechanism on the cap that allows a small amount of liquid to escape. Beyond this small category of fluids, which I infrequently use with the mug (and I’m sure others probably won’t either), it has been absolutely airtight and worry-free.

Rating 5 out of 5 stars.

Review by Chris Aldrich
Syndicated copies to:

Lecture Series Review: “Augustine: Philosopher and Saint” by Phillip Cary

Augustine: Philosopher and Saint (Great Courses, #611)Augustine: Philosopher and Saint byProfessor Phillip Cary, Ph.D., Eastern University (The Learning Company, 1997)

My rating: 4 of 5 stars

This series of 12 audio lectures is an excellent little overview of Augustine, his life, times, and philosophy. Most of the series focuses on his writings and philosophy as well as their evolution over time, often with discussion of the historical context in which they were created as well as some useful comparing/contrasting to extant philosophies of the day (and particularly Platonism.)

Early in the series there were some interesting and important re-definitions of some contemporary words. Cary pushes them back to an earlier time with slightly different meanings compared to their modern ones which certainly helps to frame the overarching philosophy presented. Without a close study of this vocabulary, many modern readers will become lost or certainly misdirected when reading modern translations. As examples, words like perverse, righteousness, and justice (or more specifically their Latin counterparts) have subtly different meanings in the late Roman empire than they do today, even in modern day religious settings.

My favorite part, however, has to have been the examples discussing mathematics as an extended metaphor for God and divinity to help to clarify some of Augustine’s thought. These were not only very useful, but very entertaining to me.

As an aside for those interested in mnemotechnic tradition, I’ll also mention that I’ve (re)discovered (see the reference to the Tell paper below) an excellent reference to the modern day “memory palace” (referenced most recently in the book Moonwalking with Einstein: The Art and Science of Remembering Everything) squirreled away in Book X of Confessions where Augustine discusses memory as:

“fields and spacious palaces” “…where are the treasures of innumerable images, brought into it from things of all sorts perceived by the senses. There is stored up, whatsoever besides we think, either by enlarging or diminishing, or any other way varying those things which the sense hath come to; and whatever else hath been committed and laid up, which forgetfulness hath not yet swallowed up and buried.”

Those interested in memes and the history of “memoria ex locis” (of which I don’t even find a reference explicitly written in the original Rhetorica ad Herrenium) would appreciate an additional reference I subsequently found in the opening (and somewhat poetic) paragraph of a paper written by David Tell on JSTOR. The earliest specific reference to a “memory palace” I’m aware of is Matteo Ricci’s in the 16th century, but certainly other references to the construct may have come earlier. Given that Ricci was a Jesuit priest, it’s nearly certain that he would have been familiar with Augustine’s writings at the time, and it’s possible that his modification of Augustine’s mention brought the concept into its current use. Many will know memory as one of the major underpinnings of rhetoric (of which Augustine was a diligent student) as part of the original trivium.

Some may shy away from Augustine because of the religious overtones which go along with his work, but though there were occasional “preachy sounding” sections in the material, they were present only to clarify the philosophy.

I’d certainly recommend this series of lectures to anyone not closely familiar with Augustine’s work as it has had a profound and continuing affect on Western philosophy, thought, and politics.

View all my reviews

Syndicated copies to:

Brief Book Review: “Make Your Own Soda: Syrup Recipes for All-natural Pop, Floats, Cocktails, and More”

Make Your Own Soda: Syrup Recipes for All-Natural Pop, Floats, Cocktails, and MoreMake Your Own Soda: Syrup Recipes for All-Natural Pop, Floats, Cocktails, and More by Anton Nocito
My rating: 3 of 5 stars

I ran across this at random and picked it up on a whim as I often do with books about concocting drinks–particularly having recently picked up a SodaStream machine for fashioning my own seltzer and sodas at the beginning of the new year. Certainly with some lovely photographs it does a reasonable job of harkening back to an older time. There are a handful of small asides and historical facts – though not nearly enough in my mind and it could have included some photos of early and mid-century soda fountains for all its talk about them. The real star of the book has to be the litany of recipes of which I’ll begin trying a few (and hopefully posting up reviews of those over time.)

I’d put this book in a similar league with my all-time favorite Charles Schumann‘s American Bar: The Artistry of Mixing Drinks which is dense with some great information and recipes for the bartender. This book isn’t quite as “hard core” as Schumann’s, but seems to come pretty close to his rigor for the art of mixology. I read an e-book version, which was generally passable, but I would likely have given it 4 stars had I read what I’m sure is a richer experience in print.

View all my reviews on

Information Theory and Paleoanthropology

A few weeks ago I had communicated a bit with paleoanthropologist John Hawks.  I wanted to take a moment to highlight the fact that he maintains an excellent blog primarily concerning his areas of research which include anthropology, genetics and evolution.  Even more specifically, he is one of the few people in these areas with at least a passing interest in the topic of information theory as it relates to his work. I recommend everyone take a look at his information theory specific posts.

silhouette of John Hawks from his blog

I’ve previously written a brief review of John Hawks’ (in collaboration with Anthony Martin) “Major Transitions in Evolution” course from The Learning Company as part of their Great Courses series of lectures. Given my interest in the MOOC revolution in higher education, I’ll also mention that Dr. Hawks has recently begun a free Coursera class entitled “Human Evolution: Past and Future“. I’m sure his current course focuses more on the area of human evolution compared with the prior course which only dedicated a short segment on this time period.  Given Hawks’ excellent prior teaching work, I’m sure this will be of general interest to readers interested in information theory as it relates to evolution, biology, and big history.

I’d love to hear from others in the area of anthropology who are interested in information theoretical applications.


Syndicated copies to:

Why a Ph.D. in Physics is Worse Than Drugs

Jonathan I. Katz, Professor of Physics, Washington University, St. Louis, Mo.
in “Don’t Become a Scientist!”


In the essay, Dr. Katz provides a bevy of solid reasons why one shouldn’t become a researcher.  I highly recommend everyone read it and then carefully consider how we can turn these problems around.

Editor’s Note: The original article has since been moved to another server.

How might we end the war against science in America?

Syndicated copies to:

Forthcoming title on “Information Theory for Bioinformatics”

While reading today I ran across a notice on Wiley’s German-based website that Viswanathan Arunachalam has a text on Information Theory for Bioinformatics which is scheduled to be released in June 2014.

From the publisher’s website, they provide the following synopsis:

This book discusses information theory as a means of extracting data from large amounts of biological sequences. Utilizing the Shannon theory, the book explains using the information theory principles to interpret sequences and extract vital information. It provides a detailed overview of the practical applications in bioinformatics and includes coverage of diversity in nucleotide and amino acid sequences, sing-nucleotide polymorphism (SNP) and indel sites, binding sites in promoter regions, splicing sites, and more.

If I can manage to get an early copy, I’ll provide a review shortly.

Syndicated copies to:

New Routledge Text on Systems Theory

Over the holiday I ran across a press release, which follows with web links added, for a new book on systems theory. It promises to be an excellent read on the development and philosophy of systems theory for those interested in cybernetics, information theory, complexity and related topics.

Book cover image of Traditions of Systems Theory: Major Figures and Contemporary Developments

MIAMI, Fla., Dec. 19, 2013
Dr. Darrell Arnold, Assistant Professor of Philosophy and Director of the Institute for World Languages and Cultures at St. Thomas University, has published an edited volume with Routledge entitled Traditions of Systems Theory: Major Figures and Contemporary Developments. Hans-Georg Moeller, of University College Cork, Ireland, notes that the book “provides a state-of-the-art survey of the increasingly influential and fascinating field of systems theory. It is a highly useful resource for a wide range of disciplines and contributes significantly to bringing together current trends in the sciences and the humanities.” The book includes 17 articles from leading theoreticians in the field, including pieces by Ranulph Glanville, the President of the American Society for Cybernetics, as well as Debora Hammond, the former President of the International Society for Systems Sciences. It is the first comprehensive edited volume in English on the major and countervailing developments within systems theory.

Dr. Arnold writes on 19th century German philosophy, contemporary social theory, as well as technology and globalization, with a focus on how these areas relate to the environmental problematic. He has translated numerous books from German, including C. Mantzavinos’s Naturalistic Hermeneutics (Cambridge UP) and Matthias Vogel’s Media of Reason (Columbia UP). Dr. Arnold is also editor-in-chief of the Humanities and Technology Review.

For additional information on St. Thomas University academic programs and faculty publications, please contact Marivi Prado, Chief Marketing Officer, 305.474.6880;

Dr. Darrell P. Arnold
Dr. Darrell P. Arnold

I’ve ordered my copy and will be providing a review shortly.

The Two Cultures

C.P. Snow, Kt., CBE (1905 – 1980), an English physical chemist and novelist
in 1959 Rede Lecture entitled “The Two Cultures”


C. P. Snow, English physicist, author, and diplomat.
C. P. Snow, English physicist, author, and diplomat.


Syndicated copies to:

Renaissance for Information Theory in Biology

This year is the progenitor of what appears to be the biggest renaissance for the application of information theory to the area of biology since Hubert Yockey, Henry Quastler, and Robert L. Platzman’s “Symposium on Information Theory in Biology at Gatlinburg, Tennessee” in 1956. (I might argue it’s possibly even bigger than Claude Shannon’s Ph.D. thesis.)  It certainly portends to create a movement that will rapidly build upon and far surpass Norbert Weiner’s concept of Cybernetics and Ludwig von Bertalanffy’s concept of General Systems Theory.

This week John Baez has announced an upcoming three day workshop on “Entropy and Information in Biological Systems” to be hosted by the National Institute for Mathematical and Biological Synthesis in Knoxville, TN, tentatively scheduled for October 22-24, 2014.

Apparently unbeknownst to Baez, earlier this year Andrew Eckford, Toby Berger, and Peter Thomas announced a six day workshop on “Biological and Bio-Inspired Information Theory” to be hosted by the Banff International Research Station for Mathematical Innovation and Discovery scheduled for October 26-31, 2014 – just two days later!

What a bonanza!!

The BIRS workshop will be a bit more general in its approach while the NIMBioS workshop has a slightly tighter view specifically on maximum entropy as applied to biology.

Even more telling (and perhaps most promising) about the two workshops is the very heavy mathematical bent both intend to make their focus.  I have a theory that the bounds of science are held below the high water level of mathematics (aka are “bounded by” in mathematics-speak), so there is nothing more exciting than to see groups attempting to push the mathematics and its application further. It was both the lack of mathematical rigor and the general youth of biology (and specifically genetics and microbiology) in the 1950’s which heavily hampered the early growth of cybernetics as a movement. Fortunately this is no longer the case on either count. Now we just need more researchers who are more readily conversant in the two realms simultaneously.

Marsden and Weinstein’s Calculus Series

A friend tipped me off that Marsden and Weinstein’s entire Calculus series is available for free online now.

I remember using volume III in Dr. James Martino‘s class many moons ago and enjoying its style. I do however have to credit all my facility in the subject to Dr. Richard Joseph who taught it to me by means of the steepest gradient possible – that of electromagnetic theory.

Marsden & Weinstein's Calculus III textbook cover

Fundamental Difference Between Science and Politics

Nate Silver, statistician
in The Signal and the Noise: Why So Many Predictions Fail-but Some Don’t (Penguin Books, 2012)


The Signal and The Noise

Syndicated copies to:

Review of The Signal and the Noise: Why So Many Predictions Fail – But Some Don’t

The Signal and the Noise: Why So Many Predictions Fail - But Some Don't by Nate SilverNate Silver (
The Signal and the Noise: Why So Many Predictions Fail, But Some Don't Book Cover The Signal and the Noise: Why So Many Predictions Fail, But Some Don't
Nate Silver
Business & Economics
Penguin Press HC
September 27, 2012
personal library

The founder of challenges myths about predictions in subjects ranging from the financial market and weather to sports and politics, profiling the world of prediction to explain how readers can distinguish true signals from hype, in a report that also reveals the sources and societal costs of wrongful predictions.

Started Reading: May 25, 2013
Finished Reading: October 13, 2013

Given the technical nature of what Nate Silver does, and some of the early mentions of the book, I had higher hopes for the technical portions of the book. As usual for a popular text, I was left wanting a lot more. Again, the lack of any math left a lot to desire. I wish technical writers could get away with even a handful of equations, but wishing just won’t make it so.

The first few chapters were a bit more technical sounding, but eventually devolved into a more journalistic viewpoint of statistics, prediction, and forecasting in general within the areas of economics, political elections, weather forecasting, earthquakes, baseball, poker, chess, and terrorism. I have a feeling he lost a large part of his audience in the first few chapters by discussing the economic meltdown of 2008 first instead of baseball or poker and then getting into politics and economics.

While some of the discussion around each of these bigger topics are all intrinsically interesting and there were a few interesting tidbits I hadn’t heard or read about previously, on the whole it wasn’t really as novel as I had hoped it would be. I think it should be required reading for all politicians however, as I too often get the feeling that none of them think at this level.

There was some reasonably good philosophical discussion of Bayesian statistics versus Fisherian, but it was all too short and could have been fleshed out more significantly. I still prefer David Applebaum’s historical and philosophical discussion of probability in Probability and Information: An Integrated Approach though he surprisingly didn’t mention R.A. Fisher directly himself in his coverage.

It was interesting to run across additional mentions of power laws in the realms of earthquakes and terrorism after reading Melanie Mitchell’s Complexity: A Guided Tour (review here), but I’ll have to find some texts which describe the mathematics in full detail. There was surprisingly large amount of discussion skirting around the topics within complexity without delving into it in any substantive form.

For those with a pre-existing background in science and especially probability theory, I’d recommend skipping this and simply reading Daniel Kahneman’s book Thinking, Fast and Slow. Kahneman’s work is referenced several times and his book seems less intuitive than some of the material Silver presents here.

This is the kind of text which should be required reading in high school civics classes. Perhaps it might motivate more students to be interested in statistics and science related pursuits as these are almost always at the root of most political and policy related questions at the end of the day.

For me, I’d personally give this three stars, but the broader public should view it with at least four stars if not five as there is some truly great stuff here. Unfortunately a lot of it is old hat or retreaded material for me.

Syndicated copies to:

Book Review: “Complexity: A Guided Tour” by Melanie Mitchell

Complexity: A Guided Tour by Melanie MitchellMelanie Mitchell (
Complexity: A Guided Tour Book Cover Complexity: A Guided Tour
Melanie Mitchell
Popular Science
Oxford University Press
May 28, 2009

This book provides an intimate, highly readable tour of the sciences of complexity, which seek to explain how large-scale complex, organized, and adaptive behavior can emerge from simple interactions among myriad individuals. The author, a leading complex systems scientist, describes the history of ideas, current research, and future prospects in this vital scientific effort.

This is handily one of the best, most interesting, and (to me at least) the most useful popularly written science books I’ve yet to come across. Most popular science books usually bore me to tears and end up being only pedantic for their historical backgrounds, but this one is very succinct with some interesting viewpoints (some of which I agree with and some of which my intuition says are terribly wrong) on the overall structure presented.

For those interested in a general and easily readable high-level overview of some of the areas of research I’ve been interested in (information theory, thermodynamics, entropy, microbiology, evolution, genetics, along with computation, dynamics, chaos, complexity, genetic algorithms, cellular automata, etc.) for the past two decades, this is really a lovely and thought-provoking book.

At the start I was disappointed that there were almost no equations in the book to speak of – and perhaps this is why I had purchased it when it came out and it’s subsequently been sitting on my shelf for so long. The other factor that prevented me from reading it was the depth and breadth of other more technical material I’ve read which covers the majority of topics in the book. I ultimately found myself not minding so much that there weren’t any/many supporting equations aside from a few hidden in the notes at the end of the text in most part because Dr. Mitchell does a fantastic job of pointing out some great subtleties within the various subjects which comprise the broader concept of complexity which one generally would take several years to come to on one’s own and at far greater expense of their time. Here she provides a much stronger picture of the overall subjects covered and this far outweighed the lack of specificity. I honestly wished I had read the book when it was released and it may have helped me to me more specific in my own research. Fortunately she does bring up several areas I will need to delve more deeply into and raised several questions which will significantly inform my future work.

In general, I wish there were more references I hadn’t read or been aware of yet, but towards the end there were a handful of topics relating to fractals, chaos, computer science, and cellular automata which I have been either ignorant of or which are further down my reading lists and may need to move closer to the top. I look forward to delving into many of these shortly. As a simple example, I’ve seen Zipf’s law separately from the perspectives of information theory, linguistics, and even evolution, but this is the first time I’ve seen it related to power laws and fractals.

I definitely appreciated the fact that Dr. Mitchell took the time to point out her own personal feelings on several topics and more so that she explicitly pointed them out as her own gut instincts instead of mentioning them passingly as if they were provable science which is what far too many other authors would have likely done. There are many viewpoints she takes which I certainly don’t agree with, but I suspect that it’s because I’m coming at things from the viewpoint of an electrical engineer with a stronger background in information theory and microbiology while hers is closer to that of computer science. She does mention that her undergraduate background was in mathematics, but I’m curious what areas she specifically studied to have a better understanding of her specific viewpoints.

Her final chapter looking at some of the pros and cons of the topic(s) was very welcome, particularly in light of previous philosophic attempts like cybernetics and general systems theory which I (also) think failed because of their lack of specificity. These caveats certainly help to place the scientific philosophy of complexity into a much larger context. I will generally heartily agree with her viewpoint (and that of others) that there needs to be a more rigorous mathematical theory underpinning the overall effort. I’m sure we’re all wondering “Where is our Newton?” or to use her clever aphorism that we’re “waiting for Carnot.” (Sounds like it should be a Tom Stoppard play title, doesn’t it?)

I might question her brief inclusion of her own Ph.D. thesis work in the text, but it did actually provide a nice specific and self-contained example within the broader context and also helped to tie several of the chapters together.

My one slight criticism of the work would be the lack of better footnoting within the text. Though many feel that footnote numbers within the text or inclusion at the bottom of the pages detracts from the “flow” of the work, I found myself wishing that she had done so here, particularly as I’m one of the few who actually cares about the footnotes and wants to know the specific references as I read. I hope that Oxford eventually publishes an e-book version that includes cross-linked footnotes in the future for the benefit of others.

I can heartily recommend this book to any fan of science, but I would specifically recommend it to any undergraduate science or engineering major who is unsure of what they’d specifically like to study and might need some interesting areas to take a look at. I will mention that one of the tough parts of the concept of complexity is that it is so broad and general that it encompasses over a dozen other fields of study each of which one could get a Ph.D. in without completely knowing the full depth of just one of them much less the full depth of all of them. The book is so well written that I’d even recommend it to senior researchers in any of the above mentioned fields as it is certainly sure to provide not only some excellent overview history of each, but it is sure to bring up questions and thoughts that they’ll want to include in their future researches in their own specific sub-areas of expertise.

Syndicated copies to: