Introduction to Information Theory | SFI’s Complexity Explorer

The Santa Fe Institute's free online course "Introduction to Information Theory" taught by Seth Lloyd via Complexity Explorer.

Many readers often ask me for resources for delving into the basics of information theory. I hadn’t posted it before, but the Santa Fe Institute recently had an online course Introduction to Information Theory through their Complexity Explorer, which has some other excellent offerings. It included videos, fora, and other resources and was taught by the esteemed physicist and professor Seth Lloyd. There are a number of currently active students still learning and posting there.

Introduction to Information Theory

About the Tutorial:

This tutorial introduces fundamental concepts in information theory. Information theory has made considerable impact in complex systems, and has in part co-evolved with complexity science. Research areas ranging from ecology and biology to aerospace and information technology have all seen benefits from the growth of information theory.

In this tutorial, students will follow the development of information theory from bits to modern application in computing and communication. Along the way Seth Lloyd introduces valuable topics in information theory such as mutual information, boolean logic, channel capacity, and the natural relationship between information and entropy.

Lloyd coherently covers a substantial amount of material while limiting discussion of the mathematics involved. When formulas or derivations are considered, Lloyd describes the mathematics such that less advanced math students will find the tutorial accessible. Prerequisites for this tutorial are an understanding of logarithms, and at least a year of high-school algebra.

About the Instructor(s):

Professor Seth Lloyd is a principal investigator in the Research Laboratory of Electronics (RLE) at the Massachusetts Institute of Technology (MIT). He received his A.B. from Harvard College in 1982, the Certificate of Advanced Study in Mathematics (Part III) and an M. Phil. in Philosophy of Science from Cambridge University in 1983 and 1984 under a Marshall Fellowship, and a Ph.D. in Physics in 1988 from Rockefeller University under the supervision of Professor Heinz Pagels.

From 1988 to 1991, Professor Lloyd was a postdoctoral fellow in the High Energy Physics Department at the California Institute of Technology, where he worked with Professor Murray Gell-Mann on applications of information to quantum-mechanical systems. From 1991 to 1994, he was a postdoctoral fellow at Los Alamos National Laboratory, where he worked at the Center for Nonlinear Systems on quantum computation. In 1994, he joined the faculty of the Department of Mechanical Engineering at MIT. Since 1988, Professor Lloyd has also been an adjunct faculty member at the Sante Fe Institute.

Professor Lloyd has performed seminal work in the fields of quantum computation and quantum communications, including proposing the first technologically feasible design for a quantum computer, demonstrating the viability of quantum analog computation, proving quantum analogs of Shannon’s noisy channel theorem, and designing novel methods for quantum error correction and noise reduction.

Professor Lloyd is a member of the American Physical Society and the Amercian Society of Mechanical Engineers.

Tutorial Team:

Yoav Kallus is an Omidyar Fellow at the Santa Fe Institute. His research at the boundary of statistical physics and geometry looks at how and when simple interactions lead to the formation of complex order in materials and when preferred local order leads to system-wide disorder. Yoav holds a B.Sc. in physics from Rice University and a Ph.D. in physics from Cornell University. Before joining the Santa Fe Institute, Yoav was a postdoctoral fellow at the Princeton Center for Theoretical Science in Princeton University.

How to use Complexity Explorer: How to use Complexity Explore
Prerequisites: At least one year of high-school algebra
Like this tutorial? 


  1. Introduction
  2. Forms of Information
  3. Information and Probability
  4. Fundamental Formula of Information
  5. Computation and Logic: Information Processing
  6. Mutual Information
  7. Communication Capacity
  8. Shannon’s Coding Theorem
  9. The Manifold Things Information Measures
  10. Homework
Syndicated copies to:

Dr. Michael Miller Math Class Hints and Tips | UCLA Extension

An informal orientation for those taking math classes from Dr. Michael Miller through UCLA Extension.

Congratulations on your new math class, and welcome to the “family”!

Beginners Welcome!

Invariably the handful of new students every year eventually figure the logistics of campus out, but it’s easier and more fun to know some of the options available before you’re comfortable halfway through the class. To help get you over the initial hump, I’ll share a few of the common questions and tips to help get you oriented. Others are welcome to add comments and suggestions below. If you have any questions, feel free to ask anyone in the class, we’re all happy to help.

First things first, for those who’ve never visited UCLA before, here’s a map of campus to help you orient yourself. Using the Waze app on your smartphone can also be incredibly helpful in getting to campus more quickly through the tail end of rush hour traffic.

Whether you’re a professional mathematician, engineer, physicist, physician, or even a hobbyist interested in mathematics you’ll be sure to get something interesting out of Dr. Miller’s math courses, not to mention the camaraderie of 20-30 other “regulars” with widely varying backgrounds (from actors to surgeons and evolutionary theorists to engineers) who’ve been taking almost everything Mike has offered over the years (and yes, he’s THAT good — we’re sure you’ll be addicted too.) Whether you’ve been away from serious math for decades or use it every day or even if you’ve never gone past Calculus or Linear Algebra, this is bound to be the most entertaining thing you can do with your Tuesday nights in the Autumn and Winter. If you’re not sure what you’re getting into (or are scared a bit by the course description), I highly encourage to come and join us for at least the first class before you pass up on the opportunity. I’ll mention that the greater majority of new students to Mike’s classes join the ever-growing group of regulars who take almost everything he teaches subsequently.

Don’t be intimidated if you feel like everyone in the class knows each other fairly well — most of us do. Dr. Miller and mathematics can be addictive so many of us have been taking classes from him for 5-20+ years, and over time we’ve come to know each other.

Tone of Class

If you’ve never been to one of Dr. Miller’s classes before, they’re fairly informal and he’s very open to questions from those who don’t understand any of the concepts or follow his reasoning. He’s a retired mathematician from RAND and long-time math professor at UCLA. Students run the gamut from the very serious who read multiple textbooks and do every homework problem to hobbyists who enjoy listening to the lectures and don’t take the class for a grade of any sort (and nearly every stripe in between). He’ll often recommend a textbook that he intends to follow, but it’s never been a “requirement” and more often that not, the bookstore doesn’t list or carry his textbook until the week before class. (Class insiders will usually find out about the book months before class and post it to the Google Group – see below).

His class notes are more than sufficient for making it through the class and doing the assigned (optional) homework. He typically hands out homework in handout form, so the textbook is rarely, if ever, required to make it through the class. Many students will often be seen reading various other texts relating to the topic at hand as they desire. Usually he’ll spend an 45-60 minutes at the opening of each class after the first to go over homework problems or questions that anyone has.

For those taking the class for a grade or pass/fail, his usual policy is to assign a take home problem set around week 9 or 10 to be handed in at the penultimate class. [As a caveat, make sure you check his current policy on grading as things may change, but the preceding has been the usual policy for a decade or more.]

Parking Options

Lot 9 – Located at the northern terminus of Westwood Boulevard, one can purchase a parking pass for about $12 a day at the kiosk in the middle of the street just before Westwood Blvd. ends. The kiosk is also conveniently located right next to the parking structure. If there’s a basketball game or some other major event, Lot 8 is just across the street as well, though it’s just a tad further away from the Math Sciences Building. Since more of the class uses this as their parking structure of choice, there is always a fairly large group walking back there after class for the more security conscious.

Lot 2 – Located off of Hilgard Avenue, this is another common option for easy parking as well. While fairly close to class, not as many use it as it’s on the quieter/darker side of campus and can be a bit more of a security issue for the reticent.

Tip: For those opting for on-campus parking, one can usually purchase a quarter-long parking pass for a small discount at the beginning of the term.

Westwood Village and Neighborhood – Those looking for less expensive options street parking is available in the surrounding community, but use care to check signs and parking meters as you assuredly will get a ticket. Most meters in the surrounding neighborhoods end at either 6pm or 8pm making parking virtually free (assuming you’re willing to circle the neighborhood to find one of the few open spots.)

There are a huge variety of lots available in the Village for a range of prices, but the two most common, inexpensive, and closer options seem to be:

  • Broxton Avenue Public Parking at 1036 Broxton Avenue just across from the Fox Village and Bruin Theaters – $3 for entering after 6pm / $9 max for the day
  • Geffen Playhouse Parking at 10928 Le Conte Ave. between Broxton and Westwood – price varies based on the time of day and potential events (screenings/plays in Westwood Village) but is usually $5 in the afternoon and throughout the evening

Dining Options

More often than not a group of between 4 and 15 students will get together every evening before class for a quick bit to eat and to catch up and chat. This has always been an informal group and anyone from class is more than welcome to join. Typically we’ll all meet in the main dining hall of Ackerman Union (Terrace Foodcourt, Ackerman Level 1) between 6 and 6:30 (some with longer commutes will arrive as early as 3-4pm, but this can vary) and dine until about 6:55pm at which time we walk over to class.

The food options on Ackerman Level 1 include Panda Express, Rubio’s Tacos, Sbarro, Wolfgang Puck, and Greenhouse along with some snack options including Wetzel’s Pretzels and a candy store. One level down on Ackerman A-level is a Taco Bell, Carl’s Jr., Jamba Juice, Kikka, Buzz, and Curbside, though one could get takeout and meet the rest of the “gang” upstairs.

There are also a number of other on-campus options as well though many are a reasonable hike from the class location. The second-closest to class is the Court of Sciences Student Center with a Subway, Yoshinoya, Bombshelter Bistro, and Fusion.

Naturally, for those walking up from Westwood Village, there are additional fast food options like In-N-Out, Chick-fil-A, Subway, and many others.

Killing Time

For those who’ve already eaten or aren’t hungry, you’ll often find one or more of us browsing the math and science sections of the campus bookstore on the ground level of Ackerman Union to kill time before class. Otherwise there are usually a handful of us who arrive a half an hour early and camp out in the classroom itself (though this can often be dauntingly quiet as most use the chance to catch up on reading here.) If you arrive really early, there are a number of libraries and study places on campus. Boelter Hall has a nice math/science library on the 8th Floor.

Mid-class Break Options

Usually about halfway through class we’ll take a 10-12 minute coffee break. For those with a caffeine habit or snacking urges, there are a few options:

Kerckhoff Hall Coffee Shop is just a building or two over and is open late as snack stop and study location. They offer coffee and various beverages as well as snacks, bagels, pastries, and ice cream. Usually 5-10 people will wander over as a group to pick up something quick.

The Math Sciences Breezeway, just outside of class, has a variety of soda, coffee, and vending machines with a range of beverages and snacks. Just a short walk around the corner will reveal another bank of vending machines if your vice isn’t covered. The majority of class will congregate in the breezeway to chat informally during the break.

The Court of Sciences Student Center, a four minute walk South, with the restaurant options noted above if you need something quick and more substantial, though few students use this option at the break.

Bathrooms – The closest bathrooms to class are typically on the 5th floor of the Math Sciences Building. The women’s is just inside the breezeway doors and slightly to the left. The men’s rooms are a bit further and are either upstairs on the 6th floor (above the women’s), or a hike down the hall to the left and into Boelter hall. I’m sure the adventurous may find others, but take care not to get lost.

Informal Class Resources

Google Group

Over the years, as an informal resource, members of the class have created and joined a private Google Group (essentially an email list-serv) to share thoughts, ideas, events, and ask questions of each other. There are over 50 people in the group, most of whom are past Miller students, though there are a few other various mathematicians, physicists, engineers, and even professors. You can request to join the private group to see the resources available there. We only ask that you keep things professional and civil and remember that replying to all reaches a fairly large group of friends. Browsing through past messages will give you an idea of the types of posts you can expect. The interface allows you to set your receipt preferences to one email per message posted, daily digest, weekly digest, or no email (you’re responsible for checking the web yourself), so be sure you have the setting you require as some messages are more timely than others. There are usually only 1-2 posts per week, so don’t expect to be inundated.

Study Groups

Depending on students’ moods, time requirements, and interests, we’ve arranged informal study groups for class through the Google Group above. Additionally, since Dr. Miller only teaches during the Fall and Winter quarters, some of us also take the opportunity to set up informal courses during the Spring/Summer depending on interests. In the past, we’ve informally studied Lie Groups, Quantum Mechanics, Algebraic Geometry, and Category Theory in smaller groups on the side.


As a class resource, some of us share a document repository via Dropbox. If you’d like access, please make a post to the Google Group.

Class Notes

Many people within the class use digital pens to capture not only the written notes but the audio discussion that occurred in class as well (the technology also links the two together to make it easier to jump around within a particular lecture). If it helps to have a copy of these notes, please let one of the users know you’d like them – we’re usually pretty happy to share. If you miss a class (sick, traveling, etc.) please let one of us know as the notes are so unique that it will be almost like you didn’t miss anything at all.

You can typically receive a link to the downloadable version of the notes in Livescribe’s Pencast .pdf format. This is a special .pdf file but it’s a bit larger in size because it has an embedded audio file in it that is playable with the more recent version of Adobe Reader X (or above) installed. (This means to get the most out of the file you have to download the file and open it in Reader X to get the audio portion. You can view the written portion in most clients, you’ll just be missing out on all the real fun and value of the full file.) With the notes, you should be able to toggle the settings in the file to read and listen to the notes almost as if you were attending the class live.

Viewing and Playing a Pencast PDF

Pencast PDF is a new format of notes and audio that can play in Adobe Reader X or above.

You can open a Pencast PDF as you would other PDF files in Adobe Reader X. The main difference is that a Pencast PDF can contain ink that has associated audio—called “active ink”. Click active ink to play its audio. This is just like playing a Pencast from Livescribe Online or in Livescribe Desktop. When you first view a notebook page, active ink appears in green type. When you click active ink, it turns gray and the audio starts playing. As audio playback continues, the gray ink turns green in synchronization with the audio. Non-active ink (ink without audio) is black and does not change appearance.

Audio Control Bar

Pencast PDFs have an audio control bar for playing, pausing, and stopping audio playback. The control bar also has jump controls, bookmarks (stars), and an audio timeline control.

Active Ink View Button

There is also an active ink view button. Click this button to toggle the “unwritten” color of active ink from gray to invisible. In the default (gray) setting, the gray words turn green as the audio plays. In the invisible setting, green words seem to write themselves on blank paper as the audio plays.


For those interested in past years’ topics, here’s the list I’ve been able to put together thus far:

Fall 2006: Complex Analysis
Winter 2007: Field Theory
Fall 2007: Algebraic Topology
Winter 2008: Integer Partitions
Fall 2008: Calculus on Manifolds
Winter 2009: Calculus on Manifolds: The Sequel
Fall 2009: Group Theory
Winter 2010: Galois Theory
Fall 2010: Differential Geometry
Winter 2011: Differential Geometry II
Fall 2011: p-Adic Analysis
Winter 2012: Group Representations
Fall 2012: Set Theory
Winter 2013: Functional Analysis
Fall 2013: Number Theory (Skipped)
Winter 2014: Measure Theory
Fall 2014: Introduction to Lie Groups and Lie Algebras Part I
Winter 2015: Introduction to Lie Groups and Lie Algebras Part II
Fall 2015: Algebraic Number Theory
Winter 2016: Algebraic Number Theory: The Sequel
Fall 2016: Introduction to Complex Analysis, Part I
Winter 2017: Introduction to Complex Analysis, Part II

Syndicated copies to: