Complexity Explorer Santa Fe Institute

Your progress is not being saved! Enroll now or log in to track your progress or submit homework.

Introduction to Information Theory

Lead instructor:

About the Tutorial:

This tutorial introduces fundamental concepts in information theory. Information theory has made considerable impact in complex systems, and has in part co-evolved with complexity science. Research areas ranging from ecology and biology to aerospace and information technology have all seen benefits from the growth of information theory.

In this tutorial, students will follow the development of information theory from bits to modern application in computing and communication. Along the way Seth Lloyd introduces valuable topics in information theory such as mutual information, boolean logic, channel capacity, and the natural relationship between information and entropy.

The tutorial covers a substantial amount of material while limiting discussion of the mathematics involved. When formulas or derivations are considered, the mathematics are described such that less advanced math students will find the tutorial accessible. Prerequisites for this tutorial are an understanding of logarithms, and at least a year of high-school algebra.

About the Instructor(s):

 Seth Lloyd is Professor of Mechanical Engineering at the Massachusetts Institute of Technology (MIT), currently on administrative leave.  He received his A.B. from Harvard College in 1982, the Certificate of Advanced Study in Mathematics (Part III) and an M. Phil. in Philosophy of Science from Cambridge University in 1983 and 1984 under a Marshall Fellowship, and a Ph.D. in Physics in 1988 from Rockefeller University under the supervision of Professor Heinz Pagels.

From 1988 to 1991, Professor Lloyd was a postdoctoral fellow in the High Energy Physics Department at the California Institute of Technology, where he worked with Professor Murray Gell-Mann on applications of information to quantum-mechanical systems. From 1991 to 1994, he was a postdoctoral fellow at Los Alamos National Laboratory, where he worked at the Center for Nonlinear Systems on quantum computation. In 1994, he joined the faculty of the Department of Mechanical Engineering at MIT.

 

A FURTHER NOTE ABOUT THIS TUTORIAL 

ComplexityExplorer has carefully considered whether or not to continue to make available this tutorial, due to instructor Seth Lloyd's association with the late Jeffrey Epstein. In consultation with SFI's leadership and ComplexityExplorer's founding faculty member, it was decided to leave the tutorial available for those who may find the content instructive. The scientific content remains, as before, valid and rigorous. 

This does not suggest that ComplexityExplorer in any way excuses or condones the actions of Dr. Lloyd with respect to Mr. Epstein. We herewith offer full transparency of the allegations facing Dr. Lloyd in this case and additionally note that he has been placed on administrative leave from his position at MIT pending the findings of a full legal investigation.

More details on the case can be found at the following:
http://factfindingjan2020.mit.edu/
https://news.mit.edu/2020/mit-releases-results-fact-finding-report-jeffrey-epstein-0110
https://www.nature.com/articles/d41586-020-00072-x

On balance, readers may also consider Dr. Lloyd's responses:
https://medium.com/@sqlloyd/i-am-writing-to-apologize-to-jeffrey-epsteins-victims-eee805c4f13
https://medium.com/@sqlloyd/what-i-told-mit-about-epsteins-donations-feec0beb26a

In the spirit of full transparency, also read the Santa Fe Institute’s statement on accepting funds from Epstein’s foundations, and its institutional response:
https://www.buzzfeednews.com/article/peteraldhous/jeffrey-epstein-sex-trafficking-science-donations
https://www.santafe.edu/news-center/news/sfi-gives-25k-solace-crisis-treatment-center

How to use Complexity Explorer
Enrolled students:

3,003

View Participant Map

Prerequisites:

At least one year of high-school algebra

Like this tutorial?

9 Student
Ratings


Syllabus

  1. Introduction
  2. Forms of Information
  3. Information and Probability
  4. Fundamental Formula of Information
  5. Computation and Logic: Information Processing
  6. Mutual Information
  7. Communication Capacity in a Noisy Channel
  8. Shannon's Coding Theorem
  9. The Manifold Things Information Measures
  10. Homework
  11. Supplementary Materials