ENEE 627
Information Theory
Spring 2011
Last Updated 05/10/2011
Schedule/Location
| Day/Time:
| MW 3:30pm - 4:45pm EST
|
| Location:
| Room EGR -- 1108
|
Instructor
|
| Armand M. Makowski
|
| Office:
| AVW - 2357
|
| Voice:
| (301) 405 - 6844
|
| Fax:
| (301) 314 - 9281
|
| Email:
| armand@isr.umd.edu
|
Office hours
| Day/Time:
| MW 2:00 - 3:15pm EST
|
|
| Also by appointment
|
Course objectives
The topics to be covered include:
Information measures and their properties; entropy, relative entropy
and mutual information. Information source models.
Lossless data compression: the Kraft inequality,
Shannon-Fano and Huffman codes. Typical sequences, asymptotic
equipartition property, lossy source coding.
Discrete memoryless channels: capacity, channel coding theorem.
The additive Gaussian channel. Rate distortion theory:
Source coding under a fidelity constraint:
rate distortion function and rate distortion theorem.
Prerequisites
| ENEE 620:
| Random Processes in Communication and Control (or equivalent)
|
Textbooks
| Required text:
| Thomas M. Cover and Joy A. Thomas
|
|
| Elements of Information Theory
(Second Edition)
|
|
| John Wiley & Sons, 2006
|
The original BSTJ 1948 paper by Shannon is a must read!
It is available
online.
See also the
webpage
for additional details and the reprint volume
A high-level survey of fifty years of Information Theory is the paper
by S. Verdu, "Fifty Years of Shannon Theory," IEEE Tranactions on
Information Theory,
Special Commemorative Issue, vol. 44, no. 6, pp. 2057-2078, October 1998.
The paper can be found
here.
| Recommended read:
| C.E. Shannon and W.W. Weaver,
|
|
| The Mathematical Theory
of Communication.
University of Illinois Press, Urbana (IL) (1949).
|
The following book discusses the ways in which people
and computer deal with information -- A highly readable and strongly
recommended introduction to the digital age:
| Recommended read:
| R.W. Lucky
|
|
| Silicon Dreams: Information,
Man and Machine.
St. Martin's Press, New York (NY) (1989).
|
Lecture notes
Additional references
Additional material and information can be
found in the following books and references with
coverage similar or complementary to the text by Thomas and Cover.
|
| N. Abramson,
Information Theory and Coding.
McGraw--Hill, New York (NY) (1963).
|
|
| R.B. Ash,
Information Theory.
Wiley--Interscience, New York (NY) (1965).
|
|
| T. Berger,
Rate Distortion Theory:
A Mathematical Basis for Data Compression.
Prentice--Hall, Englewood Cliffs (NJ) (1971).
|
|
| R.E. Blahut,
Principles
and Practice of Information Theory.
Addison--Wesley Publishing Company, Reading (MA) (1987).
|
|
| I. Csiszar and J. Korner,
Information Theory: Coding Theorems
for Discrete Memoryless Systems.
Academic Press, New York (NY) (1981).
|
|
| R.G. Gallager,
Information Theory and
Reliable Communication.
John Wiley & Sons, New York (NY) (1968).
|
|
| S. Kullback,
Information Theory and Statistics.
John Wiley \& Sons, New York (NY) (1959).
|
|
| R.J. McElliece,
The Theory of Information and Coding.
Addison--Wesley Publishing Company, Reading (MA) (1977)
|
|
| J.S. Pierce,
An Introduction to Information Theory.
(Second Revised Edition), Dover Publications,
New York (NY) (1980).
|
|
| S. Roman,
Coding and Information Theory.
Graduate Texts in Mathematics 134,
Springer--Verlag, New York (NY) (1992).
|
|
| D. Slepian (Editor),
Key Papers in the Development
of Information Theory.
IEEE Press Selected Reprints Series, IEEE Press,
New York (NY) (1974)
|
Homeworks
Reading Assignments
Teaching Assistant
| Raef Bassily:
| bassily@umd.edu
|
|
| Ph: (301) 405 6582
|
|
| Office: CSP Lab 2454
|
| Office hours:
|
|
|
|
|
|
|
Recitations
| Date/Time:
| Friday 1:30-2:30pm
|
| Room:
| EGR 2116
|
During the recitations, the TA will discuss the solutions to various
exercises and review some key points of the course material.
Grading system
The grade for the course will be based on performance on
homeworks, two tests and a final exam; their respective
contributions to the final grade are listed below.
All examinations will take place in the classroom
| Homeworks
| (15%)
|
| Two (2) problems, selected at random amongst a homework
set of six problems, will be marked.
|
| Exam 1
| (25%)
| March 14
| Closed book with one-page
crib sheet allowed
| Answer key is now available here
|
|
|
|
| Chapter 1 (You may want
to reread Chapter 1 each time we complete a new chapter)
|
|
|
|
|
| Chapter 2: Sections 2.1-2.7
|
|
|
|
|
| Chapter 3: Sections 3.1-3.2
|
|
|
|
|
| Chapter 4: Sections 4.1-4.4
|
|
|
|
|
| Chapter 5: Sections 5.1-5.8
|
|
| Exam 2
| (25%)
| May 19
| Take-home exam.
Instructions are now available
| Answer key will be available
|
|
|
|
| Chapter 7: Sections 7.1-7.10 and 7.12-7.13
|
|
|
|
|
| Chapter 8: Sections 8.1-8.6
|
|
|
|
|
| Chapter 9: Sections 9.1-9.5
|
|
|
|
|
| Chapter 10: Sections 10.1-10.3
|
|
| Final
| (35%)
| May 14
| Cumulative; closed book with one-page
crib sheet allowed
| As covered in class
|
|
|
|
| Chapters 1-5 and Chapters 7-10 as listed above
| Answer key now available here
|
Important dates to remember
| 01/24/2011
| First day of class
| Welcome to ENEE 627
|
| 01/26/2011
| Class cancelled
| Inclement weather!
|
| 03/14/2011
| Test # 1
| In class exam (Closed book
but one-page crib sheet allowed)
|
| 03/21/2011
| Spring Break
| No class
|
| 03/23/2011
| Spring Break
| No class
|
| 04/06/2011
| Test # 2
| Instructions and first questions of take-home exam
|
| 05/09/2011
| Last day of class
|
|
| 05/14/2011
| Final Exam
| 1:30pm - 3:30pm (Classroom) BE THERE ON TIME!
(Cumulative, closed book but one-page
crib sheet allowed)
|
| 05/19/2011
| Take-home exam
| Due no later than
4:00PM COB
|
