Spaced-repetition flashcards for Information Theory.
| File | Content |
|---|---|
01_foundations.md |
What Information Theory Studies, Probability Review, Joint and Conditional Probability, Independence, The Self-Information, Units of Information, … |
02_entropy.md |
Defining Entropy, Why Shannon's Formula, Properties of Entropy, The Binary Entropy Function, Entropy of Common Distributions, Entropy of Pairs and Joint Entropy, … |
03_joint_conditional_entropy.md |
Conditional Entropy, Chain Rule for Entropy, Conditioning Reduces Entropy, Subadditivity and Independence Bound, Fano's Inequality, Conditioning on a Function, … |
04_kl_mutual_information.md |
Kullback-Leibler Divergence, Properties of KL Divergence, Mutual Information, Mutual Information and Entropy, Non-Negativity of Mutual Information, Conditional Mutual Information, … |
05_source_coding.md |
The Source Coding Problem, Codes and Codewords, Code Types, Kraft's Inequality, Optimal Code Length, Shannon's Source Coding Theorem, … |
06_huffman_arithmetic.md |
Huffman Coding, Properties of Huffman Codes, Block Huffman Coding, Huffman vs Shannon-Fano, Arithmetic Coding, Implementation of Arithmetic Coding, … |
07_channels_capacity.md |
The Channel Model, Channel Capacity, The Binary Symmetric Channel, The Binary Erasure Channel, Other Important Channels, Symmetric Channels, … |
08_noisy_channel_coding.md |
Shannon's Noisy Channel Coding Theorem, Random Coding, Joint Typical Set, Achievability Proof Sketch, Converse: Why You Can't Exceed Capacity, Fano's Inequality in Converse, … |
09_gaussian_rate_distortion.md |
Differential Entropy, Gaussian Maximizes Entropy, The AWGN Channel, Shannon-Hartley Theorem, Parallel Gaussian Channels, Rate-Distortion: Lossy Compression, … |
10_error_correcting_codes.md |
Why Error-Correcting Codes, Hamming Distance, Hamming Bound, Hamming Codes, Linear Codes, Minimum Distance and Singleton Bound, … |
11_kolmogorov_complexity.md |
Algorithmic Information, Kolmogorov Complexity Definition, Incompressibility, Relationship to Shannon Entropy, The Coding Theorem, Undecidability of |
12_information_ml.md |
The Information-Theoretic Lens on ML, Cross-Entropy Loss, KL Divergence as Regularization, Maximum Entropy Principle, Mutual Information for Feature Selection, Information Bottleneck, … |
Cards use the hashcards format:
Q: Question
A: Answer
C: Text with [cloze deletion] inline
Problem cards (P:/S:) walk through IDENTIFY / PLAN / EXECUTE / EVALUATE.
Card quality rules live in CARD_STANDARD.md.
Add this repository to flashcards while logged in with your GitHub account.