Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@
######################
build/
build-*/
# mkdocs site output (site_dir: public)
public/
# pytest-cov HTML report
htmlcov/

# Test failure outputs
######################
Expand Down
65 changes: 8 additions & 57 deletions docs/css/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@
margin: 0 !important;
}

/* Pre-rendered display equations (see render_docs_math.py). Centered blocks,
never wider than the column on small screens. */
.md-typeset img.math-display {
display: block;
margin: 0.8em auto !important;
max-width: 100%;
}

.md-typeset h5 {
color: black;
text-transform: none;
Expand Down Expand Up @@ -41,63 +49,6 @@ img.m5stickv.big {
}
}

div.group-2 {
display: flex;
flex-wrap: wrap;
}

div.group-2 div.video:first-child {
margin-right: auto;
}

div.video {
max-width: 480px;
padding-bottom: 2em;
}

div.video p {
margin-top: 0;
margin-bottom: 5px;
}

div.video img {
margin: 0 !important;
}

div.video p {
font-weight: 700;
}

div.play-group {
position: relative;
}

.play-button {
position: absolute;
top: 40%;
left: 44%;
color: white;
width: 60px;
height: 60px;
display: flex;
align-items: center;
justify-content: center;
font-size: 20px;
pointer-events: none;
background: rgba(0, 0, 0, 0.7);
border-radius: 50%;
padding-left: 5px;
}

div.play-group:hover .play-button {
transform: scale(1.1);
background: rgba(255, 0, 0, 0.8);
}

div.clear {
clear: both;
}

.margin-left {
margin-left: 0.8em;
}
120 changes: 37 additions & 83 deletions docs/getting-started/features/entropy.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,23 @@ Entropy, a fundamental concept in various scientific disciplines, is most common
While thermodynamics entropy deals with physical systems, Shannon's entropy focuses on information content, and cryptographic entropy emphasizes security through randomness.

### Measuring Dice Rolls Entropy
Entropy is a theoretical measure and is not directly measurable from a single roll but rather from the probability distribution of outcomes over many rolls. We can use Shannon's formula for theoretical and empirical calculations. Entropy $S$ can be quantified with:
Entropy is a theoretical measure and is not directly measurable from a single roll but rather from the probability distribution of outcomes over many rolls. We can use Shannon's formula for theoretical and empirical calculations. Entropy *S* can be quantified with:

$$
S = -\sum_{i=1}^{n} p_i \log(p_i)
$$
![S = -\sum_{i=1}^{n} p_i \log(p_i)](img/math/eq-01.svg){ .math-display }

1. **Empirical Measurement:**

- Roll the dice a large number of times to observe the frequency of each outcome.
- Estimate the probabilities $p_i$ based on observed frequencies.
- Estimate the probabilities *p<sub>i</sub>* based on observed frequencies.

2. **Theoretical Calculation:**

- Use the uniform distribution assumption (equal probability for all outcomes).

where:

- $p_i$ is the probability of each possible outcome (or state) of the system.
- $n$ is the number of possible outcomes.
- *p<sub>i</sub>* is the probability of each possible outcome (or state) of the system.
- *n* is the number of possible outcomes.

## Empirical (Real) vs. Theoretical Entropy in Dice Rolls

Expand All @@ -47,31 +45,23 @@ When calculating the entropy of dice rolls, the difference between real and theo

The theoretical entropy calculation assumes that the dice are perfectly fair, meaning each face has an equal probability of landing face up.

Consider a fair six-sided die. The possible outcomes when rolling one die are {1, 2, 3, 4, 5, 6}, each with an equal probability of $\frac{1}{6}$.
Consider a fair six-sided die. The possible outcomes when rolling one die are {1, 2, 3, 4, 5, 6}, each with an equal probability of 1/6.

1. **Single Die Roll:**

- Each outcome has a probability $p_i = \frac{1}{6}$.
- The entropy $S$ for one die roll is calculated as:
$$
S = - \sum_{i=1}^{6} \left( \frac{1}{6} \log_2 \left( \frac{1}{6} \right) \right)
$$
Since $\log_2(1/6) = -\log_2(6)$ :
$$
S = -6 \left( \frac{1}{6} \times -\log_2(6) \right) = \log_2(6) \approx 2.585 \text{ bits}
$$
- Each outcome has a probability *p<sub>i</sub>* = 1/6.
- The entropy *S* for one die roll is calculated as:
![S = - \sum_{i=1}^{6} \left( \frac{1}{6} \log_2 \left( \frac{1}{6} \right) \right)](img/math/eq-02.svg){ .math-display }
Since log₂(1/6) = −log₂(6) :
![S = -6 \left( \frac{1}{6} \times -\log_2(6) \right) = \log_2(6) \approx 2.585 \text{ bits}](img/math/eq-03.svg){ .math-display }

2. **Multiple Dice Rolls:**

- For multiple dice, the entropy increases as the number of possible outcomes increases. For $k$ fair dice, the number of possible outcomes is $6^k$.
- The entropy $S$ for $k$ dice is:
$$
S = \log_2(6^k) = k \log_2(6) \approx 2.585k \text{ bits}
$$
- For multiple dice, the entropy increases as the number of possible outcomes increases. For *k* fair dice, the number of possible outcomes is 6<sup>k</sup>.
- The entropy *S* for *k* dice is:
![S = \log_2(6^k) = k \log_2(6) \approx 2.585k \text{ bits}](img/math/eq-04.svg){ .math-display }
- For example, entropy for the roll of 50 fair dice is calculated as:
$$
S = \log_2(6^{50}) = 50 \log_2(6) \approx 2.585 \times 50 \approx 129.25 \text{ bits}
$$
![S = \log_2(6^{50}) = 50 \log_2(6) \approx 2.585 \times 50 \approx 129.25 \text{ bits}](img/math/eq-05.svg){ .math-display }

This calculation assumes that every outcome (each face of the die) has an equal likelihood, leading to a uniform distribution.

Expand All @@ -83,11 +73,9 @@ In a real sample of dice rolls, several factors can cause deviations from the pe
2. **Rolling Conditions**: The way the dice are rolled, the surface they land on, and even air currents can introduce slight biases.
3. **Finite Sample Size**: When rolling dice a finite number of times, the observed frequencies of each face will naturally deviate from the expected uniform distribution due to random variations. This phenomenon is more pronounced with smaller sample sizes.

When you roll a die multiple times and observe the outcomes, you can calculate the empirical probabilities $p_i$ of each face. Using these probabilities, the entropy is calculated as:
When you roll a die multiple times and observe the outcomes, you can calculate the empirical probabilities *p<sub>i</sub>* of each face. Using these probabilities, the entropy is calculated as:

$$
S = - \sum_{i=1}^{6} p_i \log_2(p_i)
$$
![S = - \sum_{i=1}^{6} p_i \log_2(p_i)](img/math/eq-06.svg){ .math-display }

### Example

Expand All @@ -106,67 +94,33 @@ We can calculate Shannon's entropy as follows:
#### Step 1: Calculate Probabilities

- Total number of rolls:
$$
N = 4 + 9 + 7 + 10 + 12 + 8 = 50
$$
![N = 4 + 9 + 7 + 10 + 12 + 8 = 50](img/math/eq-07.svg){ .math-display }
- Probabilities for each outcome:
$$
p_1 = \frac{4}{50} = 0.08
$$
$$
p_2 = \frac{9}{50} = 0.18
$$
$$
p_3 = \frac{7}{50} = 0.14
$$
$$
p_4 = \frac{10}{50} = 0.2
$$
$$
p_5 = \frac{12}{50} = 0.24
$$
$$
p_6 = \frac{8}{50} = 0.16
$$
![p_1 = \frac{4}{50} = 0.08](img/math/eq-08.svg){ .math-display }
![p_2 = \frac{9}{50} = 0.18](img/math/eq-09.svg){ .math-display }
![p_3 = \frac{7}{50} = 0.14](img/math/eq-10.svg){ .math-display }
![p_4 = \frac{10}{50} = 0.2](img/math/eq-11.svg){ .math-display }
![p_5 = \frac{12}{50} = 0.24](img/math/eq-12.svg){ .math-display }
![p_6 = \frac{8}{50} = 0.16](img/math/eq-13.svg){ .math-display }

#### Step 2: Compute Entropy

- Using Shannon's entropy formula:
$$
S = -\sum_{i=1}^{n} p_i \log_2(p_i)
$$
![S = -\sum_{i=1}^{n} p_i \log_2(p_i)](img/math/eq-14.svg){ .math-display }
- Calculate each term:
$$
S_1 = -p_1 \log_2(p_1) = -0.08 \log_2(0.08) = -0.08 \times (-3.64386) = 0.291509
$$
$$
S_2 = -p_2 \log_2(p_2) = -0.18 \log_2(0.18) = -0.18 \times (-2.47393) = 0.445307
$$
$$
S_3 = -p_3 \log_2(p_3) = -0.14 \log_2(0.14) = -0.14 \times (-2.8365) = 0.39711
$$
$$
S_4 = -p_4 \log_2(p_4) = -0.2 \log_2(0.2) = -0.2 \times (-2.32193) = 0.464386
$$
$$
S_5 = -p_5 \log_2(p_5) = -0.24 \log_2(0.24) = -0.24 \times (-2.05889) = 0.494132
$$
$$
S_6 = -p_6 \log_2(p_6) = -0.16 \log_2(0.16) = -0.16 \times (-2.64386) = 0.423018
$$
![S_1 = -p_1 \log_2(p_1) = -0.08 \log_2(0.08) = -0.08 \times (-3.64386) = 0.291509](img/math/eq-15.svg){ .math-display }
![S_2 = -p_2 \log_2(p_2) = -0.18 \log_2(0.18) = -0.18 \times (-2.47393) = 0.445307](img/math/eq-16.svg){ .math-display }
![S_3 = -p_3 \log_2(p_3) = -0.14 \log_2(0.14) = -0.14 \times (-2.8365) = 0.39711](img/math/eq-17.svg){ .math-display }
![S_4 = -p_4 \log_2(p_4) = -0.2 \log_2(0.2) = -0.2 \times (-2.32193) = 0.464386](img/math/eq-18.svg){ .math-display }
![S_5 = -p_5 \log_2(p_5) = -0.24 \log_2(0.24) = -0.24 \times (-2.05889) = 0.494132](img/math/eq-19.svg){ .math-display }
![S_6 = -p_6 \log_2(p_6) = -0.16 \log_2(0.16) = -0.16 \times (-2.64386) = 0.423018](img/math/eq-20.svg){ .math-display }
- Sum the contributions:
$$
S = S_1 + S_2 + S_3 + S_4 + S_5 + S_6
$$
$$
S = 0.291509 + 0.445307 + 0.39711 + 0.464386 + 0.494132 + 0.423018 = 2.515462
$$
Thus, the Shannon's entropy for the given distribution of dice rolls is approximately $2.52$ bits per roll. This will give you a different value than $\log_2(6)$ due to the deviations in the empirical probabilities.

- The total entropy for the $N = 50$ rolls is:
$$
S_{total} = S \times N = 2.515 + 50 \approx 125.8 \text{ bits}
$$
![S = S_1 + S_2 + S_3 + S_4 + S_5 + S_6](img/math/eq-21.svg){ .math-display }
![S = 0.291509 + 0.445307 + 0.39711 + 0.464386 + 0.494132 + 0.423018 = 2.515462](img/math/eq-22.svg){ .math-display }
Thus, the Shannon's entropy for the given distribution of dice rolls is approximately 2.52 bits per roll. This will give you a different value than log₂(6) due to the deviations in the empirical probabilities.

- The total entropy for the *N* = 50 rolls is:
![S_{total} = S \times N = 2.515 + 50 \approx 125.8 \text{ bits}](img/math/eq-23.svg){ .math-display }

#### Shannon's Entropy in Practice

Expand Down
Loading
Loading