A picture is where the bias becomes visible. The plotting code lives in src/primes_in_intervals/plotting.py (pii plot and pii animate from the shell), and the heart of it is one function, plot_distribution_frame, which draws a single checkpoint’s distribution onto an axis.
The data are drawn as translucent bars, with a red dot on top of each. The predictions of Chapter 8 are drawn as markers at the integers \(m\), each model with its own colour and marker: by default \(F\) (green squares), \(F_0\) (blue triangles) and \(\mathrm{Binom}(H, \mu/H)\) (orange diamonds) for overlapping intervals, and only the Binomial for the other interval types, since \(F\) and \(F_0\) were derived for overlapping intervals. The argument models chooses others, including the local expressions \(Q(m; \mu, H)\) and \(Q(m; \lambda, H)\). The predictions are those for the checkpoint’s own range of starting points (\(M = A\) and \(N = c - A\) for a checkpoint \(c\) of a dataset with lower bound \(A\)), computed by the same predict_all that compare uses, so a plot and a table can never disagree.
Only the integers \(m\) carry a meaning, which is why the predictions are markers. A dashed guide curve runs through each set of markers to help the eye; it interpolates \(m!\) by the gamma function and means nothing between the integers (pass guides=False to leave it out). When a prediction is negative, which \(F\) is in the tails, the vertical axis extends below zero and the least value is noted in the corner, rather than being cut off at zero.
A text box shows the parameters and the summary statistics of the frame, including \(\mu\) and \(\lambda\); the legend sits in the upper left, and a grid is drawn underneath everything else. The axis is cleared first, so the same function serves as the frame function of an animation: animate_distribution runs it across the checkpoints (skipping the trivial first one), holding the axes fixed so the frames are comparable, and returns the figure and the animation; save_gif and save_mp4 write the result to a file.
Every chapter runs in its own session; the folded cells below rebuild the running datasets of Chapter 6 and apply analyze to each, which is all the plots need.
Setup: rebuild the running datasets from the database
import numpy as np# NumPy 2 displays scalars inside containers as np.float64(...); restore the# plain numeric display so statistics read cleanly.np.set_printoptions(legacy="1.25")from primes_in_intervals import nest, partition, retrievedef pick(found, lower_bound):"""Return the retrieved dataset whose data begins at the given lower bound. Introduced in the Datasets chapter: accepts either return shape of ``retrieve`` (one dataset, or a list of them). """ifisinstance(found, dict): found = [found]returnnext(d for d in found if d['header']['lower_bound'] == lower_bound)# Example 1: interval length 100 over (2 * 10**6, 3 * 10**6].A1, B1, H1 =2*10**6, 3*10**6, 100retrieve_data1disj = retrieve(H1, 'disjoint')retrieve_data1olap = pick(retrieve(H1, 'overlap'), A1)partition(retrieve_data1disj)partition(retrieve_data1olap)nest_data1disj = nest(retrieve(H1, 'disjoint'))nest_data1olap = nest(pick(retrieve(H1, 'overlap'), A1))# Example 2: interval length 85 about exp(18), and the companion length-90# datasets used for nesting.N2, H2 =int(np.exp(18)), 85retrieve_data2disj = retrieve(H2, 'disjoint')retrieve_data2olap = retrieve(H2, 'overlap')partition(retrieve_data2disj)partition(retrieve_data2olap)nest_data2disj = nest(retrieve(90, 'disjoint'))nest_data2olap = nest(pick(retrieve(90, 'overlap'), 65559979))
Setup: analyze the running datasets
from primes_in_intervals import analyzefor _ds in (retrieve_data1disj, retrieve_data1olap, retrieve_data2disj, retrieve_data2olap, nest_data1disj, nest_data1olap, nest_data2disj, nest_data2olap): analyze(_ds)
Example 1
The final frame for the overlapping intervals of Example 1: a million intervals \((a, a + 100]\), with \(a\) between two and three million.
The distribution is more concentrated about its centre than either \(F_0\) or the Binomial allows, and the green squares of \(F\) follow it much more closely: the bias of the book’s title, on real data.
Example 2
The final frame for Example 2’s overlapping intervals about \(e^{18}\), \(H = 85\). This time we write our own text box, describing the range as a window centred at \(x_0 = [e^{18}]\). The argument overlay accepts a function, which receives the dataset, the checkpoint, \(H\), and the predictions being drawn, and returns the text; that way the box can quote \(\mu\) from the very predictions on the plot.
import numpy as npX = retrieve_data2olapC =list(X['distribution'].keys())x0 =int(np.exp(18))def centred_overlay(X, c, H, pred): s = X['statistics'][c] med =int(s['med']) if s['med'] ==int(s['med']) else s['med']return (fr'$X = \pi(a + H) - \pi(a)$'+'\n\n'+fr'$x_0 - K < a \leq x_0 + K$'+'\n\n'+fr'$H = {H}$, $x_0 = [\exp(18)]$, $K = {c - x0}$'+'\n\n'+fr'$\mu = {pred["mu"]:.5f}$'+'\n\n'+r'$\mathbb{E}[X] = $'+f"{s['mean']:.5f}"+'\n\n'+r'$\mathrm{Var}(X) = $'+f"{s['var']:.5f}"+'\n\n'+fr'median : ${med}$'+'\n\n'+fr"mode(s): ${s['mode']}$")fig, ax = plt.subplots(figsize=(22, 11))fig.suptitle('Primes in intervals')plot_distribution_frame(ax, X, C[-1], overlay=centred_overlay, overlay_position=(0.74, 0.18))plt.show()
The window is narrow and far from the origin, so the density of primes hardly changes across it: \(\mu\) is \(85/18\) to five decimal places.
The full animations step through every checkpoint of these figures, the sample growing frame by frame; they are produced with animate_distribution followed by save_gif or save_mp4, or in one line from the shell with pii animate. The worked examples to come lean on them.
# Plot & animate {#sec-plot}A picture is where the bias becomes visible. The plotting code lives in `src/primes_in_intervals/plotting.py` (`pii plot` and `pii animate` from the shell), and the heart of it is one function, `plot_distribution_frame`, which draws a single checkpoint's distribution onto an axis.The data are drawn as translucent bars, with a red dot on top of each. The predictions of @sec-compare are drawn as markers at the integers $m$, each model with its own colour and marker: by default $F$ (green squares), $F_0$ (blue triangles) and $\mathrm{Binom}(H, \mu/H)$ (orange diamonds) for overlapping intervals, and only the Binomial for the other interval types, since $F$ and $F_0$ were derived for overlapping intervals. The argument `models` chooses others, including the local expressions $Q(m; \mu, H)$ and $Q(m; \lambda, H)$. The predictions are those for the checkpoint's own range of starting points ($M = A$ and $N = c - A$ for a checkpoint $c$ of a dataset with lower bound $A$), computed by the same `predict_all` that `compare` uses, so a plot and a table can never disagree.Only the integers $m$ carry a meaning, which is why the predictions are markers. A dashed guide curve runs through each set of markers to help the eye; it interpolates $m!$ by the gamma function and means nothing between the integers (pass `guides=False` to leave it out). When a prediction is negative, which $F$ is in the tails, the vertical axis extends below zero and the least value is noted in the corner, rather than being cut off at zero.A text box shows the parameters and the summary statistics of the frame, including $\mu$ and $\lambda$; the legend sits in the upper left, and a grid is drawn underneath everything else. The axis is cleared first, so the same function serves as the frame function of an animation: `animate_distribution` runs it across the checkpoints (skipping the trivial first one), holding the axes fixed so the frames are comparable, and returns the figure and the animation; `save_gif` and `save_mp4` write the result to a file.Every chapter runs in its own session; the folded cells below rebuild the running datasets of @sec-raw_data and apply `analyze` to each, which is all the plots need.{{< include _setup_analyzed.qmd >}}### Example 1 {#sec-eg1plot}The final frame for the overlapping intervals of Example 1: a million intervals $(a, a + 100]$, with $a$ between two and three million.```{python}import matplotlib.pyplot as pltfrom primes_in_intervals import plot_distribution_frameX = retrieve_data1olapC =list(X['distribution'].keys())plt.rcParams.update({'font.size': 22})fig, ax = plt.subplots(figsize=(22, 11))fig.suptitle('Primes in intervals')plot_distribution_frame(ax, X, C[-1], x_pad=0)plt.rcParams['savefig.facecolor'] ='white'plt.show()```The distribution is more concentrated about its centre than either $F_0$ or the Binomial allows, and the green squares of $F$ follow it much more closely: the bias of the book's title, on real data.### Example 2 {#sec-eg2plot}The final frame for Example 2's overlapping intervals about $e^{18}$, $H = 85$. This time we write our own text box, describing the range as a window centred at $x_0 = [e^{18}]$. The argument `overlay` accepts a function, which receives the dataset, the checkpoint, $H$, and the predictions being drawn, and returns the text; that way the box can quote $\mu$ from the very predictions on the plot.```{python}import numpy as npX = retrieve_data2olapC =list(X['distribution'].keys())x0 =int(np.exp(18))def centred_overlay(X, c, H, pred): s = X['statistics'][c] med =int(s['med']) if s['med'] ==int(s['med']) else s['med']return (fr'$X = \pi(a + H) - \pi(a)$'+'\n\n'+fr'$x_0 - K < a \leq x_0 + K$'+'\n\n'+fr'$H = {H}$, $x_0 = [\exp(18)]$, $K = {c - x0}$'+'\n\n'+fr'$\mu = {pred["mu"]:.5f}$'+'\n\n'+r'$\mathbb{E}[X] = $'+f"{s['mean']:.5f}"+'\n\n'+r'$\mathrm{Var}(X) = $'+f"{s['var']:.5f}"+'\n\n'+fr'median : ${med}$'+'\n\n'+fr"mode(s): ${s['mode']}$")fig, ax = plt.subplots(figsize=(22, 11))fig.suptitle('Primes in intervals')plot_distribution_frame(ax, X, C[-1], overlay=centred_overlay, overlay_position=(0.74, 0.18))plt.show()```The window is narrow and far from the origin, so the density of primes hardly changes across it: $\mu$ is $85/18$ to five decimal places.The full animations step through every checkpoint of these figures, the sample growing frame by frame; they are produced with `animate_distribution` followed by `save_gif` or `save_mp4`, or in one line from the shell with `pii animate`. The worked examples to come lean on them.