# Generic chaining and Dudley's entropy bound (Wed, Jan 11)

Recall that we consider a subgaussian process $\{X_t : t \in T\}$ and equip $T$ with the distance $d(s,t) = \sqrt{\E(X_s-X_t)^2}$.

• Chaining: Instead of applying a naive union bound to control $X_t - X_{t_0}$ for every $t \in T$, we can break every such difference into parts and then bound each part. This potentially reduces the number of events we need to take a union bound over.

For instance, given three random variables $X_0,X_1,X_2,X_3$, we could write

$\begin{eqnarray*} X_3 - X_0 &=& (X_3 - X_1) + (X_1 - X_0) \\ X_2 - X_0 &=& (X_2 - X_1) + (X_1 - X_0) \\ X_1 - X_0 &=& X_1 - X_0\,. \end{eqnarray*}$
• Generic chaining upper bound: For sub-gaussian processes, we can control $\E \max_{t \in T} X_t$ using chaining.

Fix some $t_0 \in T$ and a sequence of finite sets

$\{t_0\} = T_0 \subseteq T_1 \subseteq T_2 \subseteq \cdots \subseteq T_n \subseteq \cdots \subseteq T\,,$

with cardinalities bounded by $\abs{T_n} \leq 2^{2^n}$ for every $n \geq 1$.

Then we have:

$$$\label{eq:gc-ub} \E \max_{t \in T} X_t \lesssim \max_{t \in T} \sum_{n \geq 0} 2^{n/2} d(t,T_n)\,,$$$

where $d(t,T_n) = \min_{s \in T_n} d(s,t)$.

• Dudley’s entropy bound: If $\{X_t : t \in T\}$ is a subgaussian process, then

$$$\label{eq:dudley} \E \max_{t \in T} X_t \lesssim \sum_{n \geq 1} 2^{n/2} e_n(T)\,,$$$

where $e_n(T)$ is the largest distance $d$ such that there are $2^{2^n}$ points $y_1,y_2,\ldots,y_{2^{2^n}}$ in $T$ with $d(y_i,y_j) \geq d$ for all $i \neq j$.

Easy exercise: Show that Dudley’s entropy bound follows from the generic chaining upper bound \eqref{eq:gc-ub}.

• Covering numbers: Let $N(T,d,\e)$ denote the smallest number $N$ such that the metric space $(T,d)$ can be covered by $N$ balls of radius $\e$, and consider the quantity

$C_T \seteq \sum_{h=-\infty}^{\infty} 2^h \sqrt{\log N(T,d,2^{h})}\,.$

Up to a factor of $2$, this is equivalent to the more elegant expression $\int_0^{\infty} \sqrt{\log N(T,d,\e)} \,d\e$. It is an exercise to show that

$C_T \asymp \sum_{n \geq 1} 2^{n/2} e_n(T)\,.$

Exercise: Show that for any metric space $(T,d)$, it holds that

$$$\label{eq:CT} \sum_{n \geq 1} 2^{n/2} e_n(T) \asymp \sum_{h=-\infty}^{\infty} 2^h \sqrt{\log N(T,d,2^h)}\,.$$$
• Analysis of the probability simplex

Let $T = \{ x \in \R^n : x_1,\ldots,x_n \geq 0, x_1+\cdots+x_n = 1 \}$ denote the probability simplex.

Lemma 1: For this $T$, let $X_t = \langle g,t\rangle$, where $g=(g_1,\ldots,g_n)$ is a standard $n$-dimensional Gaussian. Then $\E \max_{t \in T} X_t \asymp \sqrt{\log n}$.

This is because the maximum of the linear functional $t \mapsto \langle g,t\rangle$ is achieved at a vertex of the convex body $T$ which is equal to the convex hull of the standard basis vectors: $T = \mathrm{conv}(e_1,\ldots,e_n)$. Thus $\max_{t \in T} \langle g,t\rangle = \max \{ g_i : i =1,\ldots,n\}$, and therefore

$\E \max_{t \in T} \langle g,t\rangle = \E \max(g_1,\ldots,g_n) \asymp \sqrt{\log n}.$

Let us now see that the entropy bound \eqref{eq:dudley} is far from tight in this simple case. We will use the formulation via covering numbers coming from the equivalence \eqref{eq:CT}.

Consider vectors $t \in T$ with $t_i \in \{0,1/k\}$ for $i=1,\ldots,n$. There are ${n \choose k} \geq \left(\frac{n}{k}\right)^k$ such vectors. For $k \leq \sqrt{n}$, we have $(n/k)^k \gtrsim 2^{k \log n}$, and it is possible to choose a proportional subset of those vectors (e.g., a random subset will suffice) such that

$d(t_i,t_j) \gtrsim \left(k \cdot \frac{1}{k^2} \right)^{1/2} \asymp k^{-1/2}, \qquad i \neq j\,.$

It follows that for $k \leq \sqrt{n}$, we have $\log N(T,d,k^{-1/2}) \gtrsim k \log n$. Consider then the $\approx \log n$ values of $h$ for which $1 \leq 2^h \leq \sqrt{n}$.

This gives us $\approx \log n$ values of $h$ for which

$2^{-h} \sqrt{\log N(T,d,2^{-h})} \gtrsim 2^{-h} \cdot 2^h \sqrt{\log n} = \sqrt{\log n}\,.$

Therefore Dudley’s bound \eqref{eq:CT} is $\gtrsim (\log n) \cdot \sqrt{\log n} = (\log n)^{3/2}$. (Recall that the correct bound, given by Lemma 1, is $\sqrt{\log n}$.)