Forsythe Conjecture & Restarted Krylov Methods

• Forsythe Conjecture is a hypothesis on the asymptotic behavior of restarted Krylov methods, focusing on convergence patterns of normalized residuals.
• Modern interpretations leverage Krylov subspace and Arnoldi projections to extend the conjecture to symmetric, orthogonal, and nonsymmetric matrices.
• The conjecture remains largely open for restart lengths s ≥ 2, presenting challenges in proving convergence uniqueness and characterizing limit sets.

The Forsythe Conjecture concerns the asymptotic behavior of restarted Krylov subspace methods, specifically the conjugate gradient (CG) method, for solving linear systems or minimizing quadratic forms. Originally formulated by G.E. Forsythe in 1968 for symmetric positive-definite (SPD) matrices, it hypothesizes strict regularities in the convergence pattern of the normalized residuals under periodic CG restarting, with significant implications for the structure and performance of iterative solvers. Modern developments reinterpret and generalize the conjecture to cover symmetric, orthogonal, and nonsymmetric matrices via advanced Krylov-based and Arnoldi-type projection frameworks (Faber et al., 2022).

1. Original Formulation and Historical Context

Forsythe's initial setting was the minimization of $$f(x)=\frac{1}{2}x^{\mathrm{T}}A x - x^{\mathrm{T}}b$$ with $A \in \mathbb{R}^{n \times n}$ symmetric positive definite and $b \in \mathbb{R}^n$, corresponding to solving $Ax = b$ via the CG method. He introduced periodic restarting of CG every $s$ steps, known as the optimal $s$-gradient method. Iterates $x_k$ and residuals $r_k = b - Ax_k$ are generated, with $y_k = r_k/\|r_k\|$.

For $s=1$, Forsythe established rigorous convergence: the normalized residuals $\{y_k\}$ alternate between two limiting directions. For $s=2$, numerical evidence showed a similar two-directional limit cycle but a general proof was absent. The conjecture posits that for $2 \leq s < d(A)$, where $d(A)$ is the minimal polynomial degree of $A$, both subsequences $\{y_{2k}\}$ and $\{y_{2k+1}\}$ each converge to a single unit vector. This assertion remains largely unproven except for certain low values of $s$ and specific matrix classes (Faber et al., 2022).

2. Modern Krylov-Subspace Interpretation

The conjecture has been translated into Krylov-subspace language, aligning with contemporary understanding of iterative solvers. The restarted CG process is recast as follows: at each restart, $x_{k+1}$ is the $A$-inner-product orthogonal projection of the true solution onto the affine subspace $x_k + \mathcal{K}_s(A, r_k)$, where $\mathcal{K}_s(A, r_k)$ is the Krylov subspace of order $s$ generated by $A$ and $r_k$. Explicitly,

$$x_{k+1} = \arg\min_{z \in x_k + \mathcal{K}_s(A, r_k)} \|x - z\|_A$$

with corresponding residuals and normalized directions.

The modern Forsythe conjecture (for symmetric $A$) states: given $1 \leq s < d(A)$ and initial $x_0$ such that $d(A, r_0) \geq s + 1$, the sequences $\{y_{2k}\}$ and $\{y_{2k+1}\}$ converge to unit vectors $y_{\text{even}}$ and $y_{\text{odd}}$ respectively as $k \to \infty$, so the residual directions asymptotically oscillate between two vectors (Faber et al., 2022).

3. Generalization via Arnoldi Cross Iteration

Extending beyond SPD matrices, the conjecture has been generalized using projections inspired by the Arnoldi (and for symmetric cases, the Lanczos) algorithm. For arbitrary $A \in \mathbb{R}^{n \times n}$ and $v$ with $d(A, v) > s$, the Arnoldi projection $P_s(A; v)$ yields $w = q(A)v$ where $q \in \mathcal{M}_s$ is a monic polynomial of degree $s$ minimizing norm, and $w$ is orthogonal to $\mathcal{K}_s(A, v)$.

Algorithmically, the Arnoldi Cross Iteration with restart $s$ ("ACI(s)") alternates projections with $A$ and $A^\mathrm{T}$:

1. $w_k = P_s(A; v_k) v_k / \|P_s(A; v_k) v_k\|$
2. $$v_{k+1} = P_s(A^\mathrm{T}; w_k) w_k / \|P_s(A^\mathrm{T}; w_k) w_k\|$$

For symmetric $A$ this reduces to a Lanczos-type projection sequence on the sphere. The generalized conjecture claims that, under the same degree condition, the sequence $\{v_k\}$ converges to a single limit vector even in nonsymmetric settings, although results for $s \geq 2$ remain largely open (Faber et al., 2022).

4. Theoretical Results and Limit Set Structure

Key verified theoretical results concerning ACI(s) and the Forsythe conjecture include:

• Monotonicity and Bound (Theorem 2.3): All vectors in ACI(s) sequences are well defined, and $$\|w_k\| \leq \|v_{k+1}\| \leq \|w_{k+1}\| \leq \|v_{k+2}\| \leq \Phi_s(A)$$ for all $k \geq 0$, with $$\Phi_s(A) = \max_{\|v\|=1} \min_{q \in \mathcal{M}_s} \|q(A)v\|$$.
• Vanishing Differences (Theorem 2.5): $\|w_{k+1} - w_k\| \to 0$ and $\|v_{k+1} - v_k\| \to 0$ as $k \to \infty$.
• Limit Set Properties (Theorem 3.8): The set $\Sigma_*^A$ of all limit points of $\{v_k\}$ is nonempty, closed, and connected on the unit sphere. Every limit $v_*$ must satisfy $v_* = T_{A^\mathrm{T}}(T_A(v_*))$, where $T_A(v) = P_s(A; v) v / \|P_s(A; v) v\|$.

For symmetric $A$: - If $d(A,v_0) = s+1$, then $v_0 = v_2 = v_4 = \ldots$ (Theorem 4.2). - Every limit vector $v_*$ satisfies $(Q_{2s}(A; v_*) - \tau^2 I) v_* = 0$ for $s < d(A, v_*) \leq 2s$ (Theorem 4.4), an interpolation relation. - For $s=1$, the sequence $\{v_{2k}\}$ converges uniquely (Theorem 4.7).

For orthogonal $A$ with $s=1$ and $0 \notin F(A)$, any limit $v_*$ has grade 2 and corresponds to the real part of a conjugate-pair of eigenvalues (Lemma 4.9), and the full sequence converges under mild conditions (Theorem 4.12) (Faber et al., 2022).

5. Open Problems and Research Directions

Despite advances for the $s=1$ case, key challenges persist, particularly for higher restart lengths ($s \geq 2$):

• For $s \geq 2$ and symmetric $A$, uniqueness of the limit point is unproved in general. Previous arguments for $s=2$ rely on a coefficient-convergence result attributed to Zabolotskaya (1979), whose full rigor remains controversial (Faber et al., 2022).
• For nonsymmetric ACI(s), convergence beyond $s=1$ is open, due in part to lack of simple monotonicity in norm and the possibility of higher-dimensional limit sets.
• A promising avenue combines finite enumeration of candidate limit polynomials (of degree at most $2s$) with proof that coefficient convergence implies vector convergence.
• Connections to two-sided Rayleigh quotient and Alternating Rayleigh Quotient Iteration (ARQI) methodologies may provide new monotonicity or contraction arguments.

A plausible implication is that resolving these open questions would significantly advance understanding of the limiting dynamics of restarted Krylov methods.

6. Current State of Knowledge

The state of results on the Forsythe conjecture and its generalizations can be summarized as follows:

Case	Status	Notes/Provenance
$s=1$, $A = A^\mathrm{T}$ (symmetric)	Proven (unique 2-cycle)	Akaike (1959), Afanasjew et al. (2008), (Faber et al., 2022)
$s=1$, $A$ orthogonal, $0 \notin F(A)$	Proven (full convergence)	(Faber et al., 2022)
$s\geq 2$, symmetric $A$	Largely open	Evidence for $s=2$, full proof lacking
$s\geq 2$, nonsymmetric $A$ (ACI(s))	Open	Limit set structure may be higher-dimensional

Although the case $s=1$ is now rigorously understood in both SPD and some orthogonal settings, the Forsythe conjecture for $s \geq 2$ remains a central open problem, whose resolution likely requires novel insights into the algebraic structure and convergence properties of Krylov-projection polynomials and their limits (Faber et al., 2022).