Dominated Novelty Search: Rethinking Local Competition in Quality-Diversity

Abstract: Quality-Diversity is a family of evolutionary algorithms that generate diverse, high-performing solutions through local competition principles inspired by natural evolution. While research has focused on improving specific aspects of Quality-Diversity algorithms, surprisingly little attention has been paid to investigating alternative formulations of local competition itself -- the core mechanism distinguishing Quality-Diversity from traditional evolutionary algorithms. Most approaches implement local competition through explicit collection mechanisms like fixed grids or unstructured archives, imposing artificial constraints that require predefined bounds or hard-to-tune parameters. We show that Quality-Diversity methods can be reformulated as Genetic Algorithms where local competition occurs through fitness transformations rather than explicit collection mechanisms. Building on this insight, we introduce Dominated Novelty Search, a Quality-Diversity algorithm that implements local competition through dynamic fitness transformations, eliminating the need for predefined bounds or parameters. Our experiments show that Dominated Novelty Search significantly outperforms existing approaches across standard Quality-Diversity benchmarks, while maintaining its advantage in challenging scenarios like high-dimensional and unsupervised spaces.

• The paper introduces Dominated Novelty Search (DNS), a novel Quality-Diversity algorithm that reformulates local competition in Genetic Algorithms through dynamic fitness transformations.
• DNS dynamically adapts competition based on distances to fitter solutions in descriptor space, achieving state-of-the-art results on standard benchmarks.
• Unlike traditional methods, DNS requires no predefined bounds, grid structures, or fixed distance thresholds, offering a more principled approach to maintaining population diversity.

The paper introduces Dominated Novelty Search (DNS), a novel Quality-Diversity (QD) algorithm that reframes local competition in genetic algorithms through dynamic fitness transformations, negating the necessity for predefined bounds or parameters. The authors posit that QD algorithms can be conceptualized as Genetic Algorithms (GAs) where local competition arises from fitness transformations rather than explicit collection mechanisms.

The paper's central thesis revolves around the concept that QD algorithms can be reformulated as GAs, where local competition is enacted via specialized fitness transformations rather than through explicit collection mechanisms like grids or archives. This perspective contrasts with the traditional view of QD algorithms as distinct from GAs.

Key tenets and contributions of the paper include:

• Reformulation of QD as GA with Local Competition: The authors reformulate QD algorithms as GAs augmented with specialized fitness transformations that implement local competition. This perspective shifts focus from the storage mechanisms to the underlying competition dynamics that drive evolutionary search.
• Dominated Novelty Search Algorithm: DNS dynamically adapts to the structure of the solution space through a competition mechanism based on relative distances between solutions of different fitness levels. The algorithm rewards solutions that either outperform their neighbors or find unique behaviors compared to better-performing solutions.
• Competition Function: The competition function in DNS computes modified fitness values based on how distant solutions are from other better-performing solutions. Given a population with fitness values $f = (f_i)_{i=1}^N$ and descriptors $d = (d_i)_{i=1}^N$, the competition function operates in three steps:

1. For each solution $i$, identify all solutions with superior fitness:

$$\mathcal{D}_i = \{j \in \{1,\ldots,N\} \mid f_j > f_i\}$$

where: * $\mathcal{D}_i$ is the set of indices of solutions fitter than solution $i$ * $N$ is the total number of solutions in the population * $f_j$ is the fitness of solution $j$ * $f_i$ is the fitness of solution $i$ 2. Compute pairwise distances in descriptor space between the solution $i$ and all the fitter solutions:

$$d_{ij} = \lVert d_i - d_j \rVert \quad \forall j \in \mathcal{D}_i$$

where: * $d_{ij}$ is the Euclidean distance between the descriptor of solution $i$ and the descriptor of solution $j$ * $d_i$ is the descriptor vector of solution $i$ * $d_j$ is the descriptor vector of solution $j$ 3. Calculate the competition fitness $\tilde{f}_i$ as the dominated novelty score – the average distance to the $k$-nearest-fitter solutions:

$$\tilde{f}_i = \begin{cases} \frac{1}{k}\sum_{j \in \mathcal{K}_i} d_{ij} & \text{if } |\mathcal{D}_i| > 0 \ +\infty & \text{otherwise} \end{cases}$$

where: * $\tilde{f}_i$ is the competition fitness of solution $i$ * $k$ is the number of nearest fitter solutions to consider * $\mathcal{K}_i$ is the set of indices of the $k$ nearest fitter solutions to solution $i$ * $d_{ij}$ is the distance between solution $i$ and fitter solution $j$

• Experimental Validation: The authors conduct experiments across multiple domains, including continuous control tasks, environments with discontinuous descriptor spaces, and high-dimensional descriptor spaces. The experiments compare DNS against standard QD algorithms like MAP-Elites and Novelty Search with Local Competition (NSLC).
• Performance Metrics: The paper uses Quality-Diversity (QD) score and coverage as evaluation metrics. Traditional implementations of these metrics assume a structured archive, typically the grid used in MAP-Elites. To ensure comparability across different methods, the authors project the individuals onto a predefined grid-based archive to compute metrics.

The experimental results demonstrate that DNS achieves state-of-the-art performance on standard QD benchmarks. Specifically, DNS outperforms MAP-Elites in well-defined descriptor spaces, is robust to increasing dimensionality in high-dimensional descriptor spaces, and adapts effectively to learned representations in unsupervised descriptor spaces.

The authors claim that by eliminating the need for predefined bounds, grid structures, or fixed distance thresholds, DNS offers a more principled solution to maintaining diversity in evolved populations. The algorithm's ability to dynamically adapt to the natural structure of the solution space is particularly valuable in scenarios where the shape and bounds of the descriptor space cannot be known in advance.