Intrinsic Plausible Novelty Score

• IPNS is a quantitative metric that evaluates novelty by combining state surprisal in reinforcement learning with average token-level language model surprise in texts.
• It utilizes an autoencoder-based state encoding, density estimation, and value regression to compute a product of state novelty and expected benefit.
• Empirical results show IPNS enhances sample efficiency in RL and reliably identifies atypical content in manuscripts through statistical surprisal analysis.

The Intrinsic Plausible Novelty Score (IPNS) is a quantitative metric designed to measure the degree of novelty attributed to individual states encountered by a reinforcement learning agent, or the atypicality of scholarly manuscripts based on language-model surprise. The term “intrinsic plausible novelty” originated in deep reinforcement learning, where it was used to incentivize exploration towards states that are both under-visited and likely beneficial for policy improvement (Banerjee et al., 2022). In the context of scholarly text analysis, IPNS has been operationalized as the average token-wise surprisal under a large causal LLM trained on the academic corpus, serving as an interpretable proxy for the statistical rarity or novelty of word combinations present in the document (Wang, 2024). Both frameworks emphasize the use of data-driven, interpretable novelty signals for optimizing search (exploration) or evaluation procedures.

1. Mathematical Formulation of IPNS

In reinforcement learning applications, IPNS is defined as a product of two factors: state-novelty and expected benefit. Specifically, given current state $s_t \in \mathcal{S}$ and its low-dimensional autoencoder representation $z_t \in \mathbb{R}^{m'}$, and a buffer $\mathcal{Z}_n = \{z_1, \dots, z_n\}$ containing previously encoded states, the primary quantities are:

• State-novelty score: $$\eta(z_t, \mathcal{Z}_n) = \|z_t - \mathrm{HVD}(\mathcal{Z}_n)\|_2$$, where HVD denotes the high-visitation-density point in the state-embedding buffer.
• Expected benefit score: $V_{\theta_n}(s_t)$, learned via temporal-difference (TD) regression.
• Plausible novelty: $$\xi(s_t, z_t, \mathcal{Z}_n) = \eta(z_t, \mathcal{Z}_n) \cdot V_{\theta_n}(s_t)$$.
• Final intrinsic reward: $$r_t^{\text{aug}} = (1-\beta) r_t + \beta \zeta(s_t, z_t, \mathcal{Z}_n)$$, where $\zeta$ applies local normalization over $K$ noisy code neighbors according to $\zeta = 2/(e^{\tilde\xi} + e^{-\tilde\xi})$ with $\tilde\xi = \xi_{\text{max}}(z_t) - \xi$.

For scholarly manuscript evaluation, IPNS is defined by the average token-level surprisal under a probabilistic surrogate $Q$: $$\overline{I}_Q(x_{1:n}) = -\frac{1}{n} \sum_{i=1}^n \log_2 Q(x_i \mid x_{1:i-1}),$$ where $Q(x_i \mid x_{1:i-1})$ is the language-model probability assigned to token $x_i$ given previous context. The IPNS of a document is thus its mean surprisal in bits per token (Wang, 2024).

2. Algorithmic Components and Data Inputs

The RL-based IPNS framework comprises three modules:

• State Encoding (SE): An autoencoder is pretrained on random trajectories to produce compact codes $z$ via $z=\mathrm{sigmoid}(\mathrm{enc}(s))$.
• State-Novelty Scoring (SNS): HVD estimation leverages density estimation over minibatches in embedding space, updating the HVD every $M$ timesteps based on visitation patterns.
• Value Estimation (PNS): A dedicated value network $V_\theta$ is concurrently updated by TD error to estimate expected returns.

In the context of academic novelty assessment, the required inputs are:

• Document tokenization (BPE over plain text)
• A causal Transformer trained on a reference corpus (Wikipedia) whose probability assignments $Q$ serve as a proxy for “scholarly discourse”
• Length and context-window parameters for reliable conditioning.

Both applications require substantial pretraining and buffer management to aggregate relevant data for accurate novelty estimation (Banerjee et al., 2022, Wang, 2024).

3. Computational Workflow and Hyperparameterization

In RL, the IPNS computation is performed per timestep in a standard actor-critic loop:

• Encode the current state.
• Update buffers and state-density statistics as needed.
• Score novelty and expected benefit.
• Normalize plausible novelty across noisy code variants.
• Augment the extrinsic reward with the intrinsic IPNS-based reward.
• Update the policy and value networks using the augmented reward.

Key hyperparameters include bottleneck dimension ($m'$), frequency and volume of HVD update steps ($M, J, I$), noise normalization parameters ($K, \rho$), intrinsic reward mixing coefficient ($\beta$), and baseline algorithm settings.

For manuscript novelty scoring, the workflow is as follows:

• Tokenize the abstract and body.
• For each token starting from a minimal history of 256, compute negative log-probability under a context window of 1024.
• Average the cumulative surprisal over all eligible tokens to produce the document’s IPNS.

The principal hyperparameters are context window, minimal history, and the surrogate corpus/model used for training $Q$ (Wang, 2024).

4. Theoretical Rationale and Interpretation

The RL-based IPNS identifies regions in the state space that are both underexplored and promising, operationalizing exploration as a trade-off between novelty and value. By focusing intrinsic rewards on states with high plausible novelty, the agent avoids wasteful exploration of unpromising regions and prevents degenerate behavior with familiar states. Local normalization accentuates novel directions while maintaining stability in reward assignment, and the mixing of intrinsic/extrinsic rewards via $\beta$ enables adjustable exploration-exploitation balance.

In manuscript evaluation, IPNS captures the degree of “surprise” a paper’s content delivers relative to a probabilistic baseline of existing discourse. High bits-per-token indicate the document deviates from stylistic and lexical expectations, which correlates empirically with expert judgments of novelty. This method directly quantifies the statistical atypicality of word combinations, and is platform-agnostic with respect to domain and training corpus, providing transparency and reproducibility in novelty measurement (Wang, 2024).

5. Empirical Validation and Practical Outcomes

In continuous-control RL benchmarks (MuJoCo environments: InvertedDoublePendulum-v2, Reacher-v2, Hopper-v2), integrating IPNS-based intrinsic reward led to:

• Faster policy learning (enhanced sample efficiency)
• Lower performance variance across runs
• Statistically higher final returns relative to baseline actor-critic algorithms (SAC, DDPG, TD3) For example, SAC+IPNS achieved $9,348 \pm 3.7$ average return on InvertedDoublePendulum-v2 vs. $7,551 \pm 3,197$ for vanilla SAC.

For manuscript novelty, the IPNS measure was tested for face and construct validity:

• Token-level analysis: context-sensitive surprisal matched scientific intuition (e.g., “room-temperature entanglement” received higher novelty than “low-temperature”).
• Section-level expert alignment: Papers binarily rated “novel” had significantly higher mean IPNS scores (Welch’s $t$, $p=0.005$). A plausible implication is that IPNS allows robust, automated screening of manuscripts for atypical content, supporting editorial and policy decisions (Wang, 2024).

6. Limitations and Assumptions

RL IPNS implementation presupposes availability of sufficient buffer samples for density and value estimation, and assumes stable autoencoder training for reliable code generation. Hyperparameter tuning remains environment-specific; adaptive strategies for $\beta$ or $\epsilon$ may be required for robustness in highly heterogeneous domains.

For the text-based IPNS, the surrogate probability distribution $Q$ is typically trained on general language, which may increase surprisal for rare but standard field-specific terms. The fixed context length constrains evaluation for very short or excessively long documents, and artifacts in tokenization (mathematical symbols, parsing errors) can artificially inflate novelty scores. Calibration against domain-specific corpora or explicit thresholding schemes is necessary to operationalize IPNS for screening or comparative evaluation. The method measures only statistical word-level novelty and does not capture conceptual, methodological, or structural innovation within manuscripts.

7. Contextual Significance and Future Directions

IPNS represents a paradigm shift toward scalable, interpretable, and data-driven novelty quantification. In RL, intrinsic plausible novelty aligns exploration incentives with meaningful progress by directly linking state-space rarity with policy improvement. In scholarly text analysis, IPNS enables objective measurement of innovation potential, complementing peer review and supporting meta-scientific inquiry into the dynamics of creativity and progress. Future research directions include task-adaptive mixing strategies, reinforcement of conceptual novelty signals, and retraining surrogate models on discipline-specific corpora for refined domain-oriented evaluations (Banerjee et al., 2022, Wang, 2024).