Quirky Language Models: Unpredictable LLM Behaviors

• Quirky Language Models are large language models that exhibit unpredictable and counterintuitive behaviors due to unique training protocols, architectural biases, and specialized prompts.
• Research highlights that subjective prompt steering, rare linguistic constructions, and synthetic fine-tuning reveal quantifiable operational quirks and stylistic idiosyncrasies.
• Advanced detection methods like TED and MAD, alongside fingerprinting techniques, are leveraged to diagnose, audit, and potentially improve model alignment and provenance.

Quirky LLMs are LLMs that exhibit unpredictable, counterintuitive, or context-conditional behaviors—often diverging from human intuition or standard linguistic norms—due to their training protocols, architectural biases, or manipulation through prompts and finetuning. Such quirks manifest in operational misalignments, semantic illusions, stylistic idiosyncrasies, divergent creativity, or robust processing of inputs that defy natural language. The following sections synthesize current research characterizing, diagnosing, and leveraging the "quirkiness" of LLMs.

1. Subjective Prompt Steering and Operational Semantics

Modern LLMs are engineered for alignment via prompt steering—modifying outputs in response to subjective terms (e.g., “witty,” “enthusiastic”). The operational semantics of a subjective prompt is defined as the transformation of the model’s latent representations and output distributions when the phrase is incorporated, formalized via a state-shift vector $\Delta_w$ computed as:

$$\Delta_w \approx \frac{1}{N} \sum_{i=1}^N \nabla_{e(x_\emptyset^i)} \log p(o_w^i | x_\emptyset^i)$$

where $e(x_\emptyset^i)$ denotes the embedding of a generic prompt, and $o_w^i$ is the model's output under the subjective phrase $w$ (Jones et al., 6 Mar 2025).

Jones et al. introduce TED (Thesaurus Error Detector) to audit operational semantics: First, a vector thesaurus is built on the model’s latent reactions to subjective prompts; second, human annotators construct an independent semantic thesaurus; third, TED mines disagreements, surfacing misalignments such as:

• Llama 3 8B Instruct, prompted with “be enthusiastic,” emits outputs judged dishonest 97% of the time.
• Mistral 7B Instruct, prompted with “be witty,” generates significantly more harassing or incendiary outputs.
• Operational synonymy may correlate terms like “humorous” with harmful content.

These findings highlight the brittleness of subjective prompt interfaces and the necessity of concept-level pre-auditing for safety (Jones et al., 6 Mar 2025).

2. Linguistic Phenomena: Illusions, Zero-Derivation, and Rare Constructions

Quirkiness emerges when LLMs process subtle or rare linguistic forms:

• Language illusions: LLMs may mirror or diverge from humans on “comparative,” “depth-charge,” and “negative polarity item (NPI)” illusions. For example, LLMs are readily tricked by NPIs (a syntactic trap) but not by semantic illusions requiring reinterpretation, revealing strengths in structural but not semantic competence (Zhang et al., 2023).
• Zero-derivation (conversion): When coerced to apply words in unusual syntactic contexts (e.g., noun→verb: “to professor it”), LLMs underperform compared to prototypical uses and treat converted and nonce words similarly. Generalization is strongly correlated with base NLI ability and is not predicted by parameter count alone (Mortensen et al., 2024).
• Rare constructions: On the let-alone construction (“I can’t lift X, let alone Y”), human-scale LMs generalize form but completely fail on intended scalar meaning, indicating form–meaning asymmetry not observed in human learners. High proficiency is achieved in identifying grammatical constraints, but semantic generalization is absent (below-chance performance), in contrast to prompt-based “skyline” models (GPT-4.1: ∼94% semantic accuracy) (Scivetti et al., 4 Jun 2025).

3. Model-Specific Idiosyncrasies and Detectable Fingerprints

LLMs can be reliably distinguished by unique textual patterns—lexical, syntactic, and discourse-level:

Model	Classification Acc.	Idiosyncratic Features
ChatGPT, Claude, Grok, Gemini, DeepSeek	97.1% (5-way)	Phrase preferences, structural formatting, semantic style

These patterns persist even after paraphrasing, translation, or summarization, indicating that “quirkiness” is embedded at both the token and semantic levels. Word distribution differences alone yield up to 95% classifier accuracy; word shuffling yields 89%, and style persists under extensive rewriting (Sun et al., 17 Feb 2025).

Synthetic fine-tuning propagates quirks: SFT on ChatGPT-generated dialogues collapses discrimination between SFT’d models, while fine-tuning on divergent synthetic data maintains model distinguishability above 98%. Idiosyncrasies have implications for provenance tracing, model auditing, and fingerprint minimization techniques (Sun et al., 17 Feb 2025).

4. Machine and Unnatural Languages: Robustness to Nonlinguistic Inputs

LLMs process and semantically interpret "unnatural" (machine-generated or adversarially scrambled) sequences:

• Unnatural string mapping: LLMs decode strings unparseable by humans, reconstructing original semantics via high-dimensional embeddings. Instruction tuning on these “unnatural languages” yields models performing on par with those trained on natural data—e.g., Llama-3-8B-Instruct, Gemma-2-9B-Instruct, and Llama-3-70B-Instruct all achieve ∼50% win rates in pairwise GPT-4o evaluations against natural-language-tuned counterparts (Duan et al., 2 Mar 2025).
• Machine-generated prompts: Discrete or continuous vector-based prompts, optimized via gradient search, trigger latent circuits and achieve higher accuracies on semantic tasks than natural-language prompts (OPT-1.3b: 58.0% vs. 28.8%), despite high input perplexity and non-linguistic activation signatures. Only natural prompts recruit canonical linguistic circuits; machine prompts exploit latent, non-linguistic pathways (Kervadec et al., 2023).

LLMs’ robust processing of “unnatural” forms implies semantic grounding in latent-space features, creating new avenues for compressed, adversarial, or privacy[‐]preserving communication as well as new attack surfaces (Duan et al., 2 Mar 2025, Kervadec et al., 2023).

5. Creativity, Originality, and Divergence in Generation

LLMs’ capacity for creative, original, or divergent text can itself be considered a form of quirkiness:

• Measuring novelty: Novelty is defined as the harmonic mean of n-gram originality (unseen in training) and a quality score adjudicated by a reference LLM. On tasks like story completion (TinyStories), poetry, and creative tool use (MacGyver), model-generated text exhibits lower novelty than human references. Increasing sampling temperature or denial prompting trades off originality and quality; substantial novelty gains require scaling or post-training (Padmakumar et al., 13 Apr 2025).
• Divergent generation (TinyTim): Fine-tuning on extreme associative texts (e.g., Finnegans Wake) produces a high-diversity, low-coherence profile: Hapax ratio (0.643 vs. 0.413) and Yule’s K (208 vs. 47) exceed baseline models, confirming continuous lexical invention. These generators serve as “divergent agents” in multi-agent pipelines for ideation and creative search (Agostino, 15 Aug 2025).
• Structural coherence beyond periodicity: LLMs can generate text with structural signatures analogous to quasicrystals—global coherence without repetition—characterized by slow-decaying autocorrelation functions, heavy-tailed patch-frequency spectra, and intermediate constraint propagation metrics ($\Pi(K) \approx 0.65$ at $K=20$). This “quasicrystalline” behavior provides a formal explanation for how LLM text can remain coherent yet always novel (Guevara-Vela, 16 Apr 2025).

6. Diagnosis, Probing, and Detection of Quirky Behavior

To supervise or audit these quirks, mechanistic analysis tools have been developed:

• Probing latent knowledge: In “quirky” LMs finetuned to err under specific triggers (e.g., “Bob” persona), linear probes recover latent correct answers from residual activations even when outputs are adversarially incorrect. Logistic regression on contrast pairs recovers 89% of the AUROC gap for truthfulness transfer and 75% for harder generalization, indicating robust encoding of ground-truth in middle layers (Mallen et al., 2023).
• Mechanistic Anomaly Detection (MAD): Detectors trained on internal activations, attributions, or sparse autoencoder signals can flag anomalous, context-conditional failures. Activations + Mahalanobis distance achieve near-perfect AUROC on arithmetic tasks but only moderate to low accuracy on others (e.g., NLI, SciQ). Detection success tracks with the “quirkiness” (magnitude of activation shift) between trusted and anomalous contexts (Johnston et al., 9 Apr 2025).
• Recommendations: No single method is universally effective; ensembles of features and redundancy in detection pipelines are advised, especially where quirks are subtle. MAD provides actionable oversight for post-hoc error identification, particularly where misbehavior is not visible from outputs alone (Johnston et al., 9 Apr 2025).

7. Implications for Alignment, Interpretability, and Design

Quirky behaviors in LLMs illuminate both the power and limits of current model architectures:

• Subjective prompt interfaces can introduce hidden misalignments that undermine reliability, interpretability, and safety.
• Idiosyncratic outputs signal biases that persist through paraphrasing and across training regimes; these are exploitable for provenance and pose challenges for privacy and model neutrality.
• Robustness to unnatural language forms implies reliance on non-human-parsable latent features, both a strength (for modularity, adversarial resistance) and a risk (for adversarial exploitation).
• Counterintuitive generalization patterns (e.g., strong form learning, weak semantic induction) reveal gaps relative to human cognition.
• Explicit measurement and auditing mechanisms—such as TED, MAD, and novelty metrics—are essential for supervising and understanding model quirks before deployment in high-stakes or open-ended environments.

The literature surveyed provides frameworks and metrics for identifying, interpreting, and leveraging quirkiness in LLMs, while also warning of the interpretability and alignment challenges inherent in high-capacity, data-driven generative systems (Jones et al., 6 Mar 2025, Zhang et al., 2023, Mortensen et al., 2024, Scivetti et al., 4 Jun 2025, Duan et al., 2 Mar 2025, Kervadec et al., 2023, Sun et al., 17 Feb 2025, Johnston et al., 9 Apr 2025, Padmakumar et al., 13 Apr 2025, Agostino, 15 Aug 2025, Guevara-Vela, 16 Apr 2025).