Zero-Shot Skill Extraction Framework

• Zero-Shot Skill Extraction Framework is an automated approach that infers task-relevant skills without labeled data by leveraging semantic grounding and compositional methods.
• It employs multi-modal encoders, foundation models, and contrastive training with synthetic data to extract and chain atomic skills for diverse tasks.
• Empirical evaluations in robotics, medical assessments, and labor market analytics demonstrate its robust performance and highlight areas for further refinement.

A zero-shot skill extraction framework is an automated system that infers and operationalizes task-relevant skills using foundation models, contrastive learning, or modular controllers, without explicit supervision or human-labeled data for the target domain or skill taxonomy. These frameworks enable transfer and generalization across domains, long-horizon tasks, and unseen skill descriptions by leveraging semantic grounding, compositional structures, multi-modal inputs, or large-scale synthetic data. Key instantiations span domains from minimally supervised medical skill assessment to labor market analytics, robot imitation, multi-task manipulation, and cross-domain policy adaptation.

1. Foundational Principles of Zero-Shot Skill Extraction

Zero-shot skill extraction frameworks fundamentally address the challenge of mapping raw, unannotated input—visual data, text, or multi-modal snippets—to actionable skill representations or execution plans in the absence of supervised task-labeled data. Hallmark characteristics include:

• Foundation Model Utilization: Off-the-shelf models (SAM, CLIP, Grounded DINO, BERT) remain frozen or are prompt-tuned for semantic segmentation, skill grounding, or text–image alignment (Kondo, 2024, Shin et al., 2024, Seker et al., 16 May 2025, Clavié et al., 2023).
• Semantic Compositionality: Tasks are decomposed into atomic or modular skills, allowing for compositional generalization (via segmentation, sequence modeling, or controller lists) (Chen et al., 1 May 2025, Seker et al., 16 May 2025, Shin et al., 2024).
• Synthetic Data Generation: LLMs generate broad, diverse positive and negative samples for each skill, yielding a discriminative training corpus for classifier or embedding models (Clavié et al., 2023, Sun, 14 Jan 2026).
• Contrastive and Hierarchical Training: Models are trained with contrastive objectives (positive and hard negative pairs) and hierarchical constraints that improve semantic consistency and discriminability, especially in multi-label contexts (Shin et al., 2024, Sun, 14 Jan 2026).

These principles underpin frameworks that achieve robust zero-shot transfer in real-world labor analysis, robotic control, medical skill assessment, and cross-domain policy adaptation.

2. Architectural Components and Algorithmic Structure

Zero-shot skill extraction frameworks display a common multi-stage architecture, frequently consisting of:

• Input Representation: Multi-modal encoders process images, video, language, sensor data, forming latent skill or semantic instruction spaces (Shin et al., 2024, Shin et al., 2024).
• Skill or Segmentation Inference:
  • Visual Segmentation: Foundation models extract segmentation masks for instruments or objects via text prompts, producing foreground/background features (ZEAL) (Kondo, 2024).
  • Embedding-based Candidates: Sentence or input embeddings are generated using BERT, CLIP, or similar, serving as query vectors for skill retrieval (Clavié et al., 2023, Sun, 14 Jan 2026).
  • Atomic Skill Detection: Demonstration segmentation identifies physically grounded atomic sub-tasks based on agent signals (e.g., gripper cycles) (Chen et al., 1 May 2025).
• Classifier/Retriever & Scoring:
  • Logistic regression classifiers or bi-encoders assign relevance to candidate skills; similarity search retrieves matching entries (Clavié et al., 2023, Sun, 14 Jan 2026).
  • Sequence modeling via LSTMs captures dynamics and temporal dependencies for skill scoring (Kondo, 2024).
• Skill Scheduling/Chaining & Execution:
  • Vision-LLMs or PLMs sequence atomic skills for long-horizon tasks; chaining modules optimize pose transitions to avoid collisions (Chen et al., 1 May 2025).
  • Controllers are grounded in task axes and keypoints using foundation models for geometric transfer in manipulation tasks (Seker et al., 16 May 2025).
• (Optional) LLM Re-ranking:
  • Candidate skills are re-ranked with a second LLM, often with mock-programming prompts for improved discrimination (Clavié et al., 2023).
• Evaluation & Metrics:
  • Empirical assessment uses task-specific metrics: RP@10, F1@5, normalized returns, relative L2 distance, and subtask completion rates (Kondo, 2024, Clavié et al., 2023, Sun, 14 Jan 2026).

The following table summarizes prominent components in representative frameworks:

Framework	Input Modality	Skill Extraction Module	Training Supervision
ZEAL (Kondo, 2024)	Surgical video images	Text-prompted segmentation + sparse CNN + BiLSTM	Annotated skill scores
ESCO LLM (Clavié et al., 2023)	Job post text	Synthetic sentences + classifier + LLM rerank	Synthetic only
GTA (Seker et al., 16 May 2025)	RGB-D scene, robot state	Foundation model keypoints + axis controllers	None (semantic grounding)
DeCo (Chen et al., 1 May 2025)	3D images + robot gripper state	Atomic task segmentation + multi-task IL	Demonstration only
SemTra (Shin et al., 2024)	Multi-modal (video/sensor/text)	Skill extractor + seq2seq PLM translation + skill adapter	Cross-domain demo
BiEncoder (Sun, 14 Jan 2026)	Job post text	Synthetic samples + RoBERTa filter + bi-encoder	Synthetic only

This layered, modular structure allows the frameworks to operate in zero-shot settings, propagate semantic and physical constraints, and avoid dependence on costly labeled data.

3. Semantic Grounding and Domain Transfer Mechanisms

Zero-shot skill extraction frameworks achieve domain generalization by grounding skills in shared semantic or geometric spaces:

• Textual and Visual Semantic Alignment: Pretrained encoders map both language and visual snippets to a shared latent space (e.g., CLIP, V-CLIP), enabling skill inference across modalities or domains (Shin et al., 2024, Shin et al., 2024).
• Modular Skill Composition: Decomposition into atomic skills or task-axis controllers allows reuse, combinatorial generalization, and efficient skill scheduling in unseen scenarios (Chen et al., 1 May 2025, Seker et al., 16 May 2025).
• Hierarchical Sequencing and Adaptation: Seq-to-seq models translate extracted skill sequences into domain-agnostic or cross-contextual instructions, instantiated by context encoders (Shin et al., 2024).
• Foundation Model Grounding: Visual foundation models (SD-DINO, Grounded SAM) locate keypoints, axes, or object masks with semantic similarity, supporting example-based transfer in manipulation and control (Seker et al., 16 May 2025, Kondo, 2024).

A plausible implication is that leveraging strong semantic priors—whether language or geometry—within compositional framework scaffolds is essential for robust zero-shot adaptation across domains and modalities.

4. Training Paradigms: Synthetic Data, Contrastive Learning, and Classifier Construction

Most frameworks circumvent real data annotation scarcity via synthetic data generation and contrastive learning:

• Synthetic Corpus Generation: LLMs (GPT-3.5, GPT-4, DeepSeek) produce thousands of job-ad-like sentences for each skill and semantically coherent multi-label pairs via hierarchical constraints (e.g., ESCO Level-2 categories) (Clavié et al., 2023, Sun, 14 Jan 2026).
• Contrastive Bi-Encoder/Classifier Training: Siamese or bi-encoder architectures align input sentences and skill descriptions in shared embedding spaces, using hard negative sampling for enhanced retrieval accuracy (Sun, 14 Jan 2026).
• Hierarchical Constraints: Imposing taxonomy-based co-occurrence (Level-2 or higher) improves fluency and discriminability over unconstrained random pairing, as shown by perplexity and ROC separation metrics (Sun, 14 Jan 2026).
• Loss Functions and Margins: Training objectives use margin-based contrastive loss, binary cross-entropy for filters (skill vs non-skill), and regression/classification (mean squared error, cross-entropy) as needed by task (Sun, 14 Jan 2026, Kondo, 2024).
• Ablation and Prompt Tuning: Prompt style (mock Python vs direct natural language) and ablation of architecture components (BiLSTM, attention, negatives K) significantly affect downstream discriminability and RP@10/F1@5 (Clavié et al., 2023, Sun, 14 Jan 2026).

These training paradigms yield models that are robust to domain and language shifts, compositionally expressive, and effective even in zero-shot multi-label settings.

5. Evaluation Results, Empirical Performance, and Limitations

Zero-shot skill extraction frameworks demonstrate competitive or superior performance on multiple benchmarks and real-world tasks:

• Surgical Skill Assessment: ZEAL (Kondo, 2024) achieves R-ℓ₂ × 100 of 3.17, outperforming prior zero-exemplar and supervised SOTA, with Spearman’s ρ (0.61) indicating room for improvement in rank-correlation losses.
• Labor Market Analytics: ESCO framework (Clavié et al., 2023) records RP@10 of 61.02–68.94 for general/tech (GPT-4 rerank), a ≥22-point gain over distant supervision, with mock-programming prompt substantially boosting recall for GPT-3.5.
• Contrastive Bi-Encoder (Chinese): F1@5 ≈ 0.80 (zero-shot on real ads), AUPRC = 0.90, outperforming TF-IDF (F1@5 ≈ 0.72) (Sun, 14 Jan 2026).
• Robotic Manipulation: GTA (Seker et al., 16 May 2025) delivers keypoint error ≤1 cm, axis error ≤3°; success rates of ~90–95% in real-robot tasks, with no retraining required for new objects.
• Multi-Task IL (DeCo): Extreme gains in zero-shot compositional tasks: RVT-2 (0%→66.67%), 3DDA (0%→21.53%), ARP (0.14%→58.06%), persistent in real-robot drawer tasks (53.33%) (Chen et al., 1 May 2025).
• Cross-Domain Imitation (OnIS, SemTra): Robust performance on dynamic environments and multiple platforms; OnIS shows >20–50 pp gain in one-shot imitation (Meta-World) and SemTra achieves normalized returns 66–83% on complex cross-domain tasks (Shin et al., 2024, Shin et al., 2024).

Limitations include residual false negatives for domain-specific jargon, incomplete taxonomy alignment, sensitivity to VLM planning quality, scalability as skill libraries grow, and brittleness to exploration gaps or visual domain shifts (Chen et al., 1 May 2025Sun, 14 Jan 2026Pathak et al., 2018).

6. Open Challenges and Future Directions

Despite significant advances, several issues remain:

• Taxonomy Alignment: False negatives for rare/domain-specific skills and occasional misalignment in multi-label contexts suggest further refinement with graph-aware objectives and cross-lingual signal integration (Sun, 14 Jan 2026).
• Skill Scheduling and Completion Detection: Existing reliance on low-level signals (e.g., gripper state) may be augmented with tactile, force, or sensor fusion for intricate manipulation (Chen et al., 1 May 2025).
• Scaling to Hierarchical/Long-Horizon Planning: As libraries expand, skill retrieval and scheduling could benefit from hierarchical, graph-based, or symbolic planners (Chen et al., 1 May 2025, Shin et al., 2024).
• Robustness Across Modality and Domain Shifts: Enhanced exploration policies, domain-adaptive contrastive embeddings, and advanced contextual encoders should be explored to support more extreme zero-shot generalization (Pathak et al., 2018, Shin et al., 2024).
• Integration with Next-Generation Foundation Models: Denser or multimodal correspondence, semantic grounding and dynamic policy instantiation via high-capacity foundation models remain open for investigation (Seker et al., 16 May 2025, Shin et al., 2024).

This suggests that the field is evolving toward unified, compositional, and robust frameworks for skill extraction and transfer, grounded in semantic reasoning, modular inference, and scalable data-efficient training protocols.