Impact Scope Metric (ISM) Overview

• Impact Scope Metric (ISM) is a suite of methods that quantifies research contributions by assessing both conceptual depth and interdisciplinary breadth.
• The taxonomy-based approach evaluates contributions by mapping research outputs to hierarchical domain nodes, using depth and penalty metrics to emphasize innovation.
• The bibliometric (H, M) approach integrates the h-index with journal diversity to measure overall impact and multidisciplinary outreach in research.

The Impact Scope Metric (ISM) encompasses a family of methodologies for quantifying not only the impact but also the scope, breadth, and diversity of scientific contributions across research domains. Traditional citation-based indices such as the h-index measure research output and citation impact but are silent on conceptual scope, multidisciplinarity, or reach. The ISM concept formalizes supplementary metrics that capture facets such as the conceptual layer of innovation within a research domain hierarchy or the diversity of journals or topics a researcher has influenced. The ISM framework includes both domain-taxonomy-based metrics—assigning researchers to structural nodes within field-specific hierarchies—and bibliometric approaches pairing citation counts with publication dispersion across journals. These approaches yield quantitative, model-driven tools for nuanced evaluation of individual, group, or institutional research performance.

1. Formal Definition and Mathematical Construction

Two principal forms of the ISM have been articulated, each with a distinct mathematical underpinning:

a. Taxonomic-Rank-Based ISM

This approach, presented by Mirkin and Nikolaev, assigns impact scope through a taxonomy tree $T=(V,E)$, where $V$ is the set of domain nodes and $E$ the set of directed edges (parent $\to$ child), with each node $v$ located at level $\ell(v)\in\{0,1,\dots,L_{max}\}$. For a given researcher $r$, let $N_r\subset V$ denote the set of nodes they have created or substantially transformed. Then,

• Base rank: $\ell_{\min}(r) = \min_{v\in N_r} \ell(v)$ (i.e., closest-to-root node),
• $n_=$: Number of impacted nodes at this best level: $V$0,
• $V$1: Number of impacted nodes deeper in the tree: $V$2,

Defining taxonomic rank: $V$3

Finally, $V$4 is normalized among a sample of researchers, mapping the lowest $V$5 (most conceptually impactful) to $V$6 and the highest (least impactful by these criteria) to $V$7: $V$8

b. Journal-Diversity-Based ISM

Adda-Bedia and Lechenault define scope using bibliometric indices:

• $V$9: h-index, the maximal integer such that at least $E$0 papers are cited at least $E$1 times,
• $E$2: number of distinct journals where the researcher has published.

These are combined into a two-dimensional index $E$3: $E$4 Yielding explicit forms: $E$5 Here, $E$6 represents overall output and scope (extensive), $E$7 quantifies multidisciplinarity or outreach (intensive).

2. Methodological Workflow

Taxonomy-Based ISM (Mirkin & Nikolaev)

1. Domain Taxonomy Construction: Build or extract a rooted tree $E$8 representing the conceptual structure of the field, with depths $E$9 assigned.
2. Selection of Key Contributions: For each researcher, select a minimal yet representative set of seminal works ($\to$0).
3. Node Assignment: By expert judgement or semi-automated methods, map each output $\to$1 to one or more taxonomy nodes it created or transformed; deduplicate to generate $\to$2.
4. Computation: Evaluate $\to$3, $\to$4, $\to$5, then compute $\to$6 and $\to$7 per above equations.
5. Stratification (Optional): Partition the normalized scores into strata (e.g., via $\to$8-means), yielding bands (e.g., “top,” “middle,” “lower”).

Bibliometric ISM (Adda-Bedia & Lechenault)

1. Data Acquisition: Extract $\to$9 and $v$0 for each researcher from bibliographic databases.
2. Computation: For each researcher, calculate $v$1 and $v$2 per the equations above.
3. Profiling: Analyze $v$3 jointly to assess combined impact and scope, and to categorize specialist vs. broadcaster profiles.
4. Aggregation: Apply at scale to research groups or institutions by summing $v$4 and $v$5.

3. Interpretation and Illustrative Examples

Taxonomy-Based ISM

• Depth as Primary Determinant: Innovation at higher layers ($v$6 small, closer to root) reflects large-scale conceptual advances, accorded greater weight.
• Breadth as Secondary Modifier: Additional nodes at the same depth incur a penalty of $v$7 per node; deeper nodes incur $v$8 per node, favoring impactful breadth but not at the expense of depth.
• Examples: In the data-analysis pilot, researcher S₂ mapped five influential works to nodes all at depth $v$9, yielding $\ell(v)\in\{0,1,\dots,L_{max}\}$0 (best in sample, $\ell(v)\in\{0,1,\dots,L_{max}\}$1). S₁, impacting both level $\ell(v)\in\{0,1,\dots,L_{max}\}$2 and level $\ell(v)\in\{0,1,\dots,L_{max}\}$3 nodes, had $\ell(v)\in\{0,1,\dots,L_{max}\}$4.

(H, M) ISM

• Extremes: $\ell(v)\in\{0,1,\dots,L_{max}\}$5, $\ell(v)\in\{0,1,\dots,L_{max}\}$6 yields $\ell(v)\in\{0,1,\dots,L_{max}\}$7, $\ell(v)\in\{0,1,\dots,L_{max}\}$8 (broad scope, little impact); $\ell(v)\in\{0,1,\dots,L_{max}\}$9, $r$0 yields same $r$1 but $r$2 (high impact, narrow scope).
• Typical Case: $r$3, $r$4 gives $r$5, $r$6, reflecting both substantial impact and outreach.
• Group Discrimination: Calculated $r$7 distinguished sub-departments by multidisciplinarity.

4. Sensitivity to Scope: Breadth Versus Depth

Taxonomy-Based ISM

• Prioritizes depth: major advances at higher (more general) taxonomy levels dominate the score.
• Recognizes breadth: multiple landmark contributions at the same conceptual level further lower $r$8, while additional, but deeper, innovations count for less.
• Penalties for breadth ($r$9 for same-depth, $N_r\subset V$0 for deeper) provide tuneable smooth trade-off without sharp thresholds; these may be optimized or calibrated per domain.

(H, M) Approach

• $N_r\subset V$1 acts as a normalized angular metric: $N_r\subset V$2 for narrow, highly cited research; $N_r\subset V$3 for broad, less-cited multidisciplinary work.
• $N_r\subset V$4 ensures gross productivity is considered independent of the profile.

5. Properties, Advantages, and Limitations

Aspect	Taxonomy-based ISM	(H, M) Bibliometric ISM
Transparency	Taxonomy and mappings are explicit	Calculation uses published data
Scope capture	Conceptual hierarchy, field-wide	Journal diversity; outreach
Breadth-vs-Depth	Depth prioritized, breadth tunable	Multidisciplinarity quantified
Subjectivity	Node mapping is expert-driven	Minimal, database-driven
Dependence on taxonomy	Strong	None (unless journal ≈ field)
Manual effort	High	Low

Advantages (both):

• Provide complementary information to citations or h-index, quantifying scope and multidisciplinarity explicitly.
• Enable cross-comparison by deploying field-agnostic scaling ($N_r\subset V$5, $N_r\subset V$6).
• Allow institutions or evaluators to weight depth (conceptual origins) versus breadth (interdisciplinary or outreach potential), matching appraisal criteria to goals.

Limitations:

• Taxonomy-based ISM: Labor-intensive, subject to expert bias, penalty values are ad hoc, sensitive to changes in taxonomy structure, and blind to pure volume of output.
• Journal-diversity ISM: Does not distinguish between truly interdisciplinary work and dispersion across similar journals; subject to the granularity of journal indexing; does not adjust for scope/impact of journals; potential over-counting or “salami slicing.”

6. Implementation, Applications, and Future Directions

Implementation of taxonomy-based ISM requires: domain taxonomy curation (e.g., ACM-CCS), development of expert panels or automated text-matching tools for paper-to-node assignments, and normalization protocols for inter-group comparison. A plausible implication is that large-scale deployment would require semi-automated or machine-learning-based assignment pipelines coupled with periodic taxonomy updates.

The $N_r\subset V$7 framework is immediately implementable wherever citation and journal data are indexed, permitting rapid benchmarking of individuals, groups, or departments. Key applications include hiring and promotion, grant allocation (where multidisciplinarity or outreach is prized), or longitudinal studies tracking evolution of research profile.

Potential avenues for refinement include weighting journal diversity by journal scope or impact, mapping papers to subjects/keywords rather than venues, and combining ISM with volume or citation-based scores in multi-criteria frameworks via least-squares stratification for optimal metric blending.

7. Relations to Established Research Metrics and Open Issues

The ISM constitutes a well-justified enhancement to the landscape of quantitative research assessment. While traditional metrics such as the h-index focus on vertical (temporal or citation) aggregation, ISM explicitly quantifies horizontal (conceptual or disciplinary) reach within or across research domains. Open-ended challenges include stabilizing penalty parameter choices, minimizing mapping subjectivity, and balancing the tension between transparency and automation. Peer-based qualitative assessment remains essential, and ISM is proposed as a transparent, reproducible complement to rather than replacement for existing evaluative practices (Murtagh et al., 2016, Adda-Bedia et al., 2017).