MorphTable Construction Overview

• MorphTable Construction is a unified approach that transforms irregular data tables into structured, relational formats and encodes complex morphological inflections.
• It employs an operator-based DSL and beam search algorithms to systematically synthesize transformation pipelines, ensuring both data normalization and accurate feature extraction.
• Applications span computational linguistics and data analytics, enabling scalable processing of linguistic inflection tables and automated normalization of wild, non-relational tables.

A MorphTable is a comprehensive, algorithmically constructed table that encodes complex structure—either morphosyntactic inflection in the context of linguistic databases such as UniMorph (Batsuren et al., 2022), or, in computational table processing, the transformation pipeline that restructures arbitrary "wild" tables into normalized relational tables as in Auto-Tables (Li et al., 2023). This article provides a thorough perspective on both paradigms of MorphTable construction, connecting the data-driven methods from NLP/linguistics and the synthesis-driven approaches for data engineering and analytics.

1. Formal Foundations of MorphTable Construction

MorphTable construction is grounded in rigorous definitions of the input and output table forms, operator sets, and feature space.

In the context of data transformation, a wild (non-relational) table $T$ is formally defined as an $n \times m$ grid possibly violating First Normal Form, column homogeneity, single-header conventions, and atomicity; relationalization seeks a table $R$ where every cell is atomic, all metadata occupy a single header row, every column is semantically homogeneous, and no repeating groups occur (Li et al., 2023). The MorphTable construction task then is: given $T$ and a fixed DSL $\mathcal{O}$ with parameter domains $P$, synthesize a pipeline

$M = [O_1(p_1), O_2(p_2), \dots, O_L(p_L)]$

satisfying $M(T)$ is relational, minimizing pipeline cost.

Within UniMorph, MorphTable construction is the instantiation of all ($\text{lemma}, \text{FeatureSet})$ pairs for a language, outputting fully annotated inflection tables where features are encoded hierarchically—organizing POS, number, case, gender, possession, tense, aspect, person, mood, voice, and argument structure within a feature tree (Batsuren et al., 2022).

2. MorphTable Operator Sets and Grammar (DSL)

The transformation process is characterized by an operator-based DSL that enables structured pipeline synthesis. Auto-Tables defines eight atomic operators:

Operator	Core Function	Parameters
stack	Collapse columns into "variable" and "value" columns	start_idx, end_idx
wide_to_long	Collapse repeating column groups into rows	start_idx, end_idx, delim
transpose	Swap rows and columns	—
pivot	Convert repeating row-blocks into columns	repeat_frequency
explode	Split multi-valued cells	column_idx, delim
ffill	Forward-fill empty cells	start_idx, end_idx
subtitles	Extract subtitles into a new column	column_idx, row_filter
none	No operation (identity for relational tables)	—

Pipelines are defined by a concise BNF grammar allowing composition:

<pipeline> ::= <step> | <step> ";" <pipeline> <step> ::= <op> "(" <arg_list> ")"

This controlled operator space enables formal search, discoverability, and interpretability in both code-generation and table-normalization contexts (Li et al., 2023).

In morphological table construction, the generator

$$f : (\text{lemma}, \text{FeatureSet}) \to \text{form}$$

instantiates paradigm patterns matching the target feature bundle, using pattern templates and post-processing rules (Batsuren et al., 2022).

3. Pipeline Synthesis and Learning-Based Search

Auto-Tables employs a constraint-driven enumeration algorithm using beam search, guided by a learned scoring model that operates in two phases:

• Phase A: Training Data Generation — Clean relational tables are transformed using inverse DSL operators to generate wild/target pairs $(T, O(p))$, producing $\sim$1.4M training examples through data augmentation (random cropping, shuffling). This process is entirely self-supervised, requiring no human annotation (Li et al., 2023).
• Phase B: Ranking Model — Each table is encoded per-cell with a 423-dimensional vector: $[384d$ Sentence-BERT semantic; $39d$ hand-crafted syntactic$]$, further reduced by CNNs. Operator and parameter prediction is cast as an 8-way softmax plus per-parameter softmax, optimizing cross-entropy loss.
• Phase C: Beam-Search & Pruning — The search maintains a beam (width $k=8$) of partial pipelines, using model probabilities to extend and prune, with constraints to prevent ill-formed or duplicate operations. Termination occurs at step count $L=3–5$ or upon reaching none().
• Phase D: Input/Output Reranking — Candidate single-step outputs are re-encoded and scored by an MLP reranker to select those that most closely "look relational," based on features such as header completion and column homogeneity.

The cost model for a pipeline $M$ executing on $T$ is

$$\mathrm{Score}(M \mid T) = \sum_{i=1}^L \log \Pr(O_i(p_i) | T_{i-1})$$

and, at the reranking stage,

$$\mathrm{RerankScore}(O_i) = \mathrm{softmax}(\mathrm{MLP}([\mathrm{features}(T_i)]))$$

These steps enable efficient, highly accurate pipeline synthesis for MorphTable construction in tabular data settings (Li et al., 2023).

4. Linguistic MorphTable Construction: Schema, Extraction, and Inflection

In the UniMorph tradition, MorphTable construction is a five-stage pipeline:

1. Extraction: Parse source paradigms (Wiktionary tables, FSTs, grammar tables) into $(\text{lemma}, \text{form}, \text{raw-tags})$ triplets.
2. Normalization: Unicode normalization, orthographic unification, stripping extraneous markup.
3. Mapping & Augmentation: Map raw tags to the UniMorph schema; fill missing data (gender, macron diacritics) by rule or lexicon lookup.
4. Segmentation: Optionally segment morphemes via language-specific suffix tables.
5. Validation & Cleanup: Validate co-occurrences against gold standards (e.g., Universal Dependencies), resolve conflicts, and output final triples in TSV format.

The feature schema uses a hierarchical tree, in which categories such as “Argument” support polypersonal agreement and stacking (e.g., ALL(COM(SG)))—a crucial advance for highly inflected languages (Batsuren et al., 2022). Each paradigm class specifies a mapping from feature bundles to templatic patterns, and form generation proceeds by applying the paradigm class pattern (with fallback if no direct match occurs).

5. Benchmarking, Evaluation, and Practical Performance

The effectiveness of MorphTable construction in table normalization is quantified using ATBench, a real-world benchmark of 244 test cases spanning forum requests, existing notebook scripts, and web-extracted tables. In ATBench:

• Auto-Tables achieves top-1 accuracy of 0.57, top-2 of 0.697, and top-3 of 0.75.
• Alternative models such as TaBERT-VS, TURL-VS, and GPT-3.5-fs perform significantly lower (e.g., TaBERT-VS at 0.193 Hit@1) (Li et al., 2023).
• Auto-Tables average latency per case is 0.224 s, surpassing by-example tools (Foofah, FlashRelate), which require user annotation plus longer runtimes.

For UniMorph MorphTables, coverage is maintained through schema expansion and automated pipeline improvements, enabling database construction for hundreds of languages, including mechanisms for handling missing features and orthographic complexity (Batsuren et al., 2022).

6. Integration, Extensibility, and Application Contexts

MorphTable construction workflows support direct integration into Python (Pandas), SQL, and other analytics environments. An example of the Python integration is:

df1 = df.transpose() df2 = pd.melt(df1, id_vars=['GroupID'], value_vars=['2015','2016','2017','2018','2019','2020'], var_name='Year', value_name='Value')

Equivalent transformations are synthesized into SQL, using constructs like UNPIVOT for attribute flattening.

Both frameworks feature extensibility mechanisms:

• In table normalization, new custom operators (e.g., “shift-cells,” “merge header rows”) can be appended to the DSL with corresponding inverses for self-supervised learning (Li et al., 2023).
• In morphological inflection, new languages are added via data-driven extraction, bespoke mapping scripts, validation heuristics, and community review, supported by LaTeX templates for rapid table instantiation (Batsuren et al., 2022).

A plausible implication is that the principled separation of operator space and hierarchical schema enables scalable, interpretable, and robust MorphTable construction across both database normalization and linguistic modeling domains.

7. Worked Examples and Templates

For linguistic MorphTables, canonical LaTeX templates encode the tabularization of inflectional data. A worked example for Spanish "hablar" (present indicative and subjunctive) is as follows:

\begin{table}[ht] \small \centering \begin{tabular}{lcc} \toprule \multicolumn{1}{c}{\bf Person} & \bf SG & \bf PL \ \midrule %–––– Present Indicative \multicolumn{3}{l}{\it Indicative – Present (\textsf{V;PRS;IND})} \ 1\textsuperscript{st} & hablo & hablamos \ 2\textsuperscript{nd} & hablas & habláis \ 3\textsuperscript{rd} & habla & hablan \[3pt] %–––– Present Subjunctive \multicolumn{3}{l}{\it Subjunctive – Present (\textsf{V;PRS;SUB})} \ 1\textsuperscript{st} & hable & hablemos \ 2\textsuperscript{nd} & hables & habléis \ 3\textsuperscript{rd} & hable & hablen \ \bottomrule \end{tabular} \caption{MorphTable for Spanish {\it hablar}, Present Indicative and Present Subjunctive.} \label{tab:sp-hablar} \end{table}

Ready-to-use templates facilitate systematic collection of inflectional tables for new languages (Batsuren et al., 2022).

---

MorphTable construction, whether as the synthesis of relationalizing transformations for arbitrary tables (Li et al., 2023) or as the systematic tabulation of inflectional morphology (Batsuren et al., 2022), is defined by schema rigor, programmatic transformation, and automated evaluation, providing foundational infrastructure for computational linguistics and data analytics.