Benchmarking PyCaret AutoML Against IndoBERT Fine-Tuning for Sentiment Analysis on Indonesian IKN Twitter Data

Abstract: This paper benchmarks a classical machine learning approach based on PyCaret AutoML against a deep learning approach based on IndoBERT fine-tuning for binary sentiment analysis of Indonesian-language Twitter comments related to Ibu Kota Nusantara (IKN). The dataset contains 1,472 manually labeled samples, consisting of 780 negative and 692 positive comments. In the machine learning setting, Logistic Regression, Naive Bayes, and Support Vector Machine were evaluated using 10-fold cross-validation, with Logistic Regression achieving the best performance among the classical models at 77.57% accuracy and 77.17% F1-score. In the deep learning setting, the indobenchmark/indobert-base-p1 model was fine-tuned for five epochs and achieved 89.59% test accuracy and 89.37% F1-score. The results show that IndoBERT substantially outperforms the machine learning baselines, highlighting the effectiveness of Transformer-based contextual representations for informal Indonesian social media text.

• The paper demonstrates that IndoBERT fine-tuning achieves 89.59% accuracy, outperforming PyCaret AutoML by over 12 percentage points.
• The methodology rigorously compares traditional ML models (Logistic Regression, Naive Bayes, SVM) via PyCaret with Transformer-based IndoBERT for sentiment classification on noisy Indonesian social media data.
• The study highlights that contextual representations enhance sentiment analysis in informal texts, though at the cost of increased computational resources.

Benchmarking PyCaret AutoML and IndoBERT Fine-Tuning for Sentiment Analysis of Indonesian IKN Twitter Data

Introduction

This work presents a rigorous benchmark of traditional AutoML-based machine learning and Transformer-based fine-tuning for sentiment classification in the context of Indonesian social media. The focus is on Twitter comments related to Indonesia’s Ibu Kota Nusantara (IKN) initiative, which presents unique challenges due to its informal textual structure and Indonesian-specific vernacular. The comparison involves three classical models—Logistic Regression, Naive Bayes, and SVM—automated via PyCaret, and a fine-tuned IndoBERT-base architecture on a curated binary-labeled dataset. The comparative analysis identifies the representational and operational differences in machine learning and deep learning paradigms for low-resource, informal text sentiment analysis.

Dataset and Preprocessing

The dataset consists of 1,472 Indonesian-language Twitter comments related to IKN, partitioned nearly evenly into 780 negative and 692 positive samples. The character-level analysis reveals a mean text length of 50.3 and a high standard deviation, indicative of the brevity and variance typical in social media.

Preprocessing includes lowercasing, NaN removal, and feature computation for text length. These procedures ensure consistency and integrity, vital for both bag-of-words (BoW) and context-sensitive models. The textual diversity and informality emphasize the necessity for robust tokenization and representation, especially critical for downstream BERT-based architectures.

Methodology

Machine Learning Pipeline—PyCaret AutoML

PyCaret 3.x functions as the automated ML orchestration framework. TF-IDF is adopted for static feature representation. Model selection incorporates:

• Logistic Regression with L2 regularization and LBFGS optimization,
• Naive Bayes for probabilistic text classification,
• SVM with a linear kernel.

Hyperparameter optimization leverages Optuna-based Bayesian search. Evaluation uses 10-fold stratified cross-validation and aggregates all key metrics accordingly.

IndoBERT Fine-Tuning Pipeline

IndoBERT, comprising 12 Transformer encoder layers, 768 hidden units, and 12 attention heads (111M parameters), is fine-tuned using AdamW (learning rate $2 \times 10^{-5}$, weight decay 0.01), with gradient clipping and a linear warmup scheduler. The classification head is a single linear layer atop the [CLS] token. Data partitioning allocates 70% training, 15% validation, and 15% test samples, maintaining class balance.

Experimental Results

PyCaret AutoML Results

Logistic Regression yields the highest ML accuracy (77.57%) and F1 (77.17%), Naive Bayes follows at 70.58%, while SVM—unusually—lags significantly at 52.72%, likely attributable to suboptimal alignment between PyCaret’s TF-IDF pipeline and high-vocabulary variation in informal social media. After extensive tuning, SVM improves to 71.17%. These outcomes reinforce the inherent limitations of standard BoW models under extreme lexical heterogeneity.

IndoBERT Fine-Tuning Results

IndoBERT fine-tuning achieves 89.59% test accuracy and 89.37% macro-averaged F1, marking a 12.02 percentage point gain over the best-performing ML baseline. Training histories manifest rapid convergence but incipient overfitting beyond the initial epoch, with validation accuracy plateauing around 85% and validation loss incrementing with further epochs.

Figure 1: Training and validation curves for loss and accuracy during IndoBERT fine-tuning; rapid overfitting is evident, typical for BERT-based models on medium-scale datasets.

Confusion matrix analysis demonstrates high recall for the negative class (95.7%) versus the positive class (82.7%). Misclassified instances are concentrated in positive-to-negative predictions, suggesting challenges in capturing indirect or ironic positive sentiment—potentially an artifact of limited positive-class diversity or nuanced class boundaries.

Figure 2: Confusion matrix for IndoBERT on the test set; recall notably diverges between negative and positive sentiment classes.

Analysis and Discussion

IndoBERT’s performance dominance underscores several theoretical and applied insights:

• Contextual Representations: IndoBERT’s bidirectional encoding captures semantic dependencies and polysemy, unattainable with TF-IDF. This is critical for informal, code-switched, and slang-laden Indonesian social media.
• Pre-training on Indonesian Corpora: The language-specific pre-training allows IndoBERT to model nonstandard expressions, OOV terms, and local phraseology effectively, rectifying the OOV and sparsity issues that classical ML featureization suffers.
• Tokenization Robustness: WordPiece tokenization endows IndoBERT with the ability to generalize over rare or novel words, substantially enhancing resilience to informal language artifacts.

Resource analysis indicates that IndoBERT’s fine-tuning and inference are substantially costlier than PyCaret-based alternatives, especially without dedicated acceleration hardware. For latency or deployment-constrained scenarios, Logistic Regression remains a viable, albeit lower-performing, operational baseline.

Implications and Future Directions

The findings reinforce the imperative of domain-adaptive, Transformer-based architectures for informal and low-resource language analysis tasks. The pronounced accuracy and F1 gains over classical pipelines catalyze further research into model compression (e.g., DistilBERT, ALBERT), data augmentation to mitigate BERT overfitting, and robust evaluation on truly out-of-distribution social media phenomena (e.g., sarcasm, code-mixing).

On the production side, practitioners must weigh accuracy-resource trade-offs; for resource-rich environments, Transformer fine-tuning sets the state-of-the-art, while for edge or high-throughput applications, Lean ML models benefit from rapid, automated deployment via frameworks like PyCaret.

Conclusion

The benchmark demonstrates that IndoBERT fine-tuning decisively surpasses PyCaret-based classical machine learning on Indonesian IKN sentiment analysis, improving accuracy by over 12 percentage points and substantiating the importance of contextual modeling in noisy, informal text settings. Logistic Regression remains an efficient fallback for constrained scenarios. Future work should probe data augmentation, regularizing strategies to abate overfitting in Transformers, and pragmatic trade-offs among model expressivity, speed, and deployment complexity.