Sentiment Analysis of AI Adoption in Indonesian Higher Education Using Machine Learning and Transformer-Based Models

Abstract: This study analyzes Indonesian student opinions on the adoption of artificial intelligence in higher education using two approaches: TF-IDF-based machine learning and Transformer-based deep learning. The dataset consists of 2,295 labeled samples, combining 1,154 student opinions with additional lexical sentiment data. LightGBM, Random Forest, and Support Vector Machine (SVM) are evaluated as machine learning models, while DistilBERT is fine-tuned for binary sentiment classification. The results show that SVM achieves the best performance among the machine learning models with 82.14% test accuracy and F1-score, while DistilBERT performs best overall with 84.78% accuracy and 84.75% F1-score. These findings indicate that Transformer-based models better capture contextual information, although SVM remains a competitive and efficient alternative for sentiment classification.

• The paper demonstrates that SVM with TF-IDF achieves high accuracy (82.14% test accuracy) for sentiment classification on Indonesian student data.
• The study shows that fine-tuned DistilBERT improves performance (84.78% test accuracy) by leveraging contextual embeddings despite higher computational needs.
• The comparative analysis supports practical AI sentiment monitoring in higher education, offering both cost-effective classical methods and state-of-the-art Transformer-based solutions.

Sentiment Analysis of AI Adoption in Indonesian Higher Education: Comparative Evaluation of Machine Learning and Transformer Models

Introduction

This work conducts a comparative investigation of sentiment analysis techniques to assess student perceptions of AI adoption in Indonesian higher education. The study focuses on the efficacy of classical machine learning models employing TF-IDF feature extraction, versus Transformer-based architectures, specifically DistilBERT, for classifying sentiment polarity in Indonesian-language data. The research context is driven by the proliferation of AI technologies such as generative LLMs (e.g., ChatGPT) and growing scrutiny regarding their impact on academic integrity and learning behaviors.

Methodology and Experimental Setup

A balanced dataset of 2,295 samples was constructed by combining 1,154 authentic Indonesian student opinion texts regarding AI in higher education with 1,141 samples from sentiment lexicons. Preprocessing involved comprehensive text normalization—lowercasing, noise (URL, mention, punctuation) removal, and standardization of non-standard terms—to mitigate out-of-vocabulary and sparsity issues.

For the classical pipeline, unigrams and bigrams were extracted using TF-IDF, followed by model selection among LightGBM, Random Forest, and SVM. SVM was applied with an RBF kernel and a regularization parameter $C=1.0$. All classical classifiers shared consistent data splits (80:10:10 train:val:test) and hyperparameters conducive to reproducibility.

In parallel, the deep learning approach involved fine-tuning distilbert-base-uncased, with inputs limited to 128 tokens, batch size 64, $\text{learning rate}=1\times10^{-6}$, dropout 0.5, weight decay 0.001, and AdamW optimization under early stopping (patience=5). Evaluation metrics included accuracy, precision, recall, and F1-score, with test set performance emphasized for model comparison.

Results

Machine Learning Models

LightGBM, while computationally fast, exhibited significant degradation on the sparse, high-dimensional TF-IDF features (test accuracy 49.67%, F1 48.46%), indicating poor suitability for this domain. Random Forest modestly improved upon LightGBM (test accuracy 58.61%, F1 56.48%) but remained suboptimal relative to SVM.

SVM achieved the leading classical results: 87.25% validation accuracy, and 82.14% test accuracy and F1-score. The SVM confusion matrix illustrated balanced error distribution, signifying stable sensitivity and specificity for both sentiment classes. The findings underscore SVM's robustness in high-dimensional sparse vector spaces characteristic of linguistic text data.

DistilBERT

DistilBERT outperformed all classical baselines, reaching 84.78% test accuracy and 84.75% weighted F1-score. The model exhibited balanced per-class F1 results (negative: 0.8548, positive: 0.8402), with only minor variance in sensitivity and precision across polarity. Training and validation loss curves evidenced minimal overfitting and strong generalization, despite a moderate compute footprint (22.8 minutes train time).

The improvement of DistilBERT over SVM—2.64% in absolute accuracy and 2.61% in F1-score—validates the advantage of contextual Transformer embeddings for Indonesian sentiment classification tasks. Nevertheless, the increased computational requirements of DistilBERT remain material, particularly in low-resource deployment environments.

Comparative Summary

Model	Approach	Test Accuracy	Test F1-score	Relative Strength
LightGBM	Gradient boosted trees	49.67%	48.46%	Fast, but poor on TF-IDF
Random Forest	Bagged decision trees	58.61%	56.48%	Modest improvement
SVM	Kernel SVM + TF-IDF	82.14%	82.14%	High accuracy/efficiency
DistilBERT	Transformer-based DL	84.78%	84.75%	Best overall, contextual

This table highlights the consistent superiority of context-aware deep LLMs over classical vector space models, but also that SVM remains highly competitive when computational or latency constraints are present.

Implications and Future Directions

The results demonstrate the feasibility and relative merits of both classical and deep models for Indonesian sentiment analysis in educational technology adoption contexts. The strong performance of SVM provides a cost-efficient solution in resource-constrained inference settings. Conversely, DistilBERT confirms state-of-the-art Transformer advantages for language understanding, with only modest additional computational overhead for significant accuracy gains.

Practically, institutions can leverage these findings to deploy automated sentiment analysis tools for continuous monitoring of student opinion as AI integration intensifies. The deployment of a Gradio-based web application as described enables scalable sentiment collection and analysis, promoting responsive policy formation.

Theoretically, the study emphasizes the need for ongoing local LLM benchmarking. Future extensions should consider larger, more nuanced datasets—including multi-class (neutral/ambivalent) sentiment and aspect-based annotation. Employing Indonesian-specific pre-trained LMs (e.g., IndoBERT, XLNet-ID) could further optimize contextual performance. Moreover, longitudinal sentiment tracking may uncover shifts in perception correlating with policy changes, new AI tool rollouts, or broader educational reforms.

Conclusion

The comparative evaluation substantiates that while SVM with TF-IDF yields strong, efficient sentiment classification for Indonesian student opinion, fine-tuned DistilBERT offers the best overall performance and generalization for this domain. The findings support both practical applications of sentiment monitoring in Indonesian higher education and continued methodological innovation, particularly with targeted Indonesian language resources and aspect-based transformer modeling. Future research will benefit from larger datasets and the application of more nuanced, task-specific LLMs to capture evolving sentiment regarding AI in education.

[Refer to "Sentiment Analysis of AI Adoption in Indonesian Higher Education Using Machine Learning and Transformer-Based Models" (2604.27439) for full methodological details.]