Sentiment Analysis on Drug Reviews: Fine-Tuning DeBERTa for Healthcare NLP

How I fine-tuned DeBERTa-v3-base for 3-class sentiment classification on complex pharmaceutical drug reviews.

Why Drug Review Sentiment Matters

Pharmaceutical companies, healthcare providers, and patients all benefit from understanding drug review sentiment at scale. Patient feedback contains invaluable signals about drug efficacy, side effects, and overall satisfaction — but manually reading thousands of reviews is impractical.

I built an NLP pipeline to automatically classify drug reviews into positive, negative, and neutral categories, enabling rapid analysis of patient sentiment patterns across different medications.

Why DeBERTa-v3-base?

DeBERTa (Decoding-enhanced BERT with disentangled attention) is Microsoft's improvement on the BERT architecture. The v3-base variant uses ELECTRA-style pre-training and disentangled attention mechanisms, making it particularly strong at understanding nuanced text — exactly what's needed for medical domain language.

Compared to standard BERT, DeBERTa's disentangled attention separately encodes content and position information, allowing better understanding of word relationships in complex medical sentences like "The medication helped with pain but caused severe nausea."

Challenges in Healthcare NLP

Drug reviews present unique NLP challenges:

• Medical terminology — Patients use a mix of technical and colloquial terms ("the SSRIs gave me brain zaps" vs. "experienced discontinuation syndrome")
• Mixed sentiment — Reviews often contain both positive and negative aspects ("great for anxiety but terrible side effects")
• Class imbalance — Reviews tend to skew strongly positive or negative, with fewer neutral reviews
• Context sensitivity — "This drug changed my life" could be positive (relief) or negative (harmful side effects) depending on context

Training Pipeline

The fine-tuning pipeline used Hugging Face Transformers with the following approach:

1. Data preprocessing — Cleaned review text, handled HTML entities, normalized medical abbreviations, and applied stratified train/val/test splitting
2. Tokenization — Used DeBERTa's SentencePiece tokenizer with dynamic padding and truncation at 512 tokens
3. Class weighting — Applied inverse frequency weighting in the loss function to handle class imbalance
4. Fine-tuning — Trained with learning rate warmup, linear decay, and early stopping on validation F1-score

Results & Learnings

The fine-tuned DeBERTa model achieved state-of-the-art accuracy on the drug review dataset, significantly outperforming baseline models like logistic regression on TF-IDF features and even fine-tuned BERT-base.

Key takeaways from this project: domain-specific fine-tuning is essential for medical NLP, disentangled attention architectures handle nuanced sentiment well, and proper handling of class imbalance is critical for real-world healthcare applications.