Publications
Peer-reviewed work on Bengali and low-resource language models. Newest first. My name is in bold in each author list.
-
Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms
Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta, Sabik Bin Sultan, Abdullah Khan Zehady
First three authors contributed equally.
879 harmful requests written natively in Bengali rather than translated, sorted into 17 harm categories anchored in Bangladeshi law, and asked five ways. Across 18 models and 15,822 judged responses, the same request written as newspaper prose is answered 17.5 points more often than written as a casual message, in 17 of the 18.
PaperarXivProjectDataLeaderboardCodePackage
The same harmful request, written five ways. Figure from the paper. Open it full size -
BanglaLlama: LLaMA for Bangla Language
Abdullah Khan Zehady, Shubhashis Roy Dipta, Naymul Islam, Safi Al Mamun, Santu Karmaker
Bangla has around 240 million native speakers and is still treated as low-resource. The paper releases 224,000 translated Bangla instruction examples, Bangla-Orca (172k) and Bangla-Alpaca (52k), and BanglaLlama, an open family of five base and instruct models trained on them.
-
Read Between the Lines: A Benchmark for Uncovering Political Bias in Bangla News Articles
Nusrat Jahan Lia, Shubhashis Roy Dipta, Abdullah Khan Zehady, Naymul Islam, Madhusodan Chakraborty, Abdullah Al Wasif
The first benchmark of 200 politically contested Bangla news articles, each labelled government-leaning, government-critique or neutral by three annotators. Across 28 proprietary and open models, F1 reaches 0.83 on government-critique but falls as low as 0.00 on neutral, and models over-predict government-leaning stances.
F1 by stance across model sizes. Neutral is the hard class. Figure from the paper. Open it full size