Analysis of National Day Rally speeches

The National Day Rally speeches serve as crucial historical documents reflecting Singapore’s journey, challenges, and aspirations. However, manually analyzing 30 years of speeches to identify key themes and patterns would be an immense undertaking. We needed a data-driven approach to uncover meaningful insights about Singapore’s policy evolution and national priorities.

• Developed text preprocessing pipeline using NLTK for cleaning and standardizing speech transcripts
• Implemented topic modeling using BERTopic to identify and extract key themes
• Created temporal analysis framework to track theme evolution across decades
• Generated data visualizations to communicate findings effectively
• Co-authored commentary piece published on TODAY Online

We developed a comprehensive text analysis system leveraging Natural Language Processing and topic modeling techniques. The solution combines NLTK for text preprocessing and BERTopic for advanced topic modeling, enabling us to analyze speeches from 1991 to 2023. Our approach revealed 10 distinct themes and their evolution across three decades, providing insights into Singapore’s changing priorities and challenges.

Our analytical approach seamlessly integrated qualitative and quantitative methods to ensure comprehensive insights. We began by developing a specialized preprocessing pipeline that incorporated custom stop words specific to the Singapore context. Text standardization was achieved through careful lemmatization, preserving meaningful context while enabling consistent analysis. The core of our analysis leveraged BERTopic’s advanced capabilities for temporal topic modeling. We structured our investigation across three distinct decades (1991-2000, 2001-2010, 2011-2023), generating detailed heatmaps to visualize how different themes evolved in prominence over time.

• “Progress and Vision” emerged as a dominant theme in the 1990s during Singapore’s first leadership transition
• “Healthcare and Aging Population” showed steady increase in importance across all decades
• Social cohesion remained a consistent priority throughout the 30-year period
• Economic themes clustered with social stability concerns during major crises (Asian Financial Crisis, Post-9/11, COVID-19)

This project has yielded valuable insights that bridge technical innovation and policy understanding. Our work has demonstrated the powerful role that NLP can play in understanding policy evolution, while emphasizing the critical importance of context-aware text preprocessing in achieving meaningful results. Through our analysis, we uncovered subtle patterns in Singapore’s policy priorities that weren’t immediately apparent through traditional analysis methods. Perhaps most importantly, this project has shown how data science can make meaningful contributions to public policy understanding, creating new pathways for evidence-based policymaking.