The Rise and Fall of Data Science Trends: A 2018–2024 Conference Perspective

The field of data science has evolved dramatically over the past several years, driven by technological breakthroughs, industry demands, and shifting priorities within the community. By analyzing conference session titles and abstracts from 2018 to 2024, we can trace the rise and fall of key trends that shaped the industry. This blog explores how different topics gained momentum, which areas declined, and what this tells us about the future of data science.

1. The Rise of AI Engineering and MLOps

2018–2019: Early discussions around MLOps and AI engineering were sparse, primarily focused on general machine learning best practices.

2020–2022: As enterprises moved from experimentation to production, MLOps tools like MLflow, Kubeflow, and model monitoring solutions surged in prominence.

2023–2024: AI engineering became a hot topic, expanding beyond MLOps to include AI agents, autonomous systems, and scalable model deployment techniques.

MLOps emerged as a necessary discipline to address the challenges of deploying and maintaining machine learning models in production environments. Initially, organizations struggled with versioning, monitoring, and automating model updates. As MLOps matured, discussions shifted from simple automation to complex orchestration involving continuous integration, deployment (CI/CD), and model drift detection. AI engineering extended this by integrating AI systems more deeply into software engineering pipelines, making it a crucial field as AI applications became more sophisticated and embedded in real-world systems.

Takeaway: The industry’s focus has shifted from building models to making them robust, scalable, and maintainable.

2. The Boom of Generative AI and Large Language Models (LLMs)

2018–2020: NLP was gaining traction, with a focus on word embeddings, BERT, and sentiment analysis.

2021–2022: Transformer-based models took center stage, with GPT-3 driving conversations around text generation.

2023–2024: The emergence of GPT-4, Claude, and open-source LLMs dominated discussions, highlighting real-world applications, fine-tuning techniques, and AI safety concerns.

The explosion of generative AI and LLMs has redefined how businesses and developers interact with artificial intelligence. Initially, NLP research revolved around improving traditional language models and embeddings, but the introduction of transformers changed the landscape. By 2021, GPT-3 had demonstrated unprecedented capabilities in text generation, leading to widespread adoption. The next wave of advancements, including fine-tuned LLMs and multimodal AI, has enabled creative applications in content creation, coding assistance, and conversational agents. However, with this growth came concerns around misinformation, ethical AI usage, and data privacy, fueling discussions around responsible AI deployment.

Takeaway: The rapid evolution of LLMs suggests a shift from model development to domain-specific applications and ethical considerations.

3. The Decline of Traditional Machine Learning

2018–2020: Algorithms like random forests, SVMs, and gradient boosting were frequent discussion points.

2021–2024: Interest declined as deep learning and pre-trained models took over, automating many tasks previously handled by classical ML techniques.

While traditional machine learning remains fundamental, its dominance has waned in the face of deep learning and automated machine learning (AutoML). Early years saw extensive discussions around feature engineering, model selection, and hyperparameter tuning, but as neural networks became more powerful and accessible, interest in classical ML methods decreased. Today, many organizations leverage pre-trained models or AutoML frameworks that abstract away much of the manual tuning required for classical techniques. This shift suggests that while traditional ML is still relevant, its role is now more supportive rather than cutting-edge.

Takeaway: Traditional ML is far from obsolete but is now seen as a foundational skill rather than a frontier topic.

4. Data Engineering’s Steady Growth

2018–2021: Data engineering was often mentioned but overshadowed by modeling advancements.

2022–2024: As AI models required larger and cleaner datasets, interest in data pipelines, ETL frameworks, and real-time data processing surged.

Data engineering has transitioned from being an underappreciated aspect of AI development to a critical discipline in its own right. Early discussions treated data engineering as a prerequisite for machine learning, but as AI models grew more complex, the need for scalable and efficient data infrastructure became undeniable. Today, data engineering is a major focal point, with organizations investing in robust ETL (Extract, Transform, Load) pipelines, real-time streaming solutions, and cloud-based data platforms. The rise of technologies like Apache Spark, Snowflake, and Delta Lake highlights the increasing demand for data infrastructure capable of supporting AI-driven applications.

Takeaway: The importance of scalable data infrastructure continues to grow as organizations prioritize high-quality data over model complexity.

5. The Rise and Plateau of Data Visualization

2018–2020: Data visualization was a major focus, with tools like Tableau, Power BI, and interactive dashboards gaining traction.

2021–2024: With automated insights and AI-driven analytics improving, the emphasis shifted from visualization to explainability and storytelling.

Early years saw a heavy emphasis on interactive dashboards, reporting tools, and data-driven decision-making. Organizations invested in building visually compelling representations of data to drive insights. However, as AI-powered analytics tools improved, manual visualization lost some of its appeal. Instead of relying on human-driven exploration, modern analytics platforms now leverage AI to surface key insights automatically. This shift has led to discussions around interpretability and explainability — ensuring that AI-generated insights remain transparent and understandable for stakeholders.

Takeaway: While visualization remains critical, the field is now integrating AI-driven insights rather than relying solely on human-driven interpretation.

Conclusion: What’s Next?

Based on these trends, we predict:

• More AI agent discussions: With LLMs maturing, focus will shift toward autonomous AI systems that can take actions based on language understanding.
• A stronger emphasis on AI ethics: As AI is deployed widely, responsible AI frameworks and governance will become essential.
• Continued growth in AI infrastructure: Scalability, efficiency, and cloud-based AI solutions will dominate discussions.

The rise and fall of data science trends reflect the ever-changing nature of the field. By understanding these shifts, professionals can better prepare for the future and align their skills with emerging opportunities.

Next Up — ODSC East 2025!

We’re planning for the 10th anniversary of ODSC East to be the biggest one yet. Coming up this May 13th-15th in Boston and virtually, ODSC East is shaping up to be a good time. We’re at a new location in Boston’s Seaport, we’re going to have an entire Keynote track, and we’re planning fun celebrations. You can register for ODSC East here while tickets are still heavily discounted, and sign up for our newsletter to get all updates as they come in.