📌 Key facts
💡 Background
The integration of large language models (LLMs) with Retrieval-Augmented Generation (RAG) systems represents a significant advancement in natural language processing. LLMs like GPT-4 have demonstrated remarkable capabilities in generating human-like text, but they often struggle with issues such as hallucinations, outdated information, and difficulty handling complex queries. By combining LLMs with RAG systems, which retrieve relevant information from external sources, these challenges can be mitigated, leading to more accurate and contextually relevant outputs. This research focuses on evaluating the performance of various open-source LLMs within RAG systems, providing critical insights into their practical applications and potential improvements.
🎯 Objective
The objective of each master thesis paper within the research group is to conduct a comparative performance analysis of 3 open-source large language models (LLMs) within a Retrieval-Augmented Generation (RAG) system infrastructure. The goal is to evaluate which of these models performs best across the four key dimensions: noise robustness, negative rejection, information integration, and counterfactual robustness (four dimensions derived from Retrieval Augmented Generation Benchmark - https://doi.org/10.48550/arXiv.2309.01431).
🎓 Profile
⚠️ Essential Requirements
- Python Coding Skills 🐍: Basic to intermediate level proficiency in Python is essential. You should be comfortable writing and understanding Python code.
- Data Analysis & Data Science Background 📊: Basic knowledge of data analysis and data science is helpful. Intermediate skills in these areas will be preferred.
- Understanding of Machine Learning 🤖: A solid understanding of the fundamentals of machine learning is beneficial. Familiarity with Natural Language Processing (NLP) concepts and techniques, including experience with language models and RAG systems, is advantageous.
- Research Independence 📚: The ability to conduct independent research without collaboration with partner companies. This ensures the research remains unbiased and solely contributes to the overarching research project.
🟢 Additional Skills (Nice to Have)
- Experience with NLP Frameworks 🔧: Previous experience with NLP frameworks and libraries (like Hugging Face Transformers, spaCy, or NLTK) will be considered a plus.
- R Programming Skills 📈: Knowledge of R can be useful, especially for data analysis and result visualization.
- Knowledge of Large Language Models 💡: Familiarity with various LLMs, such as GPT-4, or similar models, and understanding their architecture and applications.
- Experience with Different Datasets 💾: Working with a variety of dataset sizes and formats, both structured and unstructured, particularly in the context of text data.
- Experimentation and Evaluation Skills 🧪: Experience in setting up and conducting experiments, including evaluating model performance using appropriate metrics.
👩🏻🎓👨🏽🎓 Personal Attributes
- Analytical Thinking 🧠: Strong analytical skills to critically evaluate the performance of different LLMs and their limitations.
- Attention to Detail 🔍: Meticulous attention to detail to ensure accuracy in your experiments and reporting.
- Proactive and Self-Motivated 💪: Proactivity and self-motivation are key, as independent research requires a high degree of self-discipline and initiative.
- Communication Skills 🗣️: Ability to clearly communicate findings and insights, both in written and verbal form, is essential for documenting and presenting your research.
📚 Further Reading
→ Please review both pages below to get a better understanding of LLMs & RAG systems
- https://www.ibm.com/topics/large-language-models
- https://cloud.google.com/use-cases/retrieval-augmented-generation?hl=en
→ Additional comprehensive material and readings will be shared during the Onboarding process.
🦾 Who We Are
The Chair for Strategy and Organization is focused on research with impact. This means we do not want to repeat old ideas and base our research solely on the research people did 10 years ago. Instead, we currently research topics that will shape the future. Topics such as Agile Organisations and Digital Disruption, Artificial Intelligence, Blockchain Technology, Creativity and Innovation, Digital Transformation and Business Model Innovation, Diversity, Education: Education Technology and Performance Management, HRTech, Leadership, and Teams. We are always early in noticing trends, technologies, strategies, and organisations that shape the future, which has its ups and downs.
More information on Nicolas Leyh can be found @Nicolas Leyh & under LinkedIn.
📝 How to Apply
If you are interested, please contact Nicolas Leyh by submitting your CV, grade report and a short motivation letter (focus on fit for this topic, e.g. data science / ML background).
Nicolas Leyh (Chair for Strategy and Organisation) 👉 nicolas.leyh@tum.de