R / Python Pipelines for Biomedical LLM Semantic Search Apps

Abstract

Leveraging Pytorch's GPU indexing and R's data management, evaluation and visualization capabilities At the CELEHS laboratory we are particularly interested by LLM-based embeddings as BGE and BERT. As the number of models increases, we need methods to compare their clinical usefulness. While some R packages exist to leverage GPU capabilities, Pytorch is by far more used for GPU computation. In contrast, R is efficient for data management and visualization. How should one build robust and reproducible pipelines incorporating them both ? My answer is well-designed pipelines with Docker, Makefile, and Elasticsearch. In this talk I will showcase my design approaches to such challenges.