About the role
AI Engineer (Evaluation) is a active engineering role at ntu in NTU Main Campus, Singapore. Open the role to review the official description and apply on the company site.
CompanyOnsite
Key Responsibilities
- Responsibilities:
- Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
- Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
- Work with partners to collect, translate and verify evaluation datasets.
- Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
- Collaborate with cross-functional teams within AI Products to design and resolve issues.
- Maintain code repository and documentation standards.
- Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.
Requirements
- Graduate from a degree program in computer science, AI, data science or related fields (or equivalent practical experience).
- Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
- LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
- Writing production level code in Python and using version control systems such as Git.
- Strong written and verbal communication skills.
- Independent learner that is capable of reading and understanding research papers.
- Fluent in English and one other Southeast Asian language for the purposes of understanding how to build quality evaluations from a multilingual and multicultural perspective.
- We regret that only shortlisted candidates will be notified.
- Hiring Institution: NTU