Information Technology - Lead Data Engineer

About This Role

The Lead Data Engineer is a senior software developer responsible for building custom open-source data ingestion and MLOps platforms while ensuring high data quality and automation.

Responsibilities

• Design and develop data engineering services using distributed databases, Apache Airflow, and Spark/Kafka.
• Develop MLOps production pipelines and provide technical support to data scientists for model deployment at scale.
• Design data models for mission-critical, high-volume, near-real-time/batch data with idempotent/atomic pipelines.
• Build automated, self-service data platform functions for business users.
• Operate scalable data pipelines on the Databricks platform and evaluate next-generation technologies.
• Provide technical guidance on cloud architecture and participate in project planning.

Requirements

• BS in Computer Science or related discipline (PhD/MS highly desirable).
• 5+ years of relevant industry experience.
• Advanced programming skills in Python with knowledge of data structures and algorithms.
• Experience building data pipelines using open-source ingestion platforms.
• Intermediate knowledge of AWS cloud components and best practices.
• Experience deploying data stores like S3, RedShift, ElastiCache, PostgreSQL, and EMR.
• Hands-on experience with Databricks workspace, cluster management, and AI Agent capabilities.
• Experience in modern software development (Web UI, Microservices, CI/CD, Scrum/Kanban).