About the role
This Senior Data Engineer role at OCBC focuses on developing and maintaining advanced data lakehouse architectures using Apache Spark and Iceberg. The successful candidate will design scalable data processing and orchestration workflows within a cloud-native environment to support Asia's leading financial services partner.
BankingOnsite
Key Responsibilities
- Develop and maintain data lakehouse architectures using Apache Iceberg table format
- Design and implement high-performance data processing pipelines using Apache Spark and PySpark
- Handle complex transformation logic through advanced SQL proficiency
- Orchestrate and automate data engineering workflows using Python and Shell scripting
- Manage storage, compute, and event ingestion across major cloud platform native services
- Implement and manage open-source data catalogue and governance frameworks
- Collaborate in a team environment to deliver high-quality data solutions for technical and non-technical audiences
- Maintain detailed documentation for data systems and engineering workflows
Requirements
- Strong hands-on experience with Apache Spark and PySpark
- Practical experience with Apache Iceberg table format for data lakehouse architectures
- Experience with open-source data catalogue/governance frameworks
- Proficiency in SQL, including handling complex transformation logic
- Strong skills in Python for data engineering, orchestration, and automation
- Solid understanding of Unix/Linux shell scripting
- Experience with one major cloud platform and its native services for storage, compute, and data processing
- Familiarity with relational databases, data lakes, and data warehouse concepts
- Good understanding of cloud-native open technology stacks and distributed data systems
- Clear communication skills for both technical and non-technical audiences
- Bachelor's degree or Diploma in Computer Science, Engineering, IT, or related field is preferred
- Familiarity with Docker, Kubernetes, and containerized data workflows is a plus
- Experience with CI/CD pipelines, DevOps workflows, and version control systems such as Git is preferred