Job Description

We are looking for an experienced Data Engineering Lead (Vendor) to support our platform modernization program. The resource will work with internal teams to migrate applications from Cloudera CDH to a Kubernetes-based global data platform and ensure timely delivery of high-quality data engineering solutions.

Key Responsibilities

• Provide technical leadership for migration projects from Cloudera (Spark, Hive, Kafka, Control-M) to Kubernetes stack (Spark 3.5, DBT, Airflow, MinIO/S3, Kafka, Solace).
• Lead a small team and internal engineers to deliver project deliverables.
• Participate in design, architecture discussions, and migration planning with internal leads.
• Build and review high-performance, production-ready pipelines.
• Ensure adherence to standards, compliance, and governance requirements.
• Provide status reporting, escalations, and delivery tracking to stakeholders.
• Design and implement migration/acceleration framework to automate end to end migration.
• Continuous enhancements to the frameworks to ensure the stability, scalability and support for diverse use cases and scenarios.
• Work with various data applications to enable and support the migration process.

Required Skills

• 9-12 years of experience in data engineering and big data ecosystems.
• Strong hands-on expertise in Spark, Hive, Kafka, Solace.
• Working experience with Kubernetes deployments and containerized data workloads.
• Proficiency in Python, Scala, and/or Java.
• Experience in orchestration tools (Airflow, Control-M) and SQL transformation frameworks (DBT preferred).
• Familiarity with object stores (S3, MinIO).
• Hands on experience of data Lakehouse formats (Iceberg, Delta Lake, Hudi).
• Prior experience leading vendor or distributed teams for enterprise projects.
  
  

Engagement Expectations

• Vendor resource will be expected to ramp up quickly and start contributing to active migration projects.
• Must coordinate with internal leads, product owners, and platform teams to ensure project timelines are met.
• Should be flexible to support multiple time zones (if required).