We are looking for a highly skilled Data Engineer to build and manage scalable data pipelines for our 4PL (Fourth-Party Logistics) Business Insights platform.
The ideal candidate will design and implement robust ingestion, transformation, and analytics-ready data infrastructure that powers AI-driven business insights and operational intelligence.
– This role will be responsible for building end-to-end pipelines from Existing Kafka spine + Debezium CDC + Apache Flink for streaming transformation along with supporting bulk ingestion from CSV and other flat-file sources
– Would need the candidate to have working experience with Apache Iceberg on Amazon S3
– Should be familiar with ClickHouse for building customer dashboards and Trino/Athena for historical queries
– Design, develop, and maintain scalable data pipelines for ingesting logistics and operational data into the analytics platform.
– Strong SQL skills and experience optimizing analytical queries.
– Familiarity with containerization and cloud-native deployments.
– Proficiency in Python, Scala, or Java.
Data Lake & Warehouse Management :
– Manage and optimize data flow from Kafka topics into S3-based storage layers.
– Build ETL/ELT pipelines to transform and load data into ClickHouse for high-performance analytical querying.
– Design partitioning, indexing, and schema strategies in ClickHouse for low-latency AI and BI workloads.
AI & Analytics Enablement :
– Enable AI agents and analytics applications to efficiently query ClickHouse datasets.
– Ensure data quality, consistency, and availability for downstream AI-driven insights.
– Collaborate with AI/ML teams to expose optimized datasets and semantic models.
Platform Reliability & Optimization :
– Monitor and optimize pipeline performance, storage efficiency, and query latency.
– Implement observability, alerting, and retry mechanisms for ingestion pipelines.
– Ensure scalability, fault tolerance, and data governance best practices.
Collaboration :
– Work closely with :
– Product teams
– Business Insights teams
1. AI/ML engineers
– Platform engineering teams
– Participate in architecture discussions and contribute to long-term data platform strategy.
Required Skills & Qualifications :
Technical Skills :
– Strong experience in building distributed data pipelines.
– Hands-on expertise with :
Data Engineering Concepts:
– ETL/ELT pipeline design
– Data modeling for analytics
– Data partitioning and indexing strategies
– Schema evolution and metadata management
– Monitoring and observability
Are you interested in this position?
Apply by clicking on the “Apply Now” button below!
#AlbionarcJobs#FintechJobs
#AsiaJobs#MiddleEastCareers
#TechTalent#FintechRecruitment
#FinanceOpportunities#
