Our Client Currently looking for Data Engineer (Snowflake to BigQuery Migration)
What you will do
Transformation migration — port the Snowflake transformation layer (Streams + Tasks CDC, stored procedures, dynamic tables) to BigQuery — primarily incremental models. SQL translation — translate Snowflake SQL and script-based jobs into dbt models and macros, using the BigQuery Migration Service for the bulk translation plus manual fixes for what does not auto-translate.
Re-architecture (as required) — re-architect constructs with no direct BigQuery equivalent (Streams, dynamic tables, zero-copy clone, JavaScript stored procedures, usage-metadata jobs) into native BigQuery patterns or Cloud Run jobs.
Ingestion rebuild — move Kafka ingestion to the BigQuery Sink connector on the existing Kubernetes footprint; replace the current managed ELT tool with open-source (Airbyte OSS etc) / managed CDC; switch BI and event sources to native BigQuery destinations.
BigQuery foundation & security — design datasets, regions (including data-residency boundaries), partitioning and clustering; implement IAM, row-level access policies, and column-level controls / authorized views.
Orchestration (TBD) — build Cloud Composer (Airflow) DAGs — or Cloud Scheduler + Workflows — to replace legacy cron-based scheduling, with dependencies, retries, backfills, and alerting.
Historical data migration — run the one-time historical backfill using unload-to-GCS loads and the BigQuery Data Transfer Service, applying the right partition/cluster design as data lands.
Validation & cutover — run BigQuery and Snowflake in parallel, reconcile results, repoint downstream consumers, and execute the freeze / final-delta / cutover.
Required skills
These capabilities most directly determine whether the project succeeds. A strong candidate is genuinely hands-on across both the transformation and platform sides.
BigQuery (expert)
Deep, hands-on BigQuery: Standard SQL, partitioning & clustering design, IAM, row-level access policies, policy tags / column masking, authorized views, on-demand vs. slot reservations (Editions / autoscaling), Storage Write API, load jobs, and the BigQuery Migration Service + Data Transfer Service.
Snowflake (Intermediate)
Practical experience with the Snowflake internals being migrated away from: Streams, Tasks, Snowpipe, stored procedures, dynamic tables, zero-copy clone, RBAC, and row-access policies.
SQL dialect translation
Fluent translation between Snowflake and BigQuery SQL, including semi-structured / VARIANT ↔ JSON/STRUCT handling, and the judgment to know what will not auto-translate.
Data ingestion / CDC
Kafka Connect (BigQuery Sink / Storage Write API), plus at least one of Airbyte, Datastream, or comparable open-source / managed CDC.
Orchestration (TBD)
Cloud Composer / Apache Airflow (DAG design, retries, backfills) — or Cloud Scheduler + Workflows — replacing legacy cron-based scheduling.
Python
Solid Python for data engineering: porting connector-based batch jobs and building Cloud Run jobs for non-SQL logic.
Are you interested in this position?
Apply by clicking on the “Apply Now” button below!
#AlbionarcJobs#FintechJobs
#AsiaJobs#MiddleEastCareers
#TechTalent#FintechRecruitment
#FinanceOpportunities#
