Senior Data Engineer @ RemoDevs
(práce v Polsku)

Senior Data Engineer @ RemoDevs

Polsko

• 3+ years of Python development experience, including Pandas

• 5+ years writing complex SQL queries with RDBMSes.

• 5+ years of Experience with developing and deploying ETL pipelines using Airflow, Prefect, or similar tools.

• Experience with cloud-based data warehouses in environments such as RDS, Redshift, or Snowflake.

• Experience with data warehouse design: OLTP, OLAP, Dimensions, and Facts.

• Experience with Cloud-based data architectures, messaging, and analytics.

Pluses: Experience with

• Docker

• Kubernetes

• CI/CD automation

• AWS lambdas/step functions

• Data partitioning

• Databricks

• Pyspark

• Cloud certifications

About:

We are seeking a highly motivated and self-driven data engineer for our growing data team who is able to work and deliver independently and as a team. In this role, you will play a crucial part in designing, building, and maintaining our ETL infrastructure and data pipelines. This position is for a Cloud Data engineer with a background in Python, DBT, SQL, and data warehousing for enterprise-level systems.

Major Responsibilities:

• Design, develop, and deploy Python scripts and ETL processes with Prefect and Airflow to prepare data for analysis.
• Model dimensional and denormalized schemas for optimal performance reporting and discovery.
• Design AI-friendly DB schemas and ontologies.
• Architect cloud ops solutions for data topologies.
• Transform and migrate data with Python, DBT, and Pandas.
• Work with event-based/streaming technologies for real-time ETL.
• Ingest and transform structured, semi-structured, and unstructured data.
• Optimize ETL jobs for performance and scalability to handle big data workloads.
• Monitor and troubleshoot ETL jobs to identify and resolve issues or bottlenecks.
• Implement best practices for data management, security, and governance with Prefect, DBT, and Pandas.
• Write SQL queries, program stored procedures, and reverse engineer existing data pipelines.
• Perform code reviews to ensure fit to requirements, optimal execution pattern,s and adherence to established standards.
• Assist with automated release management and CI/CD processes.
• Validate and cleanse data and handle error conditions gracefully.

Skills

• 3+ years of Python development experience, including Pandas
• 5+ years writing complex SQL queries with RDBMSes.
• 5+ years of Experience with developing and deploying ETL pipelines using Airflow, Prefect, or similar tools.
• Experience with cloud-based data warehouses in environments such as RDS, Redshift, or Snowflake.
• Experience with data warehouse design: OLTP, OLAP, Dimensions, and Facts.
• Experience with Cloud-based data architectures, messaging, and analytics.

Pluses: Experience with

• Docker
• Kubernetes
• CI/CD automation
• AWS lambdas/step functions
• Data partitioning
• Databricks
• Pyspark
• Cloud certifications

,[Design, develop, and deploy Python scripts and ETL processes with Prefect and Airflow to prepare data for analysis., Model dimensional and denormalized schemas for optimal performance reporting and discovery., Design AI-friendly DB schemas and ontologies., Architect cloud ops solutions for data topologies., Transform and migrate data with Python, DBT, and Pandas., Work with event-based/streaming technologies for real-time ETL., Ingest and transform structured, semi-structured, and unstructured data., Optimize ETL jobs for performance and scalability to handle big data workloads., Monitor and troubleshoot ETL jobs to identify and resolve issues or bottlenecks., Implement best practices for data management, security, and governance with Prefect, DBT, and Pandas., Write SQL queries, program stored procedures, and reverse engineer existing data pipelines., Perform code reviews to ensure fit to requirements, optimal execution pattern,s and adherence to established standards., Assist with automated release management and CI/CD processes., Validate and cleanse data and handle error conditions gracefully.] Requirements: SQL, Python, ETL

Kategorie

data

Odpovědět na inzerát