Senior Data Engineer

We are looking for a highly technical Data Engineer to manage complex data pipelines and infrastructure. You will be responsible for building the data bridge between the disparate databases like SQL Server, PostgreSQL etc. You will leverage Apache Airflow to orchestrate these workflows, ensuring that the data is synchronized accurately and efficiently for the real-time environments.

Key Responsibilities

· Design and maintain robust ETL/ELT pipelines to extract data from one type of database and load it into another type of database.

· Build and manage complex, mission critical workflows using Apache Airflow. You will be responsible for creating DAGs that handle dependencies, retries, and error monitoring across the data lifecycle.

· Implement secure, reliable data movement strategies to pump processed data between the main database and staging databases.

· You will own the data orchestration layer, ensuring reliable, secure, and performant data flow between disparate database environments.

· Assist in both the in-house and cloud deployments of our AI powered products and client systems, ensuring the data pipelines are optimized accordingly.

· Ensure high data integrity and low latency processing, specifically optimizing the ingestion and retrieval paths for our systems.

· Ensure all data transfers comply with strict security protocols, maintaining data privacy while moving information between external and internal environments.

Technical Requirements

· Expert-level proficiency in SQL Server, PostgreSQL, and other similar databases. You must be comfortable with advanced SQL, query optimization, and schema management across different database systems.

· Expert level knowledge of Apache Airflow, building, scheduling, and troubleshooting complex DAGs.

· Strong experience in writing scalable Python based data pipelines.

· Familiarity with Docker and Kubernetes for containerizing integration services.

· Understanding of the data lifecycle in the AI applications using Large Language Models (LLMs), and Retrieval-Augmented Generation (RAG) etc.

· Proven ability to work on integration projects where you do not have direct control over the source applications, requiring clear coordination and proactive problem solving.

Preferred Qualifications

· Experience in high compliance environments where accuracy and audit trails are paramount.

· Experience with automated data quality testing and reconciliation scripts.

· Ability to balance the technical requirements of two disparate environments.