(Senior Data Engineer (DataOps

Job Description

About: We are looking for an experienced Senior Data Engineer to be a key senior member of our DataOps team. This is a hands-on technical role where you will be the expert responsible for designing, building, and optimizing our most complex big data systems. You will take ownership of the end-to-end data flow, from designing large-scale ETL processes to managing our real-time streaming pipelines using Kafka. You'll orchestrate these complex data pipelines using Airflow and manage data within our data lake (HDFS and Minio). You'll apply a strong DevOps mindset to our data, using automation, containerization, and robust monitoring to build resilient and trustworthy data platforms. As a senior engineer, you'll be responsible for tackling our toughest technical challenges and ensuring our systems are scalable, performant, and reliable. Responsibilities: Design, build, and maintain our large-scale data pipelines for both real-time streaming (Kafka) and batch processing (Hadoop/Spark). Develop, schedule, and maintain complex ETL workflows using Apache Airflow, ensuring tasks run reliably and efficiently. Apply DataOps principles by building CI/CD processes for data pipelines, containerizing applications with Docker, and managing deployments on Kubernetes. Proactively monitor system health and data quality using tools like Prometheus and Grafana. You'll be responsible for deep-dive troubleshooting, resolving bottlenecks, and tuning our Spark jobs and Kafka clusters for peak performance. Serve as a subject matter expert for our big data infrastructure, including our Hadoop clusters, data lake (HDFS, Minio), Kafka, and Zookeeper. Work closely with data scientists, analysts, and other engineers to understand their needs and provide them with clean, performant data. Provide technical guidance to other team members and contribute to the development of team-wide best practices. Requirements: Bachelor's or Master's degree in Computer Science/Engineering, or equivalent practical experience in the field. 2+ years of professional experience and strong proficiency in at least one of our primary languages: Python, .NET, or Java. Proven, deep experience designing, building, and optimizing complex ETL and data pipelines using tools like Spark and Hive. Hands-on mastery of Kafka for high-throughput data streaming and solid experience managing Zookeeper for coordination. Strong experience building and managing complex data pipelines using Apache Airflow. Hands-on experience with modern data lake architectures, including distributed file systems (HDFS) and S3-compatible object storage (Minio preferred). Strong comfort with the Linux command line, plus practical experience building and running software with Docker and Kubernetes. Proven experience with monitoring, including using Grafana to build charts and Prometheus to send alerts. Experience with Ansible for automating configuration management, deployment, and infrastructure provisioning is a strong plus.