Data Engineer Job in Bangalore at PayU

PayU is hiring for the role of Data Engineer!

Responsibilities of the Candidate:

• Build & support scalable data engineering pipelines (load, extract, clean, transform, ingest, etc.) for our Data Platform and data assets.
• Build and Deploy the Data Engineering pipelines for scale using Airflow.
• Have ownership of multiple data pipelines and Data Engineering Projects.
• Review, audit, and improve the Data Engineering pipelines.
• Document and communicate technical and functional design (HLD, LLD), timelines, and plans.
• Work closely with AI/ML, DevOps, MLE, and other engineering and product teams.
• Work with the engineering and implementation team to understand business problems and implement efficient software solutions.
• Follow configurable programming practices (config files, modular programming, parameterization, etc.)
• Support Maintenance/enhancement of deployed pipelines. This includes bug fixes and performance improvement.
• Participate in organization-building activities (cross-team training, hiring, etc.) to further the PayU mission and culture.

Requirements:

• Excellent Coding skills in Python and SQL, PySpark is a must.
• Experience of working on public clouds (AWS / Azure/ GCP).
• Experience of writing/maintaining Spark code base in a production environment.
• Big Data Experience - Relational Databases (Postgres, MySQL etc.), MongoDB, Cassandra.
• Well Versed with orchestration frameworks like Airflow, Kubeflow etc.
• Build & support scalable data engineering pipelines (ETLs etc.). This will entail extracting, loading, and transforming of ‘big data’ from a wide variety of sources, both batch & streaming, using the latest data frameworks and technologies, along with real-time monitoring dashboards and alerting.
• Essential experience of working with distributed systems software development.
• Demonstrated experience of production experience in big data infrastructure and data modeling.
• Critical experience performance optimization for both data loading and data ingestion.
• Know-how of critical tools, in a developer’s toolkit such as GitHub, Dockers etc.
• Ability to work in a fast-paced and deadline-driven environment.