Data Engineer Job in Noida at PW (Physics Wallah)

Physics Wallah is hiring for the role of Data Engineer!

Responsibilities of the Candidate:

• Develop ETL (Extract, Transform, Load) processes to move and transform data between systems.
• Integrate data from various sources, ensuring consistency and accuracy.
• Monitor system performance and identify areas for optimization.
• Align architecture with business requirements and provide solutions which fit best to solve the business problems
• Build the infrastructure required for optimal ETL pipelines from a wide variety of data sources using SQL and AWS - big data- technologies.
• Evaluate and recommend new technologies to enhance data processing capabilities.
• Work closely with data scientists, analysts, and other stakeholders to understand their data needs.
• Use programming language( python, spark) and tools efficiently to collate the data
• Identify ways to improve data reliability, efficiency and quality.
• Use data to discover tasks that can be automated.

Requirements:

• Can Develop ETL (Extract, Transform, Load) processes to move and transform data between systems.
• Have good hands-on experience in any coding language Python or Scala.
• Good understanding of Data Engineering concepts such as data warehousing, datalakes, data modeling, data quality assurance, data lifecycle management, streaming data, metadata management etc.
• Strong hold on cloud-based ecosystems(AWS preferred).
• Have an understanding of data pipeline orchestration tools such as Airflow or Azkaban (Airflow Preferable).
• Can work with spark, EMR and Kubernetes cluster.
• Can write spark jobs, shell scripts and optimized data pipelines.
• Experience with relational SQL and NoSQL databases, including Postgres, Amazon Redshift and MongoDB.
• Have an understanding of OOPS-based data platforms, and can develop data platform modules.
• Good understanding of backend APIs and their use cases.