Data Scrapping Engineer Job in Bengaluru at Kira Studio

Responsibilities of the Candidate:

• Data Acquisition:
  • Design and implement efficient data scraping pipelines to extract data from a wide range of sources, including websites, APIs, and databases.
  • Navigate complex data structures and handle various data formats, such as JSON, XML, and CSV.
• Data Cleaning and Preprocessing:
  • Develop robust data cleaning processes to ensure data quality, consistency, and integrity.
  • Apply advanced data preprocessing techniques to handle missing values, outliers, and inconsistencies in the collected data.
• Data Transformation:
  • Transform and structure the cleaned data into formats compatible with our LLMs, such as text, numerical, and categorical features.
  • Optimize data representations to enhance the performance and accuracy of our AI models.
• Data Integration:
  • Integrate the processed data into our existing data pipelines and storage systems, ensuring seamless data flow and accessibility.
  • Collaborate with cross-functional teams to align data requirements and facilitate data-driven decision-making.
• Data Monitoring and Maintenance:
  • Continuously monitor the data scraping pipelines to ensure data freshness, reliability, and scalability.
  • Proactively identify and resolve data-related issues, such as data drift, inconsistencies, and performance bottlenecks.

Requirements:

• Educational Background:
  • A bachelor's degree in Computer Science, Data Science, or a related field. Advanced degrees are a plus.
• Data Scraping Expertise:
  • Extensive experience in web scraping techniques, including using libraries like BeautifulSoup, Scrapy, and Selenium.
  • Proficiency in handling dynamic web pages, authentication, and anti-scraping mechanisms.
• Data Preprocessing Skills:
  • Strong knowledge of data cleaning, normalization, and feature engineering techniques.
  • Familiarity with data preprocessing libraries like Pandas, NumPy, and scikit-learn.
• Programming Proficiency:
  • Excellent programming skills in Python, with experience in data manipulation and analysis.
  • Familiarity with SQL and NoSQL databases for data storage and retrieval.
• Problem-Solving Abilities:
  • Strong analytical and problem-solving skills, with the ability to tackle complex data challenges.
  • Attention to detail and a meticulous approach to ensuring data quality and integrity.
• Communication and Collaboration:
  • Excellent communication skills to effectively collaborate with cross-functional teams and stakeholders.
  • Ability to translate technical concepts and data requirements to non-technical audiences.