The Big Data Engineering Analyst shall have strong technology skills across the big data ecosystem. S/he shall be responsible for designing and developing big data pipelines and solutions which will help engineer data for our clients. The person should be able to acquire data, understand it, visualize it, process it using big data tools like pig, spark, kafka, hive etc. The person will also be involved in data modeling/structuring using hadoop based tools.

Location : Hyderabad
Work Hours : 10AM – 7PM
Years of Experience : 2 years
Employment Type : Full-time

Responsibilities:

• Develop code for transformation logic in Spark, Pig, Hive, Sqoop etc.
• Conduct in-depth data analysis and modeling in Hive
• Ability to use Data quality frameworks
• Ability to integrate batch and real-time source systems and validation into HDFS
• Able to track data lineage in HDFS
• Learn and research new techniques for data engineering that can be applicable to the business problems we solve
• Work across multiple ongoing projects and suggest automation and optimization opportunities

Requirements:

•  Problem solver using data and deep understanding of designing/setting up big data solutions
• Minimum of 2 years’ experience using and implementing data pipelines in a hadoop environment
• Experience working with very large data sets, grouping together data, working knowledge of Hadoop, MapReduce, Pig, Spark, Hive, Sqoop, HBase
• In depth knowledge in Data Management which includes Data Modeling, Data Scrubbing, Cleaning/validation
• Understand the concepts and implementation of Data Lake and canonical forms for individual source system
• Hadoop environment experience – preferably Hadoop with YARN – and familiarity with CDH, MapR and/or HortonWorks deployments
• Experience in message queue operation and integration experience, such as Apache Kafka, RabbitMQ, or Nifi
• Experience with public, private, and hybrid cloud implementations like AWS or Azure
• Apache Spark experience – including production experience using SparkSQL and Spark Streaming or Apache Storm a strong plus – Familiarity with data visualization tools (e.g. Tableau, Spotfire etc) is good to have
• Strong interpersonal and communication skills and flexibility to work US Hours if needed
• Be flexible to learn new technologies and apply them to solve business problems

About Us:

DataBeat.io is a data and analytics services company that provides big data, analytics and operations management services to various companies globally.

Working at DataBeat.io helps you to be at the forefront of the big data and analytics ecosystem. You will work with clients who are leading companies that develop breakthrough solutions, concepts that are shaping the technology world and cutting-edge tools. Fast growing company where your performance and contribution could move you into leadership positions fairly quickly.