Change before you have to - Jack Welch

Job Description

Data Engineer – Spark / Scala

No. of Position(s) : 1

Years of Experience: 3+ Years

Requirements:

  • Minimum 3 years of development experience in Hadoop-based applications using Apache Hadoop HDFS, Sqoop, Hive, Java and Spark, Spark SQL, Kafka, Scala, and Spark Streaming.
  • Should have a strong understanding of framework development for data ingestion, data processing, and data extraction
  • Should have strong understanding and knowledge in various file formats like orc, Avro, parquet, etc.
  • Should have experience in Spark job performance tuning
  • Experience with data stores like SQL/NoSQL databases, data warehouse, or cloud object storage.
  • Experience in developing complex SQL queries
  • Experience with any ETL /Reporting tools (ETL using scala/spark preferably Or Cognos reporting) will be a plus
  • Ability to prepare & install scripts and code promotion procedures as well as build automation
  • Good experience in building/tuning Spark pipelines in Scala/Python.
  • Good experience in writing complex Hive queries to derive business-critical insights.
  • Good Programming experience with Python or Scala or Pyspark
  • Understanding of Data Lake vs Data Warehousing concepts.
  • Good to have Experience in AWS Cloud, exposure to Lambda/EMR/Kinesis/Ec2/Redshift
Please fill up this form to get started