Big Data Engineer

Employer
Mindlance
Location
Rockville, MD
Posted
Jan 25, 2022
Closes
Jan 27, 2022
Ref
330869882
Function
Engineer, IT, QA Engineer
Hours
Full Time
Position Description:The Big Data Engineers on AWS will be responsible for analyzing requirements, prototyping data analysis solutions (primarily in Hive, Hadoop, python), collecting, preparing data for modeling and productionizing the data preparation pipelines on AWS. Candidates need to have strong capabilities in large data warehouses using relational and/or Hadoop based systems., UNIX scripting, as well as database skills ( Postgres). The complexity driving this project is the domain understanding and volume of financial data in the petabytes of disparate data sets on an integrated platform providing interactive analytics to the users. Required Skills and Responsibilities:Data Analysis and data profiling. Understand logical, physical and conceptual data models.Excellent understanding of concepts and hands-on experience on Big Data platforms, Hadoop, Hive, Spark on AWSMust have extensive experience with data preparation pipeline implementations, data storage and distribution on AWS with a security mindsetOptimization and tuning of data structures and queries on Big Data platforms on AWSA compelling track record of building large scale systems utilizing Big Data TechnologiesStrong Programming experience Python and/or Java or Scala and building user defined functionsExperience with Data Analytics tools like Databricks/Dataiku/Domino for data wrangling to ideate analyticsUnix / Shell scripting, strong SQL, including analytic functions.AWS RDS Postgres database SQL skillsExperience with AWS DevOps functionality with security mindset: Jenkins, Console, Services, Roles, Security groups, AMI refreshes, etc.Providing support to maintain, optimize, troubleshoot, and configure the Hadoop environment and configurations needs to execute the data pipelinesPreferred Skills:Understanding of statistics and machine learningExperience developing Machine learning modelsExperience with NoSQL (HBase, Redshift, Cassandra, MongoDB)Experience with Graph databaseAWS RDS skills on AuroraEducation:Bachelor degree in computer science, statistics, or related fieldBig Data certification is a plus5+ Years in Big data engineering in AWS CloudGood Communication SkillsConstant, Motivated and Quick Learner