Big Data Developer

Abbtech Professional Resources
Falls Church, VA
Feb 15, 2021
Feb 24, 2021
Full Time
Our team is seeking a talented Big Data Developer for a ground breaking project that is helping our client cut business processing time in half. As a member of the agile team, you will be responsible for designing, implementing, and analyzing of innovative solutions to clients challenging machine learning and data mining problems. You will collaboratively work with Solution Architects to deliver solutions in alignment with the strategic vision. Each person is involved in the project s entire process, from concept through design and implementation to demonstration. Our management structure gives you the freedom to create innovative designs and products. Be responsible for analyzing the requirements and taking it all the way from conceptual phase to implementation phase. Ability to develop using Spark framework in Java or Python Hands-on experience provisioning and developing Big Data (EMR, Kinesis, AWS MSK, Athena) solutions in the AWS Cloud infrastructure Requires thorough understanding of the Big Data security best practices Design, development and review of code to include documentation for all components of the Big Data stack such as Hadoop Work with test/QA team to build automated test scripts Promotes teamwork and works with other cross-functional teams Qualifications Bachelor s or master s degree Experience working in a Big Data production cluster, open source technologies and AWS services. Experience with Data Ingest Pipelines and Data lakes Experience using visualization frameworks such as ReactJS or AngularJS Experience with DevOps, CI/CD and delivery process automation tools such as Ansible, Terraform and Jenkins 5+ years of experience designing and building data ingestion pipelines using tools such as Kafka, Flume, NIFI frameworks, Kinesis Experience with Elastic search Experience with Data Science platforms such as Alterxy, KNIME, Jupyter Hub, ML Flow 5+ years of experience with data integration from multiple sources including from NoSQL databases such as HBase Ability to obtain Public Trust Clearance 4 days remote, 1 day onsite in Falls church VA after COVID, right now its remote. ABBTECH is an EOE/M/F/D/V Employer Job Requirements: Preserve and enhance the integrity and performance of Society Insurance's Oracle databases by using tools to monitor the performance and status of production, development, and QA databases Identify process improvements and provide a strategic direction to resolution Work with programmers or manually updates systems to resolve data problems Act as technical escalation lead for customer-related streaming data issues Automating the analysis, processing, and testing of big data Implement infrastructure to support market data and trading Maintain: Custom Talend jobs to ingest, enrich and distribute data in a MapR Hadoop ecosystem File, source and record level data quality checks Logging and Error Handling Develop solutions that integrate with existing architecture Stored procedures to analyze, query and manipulate data according to defined rules and procedures Integrating the front-end elements built by co-workers into the application Stored procedures to perform database tasks such as inserts, updates and deletes/li Handle ETL, transmission, encryption, and archiving of files Develop and evaluate content for interactives, with a focus on data -- government and commercial data, APIs, and databases built from scratch -- along with accompanying text Develop, maintain and support delivering Market Data Manage, train and mentor local development team members Define, document, and develop new or maintain existing interfaces between applications, databases, and third party systems to meet business requirements Evaluating and adopting new technologies such NoSQL, NewSQL and Hadoop, as challenges are pushing us more and more into the Big Data space Understand digital architectures and information flow across multiple platforms and coordinate with IT and business partners in delivering the required capabilities Build, and own the enterprise data model, help business analysts and stakeholders derive value from data, contribute to building scalable data infrastructure Detect and correct errors in existing historical datasets, and systematize the logic for automation Ensure that all enterprise database architecture decisions provide for quality data that is readily available for consumption