Our client, one of the largest Amazon Web Services (AWS) partner for data services, is looking for top talent to join their elite team of technologists to build and contribute to large-scale, innovative projects. Our client wants candidates who are eager to learn and grow with them as they work to move their market monitoring applications to modern big data platforms in the cloud where up to 130 billion market events are being processed each day. If you are a fast learner, have a sharp mathematical and technical mind and are interested in working for one of the biggest data users in the country in a fast-paced, open source environment, our client has an immediate need for a Data Engineer to coordinate changes to computer databases, test, and implement the database applying knowledge of database management systems. THE PLATFORM Our client's Data Platform is responsible for the collection, management and dissemination of data in their institution. It's an enterprise capability that services multiple business lines across the entire organization. The platform collects and processes billions of records (up to 100 billion) each day in support of their regulatory mission to protect investors and promote market integrity. They are embarking upon a large, multi-year, enterprise-wide initiative to re-platform major pieces of our existing offering, replacing legacy functionality with a set of modern, highly customer-centric products and capabilities. The ideal candidate will provide technical leadership and implementation to a team focused on delivering one or more of those modern, customer-centric products and capabilities. Data Engineer Are you passionate about technology? Do you love programming? Do you sometimes pat yourself on the back after writing a particularly elegant function? Are you an aficionado of the art of software design? Do you expect the best from yourself and those around you? Our client is looking for passionate developers who want to make a difference. If you are a rock star developer, a strong communicator, and a self-starter who can grab hold of a challenge and dive deep with it, then please join our client's delivery-focused team. Job Responsibilities Able to grasp complex business requirements Provide tech expertise and leadership for analysis, design, and modeling work Design, develop, implement, test, deploy, and support big data processing software in the AWS Cloud Establish and implement ETL and big data processing best practices to design, test, implement, and support mission critical applications Set clear expectations with engineering team and peers Provide timely and regular updates to management Ensure security and compliance requirements are met Be delivery-focused Be quality-focused Own the outcome Ship great products! Experience & Qualifications 3+ years of experience required, including some commercial/non-government work. Hands on experience designing, developing, implementing, testing, and deploying large scale ETL/Data Analytics/Java/J2EE projects. Believes in Scrum/Agile, and has deep experience delivering software when working on teams that use Scrum/Agile methodology. Strong and creative analytical and problem-solving skills. Strong communication skills. Enable long-term implementation, facilitate product vision and strategy. Required Technical Skills & Knowledge for Lead Data engineer: SPARK Hadoop, Hive, Hbase REST APIs Write complex SQL queries Performance tuning and optimization Familiarity of columnar storage formats ORC, Parquet and various compression techniques Unit testing with Junit or Scalatest Git/Maven Code Reviews Agile Testing strategy and implementation AWS Cloud experience ConsultNet is an Equal Opportunity Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, ancestry, citizenship, disability, age, military or veteran status, and other characteristics protected under federal, state and local law.