We are looking for an experienced Senior Big Data Engineer with strong expertise in Scala, Apache Spark, and Apache Kafka to design and build large-scale, high-performance data processing systems. The ideal candidate will have hands-on experience in real-time and batch data pipelines and a deep understanding of distributed computing.

Provide consulting services on new initiatives (small to large of varying complexity). Explore new emerging technologies and how they best suit our applications Develop, code, document, and execute unit tests, system, integration, and acceptance tests using different languages and testing tools for functions of high complexity. Ensure adequate technical documentation and training. Optimize spark jobs and java applications. Architect, design, and implement solutions that meet the stakeholder’s needs Participate actively in requirements gathering, data modeling, and design sessions Prepare high-level and detailed technical specifications for the projects in accordance with PLC, security, and architecture documentation objectives Develop detailed plans and accurate estimates for the completion of build, system testing, and implementation phases of a project Develop, code, document, and execute unit tests, systems, integration and acceptance tests, and testing tools for functions of high complexity Write, test, and maintain detailed programs according to specifications given by computer software engineers and systems analysts

Knowledge and Experience Strong Scala programming experience (8+ years) Apache Spark (Core, SQL, DataFrames/Datasets, Spark Streaming or Structured Streaming) Apache Kafka (Architecture, Topic design, partitioning, consumer groups, offset management) Experience with batch and real-time data processing Strong understanding of distributed systems, concurrency, and fault tolerance

What do you need to succeed? Must Have Experience in developing and optimizing Big Data applications using Scala and Spark on Cloudera/HDP. Experience in building data pipelines Experience in developing/designing micro-service architecture. Working knowledge of Jenkins CI, Git, JIRA Ability to seek improvements to all aspects of the development process Nice to have Knowledge of Credit Risk Markets.