About the role
Job Description
On behalf of our public sector client, Affinity is seeking a Senior AWS Data Engineer/Architect to provide technical leadership and guidance migrating an on-premises data warehouse to a data lake (data lakehouse) architecture on AWS.
Responsibilities
Developing data pipeline and orchestration design patterns using AWS tools including DMS, S3, AWS Glue, Managed Workflows for Apache Airflow and dbt Develop optimal patterns for data pipeline and data quality monitoring and alerting Conversion of SQL Server and SSIS data warehouse data models and code to dbt, Airflow and Redshift data models and code Ensure security and data access best practices (IAM roles, encryption, credential management) and adherence to TSBC IT security process and TSBC data governance policy Collaborate with IT and Data Management team to design a CI/CD deployment system for managing development and production AWS data lake accounts
Education And Experience
Degree in computer science, computer engineering, system engineering, big data and statistics or equivalent work experience Typically requires over 6 years data engineering, applied data science, or software engineering experience Experience with large-scale data migrations or cloud modernization projects
Knowledge And Skills
Capability to architect highly scalable distributed systems, using AWS tools. Familiarity with DevOps best practices and AI code assistance tools Understands data pipeline technologies and tools and have experience building high-performance data systems. Extensive knowledge in SQL, python and dbt Expert knowledge of data modeling and understanding of different data structures and their benefits and limitations under particular use cases. Experience using AWS technologies including S3, Redshift, AWS Glue, CloudFormation, CloudWatch Experience with integration of data from multiple data sources and multiple styles of APIs Experience working with SQL databases such as Oracle, Sql Server, MySQL, Redshift Knowledge of various ETL techniques and frameworks
Competencies
Architecting secure and scalable systems Strong communication skills for cross-team collaboration Ability to mentor and guide other team members
Affinity Earn
Know someone who’s great for this, or any of our open roles? Earn up to $4,000/year for each successful referral through Affinity Earn. You can also earn up to $50,000 for helping us find new clients. Learn about our referral program at https://affinity-group.ca/earn/ or browse our jobs & follow us at https://www.linkedin.com/company/affinity-staffing/jobs/
About Affinity
Affinity Group is a technology and business consulting and services company. We believe in creating long term relationships between clients and consultants that foster a mutually beneficial partnership. Affinity is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided on the basis of qualifications, merit and business need.
For more information on Affinity, please visit www.affinity-group.ca
Job Number: 13664
Similar Jobs
About the role
Job Description
On behalf of our public sector client, Affinity is seeking a Senior AWS Data Engineer/Architect to provide technical leadership and guidance migrating an on-premises data warehouse to a data lake (data lakehouse) architecture on AWS.
Responsibilities
Developing data pipeline and orchestration design patterns using AWS tools including DMS, S3, AWS Glue, Managed Workflows for Apache Airflow and dbt Develop optimal patterns for data pipeline and data quality monitoring and alerting Conversion of SQL Server and SSIS data warehouse data models and code to dbt, Airflow and Redshift data models and code Ensure security and data access best practices (IAM roles, encryption, credential management) and adherence to TSBC IT security process and TSBC data governance policy Collaborate with IT and Data Management team to design a CI/CD deployment system for managing development and production AWS data lake accounts
Education And Experience
Degree in computer science, computer engineering, system engineering, big data and statistics or equivalent work experience Typically requires over 6 years data engineering, applied data science, or software engineering experience Experience with large-scale data migrations or cloud modernization projects
Knowledge And Skills
Capability to architect highly scalable distributed systems, using AWS tools. Familiarity with DevOps best practices and AI code assistance tools Understands data pipeline technologies and tools and have experience building high-performance data systems. Extensive knowledge in SQL, python and dbt Expert knowledge of data modeling and understanding of different data structures and their benefits and limitations under particular use cases. Experience using AWS technologies including S3, Redshift, AWS Glue, CloudFormation, CloudWatch Experience with integration of data from multiple data sources and multiple styles of APIs Experience working with SQL databases such as Oracle, Sql Server, MySQL, Redshift Knowledge of various ETL techniques and frameworks
Competencies
Architecting secure and scalable systems Strong communication skills for cross-team collaboration Ability to mentor and guide other team members
Affinity Earn
Know someone who’s great for this, or any of our open roles? Earn up to $4,000/year for each successful referral through Affinity Earn. You can also earn up to $50,000 for helping us find new clients. Learn about our referral program at https://affinity-group.ca/earn/ or browse our jobs & follow us at https://www.linkedin.com/company/affinity-staffing/jobs/
About Affinity
Affinity Group is a technology and business consulting and services company. We believe in creating long term relationships between clients and consultants that foster a mutually beneficial partnership. Affinity is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided on the basis of qualifications, merit and business need.
For more information on Affinity, please visit www.affinity-group.ca
Job Number: 13664