Data Engineer with Spark Streaming/Flink

Job Title: Data Engineer

Location: Halifax , NS

Employment Type: (FTE )

Duration: FTE

About VLink: Started in 2006 and headquartered in Connecticut, VLink is one of the fastest growing digital technology services and consulting companies. Since its inception, our innovative team members have been solving the most complex business, and IT challenges of our global clients.

What is the role:

• Lead a group of developers building data pipelines using big data technologies (Spark, Flink, Databricks) on medium to large scale datasets for batch as well as real time streaming patterns

• Maintain existing data pipelines to support on premise clients and build the next generation ETL processes leveraging Databricks in Cloud

• Work in a creative & collaborative environment driven by agile methodologies with focus on CI/CD, Application Resiliency Standards

• Experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the team

• Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in around data ingestion & transformation onprem and in cloud

• Collaborate with product owners, business stakeholders to understand requirements, prioritize tasks and work with the scrum master/development team to meet program timelines

• Mentor team on technical aspects, participate in design/code reviews to ensure code is designed well, elegantly coded, and effectively tuned for performance.

Qualifications

• Bachelor’s Degree

• 8+ years of hands-on experience in solution, design and development of ETL applications using Spark (Pyspark preferred ) within Big Data space.

• 5+ years’ experience managing a team of developers focused on data pipelines, providing oversight on implementation, performing code reviews and managing Production rollouts

• 2+ years of experience in streaming applications using Spark Streaming / Flink .

• 2+ year’s experience building and working with rest api’s using Java or python

• Knoledge of best practices , architectural patterns, limitations within Big Data ecosystem

• 2+ years of experience working with Agile Engineering practices.

• Knowledge and experience in non-functional requirements like Security, Performance, Vulnerabilities, etc.

Nice to Have:

• Good to have some experience / knowledge on cloud-based solutions on Azure, Databricks

Employment Practices:

EEO, ADA, FMLA Compliant

VLink is an equal opportunity employer.

At VLink, we are committed to embracing diversity, multiculturalism, and inclusion. VLink does not discriminate on the basis of race, color, religion, sex, national origin, disability status, protected veteran status, or any other characteristic protected by law.

All aspects of employment including the decision to hire, promote, or discharge, will be decided on the basis of qualifications, merit, performance, and business needs.