Lead Big Data Platform Engineer

Key Skills

SQL, Python Programming, R Programming, AWS, Tableau, Scala Programming

Job Description

Are you a DevOps guru who is passionate for creating solutions using cutting-edge technologies? Do you excel under pressure? Can you summarize highly complex problems so that others can help you solve them? Does everyone turn to you for answers to tough questions? Do you instrument everything, collect data about data, and look for the correlation in everything? Do you like gathering evidence to base your decisions off of, but can use your gut, intuition and experience to make quick decisions when necessary? If you smile in the face of pressure, can work independently but are also a great team player, we're looking for you!
As a Platform Architect, you will identify, evaluate and provision big data solutions, frameworks and technologies. You will help ensure integrity, security, availability, effectiveness, and efficiency of technology infrastructure, including middleware, networks and communications. You will understand the business, customer & market requirements and apply technical and functional knowledge to solve a wide array of architectural problems. You will help establish and implement frameworks and standards for our entire data organization.

Responsibilities:

• Shape, implement, and evangelize your big data architectures.
• Build high-efficiency, secure, real-time data pipelines for ingest and processing
• Ensure the architecture supports all functional and non-functional requirements
• Establish and implement frameworks and standards for the data organization
• Help identify, analyze, and resolve infrastructure problems
• Develop and execute test plans to check infrastructure technical development; recommend improvements
• Develop technical content, such as white papers and reference architectures, that can be used by our organization to assist in deploying cloud-based analytics solutions
• Capture and share best-practice knowledge amongst the Big Data and AWS solutions architect community
• Build and maintain technical trusted advisor relationships to influence key technical decision makers on platform choices
• Build and maintain solutions to gather metrics across all our platforms

Basic Qualifications:

• Bachelor's degree in Computer Science, Engineering, Information Technology, or related field or equivalent work experience
• 2 to 4 years of relevant IT work experience.
• 3+ years of python work experience
• Leadership experience in large, cross-functional teams, influencing senior-level management and key stakeholders.
• Experience developing platform architectures with one or more big data technologies (such as HDFS, HBase, Spark, NiFi, Kafka, Glue, Lake Formation, MSK, AWS CICD Tools); General understanding of the tradeoffs that come with these technologies, such as when they are appropriate to use, and what it takes to secure and operationalize them.
• Experience designing new or enhancing existing architecture frameworks and implementing them in a cooperative setting.
• Strong conceptual understanding of security at rest and in transit.
• Passionate about delivering platform technologies and features in a team environment even if the team size is 1.
• Demonstrate the ability to quickly ramp on new technologies.
• Troubleshooting skills, ability to determine impacts, ability to resolve complex issues, and ability to exercise sound judgment and initiative in stressful situations.
• Lead and collaborate with Studio DevOps engineers to ensure the platform meet their needs
• Strong oral and written communication and interpersonal skills.

Preferred Qualifications:

• Experience with data analytic tools including AWS Glue, EMR, Sagemaker, Hive, Spark, HBase, Zeppelin, etc. General understanding of the tradeoffs that come with these technologies, such as when they are appropriate to use, and what it takes to secure and operationalize them.
• Experience with caching and queueing technologies – Kafka/Kinesis, Rabbitmq/SQS, Redis/Memcache etc.
• Experience with REST APIs, Java and Python (or another, similar, interpreted language)
• Experience working in a Linux environment such as Red Hat, CentOS or Ubuntu
• Production experience with Dev Ops Tools like: Terraform, Chef, Anisle, Docke, Jenkins, AWS CICD tools
• Familiarity with Machine Learning and Artificial Intelligence Approaches
• Familiarity with Data Analysis or Statistics, and some data wrangling (programmatically extracting databases, transforming data, modeling it in readable form, etc)
• Experience using with Code Management/CICD tools like GITHub, Jenkins is useful
• Good understanding of Lambda Architecture, along with its advantages and drawbacks
• Experience with integration of data from multiple data sources
• Experience with MlOps
• AWS/Big Data Certifications