Senior Data Engineer (Client Ops) - X Delivery

Key Skills

Apache Cassandra, Apache Hive, Apache Pig, AWS, Azure, Azure CosmosDB, Big Data Technology, Data Analytics, Data Engineering, Docker, Effective communication skills, ETL frameworks, Github, Google Cloud Platform (GCP), HBase, Kubernetes-K8s, Lambda, Machine learning techniques, MongoDB, MySQL, NoSQL, Snowflake, SparkML

Job Description

As a part of BCG’s X DELIVERY Operate team you will work closely with BCG Clients, Consulting teams and other stakeholders on a diverse range of advanced analytics topics. You will have the opportunity to leverage analytical methodologies to deliver value to BCG's Consulting (case) teams and Practice Areas (domain) through providing analytics subject matter expertise, and accelerated execution support.

You will collaborate with case teams to understand complex data pipelines and architecture to support analytic solutions serving client needs. You will provide technical support through deeper understanding of relevant data analytics solutions and processes to build high quality and efficient analytic solutions. You will be responsible to ensure smooth operations of the data pipelines and work with the client data team (when necessary) to ensure data quality control.

YOU'RE EXCELLENT AT

• Develop and implement comprehensive observability mechanisms for data pipelines, ensuring visibility into data flows, transformations, and quality metrics.
• Create custom monitoring solutions tailored to the unique needs of data processing workflows.
• Evaluate, enhance, and optimize existing monitoring frameworks to ensure real-time tracking of data pipeline health, performance, and data quality.
• Collaborate with cross-functional teams to integrate monitoring tools seamlessly into the data ecosystem.
• Lead efforts in proactive incident management, identifying potential issues before they impact data processing.
• Develop and implement alerting strategies to notify teams of potential anomalies, bottlenecks, or deviations from expected data behaviours.
• Implement automation mechanisms for self-healing and resolution of common incidents, reducing manual intervention and improving system reliability.
• Conduct in-depth root cause analysis for incidents, working closely with data engineering and operations teams to identify and address underlying issues.
• Implement corrective and preventive measures to minimize the recurrence of incidents.
• Comfortable working with different Operating Systems - Linux, MacOS, Windows
• Knows Infrastructure as a Code (IaaC) - Terraform, Biceps templates, ARM templates, Pulumi, Azure DevOps, etc.
• Experience working with Continuous Integration (CI) - GitHub Actions, CodeBuild, GitLab, Jupyter Notebooks, etc.
• Comfortable working with Containerization - Docker, Kubernetes.
• Building Monitoring and Logging Pipelines around ETL frameworks - DataDog, Prometheus, Grafana, CloudWatch, etc.
• Worked with Data Orchestration tools like Airflow, Azure Data Factory, etc.
• Experienced in using and working with online documentations like Atlassian Confluence, GitHub Repositories.
• Prior experience working in Agile methodology with ticket management tools like JIRA, ServiceNow, Rally.
• Handled impact assessment of mission critical projects within agreed SLA using CI/CD.
• Must enable in creating automated tests for each build and overall reduce the number of errors in each build.

YOU’RE GOOD AT:

• Experience in Data warehousing and ETL projects.
• Comfortable working with unified Data Analytics platforms like Databricks, Snowflake, Azure Synapse Analytics etc.
• Hands on with writing and building optimized Spark applications on Python or Scala (Preferably Python)
• Database knowledge - SQL (MYSQL, SQL Server, Oracle), NoSQL (Cosmos DB, PostgreSQL)
• MLOps
• Fine tuning, Optimizing, and deploying AI Models.

Functional Skills:

• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Enhancing/optimizing and maintaining ETL process(s) across on-premise and cloud architectures
• Monitoring performance and advising any necessary infrastructure changes

Communicating with confidence and ease:

• You will be a clear and confident communicator, able to deliver messages in a concise manner with strong and effective written and verbal communication.

Thinking Analytically:

• You should be strong in analytical solutioning with hands on experience in advanced analytics delivery, through the entire life cycle of analytics. Strong analytics skills with the ability to develop and codify knowledge and provide analytical advice where required.

What You'll Bring (Experience & Qualifications)

YOU BRING (EXPERIENCE & QUALIFICATIONS)

• Bachelor's / Master's degree in computer science engineering/technology
• At least 4+ years within relevant domain of Data Engineering across industries and work experience providing analytics solutions in a commercial setting
• Consulting experience will be considered a plus
• Proficient understanding of distributed computing principles
• Management of Spark clusters, with all included services – various implementations of Spark preferred
• Ability to solve ongoing issues with operating the cluster and optimize for efficiency
• Understanding of prevalent cloud ecosystems and its associated services AWS, Azure, Google Cloud, IBM Cloud. Expertise in at least one.
• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
• Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O
• Good understanding of Lambda Architecture, along with its advantages and drawbacks

You will work with the case team and/or client technical and border X Delivery Teams.

BCG X DELIVERY combines innovative skills in computer science, artificial intelligence, statistics, and machine learning with deep industry expertise. The BCG X DELIVERY OPERATE team is comprised of data engineers, data scientists and business consultants who specialize in the use of advanced analytics to get high-impact business results. Our teams own the full analytics value-chain end to end: framing the business problem, building the data, designing innovative algorithms, creating scale through designing tools and apps, and training colleagues and clients in new solutions. Here at BCG X DELIVERY OPERATE, you’ll have the chance to work with clients in every BCG region and every industry area. We are also a core member of a rapidly growing Digital enterprise at BCG – a constellation of teams focused on driving practical results for BCG clients by applying leading edge analytics approaches, data, and technology