Sr Data Engineer

Key Skills

ChatGPT, Data Engineering, Data Modeling, Data pipelines, Data science techniques, Effective communication skills, ETL frameworks, HDFS, Looker, Machine learning techniques, NoSQL, PowerBI, PySpark, PyTorch, Scikit-learn, Statistics, Tableau, TensorFlow

Job Description

Are you ready to help us make the future?

Join a team that is elevating our strategy to drive advanced analytics and visualization tools across the Commercial enterprise. In this role, Sr Data Engineer – Commercial, you will design, implement, and manage the data architecture, systems, and processes to effectively collect, store, process and analyze high volume, high dimensional data to provide strategic insight into complex business problems. This will involve creating and maintaining scalable, efficient, and secure data pipelines, data warehouses, and data lakes. You need to ensure consistency in data quality and availability for analysis and reporting including compliance with data governance and security standards.

Duties and Responsibilities

• Lead complex data science and analytics projects in support of the commercial organization.
• Provide guidance, mentorship, and technical leadership to junior data engineers and the broader team.
• Participate in strategic decision-making processes related to data architecture, technology selection, and long-term data engineering roadmaps to align with broader organizational goals.
• Work with product owner to identify the data requirements and design/ maintain/ optimize data pipeline to ingest, transform, and load structured and unstructured data from various sources into the data warehouse or data lake.
• Design and implement data models and schemas to support analytical and reporting requirements.
• Collaborate with data scientists and analysts to define and structure data for effective analysis and reporting.
• Develop and maintain ETL (Extract, Transform, Load) processes
• Administer, optimize, and manage databases, data warehouses, and data lakes to ensure performance, reliability, and scalability.
• Enforce data governance policies, standards, and best practices to maintain data quality, privacy, and security.
• Create and maintain comprehensive documentation for data architecture, processes, and systems.
• Troubleshoot and resolve data-related problems and optimize system performance.
• Partner with IT support team on production processes, continuous improvement, and production deployments.

YOU MUST HAVE

• Five or more years of relevant experience in data engineering, ETL development, or database administration.
• Familiarity with data visualization tools (e.g., Tableau, Power BI, Looker).
• Familiarity with basic machine learning concepts and integration of machine learning models into data pipelines.
• Knowledge of agile development methodology

WE VALUE

• Expert using scripting and querying languages, such as Python, SQL, and others. Python is preferred.
• Working with at least one NoSQL system (HBase, Cassandra, MongoDB)
• Knowledge of databases, data warehouse platforms (Snowflake) and Cloud based tools (Databricks)
• Experience in using data integration tools for ETL processes.
• Knowledge of Data Modelling techniques including schema design for both rational and NoSQL databases
• Understanding of Hadoop's ecosystem (including HDFS) and Spark for processing and analyzing large-scale datasets.
• Demonstrated experience working with cutting-edge packages such as SciKit, TensorFlow, Pytorch, GPT, PySpark, Bit bucket etc.
• Ability to develop and communicate technical vision for projects and initiatives that can be understood by customers and management.
• Proven mentoring ability to drive results and technical growth in peers.
• Effective communication skills (verbal, written, and presentation) for interacting with customers and peers.
• Demonstrated application of statistics, statistical modeling, and statistical process control.