Data Engineer

Key Skills

3D data processing, Aartificial intelligence, Azure, C# Programming, Data Engineering, Data science techniques, Database, DevOps, ETL/ELT technologies, Machine learning techniques, PowerBI, Python Programming, SQL, SQL Server

Job Description

The data engineer is the key interface in data operationalization and data analysis. The Data Engineer will also be responsible for working with key business stakeholders, IT and business experts, architecture, etc. to plan and deliver optimal analytics and data science solutions.

Additionally, data engineers will also be expected to collaborate with data scientists, data analysts and other data consumers and work on the models and algorithms developed by them in order to optimize them for data quality, security and governance and put them into production leading to potentially significant productivity gains.

Main tasks

• Design data flows, this is generally defined by data sources, movements, processing on this data and redirection to another flow or a data repository.
• Maintain, optimize as workloads evolve;
• Establish monitoring mechanisms.
• Determine the types, zones, governance and life cycles of data storage.
• The Data Engineer will be responsible for using innovative and modern tools, techniques and architectures to partially or completely automate the most common, repeatable and time-consuming data preparation and integration tasks to minimize manual and error-prone processes and improve productivity.
• The data engineer should also help renovate data management infrastructure to drive automation in data integration and management.
• The Data Engineer must work closely with data science teams and business (data) analysts to refine their data requirements for various data and analytics initiatives and their consumption needs of data.
• The data engineer should be curious and knowledgeable about new data initiatives and how to tackle them. This includes applying their data and/or domain understanding to meet new data requirements.
• It is responsible for proposing appropriate (and innovative) data ingestion, preparation, integration and operationalization techniques to optimally meet these data needs.
• The Data Engineer is responsible for training their counterparts such as Data Scientists, Data Analysts, Super Users or any data consumers regarding the logic of data flows and data preparation in general, this which will allow them to more easily integrate and consume the data they need for their own use cases.
• It is the responsibility of the data engineer to ensure that users and consumers of data use the data transmitted to them responsibly through data governance and compliance initiatives. Data engineers should help control and promote content created within the company and by data scientists using a data catalog.
• Data engineer will be considered a mix of data technician, data technology specialist, data analyst, data evangelist or data guru. This role will promote knowledge of existing data and expertise available in data analysis to business units and IT.

Education and certifications

Bachelor's degree in scientific computer science (mathematics-computer science,

computer science, software engineering, computer engineering or a related discipline).

Experience

• Five (5) years of experience as a technical lead in data processing.
• Creation of IoT solutions and real-time data processing.
• Creation of solutions with modern data analysis techniques.
• Advanced experience with Power BI.
• Experience with Microsoft Azure data processing, storage and analysis tools.
• Strong experience working with big data in building and optimizing data pipelines, storage, pipeline architectures and integrated datasets using traditional data processing technologies. data integration (ETL/ELT).
• Knowledge and experience with different languages ​​used by data scientists such as Python and R.
• Knowledge and experience with distributed data tools: MapReduce, Hadoop, Hive, Kafka.
• Knowledge and experience working in multiple deployment environments, cloud, on-premises and hybrid, containerization techniques such as Docker, Kubernetes.
• Strong experience in working and optimizing ETL processes in data preparation.
• Strong experience in working and optimizing data flows for real-time data analyses.
• Strong experience in working with Azure technologies in the following resource families: storage, databases, analysis, integration, Internet of Things in addition to the “Power” suite environment (Power BI, etc.) of Office 365.
• Strong experience working with data science teams in refining and optimizing data science and machine learning models and algorithms.
• Basic experience working with data governance/quality and data security teams.
• Strong experience supporting and working with cross-functional teams in a dynamic business environment.

• Very good knowledge and experience with SQL Server and Oracle databases.

• Very good knowledge and experience with messaging and communications technologies on Azure (for example: IoT Hub, Event Hub, Event Grid, Message Bus).
• Very good knowledge and experience with integration, business intelligence and related technologies on Azure (for example: Power BI, Azure Data Factory, Logics Apps, Azure Function, Data Gateway.
• Very good knowledge and experience with analytics and artificial intelligence technologies on Azure (for example: Power BI, Azure Synapse Analytics, Azure Machine Learning, Azure Apache Sparc, Stream Analytics)
• Very good knowledge and experience with storage technologies on Azure (e.g. example: Azure Data Lake Storage, Azure Synapse server less database, Azure Synapse dedicated database, Azure Cosmos DB, Azure SQL Database).
• Very good knowledge and experience with data governance technologies on Azure (for example: Azure Purview, Azure Monitor, Azure Logs Analytics).

General knowledge

Proficiency in spoken and written French.

• Very creative and collaborative with business and IT teams to define the business problem, refine requirements, and design and develop management deliverables of the data.
• Is a confident, energetic and technical leader with strong interpersonal skills.
• Good judgment, understands the sense of urgency and demonstrates commitment to high standards of ethics, regulatory compliance, customer service and integrity.

Specific and technical knowledge

• Mastery of the SQL language with Oracle
• Advanced mastery of the SQL language with SQL Server
• Mastery of Microsoft Power BI
• Mastery of the C# language
• Knowledge of Python
• Knowledge of Azure DevOps
• In-depth mastery of the different modes of data storage of an organization that is both cloud-based and on-site.
• In-depth mastery of different data flow techniques and architectures.
• Strong ability to design, build and manage data pipelines including data transformation, data models, schemas, metadata and workload management. The ability to work with both IT and business in data integration and analysis to support business processes.
• Very good knowledge and experience with relational (SQL Server, Oracle) and non-relational (CosmosDB and others) databases.
• Demonstrated success in working with large heterogeneous data sets to extract business value using Microsoft data preparation tools.
• Adept in agile methodologies and able to apply DevOps and increasingly DataOps principles to data pipelines to improve communication, integration, reuse and automation of data flows between data managers and consumers across an organization.

Skills:

• Artificial Intelligence
• Azure Data Factory
• Data Engineering

What you can expect from us:

Combining knowledge and doing

While technology is at the heart of our clients' digital transformation, we know that people are at the heart of business success.

When you join CGI, you become a trusted advisor, collaborating with colleagues and clients to deliver actionable ideas that produce real, lasting results. We call our employees members because they are shareholders and owners of CGI. They enjoy working and growing together to build a company that we are proud of. It's been our dream since 1976. It's brought us to where we are today – one of the largest independent information technology (IT) and management consulting firms in the world.