Job details
Job type full-time
Full job description
The data engineer is responsible for processing structured and unstructured data, validating data quality, and developing and supporting data productsWe are looking for someone with strong hands on experience in all layers of data integration and analytics! we especially need experience in using python as an etl toolThe data engineer plays a significant role in agile planning, providing advice and guidance, and monitoring emerging technologies.
Some of the technology we use:
Python
Informatica
Sql server and mysql
Vertica
Kafka
Your role:
Design, code, test, correct and document programs and scripts using agreed standards and tools to achieve a well-engineered result
Derive an overall strategy of data management, within an established information architecture that supports the development and secure operation of existing and new information and digital services
Plan effective data storage, security, sharing and publishing within the organization
Gathers and processes raw, structured, semi-structured, and unstructured data using batch and real-time data processing frameworks
Ensures data quality and implements tools and frameworks for automating the identification of data quality issues
Collaborate with internal and external data providers on data validation providing feedback and making customized changes to data feeds and data mappings
Mentor and lead data engineers providing technical guidance and oversight
Provides ongoing support, monitoring, and maintenance of deployed products
Qualifications:
Strong experience with relational databases like sql server, mysql and verticaNosql databases experience is a plus!
Strong background in data modeling, data access, and data storage techniques
Experience with design, development, and implementation of highly scalable, high-volume software systems and components, source of truth systems for different business areas, developing and maintaining web services in an agile environment
Working experience with kafka streaming layer
Experience in spark framework on both batch and real-time data processing is a plus
Experience in big data integration & analytics is a plus
Experience in supply chain and logistics data is a plus
