As a data engineer, you will be responsible for designing, building, and maintaining our company's data infrastructure. This includes implementing data pipelines, developing data storage solutions, and ensuring data integrity and accessibility for data analysts and scientists.
Responsibilities
Your primary responsibilities will include:
- Designing and building scalable data pipelines to extract, transform, and load large amounts of data from various sources, such as relational databases, NoSQL databases, flat files, and streaming data
- Developing and implementing data models, including dimensional modeling, to support the organization's data analytics and reporting needs
- Creating and maintaining data lakes and data warehouses to store and manage structured and unstructured data
- Developing and implementing data security and privacy policies to ensure the confidentiality, integrity, and availability of data
- Collaborating with data scientists and analysts to understand their data requirements and provide appropriate data solutions
- Monitoring and optimizing the performance of the data infrastructure, including identifying and resolving bottlenecks and other issues
- Staying up-to-date with the latest trends and technologies in data engineering, and continuously learning and adapting to new tools and techniques
Qualifications
- Bachelor's or Master's degree in Computer Science, Data Science, or a related field
- 5+ years of experience in data engineering or a similar role
- Proficiency in SQL and at least one programming language such as Python or Java
- Experience with big data technologies such as Hadoop and Spark
- Strong understanding of data management principles and best practices
- Excellent problem-solving and communication skills
- Ability to work independently and as part of a team
In addition to these technical skills, you should have excellent problem-solving and communication abilities. You will be working closely with cross-functional teams to identify data needs and develop solutions, so being able to effectively communicate and collaborate is essential.