Chapter 1: An Overview of Data Engineering

For those new to the field of data roles, this article aims to clarify the responsibilities of a Data Engineer. I will share my experiences working for a consulting firm in this capacity.

Photo by ThisisEngineering RAEng on Unsplash

As illustrated in the spider chart below, my initial focus was primarily on Data Engineering, Database Management, and Data Wrangling. However, as I have taken on various projects, my responsibilities have shifted toward Software Engineering, Big Data, and Programming, with occasional tasks in Data Visualization.

Image Source: 7wdata.be

Currently, I am about a year into this role, having recently completed my university education. The continuous learning aspect of this job keeps it engaging, and I genuinely look forward to each workday.

According to Gartner, "Data engineers play a crucial role in constructing and managing data pipelines, while also facilitating data and analytics use cases within business processes."

In this article, I will outline my typical activities as a Data Engineer, which can vary based on the project and the company. My weekly schedule generally includes:

• Meetings
• Planning
• Building Solutions
• Writing Reports
• Presentations
• Learning

Here’s a rough breakdown of how I allocate my time across these tasks.

Meetings: What to Expect?

Yes, meetings are a part of every job. However, I don't attend meetings daily; I must also set aside time for focused, productive work.

• Client Meetings — Updates and Understanding Business Processes

Clients typically need to dedicate time for regular meetings where we discuss project updates and gain insights into their business processes. This enables us to tailor our solutions accordingly.

• Internal Team Meetings — Brainstorming, Architectural Planning, Cost Assessments, and Updates

My colleagues and I frequently engage in brainstorming sessions to explore various problem-solving approaches. We evaluate different solutions based on several factors such as cost—determining whether Azure Synapse Analytics is necessary for building a data warehouse, or if Azure SQL would suffice—skills available within the team, project timelines, and client expectations.

After conducting our research, we reconvene to finalize our plans before presenting them to the client.

Planning: A Critical Component

Planning is a crucial aspect of my role. Contrary to my initial assumptions, a significant portion of my time is devoted to strategic planning rather than just coding, such as data wrangling and modeling.

When initiating a new project, we often have a preliminary understanding of the client's needs. However, multiple solutions may exist, so I must refine our options based on the client's requirements, our internal capabilities, and the project timeline. This usually involves data architecture planning and assessing both initial and ongoing costs, like data pipeline maintenance.

Building Solutions: The Engaging Challenge

This phase is often the most intellectually stimulating and enjoyable part of the project. I prefer to dive into hands-on tasks, whether coding in Python/Spark or utilizing no-code tools like Azure Data Factory or SSIS to create data pipelines.

During this stage, my focus intensifies, and I often choose to work from home to minimize interruptions. It's also an excellent opportunity to enhance my technical skills, whether by optimizing previously used tools or learning new ones.

Writing: An Essential Yet Challenging Task

Writing is probably my least favorite part, but it remains a necessary aspect of the job. My writing typically covers:

• Findings: Information about data types, volumes, requirements, risks, and constraints.
• Proposed Solutions: This includes intended data architecture options (cloud, on-premises, or hybrid), associated costs, assumptions, and risks.
• Documentation: After the solution is implemented, I document how data flows through the system, which aids in handing over the project to client users and serves as a reference for future troubleshooting.

Presentations: Sharing Insights

Yes, Data Engineers also engage in presentations! When I have a plan, it’s essential to effectively communicate it to both clients and team members. I often utilize tools like LucidChart or yworks to create visual diagrams, as visuals enhance understanding compared to text alone.

I invest significant time preparing presentations to ensure they are visually appealing and instill confidence in both clients and colleagues regarding my work.

Learning: Continuous Growth

The learning process never stops. I continually seek to understand new technologies, concepts, or pursue certifications related to my projects. The best part? Getting paid to expand my knowledge!

When deadlines are tight, I often resort to familiar methods. However, when time permits, I explore creative solutions. Learning about cloud computing is increasingly vital, as many companies transition to cloud solutions. Data Engineers must effectively plan and propose appropriate data architectures while executing the necessary tasks.

If you're interested in cloud computing, feel free to check out my article titled "How I Learn Cloud as a Junior Data Engineer (with zero cloud knowledge)."

In Conclusion...

I invite you to share your experiences in the comments. Don't forget to follow me on Medium and subscribe to my email list; your support motivates me to keep writing!

I appreciate your encouragement! Wishing you success in all your endeavors!

Chapter 2: Insights from Real-World Experiences

What Does a Data Engineer ACTUALLY Do?

This video provides a comprehensive overview of the daily tasks and responsibilities of a Data Engineer, shedding light on the essential skills and tools utilized in the profession.

A Day In The Life Of A Data Engineer

In this video, viewers get an inside look at the daily routine of a Data Engineer, detailing the challenges and rewards of the role in a real-world setting.