Peter Friday’s Post

With demand for data engineering skills high, companies need to focus on retaining high-performing data engineers by providing career growth opportunities and paths to advancement. A clear career ladder with multiple levels - e.g. associate data engineer, data engineer, senior data engineer, staff data engineer - helps engineers understand what is expected to progress to the next level. You should also evaluate when it’s appropriate to split ladders when different roles within the data team begin to crystallize. For example, a data engineer who is spending a lot of their time debugging data issues and working on reliability may not be “performing” at a promotion-worthy level according to the data engineering ladder rubric, but that doesn’t mean that their work isn’t valuable. What you can do instead is to create a separate ladder/role, called data reliability engineer.

Do you really know which IT career path is the most suitable for you? I created a skill set radar chart of different IT roles as below. In general, software engineers focus more on hard skills (programming ability, algorithm ability), data scientists focus on modeling, data analysts focus on business understanding, and data engineers focus on database construction and application Programming ability: Software Engineer > Data Scientist > Data Engineer > Data Analyst

What does a data scientist do? A data scientist can use a range of different techniques, tools, and technologies as part of the data science process. Based on the problem, they pick the best combinations for faster and more accurate results. A data scientist’s role and day-to-day work vary depending on the size and requirements of the organization. While they typically follow the data science process, the details may vary. In larger data science teams, a data scientist may work with other analysts, engineers, machine learning experts, and statisticians to ensure the data science process is followed end-to-end and business goals are achieved.  However, in smaller teams, a data scientist may wear several hats. Based on experience, skills, and educational background, they may perform multiple roles or overlapping roles. In this case, their daily responsibilities might include engineering, analysis, and machine learning along with core data science methodologies. 

Why aren’t there many entry-level data engineering roles? There’s a few reasons for this: - data engineers produce data that is relied on by many people. Relying on a zero years of experience person to do that is a little riskier. - data engineering requires a unique blend of communication skills and technical skills, like data science, that makes it harder for juniors to ramp up effectively. - many companies only need one or two data engineers. So there’s no mentorship/growth path for juniors at these companies. Therefore they prefer to hire senior engineers. Why else do you think there aren’t many Junior data engineering positions? #dataengineering

📊 Move Over Data Scientists: Data Engineers Take Center Stage After a decade of data science repeatedly topping “hottest jobs” lists, the tech industry is witnessing the meteoric rise of another important role... 🔍 Data Engineers Are Taking the Lead: New research reveals that the once overshadowed "data engineering" role, is now as popular as the "data scientist" role. Data engineers play a pivotal role in managing data pipelines and ensuring data quality, making them the driving force behind seamless data operations. 💡 Guardians of Data Integrity: Survey results indicate that data engineers are now at the frontline of defense when it comes to tackling data-related challenges. The responsibility of data quality has been passed on to them- with smaller contributions from data analysts. 💻 The New Software Engineers? Data engineers are embracing responsibilities similar their to software engineer counterparts. They're not just managing data; they're nurturing a data product that demands meticulous process management, maintenance, and review. What are your thoughts on the future of data science and engineering roles? 👇 Tell us in the comments!

I'm looking for data engineers, scientists, or analysts to share their stories about career growth, projects, or other topics in the data world you believe need to be heard in my newsletter with +60k readers. The data world is far too large for anyone to truly understand it all, so I'd love to share your unique experiences, lessons, and stories! Why share your story? • Reach a large, relevant audience (+60k data professionals) - happy to link to what you want in the intro blurb • Build your brand and professional footprint - online writing & LinkedIn are a kind of resume • Get better at writing - I will work closely with you on editing until it's as good as possible • Help other scale What makes a good career growth story: • Actionable Learnings - should leave readers with concrete learnings they can apply • Quality Writing - concise & engaging • Unique and transparent Storytelling - helps with engagement and how helpful the content is for others If there is someone whose story you've been curious to learn about, tag them in the comments! If you're interested in sharing your story, please fill out my short form here: https://lnkd.in/gvJ-VNmx Also, thank you Ryan Peterman for the idea!

Oh, this is so 2000! Should one learn about linear programming? Sure. How to model declaratively? Yes. But everyone and their dog can do that. Leaving it at that is like recommending to a data scientist that they master the use of sklearn. Of course they should know some tools well, but if their knowledge ends there, then you might as well hire an LLM instead of a person. Seriously, a good decision scientist should, first and foremost, know how to develop and implement efficient algorithms. This means understanding computational complexity and data structures and building solid software engineering skills. 80% of what a hands-on decision scientist actually does is software engineering plus the development of heuristics for the many problems that are not solved well by MIP or SAT or CP. (Thanks to tools like Seeker, the heuristic part is becoming much less.) Moreover, the time is long past when it was okay to consider only deterministic problems. Most real-world problem instances are based on estimated data. When FedEX plans their tours for tomorrow, they are actually not facing a vehicle routing problem but a stochastic vehicle routing problem, because who knows what the traffic times will be exactly? When Walmart builds out their network of warehouses, they are not facing a capacitated network design problem but a stochastic network design problem. They only have estimates of future demands, and it takes 12–24 months to get a new warehouse operational. Thanks to the advent of new tools like InsideOpt Seeker, we can now also use the declarative programming paradigm for complex stochastic and non-convex problems. When you hire a decision scientist, make sure they understand: 1. Software Engineering 2. Data Structures and Complexity 3. Decision-making under Uncertainty and how to build ML2OR pipelines 4. Modelling in Seeker, MIP, CP/SAT.

This is a comprehensive article about hard and soft skills needed for a successful data engineer. Helpful for those trying to ratchet up their data engineering career as well those looking to hire!

The critical piece to every successful data science project? Strong data engineering skills. But what are the most important skills to look for when hiring? If you're a data engineer, where should you be investing your professional development? From programming languages and database design to cloud-based platforms and business domain knowledge, these are the 8 skills to prioritize. Take a deep dive into the specifics in our recent blog: https://lnkd.in/g92ufT-Q #DataAnalytics #DataEngineering #BigData 

Are you considering becoming a data engineer? Or maybe you need to get up to speed on data engineering concepts quickly? Here is a free mini-course. You'll learn about what data engineering, the salaries you could make, key concepts and so much more. What Is Data Engineering - Why Is Data Engineering Important? 1. https://lnkd.in/gFmtavpu Data Engineering Salaries in 2023 - How Much Data Engineers Make 2. https://lnkd.in/gsjJyGvW Data Engineering Vocab 101 3. https://lnkd.in/gRQagRax What Skills Do Data Engineers Need? 4. https://lnkd.in/ephMRrFA Python Libraries You Should Know As A Data Engineer 5. https://lnkd.in/gKxq8v3T What Is A Data Pipeline And What Should You Consider When Building One 6. https://lnkd.in/ginyTQYM 7. What Is Change Data Capture https://lnkd.in/dnJRedgE Transactional Databases Vs Data Warehouses Vs Data Lakes 8. https://lnkd.in/gxpbCRPi How To Come Up With A Data Engineering Project? 9. https://lnkd.in/gZTGK2Cz The Harsh Reality of Being a Data Engineer 10. https://lnkd.in/gxFy43dN What are your favorite free resources to learn data engineering?