Is your Data Engineer from Mars?

Data engineers have amazing skills and a unique perspective on getting the most out of your data. But sometimes it can be hard to work with them. They speak their own language, do their own magic, and look at the world differently. They may have completed their degrees at Hogwarts (just kidding, they probably studied at a non-magical university) but they weren’t born and raised on Mars - that’s just an urban myth. (Nor, to put another urban myth to rest, are they secretly robots.) When you have mountains of data to crunch, a data engineer, aka Data Wizard, might be just what you need. 

At Inciter, we have Data Wizards - data engineers, analysts, and other data specialists - as part of our team. We understand that most nonprofits aren’t working at Facebook, Google, or Amazon, where you have thousands of these rare creatures. We also know that they are expensive and that, with limited resources, you’ll want to use them wisely. 

Some not-so-mythical reasons for working with a Data Wizard: If your data is stored in several systems and you need that data to flow from one system to another, a Data Wizard can construct a pipeline to efficiently and accurately shuttle data back and forth with nary a spreadsheet. When you and your team consistently spend too much time on manual processes to wrangle your data and share it with others, a Data Wizard can automate those processes for you, cleaning and analyzing data with specialized tools. And when you have big, vague (and maybe desperate) ideas for improving your data collection and report generation, a good Data Wizard can work with your vision and give you better data analytics capabilities than you’ve dreamed of. 

Some people (and you might be one of those people) struggle to communicate with Data Wizards, or have trouble getting the results they want, often going many rounds to get to the successful conclusion of a project. It can sometimes feel like you’re from different planets, so we’re offering some tips that, in our experience, can help.

It’s all Greek to me…. 

Communication between Data Engineers and Analysts and the rest of us non-data people is challenging because we lack a common language and mental model to talk and think about data. Often both parties use the same words to describe different things. Working with data requires precise, literal language with no assumptions. If you don’t need all the data fields currently in play, you’ll have to say so - don’t assume your Data Engineer knows your data priorities, how you ideally use your data, or the value of a particular field. You are the expert in your data’s significance, and your data engineer is an expert in data structure. It might not take a village to create the changes you envision, but your Data Engineer can’t do it alone any more than you can. The first step to understanding each other should be creating a shared “Data Dictionary” for the project, where you’ll document which words will be used, and what they will mean. Data dictionaries short-cut the process of language evolution as easily as cutting across a parking lot instead of walking around three city blocks to reach the front door of your favorite grocery store. To learn more about data dictionaries, click here. 

What kind of Data Engineer are you working with?

Is your project partner more of a data engineer or a data analyst? Engineers are more likely to focus on data infrastructure that moves and stores data like pipelines, databases, and warehouses, while analysts are more likely to use tools that crunch and deliver data. There’s lots of overlap, and both are Data Engineers in their own right. Each will lean towards solutions and infrastructure that support their speciality. 

Know what you have and what you want.

You’ll need to have a clear idea of your goals so you can share them with your Data Engineer. They will need to understand whether you just want to explore the data you’ve got, or if you are looking for specific outputs, like tables, graphs, or a data set. When you are talking with your data engineer, you want to be clear about your project constraints. For example, do you need to automate a manual process? Are you looking to process data that changes every day? Or every month? Are you working with sensitive data? Knowing the answers to these questions, as well as the more obvious constraints like timeline and budget will help you get started on the right foot with your Data Engineer. 

There’s no such thing as a stupid question!

Make a request and get clarification (don’t ever be afraid to ask clarifying questions), don’t make assumptions, push back when you hear something that sounds wrong or that you don’t understand. All of these things will help you to have a great relationship with your Data Engineer. And expect that your Data Engineer will have plenty of questions for you, too. If they don’t, bug them a little, they almost always do. If your Data Engineer isn’t the talkative type, try communicating by text or email. Like researchers and other data folks, they might be introverted. 

Get ready for an information dump.

To better understand what you need from them, your Data Engineer may ask to see forms, charts, diagrams, spreadsheets, and old reports. They’ll need to know where your data came from, and how it’s used. They’ll ask for access to your data sources and schematics. A good Data Engineer will be very nosy. 

It takes time to communicate well.

Be sure to allocate enough time for conversations with your Data Engineer - time to get to know the person, to have those conversations, to clarify. Spend the time up front, and it will save you time later.  Also remember that everyone on the team wants this to be successful. We all want to do our best work, be efficient, and achieve quality results, especially in this sector where our results impact people’s lives inside and outside the organization. 

‍

See our next post for a primer on some common terms that we use around here, and what they mean. 

Do you have some messy data problems you’d like help with? Book a meeting with Inciter’s CEO and start your first clarifying, no-such-thing-as-stupid-questions conversation with us.

‍