The Agile Data Engineer: Role Description (original) (raw)

by Scott W. Ambler

An agile data engineer is anyone who is actively involved with the creation and evolution of the data aspects of one or more software-based solutions. Sometimes called an agile data developer or agile DBA.

The Responsibilities of an Agile Data Engineer

The responsibilities of an agile data engineer varies based on the context of their current environment. For example, in a small organization they are like to be a “data jack of all trades”, whereas in a larger organization they may focus on application development or on data warehouse development. In other words, it depends. The responsibilities potentially include, but are not limited to:

Most importantly, agile data engineers recognize that they need to be agile.

What Agile Data Engineers Do

To understand the role of agile data engineers, let’s consider what they do in practice:

  1. Agile data engineers work very closely with application developers to implement and support data-oriented development efforts. Agile data engineers work closely with developers, typically supporting a single larger team or several smaller teams as the case may be. Agile data engineers can often be responsible for several data sources (e.g. databases, files, XML structures, and so on) or at least be co-responsible for them. For example, if two development teams access the same database, and each of them have different data engineers, then those people will need to work together to evolve that database over time. To interact effectively with developers, agile data engineers need to learn fundamental development techniques, encapsulating database accessand mapping objects to RDBs (O/R mapping).
  2. Agile data engineers must work in an evolutionary, if not agile, manner just as application developers do. The biggest potential change for Agile data engineers is that they will need to work in an iterative and incremental manner for with many teams. Modern development processes, such as Scrum, SAFe, or Kanban typically don’t provide detailed requirements up front nor do they focus on detailed models (and certainly not detailed data models up front). They will do some initial requirements and architecture envisioning and then evolve their models over time to reflect their changing understanding of the problem domain as well as the changing requirements of their stakeholders.
  3. Agile data engineers need to work with operations engineers to take advantage of and to help evolve corporate meta data, standards, and guidelines. When it comes to data, an operations engineer focuses on the physical aspects of data administration such as data source installation, maintenance, back up, restoration, upgrades, tuning/optimization, disaster recovery, and support. Agile data engineers will collaborate with operations engineers to help the team produce work which reflects your organizations overall strategy. They will also want to take advantage of any Master Data Management (MDM) efforts that are currently in place, if any. There is no reason, other than political, why this cannot happen in an agile manner.
  4. Agile data engineers need to work with enterprise architects to ensure that their work reflects the architectural vision. Agile data engineers will work with your enterprise architecture team, if one exists, to take advantage of existing infrastructure and strategies. Once again, this can also happen in an agile manner.

The Skills of an Agile Data Engineer

To be successful, agile data engineers require the following skillset:

Figure 1. The Agile Database Techniques Stack.

Choose Your WoW! 2nd Edition
This book, Choose Your WoW! A Disciplined Agile Approach to Optimizing Your Way of Working (WoW) – Second Edition, is an indispensable guide for agile coaches and practitioners. It overviews key aspects of the Disciplined Agile® (DA™) tool kit. Hundreds of organizations around the world have already benefited from DA, which is the only comprehensive tool kit available for guidance on building high-performance agile teams and optimizing your WoW. As a hybrid of the leading agile, lean, and traditional approaches, DA provides hundreds of strategies to help you make better decisions within your agile teams, balancing self-organization with the realities and constraints of your unique enterprise context.

I also maintain an agile database books page which overviews many books you will find interesting.