Agile essentials

Introduction

This is the first article in a series about Agile in the area of Data Science and Artificial Intelligence. It is necessary to explain the essentials first, before moving to more specific concepts. Especially that  Agile has many different interpretations, definitions and explanations and some of them even contradict each other.

If I were to describe what “Agile” means to my grandmother, one of the possible explanations would be: it is building valuable things adaptively and (usually) iteratively by respecting Agile philosophy, principles and values. This is  why I want to explain what Agile philosophy, principles and values mean, and why they are so important.

This article is written from the perspective of Research and Development / Proof of concept projects in the area of Data Science and Artificial Intelligence. I'll be focusing primarily on the perspective of a service provider.

Agile Manifesto and agile philosophy

To understand what Agile actually is, the most essential step consists of correctly interpreting the Agile Manifesto.

The Manifesto was  originally applied to software development, and it uses some specific terms that are listed below. In the world of Data Science, Artificial Intelligence, Machine Learning and Research and Development  the manifesto remains valid, but we just need to remember to interpret the terms below in a more generic way:

• software - this may also be a product, a solution, a method, a Machine Learning model or any result of the R & D process, that is, any outcome which we expect from the project;
• developer - this does not only mean Software Developers, but applies to any person who contributes to the development or research.

The Manifesto makes four statements, which define the ground rules for everything else. You must not interpret the rules dogmatically because the rules are intended to only highlight where the priority is in each particular case. This means that, where two important factors are in potential conflict, then the statements help you to decide which factor provides more value than the other factor.

• Individuals and interactions over process and tools - assuming that a team of highly motivated, empowered and talented people are working on a project, then they understand the project's aim and can constantly interact in an open, honest and respectful way. By focusing on individuals and interactions, the team can move forward both much faster and more and more in the direction that brings maximum value than would be possible if the team focuses mostly on following "robust" processes or using the "right" tools. This is not to say that no processes or particular tools have a value, but that any processes or tools must support the team in the more valuable process of creating value by facilitating interactions between individuals.

• Working software over comprehensive documentation - Given that our goal is to create a new method or model, and we are not sure if we will succeed, it makes much more sense to first get the method up and running. Starting by documenting it, and thereby pushing actual implementation to a later time, may be a pure waste of time and money. Again, this is not to say that no documentation has any value. Of course, we fully document our research and any outcomes on exactly the level required by our internal standards, our customer's requirements and common sense. We need to be sure that other teams can benefit from our research and discoveries (both successes and failures!), and that it will be possible for others to use the result of our research in the future.

• Customer collaboration over contract negotiation - No R&D project would succeed without gaining a deep understanding of the customer's context, the reason why the project was funded and expectations about any outcomes. And this is true right from Day One to the very end of any project. The key takeaway is that context, rationale, understanding and expectations all evolve during the course of a project, as we as service provider and our customer gain more knowledge and experience of the scope of work. It is neither efficient nor effective to focus on discussions about whether or not an item of work had been previously included in the "old" scope of work. Of course, both parties are businesses and contracts need to be respected, but collaboration in this case means being open to changes or shaping contracts in a more open way. This is the best mindset to ensure that neither party miss out on any important discovery, opportunity or threat when executing the project.

• Response to change over following a plan - As many people say: “No plan survives its first contact with the enemy”. Sure, we should put time and effort into planning, brainstorming and analysing alternatives, but we need to assume that the plan can (and probably will) change over time. It is always more important to inspect and adapt so as to continually act on current, measurable reality, which means neither sticking rigidly to the "plan"as written, nor taking actions based purely on guesswork.
  In my opinion, this statement is the most often misunderstood statement. Agile is sometimes criticised because some people interpret this statement as a suggestion to act without any plan at all. I strongly believe that it does not say more than “do not blindly follow the plan when reality around you suggests doing something differently”.

Agile Principles

The Agile Manifesto signatories also defined 12 principles which explain in detail which practices are most helpful in creating software in accordance with the Agile philosophy. When summarising and explaining these principles, I usually aggregate them into three major groups: Customer, Team and Practices. Each group focuses on a different aspect of Agile.

Customer

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software - Here, the focus is on demonstrating tangible results (even really small or medium size results) as early as possible, but with continuous updates. If we do not have a good and frequent feedback loop from our customer, then the longer that development work goes on (without feedback), the more likely it is that the results of development work will differ from what our customer expects. It is much more effective and efficient for both the customer and service provider to be constantly aligning changes in the software to the real needs of the project. Likewise, in many cases, it is impossible to define all the important elements of a project at the start. Inspection, which may lead to adaptation, would not happen without early and continuous delivery.
2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage - We not only respond positively to change, but genuinely welcome it. Changes in the face of projects which include a high degree of  uncertainty give us the opportunity to further adapt, and to find the most relevant solution.
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale - This is an extension to the “early and continuous delivery of valuable software” principle, but makes it clear that continuous delivery must take place fairly frequently. Of course, it is important to not get carried away and have full-scale inspections every day because, in that case, no actual work will be done! However, in principle, more inspections mean more opportunities to adapt and therefore to reduce waste. Putting effort into going in the wrong direction really doesn't make sense for anyone!
4. Working software is the primary measure of progress - This is an extension of: “Working software over comprehensive documentation”. When your goal is to have working software, a product, a method or a  model, you must measure your progress against that goal, and not against any other metrics, for example, the number of documents/ lines of code/ experiments conducted / ML trainings done.

Team

1. Business people and developers must work together daily throughout the project - If we want to deliver results frequently and continuously, and if we want our progress to be measured primarily in terms of working software, product, method, or model, then we must have instant (at least daily) access to business people so that developers can reach out for information or decisions.
2. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done - This is one of the most important principles. Following all the other Agile principles will make no difference at all if the development team is not highly motivated and skilled. When such a team fully understands the aim of the project that they are working on, are empowered to take decisions, have the necessary tools and are genuinely interested in the project, then they need little more. Certainly, there is no need for a deep hierarchy, onerous administrative tasks or micro-management. There are many possible reasons why a project is interesting to the members of the development team. Some people are driven by constantly learning new things, some need to do something super difficult, whilst others need to do things which make a real difference.
3. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation - After lock-down and the increase in working remotely, this principle sounds really old-fashioned. We all know that it is possible to achieve really great things when working 100% remotely and that this form of work is a reality in today's world. I personally think that it is not so much about being in the same physical space or room. Slack Huddles or any other similar tool now provide a really good substitute. We just need to remember about the limitations of remote communication, and probably pay even more attention to the other Agile principles. Bear in mind that forcing everyone to go back to working in the office cannot, in itself, replace feelings of mutual trust, engagement, empowerment and focus. It can, however, undermine them seriously.
4. The best architectures, requirements, and designs emerge from self-organizing teams - A self-organizing team is best placed to take decisions about how to implement software, create a model  and so on, as long as the team takes into full account customer feedback.

Practices

1. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely - As stated above, every successful project requires both developers and the business side to take an active part in early, continuous and frequent inspections followed by introducing any required  adaptations. High frequency also requires a constant, sustainable pace, otherwise it will not be possible for everyone to inspect and adapt at the same time.
2. Continuous attention to technical excellence and good design enhances agility -  Early, continuous and frequent inspections with adaptations require technical excellence. Experience shows that Agile does not have a chance to work in environments where technical excellence is not truly valued.
3. Simplicity - the art of maximizing the amount of work not done - is essential - Each product and project has an endless list of possible things to do, improvements to implement, experiments to conduct, etc. And, in most cases, we operate under time, budget, people and resource constraints. The key to prioritising is to maximise the amount of work that is not done. This means saying “no” to a lot of things to have the time, and to use ALL the energy available for the things that really matter.
4. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly - I strongly believe that a minimum viable Agile implementation must contain at least one thing, a retrospective. Sooner or later, every motivated and empowered team will implement Agile's principles and practices (at least those which are the most valuable) by carrying out regular retrospectives, that is, by reflecting as a team on how the team can work more effectively and efficiently.

Scrum Values

This is an article about Agile, and not about Scrum. I hope that I have made that clear because these two terms and concepts are not the same. However, the Scrum Values that were introduced in the 2016 revision of the Scrum Guide are a great example of how to define and maintain values. It is very important when it comes to teams working successfully to create results that meet all project requirements.

Scrum Values are:

• Commitment
• Focus
• Openness
• Respect
• Courage

I would love to elaborate more on Scrum Values, but this is an article for a blog about AI and Data Science, so first I need to receive some more data - feedback from you, the reader  :-). Please let me know what you think of the first article on Agile by either sending comments on Social Media or sending me direct messages on LinkedIn (https://www.linkedin.com/in/rurarz/).  And please also let me know if you would like me to write more about Scrum Values and about Agile in Data Science / AI / Research and Development.