Essence and Problem Solving Blog

January 3, 2021

Build a Better Scrum using Essence and the Heart of Agile

Filed under: Essence and Problem Solving — pemcmahon @ 3:46 am
Tags: ,

If you would like to read this blog, scroll down to where it says “Blog Text Starts Here.” If you would like to listen to the podcast version, click on the link and scroll down to where you can select the podcast.

Blog Text Starts Here:

Today, thousands of companies are conducting agile transformations and according to Forbes 77% of those transformations are Scrum [1].  Yet, despite its popularity, according to Jeff Sutherland, co-founder of Scrum, 58% of all Scrum implementations are late, over budget and have unhappy customers [2]. Given these sobering statistics, a natural question we must ask is:

How can we build a better Scrum increasing our chances of success?

At least part of the answer, according to Sutherland, can be found in the Essence framework [3]. Sutherland tells us he has conducted an exercise with thousands of people in hundreds of companies where the participant teams build their own Scrum using Essence Cards [4] and then the teams presents their Scrum to the other teams as in a science fair.

Sutherland says:

“These Essence Cards flush out what is Scrum, what people are doing with it and how it is working…and we now have extensive data showing which parts of Scrum are implemented well,  which are implemented poorly,  and which are implemented not at all…  And on the average we find that about a third are implemented well, a third poorly, and a third are not implemented at all…. Essence immediately flushes out where the problems are, and then the next question is how can we use Essence to fix it?” [2]  

As a simple example of how Essence can be used to fix these problems Sutherland tell us one common problem you have in a vendor/customer relationship is who is responsible for what… By laying the cards out on the table and asking what the customer is responsible for, what the vendor is responsible for, and what do they jointly have to work on together, success rate is significantly increased.

Sutherland is not the only leader in the Agile movement looking at ways to fix common problems with Agile teams.  Alistair Cockburn, one of the authors of the Agile Manifesto has observed similar issues with Scrum, stating:

Scrum is pretty good.  It has very few rules, but it doesn’t tell you if anything goes wrong how to fix it.”  [5]

Cockburn, who believes that Agile has become overly decorated, has started a new initiative referred to as the “Heart of Agile” (HOA) in which he focuses on just four words—Collaborate, Deliver, Reflect and Improve [6]  The focus of HOA seems to be on the human or emotional side of software development.  Cockburn told me:

“If you see me emphasizing the human component more and more it’s because my experiences show it affects success more than other things.” [7]

The importance of the human dimension in software development success is supported by my own experience and the experiences of others working as SEMAT [8] volunteers on the development of Essence. In one of our Essence working group meetings we heard:

“The reason Scrum works so well is because we have come a long way on the technical side; how to develop requirements, how to design, how to test.  But it is the team side where we have been lacking and where Scrum fills a big hole.” [7]

While Scrum has helped many teams become more successful in areas such as collaboration through practices including Daily Standups and Sprint Reviews, the HOA initiative seems to be poking at a deeper underlying human issue.  One place you can learn more about it is in the first five minutes of a talk Cockburn gave in 2019 at a Lean Conference in France. [9]   

He starts this talk by saying he wants to make one small correction related to Agile, based on what he heard others saying at the conference the day before.  He then says:

“…Others put their energy into processes and tools and documents and plans, whereas the ‘magic’ is actually the way people interact with each other…. The magic comes from the movement of ideas between minds…”

 He goes on to say that the value that many people leave off when they summarize Agile is the very first value of the Manifesto and it is the one he is most proud of

“Individuals and interactions over processes and tools”  [10]

What struck me listening to this talk wasn’t so much what was being said, but rather the implications to successful software development. I was certainly aware of the Agile Manifesto Values, but I hadn’t previously given a lot of thought to the consequences of not emphasizing the values when people try to communicate what is essential to Agile.  

In the Essence language, the Pattern construct is used to reflect Values.  As an example, in Scrum Essentials, the Scrum Values are captured on an Essence Pattern Card. [4] Refer to Figure 1.

Figure 1 Scrum Values Pattern Card   

Those values, as depicted on the card, are:

  • Commitment
  • Courage
  • Focus
  • Openness 
  • Respect 

Note particularly the words on the card preceding the five values:

Successful use of Scrum depends on people living the five Scrum values.

These values are clearly an essential element to the successful use of Scrum, but in my own experience– and I would bet in the experience of many of those involved in the poor Scrum implementations referred to by Sutherland that– the essential values are too often not implemented well.

Reviewing the essential values of agile caused me to reflect on the fact that values must be lived, not just written down. It also caused me to reflect on just what the heart of agile is. 

From the Heart of Agile website [6] we learn that HOA is actually a “community of ideas” and that:

“The community forming around the Heart of Agile is what is interesting these days. People exploring better ways to collaborate, to communicate, to probe, to learn, to reflect, and to have more fun at the same time as having more impact.“

It occurred to me while reading these words that the degree to which teams truly “live” their values often has a significant impact on the success of a software development effort.  This also caused me to reflect on just how HOA might work effectively with Essence and Scrum.  To aid this reflection I recalled something Ivar Jacobson, the primary visionary behind Essence, once said to me, and that is the fact that practices are not just static descriptions. They are what we do! They are alive! 

More specifically, Jacobson recently said:

”…it is high time for a new paradigm. Where practices can be explored, experimented with, adopted and adapted interactively, in the same space where teams plan and execute their work, and continuously inspect and adapt their working practices.” [11]

The value of Essence is that it provides a reference framework that gives users the freedom to compare, and mix practices and patterns to form the best solution for their specific situation. Thinking of Essence this way brings me back to Jeff Sutherland’s exercise and something we all can do, particularly anyone reading this blog who believes they can implement a better Scrum. 

We can all conduct our own ”Build a Better Scrum” exercise similar to the exercise Jeff Sutherland referred to.  If you are one of those organizations that may be implementing Scrum poorly, or failing with Scrum, I suggest you download your own set of Scrum Essential cards [4] and conduct your own Build a Better Scrum exercise.  And when conducting this exercise be sure to pay close attention to the values your team holds giving particular attention to not only the Scrum Essential Values, but also HOA patterns, such as the one that captures the first value of the Agile Manifesto. 

You can use the Heart of Agile along with Essence to help your team build a better Scrum by bringing back the focus on the real essentials—including the values– to successful software development. 

Comments are encouraged. 


[2] SIGSOFT Seminar, April, 2020,, Refer to 14, 26 and 28 minutes into presentation



[5], Refer to 38 minutes into presentation


[7], Refer to Epilogue and Chapter Five.

[8] SEMAT stands for Software Engineering Method and Theory. Refer to for more information.




January 11, 2018

Motivation for Essence in 7 Minutes

Over the next few weeks I am planning a series of short videos on selected Essence and Software Engineering topics.  Following are a few tentatively planned topics:

  • Using Essence Alphas like the Dashboard in Your Car
  • Scrum and Essence: An Example of What Practitioners Can Take with them for Life
  • Examples Demonstrating How Essence Checklists Are Different from Traditional Checklists
  • Examples Demonstrating How Essence Checklists Help Teams Become Self-Empowered
  • Examples Demonstrating How Essence Activity Spaces Can be Used in Industry
  • How Essence Can Help Teams Determine if they have a Leadership or a Management Problem
  • Examples of a Team Using Essence as a Reasoning Tool to Solve Common Problems
  • A Case Study of a Team that used Essence to Help Them Discover an Innovative Solution To a Difficult Problem
  • Examples of a Team Using Essence Competencies to Determine if They Have a Skills Problem
  • A Skill Practitioners Can Take with Them for Life: Making More Accurate Estimates using Essence

I start off this series this week with a 7 minute video motivating Essence.  As always feedback is encouraged.

If you are interested in learning more about how Scrum and Essence are partnering, checkout the latest announcement on the SEMAT website (

January 30, 2017

Avoiding two common software project troubled paths by “being agile” rather than “doing agile”

This week I want to talk to you about two common troubled project paths, and give you some simple tips to avoid both.

The first troubled path is putting too much effort into documentation. When you catch your project heading down this path stop and ask:

What is the driving need behind our documentation requirement?

Often there are two driving needs. First, your customer or senior management may be looking for some kind of assurance that the software team is doing the right things.  The second reason is to help users and maintainers once the software is deployed.

Unfortunately, traditional documentation isn’t very good at meeting either of these needs. It isn’t good at helping users and software maintainers because often it isn’t trusted to be kept up to date after delivery.  And, historically, documentation hasn’t been very useful to software teams at helping them create high quality software.

The second common trouble path is reacting to the first troubled path by declaring “we’re agile!” –and throwing out the essentials for high quality software along with the documentation.

Now, today, it is a fact that we are seeing more and more organizations trying to increase agility, but many of these same organizations are struggling to figure out how to “become agile” without jeopardizing essentials for success.  And in a moment I am going to explain more what I mean by that phrase “become agile” because I may not mean what you think.

But first, a little background. You may know that in previous blogs and publications I have talked about Essence.  Essence is a framework that captures essentials for high quality software in a simple way—and most important to today’s topic—those essentials are independent of the degree of agility you are driving for with your teams. Essence also includes a simple language to express practices on top of the essentials to give guidance to your teams.

Today I want to talk to you about three simple parts of an Essence practice and how these parts can help your teams find their right level of agility while not jeopardizing essentials for success and help you avoid both common troubled paths.

Those three parts are:

  • the purpose–or goal– of the practice;
  • the activities conducted as part of the practice;
  • the work products produced.

Activities in an Essence practice are what people actually do when carrying out a practice.  They can vary greatly in prescriptiveness. In other words, you can provide activities in your practice that just specify “what” people must do or you can be much more prescriptive specifying in detail the “how to” side of your practice activities.  Similarly, the degree of detail specified in a work product can vary greatly.

Now, let’s look at some examples from my “Upside Down” book.

In Story Two in Part I of my book you learn about a team that came up with an innovative approach to achieve the goal of their sprint review practice when they learned a key stakeholder could not attend the meeting. They actual got on an airplane and went to the client’s site to engage that customer and get their feedback.

The point I want to emphasize here is that the real goal of the sprint review practice wasn’t holding a meeting called a sprint review which is what too many teams get overly focused on when they blindly zero-in on following specified “how to” activities in highly prescriptive practices.

Stated differently, what we are seeing too often is teams falling into just “going through the motions” of prescribed activities. Other authors have referred to this as the difference between “being agile” and “doing agile”.   And, to be honest, I really didn’t understand this difference until I observed it in one of my own client organizations.

What you learn from that story I just referred to is how a team stays focused on the real goal and figures out a creative way to achieve that goal which is gaining the buy-in of key stakeholders—not holding a meeting called a sprint review.

In that same story you learn about a challenge the team faces with a tester having difficulty communicating with developers.  In this case, the team came up with an improved testing practice that included the activity of developers placing a note in the ticket to the tester.  The note in a ticket could be viewed as an example of a very lite work product—not a heavy documentthat helped the communication problem.

In this story it is important to understand that the goal of the testing practice wasn’t to create a note in a ticket.   The real goal was improving communication between developers and a tester. Placing a note in a ticket was just one possible activity that could help the team achieve the real goal of the practice.  And in this story you learn how the team understood that in certain situations if it made more sense to just get up from their workstation and go over and talk directly to the tester that was an option they could and should use.  This is a perfect example of “being agile” rather than “doing agile”.

What you learn from these stories is that where teams often get in trouble is when they start confusing the activity they conduct as part of a practice, or the work product they produce with the goal.  One simple way to help yourself is to keep asking yourself are we are getting too much into “doing agile” –  or just going through the motions of our rituals, such as sprint reviews even when the key people can’t attend?

Or are we “being agile” by keeping focused on the real goal and coming up with innovative solutions when necessary, such as finding creative ways to engage our stakeholders when they can’t make it to our sprint reviews, and getting up from our workstations and walking over to talk to a tester when their might be some risk of misunderstanding.

One caution here, if you decide to allow more choices in your team’s practice activities by making them less prescriptive– it is also highly encouraged that you provide clear limits to the activities, and coaching in these limits, so the team does not fall into chaos.

Always keep in mind that activities and work products are important parts of a practice, but they should never be confused with the goal of a practice.

When we see this confusion starting to happen it is often a sign that an organization is falling too much into the “doing agile.”

Use these simple practice tips when you are developing your practices and coaching your teams, and you might just find it easier than you think to avoid both common troubled project paths and find your right level of “being agile.”

January 26, 2017

Is Upside Down the key to Moving Forward?

Last week I promised in my next video and blog to explain two specific ways projects often get into trouble and give you some simple tips related to Essence that can help mitigate the risk of both.  I have had a busy week and so I am putting off that video and blog, but it will still be coming soon!

This week I was interviewed by Bill Fox from Container13; Exploring Forward-Thinking Workplaces

Bill asked me some very interesting questions.  One thing I loved about this interview was that I was able to tie in a strategy that may seem “upside down” that is used by the coaches of the 2016 Little League Baseball World Series Champions from Endwell, NY– which is just 6 miles from where I live.

Check out the interview at

I’ll try to get back on track soon!

December 27, 2014

Essence: What’s New and Different?

In November of this year, 2014, I gave a talk on Essence at Binghamton University to a group of Computer Science students.  You can catch the highlights of Part I of the talk on YouTube.  This is a great video to watch if you want to learn what is new and different about Essence, the new software engineering Object Management Group (OMG) standard intended specifically for software practitioners.  If you don’t have time to watch the complete Part I– which is about 44 minutes– you can find a cross-reference at the end of the video to where in the video you can find the following 23 key topics:

  1. What is Essence?
  2. The Difference Between Scrum and Essence.
  3. What Does Essence Contain?
  4. Why Did We Create the Strange Alpha Word?
  5. What is Different About Essence?
  6. The Essence CARD Deck.
  7. An Example Demonstrating How Essence is NOT Waterfall.
  8. A Question About Essence Versus Scrum.
  9. How Essence Can Power Whatever Approach Your Team is Already Using.
  10. About the Problem We Are Trying To Solve With Essence.
  11. How Does a Team Use the Essence Model?
  12. How Essence Checklists Are Different.
  13. How Do Teams Apply the Essence Checklists?
  14. On the Importance of Knowing When You Are Done.
  15. A Question on How a Team Can Fall Back.
  16. On the Order You Address States, and Decisions on Checklists that May Not Apply.
  17. An Example of a Team Deciding if a Checklist is Applicable to them.
  18. On Activity Spaces.
  19. Competencies Within Essence.
  20. How to Figure Out if You Have a Leadership or a Management Competency Issue.
  21. What if a Team Can’t Meet a Checklist Item?
  22. Why Isn’t Risk an Alpha?
  23. Why is Hardware included in the Definition of the Software System Alpha?

As always, your comments and feedback are encouraged.

November 27, 2014

Using Essence to Help Your Team Stay Fit, and Your Organization Find its Right Level of Governance

Some have raised the question:

Is the Essence Kernel ( ) just the essentials for all software endeavors? 

At a recent Essence user guide meeting this subject was discussed when Barry Myburgh raised the issue that some of the alpha state checklists might never be achieved by some teams.

He gave an example of a team with about ten developers he had been working with that never committed to when they would get the work done.  They had goals, but the team members had no idea if their goals were achievable.  He said they were incentivized to achieve the goals so as the deadline drew near the team worked hard, often late into the night, to get the job done.  When I listened to Barry describe his experience it resonated with my own experiences with many of my clients.

Examples of alpha state checklists teams might never achieve include:

Work Alpha, Under Control state:

  • Tasks are consistently completed on time and within estimates
  • Estimates are revised to reflect the team’s performance

Team Alpha, Performing state:

  • The team consistently meets its commitments
  • Wasted work, and the potential for wasted work are continuously eliminated

Some teams may never get to certain states such as Work Under Control because they don’t revise their estimates to reflect team performance. Rather they keep striving for goals that may be beyond their reach.   Ian Spence pointed out in our user guide meeting that some alpha state checklists are aspirational and are getting more at the health of an endeavor.  But when some people hear aspirational it can raise concerns.

Winifred Menezes, another Essence volunteer, pointed out one concern by asking—

What if a team is discussing their health and status and realize that they haven’t met a checklist item, wouldn’t there be a temptation to say, ”Oh, that item is only aspirational so we’re good and on track.”

Winifred raises a good point.  By calling some of the checklists aspirational are we making it easy for teams to decide these checklists are not essential and therefore require little attention?  Will this in fact dilute the value of the Essence framework as a guide to what is essential on all software endeavors? Will it cause organizations that are considering the adoption of Essence to lose confidence in Essence as an aid to help them find their right level of governance?

Toward the end of our user guide meeting Barry Myburgh after listening to the discussion said he had previously thought that on every software endeavor you needed to get through all of the alpha states because they were all essential to all software endeavors, but he now realized that was not the case.  Barry went on to draw an analogy.  He said when you use Essence it is like putting your team on a fitness program.  When a team uses Essence it brings an awareness of areas where they may have gotten out of shape, and can help motivate their team to improve in the future.

I personally like this analogy.  It reminds me of one of my Scrum clients who recently used a similar analogy by saying their team had gotten out of shape and they needed to go back to the gym.   They were doing this by giving the team some remedial training in best Scrum practices, and some additional coaching.  We heard a similar message from Cecile Peraire, another Essence volunteer, and a professor at Carnegie Mellon West where they have been conducting field studies using Essence with students.   In one of those studies a student indicated that using Essence Reflection Meetings reminded the team to think about points that otherwise would have been missed (  Similarly, when I was writing my latest book ( I thought what I was describing were fundamentals that most teams followed, but then I realized what I was actually describing was what it means to develop a high performance capability.

The concern that some teams may dismiss checklists that are viewed as aspirational is a valid one.  A simple way to answer this concern is to point out that we all need coaches at times to help remind us of our responsibilities—and to remind us when it’s time to head back to gym– if we want to stay fit.  But this simple answer may sound too glib to some, and this reaction is understandable.

At a deeper level this subject is dealing with the more fundamental issues related to trust in a team to self-manage itself versus the need for organizational governance to ensure required practices are adhered to.  Part of what the SEMAT initiative is trying to address through the Essence framework relates to helping organizations find the right balance between these two critical needs, while also helping teams stay focused on the real goal.

I would like to hear your thoughts on this subject.

Blog at