CMMI and Agile Blog

May 3, 2010

SEMAT, Theory and Measurement

Filed under: Software Engineering Method & Theory (SEMAT) — pemcmahon @ 4:58 pm

Much has been written about the conflicts surrounding Semat and its kickoff meeting in Zurich (http://sematblog.wordpress.com/2010/04/24/some-critiques-of-the-semat-initiative/).  One of the more heated discussions occurred on the afternoon of the first day related to Theory and Software Engineering.  The SEMAT vision statement reads in part:  “the kernel shall rest on a solid rigorous theoretical basis.” 

The discussion I am referring to began when Bertrand Meyer stated:  “We don’t want just any theory.  We want a theory based on mathematics.”  Alistair Cockburn responded by raising concerns about a mathematical basis given the lack of precision in our current understanding of software engineering.  

 I personally found this discussion intriguing and after the meeting looked up the word theory in the dictionary.  The first definition I came to read:   “A coherent group of general propositions used as principles of explanation for a class of phenomenon.” 

This sounded like what I expected. But it was the second definition that gave me a better idea why some may fear a mathematical based theory.  It read:  “A proposed explanation whose status is still conjectural, in contrast to well established propositions that are regarded as reporting matters of actual fact.”   I wondered as I read it, if a valid “theory” could be just a guess?

This possibility may help explain why some fear theory and why it seems to stir up such heated debates.  If you read some of the recent views expressed on Semat by Fowler( http://martinfowler.com/bliki/Semat.html ) and Cockburn (http://alistair.cockburn.us/A+Detailed+Critique+of+the+SEMAT+Initiative) there is a sense of “been there, done that”, and “don’t want to go back and do it again.”  This is easy to understand.  It is also easy to understand why some believe that Semat will become just an academic exercise with little practical value to real software practitioners. 

I too spent a great deal of effort in the 1980’s and 90’s trying to apply the latest “theories” on software engineering in real project environments and like Cockburn and Fowler I too recall the frustration.  It is natural and right to not want to repeat the failures of our past. 

Nevertheless, in this case I don’t feel the same as Cockburn and Fowler and other critics of Semat.  Part of my reason can be found in the Semat vision statement which I have read multiple times.  If you read it carefully you will find it is not advocating the resurrection of the failed theories of the past.  Specifically it states that the results expected one year after the start include: “The identification of specific theories or theoretical areas that hold potential for Semat, backed by examples of their successful application to specific software engineering practices.”   I find a recognition in these words of what we know we don’t know which in turn can help us focus our attention on what is most important to learn next. 

Bertrand Meyer who I perceive as strongly on the side of a mathematical basis appeared to listen to Alistair’s concerns during that heated discussion on Day One in Zurich,  replying with (as best I can recall):  “We don’t want precision for precision sake.  Today weather forecasting is imprecise, but it is based on mathematics and it has proven useful.”  Bertrand also pointed out that in mathematics there exists a concept known as “sufficiently complete.”   

At this point I sensed that Alistair was listening.  He replied with (as best I can recall): “Ok.  I am learning something here.”  I also recall feeling at this point that I might be witnessing an historic moment as two great thinkers were listening and learning from each other. 

It was unfortunate that over the next two days more moments like this did not occur.  There were ups and downs as is to be expected in any serious and challenging endeavor.  I have previously written about why I felt the Scott Ambler led requirements brainstorming on Day Three was a high point of the workshop (https://paulemcmahon.wordpress.com/2010/04/12/semat-kickoff-meeting-in-zurich/), but I would now like to explain why my support for Semat has continued to steadily grow since the Zurich workshop. 

When I originally read the SEMAT position papers of the Zurich attendees a number of statements by Larry Constantine caught my eye.  One of those statements was:  “Ivar Jacobson is fond of quoting Kurt Lewin, the founder of social psychology, that nothing is so practical as a good theory.” 

When I read it I thought this statement seemed counter to my own view of theory.   I always viewed “theory” as something separate and disconnected from “proven practices.”  My own Zurich position paper was based on my own practical experience helping clients do their job, and when I wrote it I didn’t think about theory. 

If I see something that works with a client I like to share it with others.  But I have also recognized that because something works for one client it doesn’t mean it will work for another.  I am careful when making recommendations to clients letting them know the context in which I have observed certain practices succeed.  In other words, this is my way of communicating the fact that I don’t know just how universal my own experiences are. 

Finding a set of “universals” is a goal of  Semat.  Clearly this won’t be easy.  But what gives us hope is our years of collective experience.  Today we know why many software projects fail.  Larry Constantine in his position paper references the work he did with Ed Yourdon on structured design and pleads guilty to missing the human experience.  He states: “I am convinced that any theory of software engineering that does not take into account at it core the fact that software is specified by humans, to be understood by humans, to be modified and extended by humans, and ultimately be used by humans is fatally flawed.” 

The criticality of the human element is something the agile community has helped bring to our attention.  This is just one example demonstrating how our collective experience has grown and can be used in the future to help build a solid and practical theory of software engineering. 

As another example, I have shared many of my own experiences with clients in a recent book titled Integrating CMMI and Agile Development: Case Studies and Proven Techniques for Faster Performance Improvement” (available August, 2010).   

And while I still don’t know how universal my own experiences are, there does exist compelling evidence that what we know today about why some software projects succeed and others fail may be more universal than we think.  Hillel Glazer, a Certified High Maturity CMMI Lead Appraiser said after reading my book: 

“Prior to reviewing the manuscript, I had never met Paul nor collaborated with him on engagements.  For us to have such similar experiences merely provides further evidence that Pareto was right: 80% of the problems can be explained by 20% of the issues.  The cases Paul describes can be easily relatable and extrapolated to many organizations.  Even if/when his studies don’t match a reader’s experience precisely, that doesn’t mean that they are not relevant or that there aren’t lessons to be learned and applied.”

During that heated debate on Day One in Zurich Richard Soley said:  “Maybe we should stop saying mathematical and just say measurable.”    I have a Bachelors and a Masters degree in Mathematics and when Richard made this statement I was both surprised and intrigued.  I enjoyed my years in school learning mathematics, but I have often told others how I felt I would have gained far more value if my professors could have related the Theorems and “Theories” we were learning to real world situations.  As strange as it may sound in all my formal years of mathematics education I had never recognized that mathematics was a way to help us describe in a numerical way the world we live in. 

 Again I was led back to the dictionary where I found that mathematics is:  “The science that treats of the measurement, properties and relations of quantities…”.  As a result I am finding that today I look at Theory, Mathematics, Measurement and “Proven Practices” differently.  I see them all in a much more connected way and a way that can help software practitioners in the future. 

As an example, while today I still may not be thinking much about “theory” when helping clients, I now recognize that a solid “practical theory” could become a valuable aid in the future to help check out my own experiences with the broader community.  This, in turn, could help my decisions become more effective for my clients.   On a related note, Larry Constantine also said in his position paper:  “…good theory is far more than practical because it also transcends and outlasts practice.  Practices change, but sound theory abides.”  These ideas have helped me learn and grow in my own understanding of how theory can help practice. 

These ideas have also led me to look differently at the work ahead of us on the Semat Assessment Track.  Measurement is a major topic the Semat Assessment Track will be discussing over the coming months.  I am helping Watts Humphrey lead this Track where our twelve month goal is:

A set of metrics, sufficient to assess software practices, products, and people, and backed by evidence of successful application to some projects.”

This raises a number of questions.  For example, how precise and how rigorous should our metrics be?  As one point of possible discussion, in the book “How To Measure Anything” the author defines “measurement” as:  “A set of observations that reduce uncertainty where the result is expressed as a quantity.”

Today the software engineering community has amassed a plethora of valuable observations.  Part of our task in the Assessment Track could be to sort through what we know today to determine which measures show the greatest potential to help software engineering in the future.  Some proposed outcomes based on the brainstorming in Zurich that the Assessment Track will be discussing in the months ahead include:

  • Explaining why successful projects succeed and failing projects fail and develop related measures to help
  • Identifying where we need metrics based on what is useful and important to projects
  • Considering “rules of thumb” to help derive sensible measures and these rules of thumb should be based on a “practical theory

We are interested in hearing more opinions from the full software community.  If you are interested in participating on the SEMAT Assessment Track email Paul McMahon at pemcmahon@acm.org.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: