Shutting the stable door…

Conversation with the Head of IT Operations…

SRE Consultant: Tell me, why are developers not allowed to access the production systems?

Head of IT:  We have to practice Segregation of Duties; it’s a regulatory requirement.

SRE Consultant: And what is the purpose of this requirement?

Head of IT:  To prevent the bad actors from bringing down our production service.  We need to be online 24x7x365.

SRE Consultant: And has this ever happened?  Has a developer ever brought down the production systems?

Head of IT:  Yes, as a matter of fact.

SRE Consultant: Really?  How many times since you started trading?

Head of IT:  Well, off the top of my head, probably about 5 or 6 times.  Sometimes they were mistakes, sometimes it was disgruntled employees.

SRE Consultant: And, since you locked them out of the production systems?  How many times?

Head of IT:  Well, possibly only a couple of times, but we’re always tightening up security, putting more and more checks in place to stop code simply flowing into production from a developer’s workstation.  We’ve locked down the developers’ workstations so that they can’t use USB stick, can’t download code or software from the Internet.  As I’m sure you know, just a couple of lines of bad code can take down a working system.

SRE Consultant: I see, and that’s all it takes, is it?  A few lines of bad code can ruin your entire production service?

Head of IT:  That’s right; that’s why we have all this security.

SRE Consultant: And you preside over this system?  Your’e responsible for security?

Head of IT:  Yes, that’s right.

SRE Consultant: You’re in charge of a customer-facing production infrastructure… that can be taken down with a few lines of bad code, but for the restrictions you’ve put in place on what developers can do?  And you’ve made sure the work they produce passes under the eyes of non-developers, who check to make sure it has been through the necessary quality gates?

Head of IT:  Um… well that’s a simplistic way of…

SRE Consultant:  There’s nothing in the production system itself to detect and remove such code before it causes any real damage?  Your system doesn’t have the fault tolerance to quickly and automatically dispose of any application, service or server that isn’t behaving the way it should?  Your production system isn’t built to make the idea of a “bad actor” developer irrelevant?

Head of IT:  Well… no, but…

SRE Consultant: You can leave the building now.  Your personal effects will be sent on to your home address.

This is a fictional conversation.  To my knowledge, it has never taken place in any company.  If only.

Synology DiskStation as DHCP and DNS Server at Home

I recently decided to take advantage of the Synology DiskStation’s DNS app, for a variety of reasons.  This led me to switch to using the DiskStation for DHCP too, as I find the reservation system on the Netgear router extremely cumbersome†.

There are many articles on the Internet covering aspects of this, but they all seemed to fall short of some critical steps, so I thought I’d try and create an absolute step-by-step guide that covers everything.  So it will be rather verbose!  That said, some familiarity with what DNS is and how it works is a must… if DNS is a complete and utter mystery to you, I don’t recommend reading any further.

Read more of this post

Why did my DevOps Transformation Project fail? Part 5

This may be the final post, unless I come across a project with some new obstacle to adopting DevOps.  I think it’s the core of the problem.  Fair warning, this is going to be pithy…

For reference, these are the previous articles in the series:

Not reading enough

It is astonishing how many organisations I go into that claim to want DevOps (or any of the relatedness working practices) that have not read even the most basic literature on the subject.  This includes books by the inventors and pioneers, their blogs, other blogs from industry luminaries and leaders.

You should at the very least know who Jez Humble, David Farley, Gene Kim, Martin Fowler and Robert C Martin are.  Because, if you don’t, you’re going to look like an idiot in front of a genuine DevOps practitioner.  And these guys write well… often at a very simple level and with humour.  There’s really no excuse.

And yet, I mention “The Phoenix Project” and I’m met with blank stares.

Why do so few people read? I have a few theories (this is the pithy part):

  1. pride – “I’ve been in the business for X years… I don’t need to read a book to know how to do my job”.    This reminds me of the well-known saying:  “The fool who knows he is a fool is at least on the path to enlightenment; the fool who thinks he is wise truly is a fool”.
  2. not caring enough – “I just come here to do the work required so I can pay the bills.” Well then, you will get the working life experience you deserve and nothing more.  Enjoy.
  3. not having enough time – “I’m far to busy to stop and read anything”.  Then carry on in ignorance… extremely busy ignorance.

Reading is valuable… even if it just reiterates things that seem like common sense to you; perhaps you’ll see it from a new angle that helps you pass on the wisdom to the people around you.

Read the books and blogs by the people who lead DevOps, Agile and Continuous Delivery.  Attend their talks if you can, or watch them on YouTube.  Read the books that inspired these guys (such as “The Goal”).  Read about the Toyota Production System.  Read and digest until you have absorbed so much information that you will bore everyone you meet at a party with DevOps principles.

At that point, you will be just about ready to hire some DevOps engineers.

Agile Bridge Building

After yet another session of re-explaining the bleeding obvious to thunderous silence, I decided on a fresh approach.

Commonly, we Agile practitioners start by explaining that software is not like building bridges.  Building bridges is expensive, labour-intensive, and there’s really only one opportunity to get it right.  “Big design up front” is deemed absolutely necessary, even though it doesn’t guarantee a successful outcome.  We might even show them the hilarious (but fake) picture of a bridge that doesn’t line up where it’s supposed to meet.

Read more of this post

SonarQube and Continuous Delivery Versioning

Sonar is a very good static/dynamic source code analysis tool, which I’ve used for a number of years; however, since evolving from Continuous Integration to Continuous Delivery, there are a couple of annoying limitations I’ve had to work around.

Read more of this post

Why did my DevOps Transformation Project Fail? Part 4

The belief that DevOps is “just a fad”

This is part 4 of a series; these are the previous articles:

So, this is a common reaction.  Even in organisations where nearly everyone sees the need for change, and is prepared to pitch in and be a part of that change… you always get one or two cynics who trot out the above line.

Sadly, it can only take a single coin on the train track to derail everything (well… if that weren’t an urban myth, but you get my point).  And I’ll repeat what I said in part one:  by “DevOps”, I’m also referring to Agile, Continuous Delivery, TDD, Lean, etc.

You will commonly here this from people who actually fear the changes you’re bringing about… fear that it may, one way or another, expose them as being an utter waste of company time and money.

So, here are the facts:

  • Agile was “invented” back in 2001, but the core concepts of iterative delivery can be traced back to 1957.
  • Continuous Delivery originated from tech talks by Jez Humble and David Farley circa 2006; they published the book “Continuous Delivery” in 2010.
  • DevOps as a term was circulated from around 2008; however, the core principles were being put into practice far earlier.  DevOps is really just a term for something smart engineers figured out for themselves.

All sounding too new?  Everything in the above 3 was inspired by “Lean Manufacturing” principles, which date back to the early 1700s.

Why did my DevOps Transformation Project Fail? Part 3

Still here? Well then there’s hope for you yet!

If you’re starting here, you might like to read these first for some context:

The belief that DevOps is magic

I deliberately didn’t surround the obvious word with double quotes… because, from what I’ve witnessed, there’s frequently a sort of “cargo cult” mentality in organisations that claim to want DevOps, but who don’t actually understand what DevOps is (or how/why it will benefit them).  They think that hiring one or more DevOps practitioners, perhaps even forming a “DevOps Team” will make all their problems disappear with no effort on their part.

DevOps is not magic. It is neither a silver bullet nor a panacea. It is not a technology solution either. There is no “off the peg” or “cookie cutter” version. You either adopt the DevOps mindset, or you blindly implement “DevOps Theatre”… which very similar what Thoughtworks have called “CI Theatre”.

“DevOps Theatre” will deliver almost no benefits to your organisation. You can install GitHub, Jira, TeamCity (or any other build server), Ansible, Docker, Kubernetes… whatever, but if you don’t implement it with a DevOps mindset, it will all create more problems than it solves. All you will have achieved is the ticking of some boxes on a spreadsheet.

And, to reiterate the previous section: everybody involved needs to be involved.

A grass roots approach will only get you so far before the “Corporate Immune System” kicks in and undermines (and thwarts) your efforts.  This change needs to be driven from near the top of the organisation (if not the actual top), as well as from the bottom.