The Agile Toolkit Podcast
I always say that DevOps in one sense is really an extension of agile principles out to everybody on the ship.
-Jeffery Payne
Bob Payne chats with Jeffery Payne, Founder of Coveros, at Lean+Agile DC 2018. The Payne Cousins (not really) chat DevOps and tips for pairing developers and testers. The discussion covers moving toward a generalized specialist model, testers showing up like a demolition crew, and the true meaning of pairing.
TRANSCRIPT
Bob Payne: [00:00:02] Hi I’m your host and technical idiot, Bob Payne. Just struggling with the equipment there for a little bit, making making the the the big newbie mistake of hitting play instead of record. So I’m here at Lean + Agile DC 2018 and I’m here with Jeff Payne of Coveros.
Jeffery Payne: [00:00:25] Your cousin right.
Bob Payne: [00:00:26] Yeah. Cousin
Jeffery Payne: [00:00:27] Yeah.
Bob Payne: [00:00:27] Yep. So Jeff what what are you talking about here today since I am out here in the hall and not not in the talks.
Jeffery Payne: [00:00:38] Yes I’m talking about dev test pairing. Okay so trying to get developers and testers to work together better. We find that that’s one of the biggest issues we see on teams when it comes from engineering perspective.
Bob Payne: [00:00:52] Yeah I mean I think the early agilists were a lot of XP teams that sort of did away with testers because everybody was considered to be a tester. I think it was also sort of a chemistry of the particular group of folks that were on that first team. And you had folks like Elizabeth Hendrickson, Lisa Crispin. A lot of folks sort of brought testing back into the Agil fold. Yeah what do you think the biggest problems you see with testing and agile teams or trying to get testers and coders to pair?
Jeffery Payne: [00:01:31] Yeah I think obviously one of the biggest problems is that they historically haven’t worked well together. They’re kind of on different sides of the fence as a check and a balance in some organizations right.
Jeffery Payne: [00:01:42] And and a lot of organizations even they prefer that their testers not even talk to their developers they want them to be independent speak because they think it’s kind of like an editor if if you haven’t seen it and then you review it another set of eyes you’re not you know you’re not influenced by the development. The other sort of clean room actually that’s the traditional approach. Of course it’s always been very late lifecycle and very manual right. None of those things work well on edge. All right.
Bob Payne: [00:02:11] Well none of those things actually work well in life. It’s not just an agile thing.
Jeffery Payne: [00:02:16] So you know how do we change that?
Bob Payne: [00:02:17] Shoot, it’s not secure and doesn’t scale. I’m glad we have 12 hours to fix this before production.
Jeffery Payne: [00:02:22] Yeah, Exactly. Here you go have it done by tonight. So yeah. And so what we try to help teams fix that.
Bob Payne: [00:02:30] Yeah.
Jeffery Payne: [00:02:30] Address those issues.
Bob Payne: [00:02:33] What are you what do you think has been most beneficial recently for helping you in that in that quest of getting folks to pair together.
Jeffery Payne: [00:02:42] Well we have some techniques and approaches that we like to use to try to get them to work together and also learn from each other because you know if you’re moving toward a generalized specialist model on your teams we like that model. Yup and you want collective code ownership and you want a whole team quality all these you know motherhood and apple pie concepts that we espouse too. You’ve got to get everybody productive during the entire Sprint or whatever you’re doing story development or whatever. And the only way you can do that is that people start learning from each other and cross fertilize. Historically you know I was a developer developers aren’t great testers for a number of reasons.
Jeffery Payne: [00:03:20] Just you know out of the gate they’re not very good testers and testers oftentimes particularly if they are manual testers they don’t have a very strong technical background they don’t know code they can’t write automation right. Those two things together don’t work very well. So we’ve found that by pairing Dev and test they can help learn from each other and become stronger teammates and collectively on the code better.
Bob Payne: [00:03:43] Now do you find that tools like cucumber or other. I don’t know if you’re running into teams using fitness but are early on fitness is one of those tools cucumber most recently specked flow help bridge that gap so that testers can blow out those scenarios a little more directly after the fixtures are done or even before the fixtures are done.
Jeffery Payne: [00:04:09] Definitely. Yeah I mean the the BDD oriented.
Bob Payne: [00:04:12] Yeah
Jeffery Payne: [00:04:12] Cucumber with Gherkin, kind of natural language approach is a great way to start moving particularly manual testers toward understanding how to automate without having to dive right in and start like you know trying to write good maintainable selenium scripts for instance or whatever. I mean it’s hard to write maintainable any kind of scripts.
Bob Payne: [00:04:33] Write would be better then record -those are a nightmare to maintain
Jeffery Payne: [00:04:39] No doubt, or record any test is a bad idea because that’s how they’re sold often so.
Bob Payne: [00:04:44] Right.
Jeffery Payne: [00:04:45] That’s how you know people think you’re supposed to use those tools. We definitely like those kinds of tools that we think they help move a a tester toward being more capable of providing automation support.
Bob Payne: [00:04:57] What sort of behavioral, I mean, You mentioned the word pairing. What does that mean when you say that because I see a lot of I see a lot of misuse of the word. I’m assuming you’re not but the mis use of the word pairing
Jeffery Payne: [00:05:09] I Might be, who knows. Maybe you’ll tell me i’m wrong, Bob.
Bob Payne: [00:05:11] And TDD, I see a lot of people misusing or not really understanding TDD. That’s most common but
Jeffery Payne: [00:05:17] Yes. Yeah. So I mean to me I’m basing it off of the definition of pair programming. Go you know getting two people together to work together collectively on some task. When you talk in dev test you’re really either talking about those two people working together on code almost pair programming and one of our techniques is to use a dev test to pair program yet which is a little different right because one of them maybe doesn’t actually know how to write code. So what does that mean. Right. In pairing. The other thing we use it for is to review each others tests. So if you’re going to ask developers to do a unit test you want them to learn how to write good unit test meaning think through not just happy path but you know the errors and boundary conditions exceptions and all those kinds of things they usually inherently don’t know how to do that a tester can by working with them help them understand how to do that better. Second if you’re asking your testers even if it’s manual to create tasks for integration for system for you know kind of the combinations of things across use cases and your business flows they often don’t they often won’t load the design. Well enough particularly if they haven’t been involved in those activities they should be but often aren’t.
Jeffery Payne: [00:06:34] Yeah and the developer can help them think through and understand how does this software all pieced together to meet the you know the flow that we’re looking for in our application and how users use it so they can help each other from a testing perspective we found.
Bob Payne: [00:06:47] And one of the other things that I think a lot of a lot of testers can help with as well is what are the business rules like oh yeah if you’re doing an under UI test which quite often happens in the developers domain you know what are the what are those conditions you know the happy path is easy and that’s usually where developers go because they know the happy path works but they don’t necessarily test those boundary conditions as or that or the business rules right if I had a whole bunch of J rules or other stuff I wouldn’t test that through the UI right.
Jeffery Payne: [00:07:26] Yeah no doubt.
Bob Payne: [00:07:28] Yeah.
Jeffery Payne: [00:07:28] And to your point about a happy path. The other thing we’ve seen is not every developer’s like this but you know a lot of developers consider what they’re building to be a work of art. Right. They’re like Michelangelo creating the Sistine Chapel in their in their mind. Yeah and they’re all about creating this beautiful incredible thing that’s going to last forever and just people are going to you and all over it even if it’s just their peers.
Bob Payne: [00:07:49] Yep.
Jeffery Payne: [00:07:50] And then the tester shows up testers like a demolition crew.
Bob Payne: [00:07:52] Yeah
Jeffery Payne: [00:07:53] Right. They’re trying to poke holes in it and figure out what’s wrong with it and it’s kind of like calling your baby ugly. If you’re asked to test your own code because you know you might have every intention but in the back of your mind you might be thinking I don’t really want my Sistine Chapel to have problems in it or look bad and changing that mindset is part of getting Dev and tests to work together to understand the best way if you want to build something great is to find any issues as fast as you can see eradicate them. That’s really about what it looks like when it gets delivered yet not what it looks like. You know while you’re making the sausage right.
Bob Payne: [00:08:27] Yeah. I find a lot of people use the term Pairing and they’re really talking about working together on just acceptance criteria or something like that that’s necessary but not sufficient. I think that deeper level of the deeper you can go in interaction and an understanding the better off your team is clearly
Jeffery Payne: [00:08:52] We’ve had good success getting developers involved in doing some exploratory testing as well.
Bob Payne: [00:08:57] Sure.
Bob Payne: [00:08:57] You know a lot of times testers get together and do you know session based exploratory testing across stories or whatever. What about the idea of just getting the Dev and test together for a story they’re working on and having an exploratory testing session where they work together and explore the product and talk about it and identify bugs. Again that gets the developers a little bit more comfortable doing testing and knowing what to look for thinking critically about the app. And of course it helps the tester better understand the app because if they’re they don’t understand something about what they’re testing they’ve got the developer right there they can ask Hey what was this supposed to do or how was this supposed to work.
Jeffery Payne: [00:09:32] Now I think the story is maybe vague did we really build the right thing or are we testing it properly. That dialogue’s very helpful.
Bob Payne: [00:09:38] Yeah. What else is exciting in your your world right now
Jeffery Payne: [00:09:42] Nothing
Bob Payne: [00:09:42] No?
Jeffery Payne: [00:09:42] Nothing. Well as you know we do a lot of DevOps work.
Bob Payne: [00:09:47] Yeah sure. Yeah it’s the new edge issue.
Jeffery Payne: [00:09:51] Yeah exactly.
Bob Payne: [00:09:52] Yeah. Actually you know we were going to be talking later with some folks talking about sort of you know in many ways Agile is sort of hit a ceiling and I’m hoping this will open up gaps where we can get to real real agility and real cause. All too often it’s seen as a fix for the delivery team not right. Not a systemic change that can build better value faster.
Jeffery Payne: [00:10:23] Yeah and I totally agree. I mean I think one of the mistakes that the founding fathers of Agile made is you know they were all about collaboration getting everybody to work together. But they forgot a key piece of the lifecycle which was delivery and release and production and production oriented.
Bob Payne: [00:10:41] And actually intake in the business side.
Jeffery Payne: [00:10:45] Exactly. You know it’s funny this group that was all about collaboration and getting everybody on the same page left all these people out right by mistake. Obviously they were creating it as they went so I understand. So I always say that dev ops in one sense is really an extension of agile principles out to everybody on the ship you know involved in the software delivery process in the full lifecycle software.
Bob Payne: [00:11:09] Yeah and agile and dev ops are both the you know great grandchildren of lean which was all about that base that whole process right.
Jeffery Payne: [00:11:21] Yes.
Bob Payne: [00:11:22] Yeah. You know this reintroduction of the concept of value streams and value team and stuff – It’s like back to the future.
Jeffery Payne: [00:11:32] I’m sure you’ve studied up on the history you know all the way back through Demming and you know all the way back to you know statistical process control and even beyond that I mean it’s clearly standing on the shoulders of giants like everything we do. It’s amazing how many people don’t understand that or take the time to find that out or understand.
Bob Payne: [00:11:50] And the idea that that actually Devops, Yeah there’s a whole bunch of cool technical stuff going on, but it’s about closing the loop to be able to learn. And my favorite Demming quote about that was learning is not compulsory. Neither is survival.
Jeffery Payne: [00:12:07] Here’s some great pithy comments. You know we’re in this. You know there was an article I read that compared it to an extinct extinction level event you know where we’ve got you know Internet of Things and big data and and organizations being able competitors being able to go extraordinarily fast and learn and reintegrate that learning. The end for the many organizations that will that will mean their doom and not going to pretend that DevIps or Agile is any silver bullet in allowing them to survive. But I just know the status quo is not the strategy I would take.
Jeffery Payne: [00:12:56] Yeah. Well yeah I mean if software is really eating the world which I think we would agree it is then you’d better figure out how to optimize how you build deploy deliver and feedback information fast because otherwise you are going to be out of business. Yeah eventually.
Bob Payne: [00:13:15] So what’s happening over at your company Coveros.
Jeffery Payne: [00:13:18] Coveros, yes! So We’re busy little busy little beavers helping people with Agile and devops just trying to get it right. And when we focus more on the engineering aspects of both of those things but I often get asked to you know help pull teams together and figure out how to make it all work.
Bob Payne: [00:13:36] Yup
Jeffery Payne: [00:13:36] But we really like the the engineering aspects as I call it you know Automation doesn’t solve all your problems right. I always say a tool with a fool is still a fool. Right. So you have to know what you’re doing and you have to collaborate work together. But automation can help and as long as you take that philosophy you can leverage test automation and then you see ICD Automation and other types of automation effectively. If your view is that automation somehow solves all your problems it’s a magic bullet right. And it all you know takes culture or you know magically make it all work then you’re going to be really upset right because it’s not going to work so that’s kind of what we’re focusing on.
Bob Payne: [00:14:16] Magical thinking is a strategy has also proved to be not the greatest..
Jeffery Payne: [00:14:21] Hope. Hope is not a strategy. One of my favorite sales books and I use that a lot. Yeah everybody says it’s not grounded in reality I would say just remember hope is not a strategy.
Bob Payne: [00:14:30] Yeah yeah yeah exactly. Well great. What what’s exciting you coming up. What do you see coming down the pike in the next. You know what I know prediction is tough especially about the future.
Jeffery Payne: [00:14:47] Yeah the future because if I could I wouldn’t be in this business or I’d be retired long ago.
Bob Payne: [00:14:52] Yeah exactly.
Jeffery Payne: [00:14:53] Well I am I’m excited about.
Bob Payne: [00:14:54] What do you actually see that’s here.
Jeffery Payne: [00:14:56] Well I was very skeptical at first but I am a little bit excited about what’s going on with integration of A.I. into Dev and test. There are some interesting things going on around how you can leverage AI capabilities to build better tests for your applications. Do testing in a better way. So what actually look interesting. Are they going to scale or are they going to work right we’ve been talking about AI and you know robots take over the world forever which of course is not going to happen.
Bob Payne: [00:15:30] The joke is AI is the next big thing and always will be.
Jeffery Payne: [00:15:34] Yeah it’s very true because you and I we probably are same same relevant age and we were coming up through the techie ranks. AI got really hot for a while.
Bob Payne: [00:15:43] I was in the computer architectures for AI master’s program so
Jeffery Payne: [00:15:47] Yeah! It was hot hot hot, VR – the first VR systems came out and everyone was talking about these awesome things and how we were going to live in alternative worlds. And all that stuff and of course then like a lot of things that it didn’t really happen and kind of disappeared but it bubbled along and now it’s kind of popped its head up again.
Bob Payne: [00:16:05] And so I’m not familiar with the uses that folks have been you know the application in the testing area what is the is this especially for like I mean if you look at big data you don’t know what’s in there necessarily. So you don’t know know what to test for like where’s the where’s the current application of.
Jeffery Payne: [00:16:33] Well there’s a couple. One is of course everybody’s trying to figure out how to even test AI-based systems whether it’s B.I or or whatever it is you know how do we know the answers right. Right. That’s the age old problem in the systems is you know how do you actually know whether what you got is true or not because you kind of need that testing right. But the other side of it which which we’re more focused on is other ways to build better approaches to automation that analyze the product analyze what you’re building and not completely write the scripts for you but take a step toward providing you test automation capabilities and scripting without having to do that on yourself. There are some new tools out on the market really small startup stuff that’s trying to take a different look completely at how we create automated tests and how we maybe do that automatically. Yeah and the software is a really hard problem.
Bob Payne: [00:17:37] Yeah I can I can I can extraordinarily easily imagine doing like really good deep progression by looking at sort of big data. Big data user behavior. You know we’ve kind of done that to heatmap. You know we really need this piece to be bulletproof because of risk. I’m sure there are folks out there that are mapping the the usage. But I could also imagine very easily just observe what folks are doing and and learn from that. I mean it’s the way to go.
[00:18:20] You know Al p haGo learned how to play go and meet you know with you know the vast majority of the learning in a system like that is not from the ruleset right. The initial ruleset it’s actually playing another copy of itself Veriga and and and going through the database of previous games which for go is actually harder than chess but apparently it never played go. But yeah it sounds easy it’s go go. How hard could it be.
Jeffery Payne: [00:18:53] Just go right. Just go. So what. What’s up with that. Just sounds a lot harder. four letters. Just kidding but
Bob Payne: [00:19:04] It is four letters is twice as many.
Jeffery Payne: [00:19:08] That’s fine. We’re just having a great time here right.
Bob Payne: [00:19:15] Yeah.
Jeffery Payne: [00:19:16] So yeah that’s what what I’m interested in that is just you know trying to take the dev ops concept to the next level. You mentioned round trip. Right. Which is you know a lot of people spent their early instantiations of automation just focusing on how do I get code you know from a change in their production as fast as possible with quality and stability as well. You have to balance those. But now I think the more sophisticated companies are saying OK well it’s great to get there but what happens if you get there and something’s wrong. What’s the fastest roundtrip approach to fixing that and addressing that. Is it rolling back. Is it going roundtrip and coming through. You know because the the other thing that’s and people say why is that important if we’re not the kind of company like you know say and Amazon who’s pushing code out every 11 seconds right.
Jeffery Payne: [00:20:05] Why do you need that we need that for security and stability and performance service level agreements. I mean if you’ve got a problem in production it cost you money every minute every second it’s down or that there is a risk out there with a security perspective you’ve got to figure out how to round trip change as fast as possible. And that’s an exciting area I think has been under looked at. You know it hasn’t really been the focal point of house is now I think starting to be. I mean this it is really ironic that the safest way to go is to be able to go fast.
Bob Payne: [00:20:41] I mean
Jeffery Payne: [00:20:41] Oh yeah.
Bob Payne: [00:20:43] I mean the level you know I remember those days where company would have to fail over to their dark side and emphasis on fail right because it would be days hours just downtime before they could you know oh shoot the Oracle logs didn’t replicate. Yeah. Or whatever. And in like extreme programming and some of the techniques there early on they were seen as risky and the real practice in the same way that drove up seems risky. If you’re doing it the way you and I think they should be doing it. It’s actually the least risky way of behaving
Jeffery Payne: [00:21:37] Right. Yeah it is. Yeah of course there are some apps that you’d like to be able to push into production quickly but maybe can’t ever fail. So you know you can’t you know this you know the Amazon concept of roll something out there doesn’t really work.
Jeffery Payne: [00:21:53] Roll it back and tune it roll back out and you’re kind of using your customer to test test and give you some time to live life critical for that. So there are certain ones that you need. You know just double down on your assurance process during your dev ops capability because it can’t fail on the field.. For a lot of others you know.
Bob Payne: [00:22:10] Well one of them one of the things that I’ve been thinking about because I quite often talked about high quality and the key is and someone came up to me and said what you’re really looking for is expected quality. So and he had an example that was was a big oil and gas company and one of the things that they said is your labels are too good. He’s like What do you mean said we need the labels to start to deteriorate immediately said we do not want to see someone pouring a lubricant into a cooking pan in Africa or in some other area where this is unfortunately a common practice with a brand spanking new company logo on the outside of that thing said is we actually need that to deteriorate. And I start to think about that because as you mentioned you know some fine you know I may not have critical transactions push something around or find a roll it back. You know that might be fine. You know canary roll out on Spotify right fine right.
Jeffery Payne: [00:23:39] Yep.
Bob Payne: [00:23:40] Canary Roll out on the firmware and in a medical device maybe not so fun.
Jeffery Payne: [00:23:46] Yeah
Bob Payne: [00:23:46] Because the Canary dies
Jeffery Payne: [00:23:48] And it’s a big Canary.
Bob Payne: [00:23:48] . Oh yeah yeah
Jeffery Payne: [00:23:55] Yeah. No. No doubt
Bob Payne: [00:23:56] Yeah.
Jeffery Payne: [00:23:56] And that and that is something that I think people misunderstand about dev ops.
[00:24:00] You know when I speak about DevOps at conferences I always well attended everybody’s interested in the topic because it’s hot
Bob Payne: [00:24:06] Right.
Jeffery Payne: [00:24:07] People have this perception and unfortunately senior management does that Dev ops means speed and speed alone. The goal no fast can I push things into production.
Bob Payne: [00:24:17] But imagine a life critical system where you could have test automation every single infrastructure. Code line Change is auditable in and you can get that level of safety. We used to put two you know extraordinary manual testing.
Jeffery Payne: [00:24:44] Yes it was very expensive.
Bob Payne: [00:24:45] And it’s prone to possibility of non repeatable results. Somebody makes a mistake. Somebody configurations off. And now with you know with tools that where you have immutable infrastructure you have software configured network you can actually know to some a greater degree of certainty than we were able to in the past that you have a Conformance Test system. And that adds a lot of safety.
Jeffery Payne: [00:25:24] It does and it helps with regulatory is yes right. I mean the one of the under the under represented aspects of dev ops is CM
Bob Payne: [00:25:34] Right.
Jeffery Payne: [00:25:34] Because if you’re doing it right everything you’re dragging your entire manifest of your software your test your environments your even your rollback your recovery procedures your monitoring capability. Dragging that all the way through production in a way that you know where everything came from and everything takes and ties together. And that’s what regulators want. Right.
Bob Payne: [00:26:00] Those that know they actually want safety they don’t care about the stack of documents they use sadly to hopefully inspect that you knew what you’re doing.
Jeffery Payne: [00:26:08] Want you to demonstrate that you have a process that delivers quality and they want to see that there’s relationships between the various things that you’re using to do that. And dev ops gives you all that if you do it right.
Bob Payne: [00:26:20] Yeah.
Jeffery Payne: [00:26:21] If you do it wrong it just you know throws your code down through there and everything around it is changing constantly and you’re never really going to get the speed or quality that you want.
Bob Payne: [00:26:30] Yep well great so anything you’d like to close out with Jeff for
Jeffery Payne: [00:26:36] Well just thanks for the chance to talk. I know you’ve been doing this a long time and it seems like a great podcast and we’re really enjoying the conference. Looking forward to the rest of it.
Bob Payne: [00:26:49] And if you can stand to hear me talk then they listen to some of the older ones I think
Bob Payne: [00:26:55] Definitely.
Jeffery Payne: [00:26:56] Ok cool
Bob Payne: [00:26:56] I’ll get some popcorn and listen to early one’s .. I wish you had started it maybe five years earlier than that right. I mean.
Bob Payne: [00:27:03] Yeah yeah
Jeffery Payne: [00:27:03] If you had started like right around 2000.
Bob Payne: [00:27:05] Yeah
Jeffery Payne: [00:27:05] Then
Bob Payne: [00:27:06] Yeah.
Jeffery Payne: [00:27:07] You know you would have had some interesting..
Bob Payne: [00:27:08] There’s a there was some gap years as well.
Jeffery Payne: [00:27:12] But Well thank thank you very much for having me.
Bob Payne: [00:27:14] Thanks.
