Carlos Rojas, Director of Technology and Operations at Fannie Mae, sat down with Bob Payne at Lean+Agile DC 2018 to discuss Business-Driven Agile Engineering. Carlos shares thoughts on Fannie Mae’s trailblazing Agile transformation – from prioritizing agility-supporting corporate shared services – recruiting, contracts & procurement, and facilities, to machine learning and a mantra of “Automate Everything.”
Transcript
Bob Payne: [00:00:01] Start there.
Carlos Rojas: [00:00:02] OK.
Bob Payne: [00:00:03] Hi I’m your host Bob Payne. I’m here with Carlos Rojas. And Carlos you’re talking about transformation. We were chatting you mentioned how to how to use shared services. So tell me a little bit about your your your talk here today at Lean Agile DC.
Carlos Rojas: [00:00:24] Absolutely. So couple of things to highlight right. One is you go and change a culture, Most people start with the processes. Most people start with their methodology and then they say we’re Agile. And I think that part of making it a transformational for an enterprise you know when you’re talking about 10000 people when you’re talking about 300 product teams that are developing software you know you have to take into account what we call the corporate shared services as well. You know think about the recruiting recruiting efforts. Think about the procurement. How do you write your contracts. Right. Think about your facilities you know. Do you have an open space that will be a reflection of what your mentality looks like. So you have to account for those things. Otherwise you know you end up with a great process but the culture is not impacted by what you’re doing. So that’s one of those things that I encourage companies to always think about.
Bob Payne: [00:01:16] Yeah. And we just I just recently talked with Jose from from from Fannie. And I know you guys are doing a very large transformation over there lot of DevOps. We do some some training with you guys.
Bob Payne: [00:01:34] And I’m am extraordinarily pleased to see how how big the transformation has been over it over Fannie because I worked there on a couple of early Agile projects in the early 2000s and it was it was bureaucratically painful to do it.
Carlos Rojas: [00:01:56] I know exactly wh at you’re saying. It was six checkpoints hundred twenty five deliverable is about nine months for a single release just to put a line of code in production. Now we’ve made a huge progress in terms of you know time time to market an average of about 30 days a month. We’ve got some project teams that are ready to deliver every two weeks. But again it comes down to Hey we incorporated our corporate shared services into the mix and then we think we we we instigated this concept of automation you know everything so far as he wasn’t even called them up so to begin with. How do we automate it. You know if it doesn’t make sense if it makes sense. How do we automate it and that’s when we come up with this concept of that paved road so far as the pay really was. What tools are we going to use that will help us expedite or automate some of those controls that were holding us back. Right. So great transformation. Tons of work behind the scenes but you get the point here is we haven’t finished. We’re not done yet.
Bob Payne: [00:02:58] And lean, Lean never sleeps.
Carlos Rojas: [00:03:03] Exactly exactly. So right now for example one of our key you know tunnel vision eye vision items that we’re trying to chase is agile engineering practices right.
[00:03:15] How do we turn code that is maintainable how do we create applications of 19 applications that are going to help us with you know a cloud-ready approach not necessarily how do we go about the methodology what tools that we use now but how do we make that an engineer solution that is like the ultimate driving machine for us. So we’re trying to look for that not perfection but that high performing team that will look into those proses areas that we’ll tap into the business for example that’s not a mission that’s going to help us have more quality right provisioning of environments that’s going to help us get faster at some point in time I think that we’re going to be looking at how do we do infrastructure as code so that we have a fully blown solution where you build your own servers if you deploy your own servers as development team and you’re responsible from beginning to end.
Bob Payne: [00:04:11] Hundred percent audible and nobody has the password to production exactly one per cent cut off. It’s not the engineer.
Carlos Rojas: [00:04:20] Exactly. But at least you can know who what when and you’re not having a checklist to do it. It’s all part of a creation of the solution right. Software.
Bob Payne: [00:04:30] Yep yep you know I think it is. It’s ironic and oddly a common theme and I don’t know whether it’s me or just the folks that are coming up to talk on the podcast today but almost every one of them we’ve talked a little bit about governance and how how this drives you to be more governable auditable reduces risk profile you know rather than being a risky play to be able to deploy twice a week or once every two weeks or it actually improves your risk profile considerably makes you much more audibly compliant with the controls. You say you have in place.
Carlos Rojas: [00:05:23] Yeah. You know it’s interesting you mention that when I started my journey at Fannie Mae I ran a governance organization over SDLC and I was in charge of connecting with at risk partners like legal audit you know Architectural Review Boards change control boards and all of that. And instead of fighting through the system what we did is we partner up with all of the governance groups and we said you know you can do it the way you’re doing it. But what if we do it this way and let’s just compare let’s explore let’s figure out if there’s a better way and if there is a thought that if it doesn’t work on the exploration says that this doesn’t make sense then we just go back and keep doing it the way we used to do it. I guess what the answer was always yeah that works better than the way we do it and we get more data. Thank you. So they were actually embracing it and promoting it for us.
Bob Payne: [00:06:12] Yeah. I don’t know if you’re familiar with Mark Schwartz from Citizenship and Immigration Service. He’s sort of a firebrand CIO and he said when he first got there you know coming from Silicon Valley like you don’t tell the CIO it can’t be done. But what he got to government is people would say well you can’t do that because of the far no federal acquisition of election.
Bob Payne: [00:06:39] And he said first he was angry and then he said I’m going to become the expert on the far and and what he what he did was that same approach was like ah here’s what you wanted. You know here’s the safety that you want from this regulation. And here’s how we do it better. Right. It’s sort of you know
Carlos Rojas: [00:07:02] The approach works. You know where you partner with the groups that are supposed to be the roadblocks or supposed to be the longest pole in the tent. If it works you know.
Bob Payne: [00:07:12] Well it’s, They’re there to keep the organization safe safe.
Carlos Rojas: [00:07:19] Yeah you know if you don’t do it it goes against what you’re trying to accomplish anyway because you’re not just trying to produce code faster you want it to be reliable maintainable with quality compliant. Right. So. So it’s a matter of just you can argue whether the legislation is right.
Bob Payne: [00:07:35] But you still need to comply with it.
Carlos Rojas: [00:07:36] Exactly. But sometimes the legislation or the government has the right intent. It’s how we approach it.
Bob Payne: [00:07:42] Yeah it usually does.
Carlos Rojas: [00:07:43] Exactly. So you know hopefully for those teams that are struggling with you know the governance aspect of this I’ll also just take one thing at a time.
Bob Payne: [00:07:53] Yeah.
Carlos Rojas: [00:07:54] Because you know I’ve seen people trying to cover it all, oh we’re going to fix all these problems right now you can use your theory of constraint was the biggest one. I just tried.
Bob Payne: [00:08:04] Protects the constraint, concentrate on that one.
Carlos Rojas: [00:08:07] Yeah. Did you read the goal.
Carlos Rojas: [00:08:10] Absolutely. That’s what I got that from.
Bob Payne: [00:08:12] Yeah the Hurby or the Brian in the Phoenix project. Yeah. Put water where the fire is and in small increments it pops up. Yeah great.
Bob Payne: [00:08:27] So what are you looking forward to over the next couple of years getting accomplished.
Carlos Rojas: [00:08:33] So it’s a phenomenal question.
Bob Payne: [00:08:35] It is also a journey it’s not an end point.
Carlos Rojas: [00:08:37] It is a journey. It’s not done so I’ll say maybe three things that we’re looking at right. So one is becoming a high performing team in terms of all the agile engineering practices. So really just looking under the hood and figuring out do we have the right automation and we have the right you know called brunching do we have the right techniques that are engineering techniques that will create a great product. That’s one. The second thing that we have started working on is the AI machine learning so introducing much learning to our software development lifecycle.
Bob Payne: [00:09:09] Okay
Carlos Rojas: [00:09:09] Right so right now we’ve been using some regression basic algorithms with machine learning to be able to look at the data on the history of releases that we have so that we can do two things one we can predict when a release is going to fail interesting. And then the second thing is if it fails you know we can start looking at the deployment scripts. Figure out hey this might be that two or three things that the reasons behind why that release is failing. Right. So we’re piloting some of that solution right now. And I think that aspect is cloud. How do we turn these applications to be called cloud ready right when we know in some cases we have monolithic applications that have been around for 23 years.
Carlos Rojas: [00:09:49] How do you take that and move it to the cloud and get the benefit of it not just a lift and shift, but saying Hey lets triangle that application that are more functionalities to micro services I think those are the three components that we’re going to be focusing on the next two years.
Bob Payne: [00:10:02] Yeah the the predictive analytics side that’s very that’s very interesting to me and I’m super curious how that sounds complex but it is probably something that I started while I had a sort of field career as computer architecture free I so far as I was doing that and then decided to go to procreate. So it doesn’t sound that complex. I think the interesting thing for me is that you need a reasonably large dataset for that to be able to to really start to bear fruit. But I’m excited to hear you guys would have it so.
Carlos Rojas: [00:10:52] So we have a reasonable amount of data on releases that go back you know years okay. But interestingly enough when we started training our morals we use a subset of six months. And when we started with that subset of data we identified about 150 fields that were important and then we had the model train for a few weeks and then we test that we said let’s just see that confidence level and we put real data against that model and then our success rate was about 10 percent. It was really bad. It’s no surprise right. So instead of saying instead of saying let’s get more data more fields more of features we say let’s take those hundred and fifty and gotten our way down to 50 and then we have a smaller subset of fields and we have a smaller subset of data for 3 months and then we trend the model again.
Carlos Rojas: [00:11:56] And then now we can tell you that we’re you know successfully predicting with a 90 percent certainty certainty rate. So we went back to the model with less data less data elements are more precise data set and our success rate went up.
Bob Payne: [00:12:12] Well I mean look at you know what we’re starting to see genetic markers for you know in biology that have a very large effect. So once you find the targets I think I think that level of that that that is amazing that you were able to how long did it take you to go from 10 to 90.
Carlos Rojas: [00:12:36] So it was about a 12 week project. But then thinking about thinking about the project I’m thinking about the data said it took us about 6 weeks. So I would say you know quarter a quarter and a half worth of you know.
Bob Payne: [00:12:49] That’s amazing, are you using, what are you using back end engine?
Carlos Rojas: [00:12:52] So we’re using TenserFlow. It’s an open source from Google, using Python to write the scripts and to be quite honest with you because we’re exploring our capabilities are just basic you know we’re not building a beautiful website that people can access. We’re just enabling the developers to say hey gone check this page and then you know pick your acid I.D. your application I.D. and some information on the model is going to give you the answers. So it’s basic. Right. It’s still in the exploratory mode. We’ve done a couple of good successful use cases and now we’re trying to roll it out to the company.
Bob Payne: [00:13:28] I am I’ve been pleasantly surprised at the resurgence of Python.
Bob Payne: [00:13:37] I was in some of the early Python conferences Python back when they were here in Northern Virginia and D.C. and now running into it in the large corporations that they also have micro Python which runs on a microcontroller. So you can you can you can do hardware programming. You were always able to buy it you needed a heavy heavy thing. Now they’re doing you know arm processors isn’t super lightweight.
Carlos Rojas: [00:14:12] But the difference is you know the user experience at the end of the day you know when you started developing maybe you know in my case 20 years ago with Cobalt or C++ or even Java. Right. You have to have some sort of computer science background to be able to kind of like and understand that and and do something with it. Nowadays with Python, It’s more of a English type of you know development where you don’t have to be computer science guru to learn it and apply it so I think that’s why it’s being so helpful for the adoption of those techniques.
Bob Payne: [00:14:49] I mean it was, there was a certain, well there’s a whole philosophy around you know Python and you know people people do argue whether they achieve that but they wanted it to be an easy learnable right. You know part LAMDA part object part you know and be able to drop down to the metal pretty quickly with assembly or see you know the whole idea that you could tie in those when you needed to if you needed to program and see were able to easily tie it in with Python.
Carlos Rojas: [00:15:30] Exactly. And to be quite honest with you we’re leveraging some of the open source you know libraries to execute on some of the you know projects that we have.
Carlos Rojas: [00:15:43] So it’s not like we’re building something from scratch anymore right. So that’s helpful as well.
Bob Payne: [00:15:48] Yeah. Well great Carlos. Thanks for coming in. It’s a pleasure chatting with you and I wish you guys all the well you don’t need luck all the hard work in the world.
Carlos Rojas: [00:16:02] Thank you for having us here, appreciate the opportunity. Thank you so much.
Bob Payne: [00:16:05] Thanks.
Listen to more Agile Toolkit.
