In this episode, I sat down with Rob Bos to talk about what’s happening across GitHub, Azure DevOps, GitHub Advanced Security, and the fast-moving world of GitHub Copilot. Rob shared how his work evolved from DevOps consulting and CI/CD into GitHub-focused training, governance, supply-chain security, and Copilot enablement. We talked about why so many teams are moving source control to GitHub first, how GitHub is pulling ahead on new features, and why products like Advanced Security, Dependabot, code scanning, and Copilot are becoming central to modern software delivery.
We also went deep on what it actually takes to use AI coding tools well. Rob made the case that handing someone a Copilot license is nowhere near enough: teams need training, realistic expectations, strong specs, good test harnesses, and better feedback loops. We discussed usage-based billing, token costs, environmental impact, AI Engineering Fluency, and why the future of engineering may belong to teams that combine business context, solid controls, and AI-native workflows instead of treating LLMs like magic.
- Rob Bos on GitHub: https://github.com/rajbos
- DevOps Journal (Rob Bos blog): https://devopsjournal.io/
- Xebia: https://xebia.com/
- Xebia Tech Hub: https://tech.xebia.ms/
- GitHub: https://github.com/
- GitHub Copilot: https://github.com/features/copilot
- GitHub Actions: https://github.com/features/actions
- GitHub Advanced Security: https://github.com/security/advanced-security
- Dependabot: https://github.com/dependabot
- Azure DevOps: https://azure.microsoft.com/en-us/products/devops
- Visual Studio Code: https://code.visualstudio.com/
- Visual Studio: https://visualstudio.microsoft.com/
- JetBrains: https://www.jetbrains.com/
- AI Engineering Fluency for VS Code (Visual Studio Marketplace): https://marketplace.visualstudio.com/items?itemName=RobBos.copilot-token-tracker
- AI Engineering Fluency for Visual Studio (Visual Studio Marketplace): https://marketplace.visualstudio.com/items?itemName=RobBos.AIEngineeringFluency
- GitHub Advanced Security (GHAS) course by Rob Bos on LinkedIn Learning: https://www.linkedin.com/learning/github-advanced-security-ghas
- Learning GitHub Advanced Security for Azure DevOps by Rob Bos on LinkedIn Learning: https://www.linkedin.com/learning/learning-github-advanced-security-for-azure-devops
- Responsible GitHub Copilot: Creating Reliable Code Ethically by Rob Bos on LinkedIn Learning: https://www.linkedin.com/learning/responsible-github-copilot-creating-reliable-code-ethically-24981582
- GitHub Actions in Action: https://www.manning.com/books/github-actions-in-action
- The GitHub Copilot Handbook: https://www.packtpub.com/en-us/product/the-github-copilot-handbook-9781806116638
[00:00:05] Unknown:
So, hey, man. It's good to see you again. Been too long. It's been a while. Indeed. Yeah. Thank you for having me. Yeah. Well,
[00:00:15] Barry Luijbregts:
we're now in such interesting times and you are an advocate for GitHub Copilot, I think online, and for your customers as well. And I'm using tools that you created as well, and lots of my customers do that as well. And I always tell them to install things that you created. So I would love to talk about that as well. But maybe to start with, could you introduce yourself to the listeners?
[00:00:42] Rob Bos:
Yes. I'm Rob based in The Netherlands. I still call myself a DevOps consultant because that really frames the way that I think and the way that I approach things. So that helps. And I tend to help do that with my customers as well. I tend to help them with whatever do we need to get code into production because that's basically the focus of everything that is there. And from that, that really comes with a devil's mindset. I switched over to the GitHub platform, I wanna say six years ago. It's probably it's even already seven when GitHub Actions became a thing. I mean, it's also been interested in CICD as well because that helps with a lot of the automation parts.
But also with the security around those platforms. And I was doing that on the Azure side or Azure Devo side, I should say. And then all of a sudden, we had good of actions, and that was new. And it has its own supply chain issues, so I dove straight into that. And it's been full time GitHub for me ever since. And still also on the focus of how can we get more out of the platform. So I tend to help customers with everything that that's purchased, basically. So that can be enterprise governance on how to set things up, for example. That can be how do we train engineers to actually understand how things work, either on how do you write your first pull request, for example, up to the point of how do we do supply chain management properly.
And and, of course, you know, the breadth of the ecosystem that GitHub is. So that is quite a lot. For the last year and a half or so, that's really been a a full time dedication almost on GitHub Copilot, of course. It is a tool for a developer in my opinion. But, yeah, everybody saw that we need to do something with AI, and then they call us.
[00:02:27] Barry Luijbregts:
Yeah. So, yeah, GitHub is huge. There's so much in there. Maybe we can stay there for a while before we dive into AI land. Like, GitHub has this thing which called advanced security and there's lots of stuff under there as well. What do you see customers using there? Like, dependent bot and stuff that's also under there. Right? Basically, everything that is there,
[00:02:52] Rob Bos:
and it stems from making your setup more secure. So that can be either your repository itself, making sure that you don't get all sorts of, weird things in there. That can be dependable for dependency scanning, for example. But there's also code scanning involved as well where you scan your own code instead of third party code for, hey, figuring out what is happening, how do certain things work, do I have SQL injection attacks in my own source code, things like that. And two weeks ago, I wanna say, maybe three already, they launched Code Quality.
That's another product owner, the same sort of name. It's not necessarily the phone security. So that becomes a whole suite of enabling things in and around your repository as well. And, again, one of my little things that I really like to to work on as well because there's so much interesting things happening in supply chain and supply chain attacks these days.
[00:03:44] Barry Luijbregts:
Yeah. Yeah. That's yeah. That's very important that you don't have, let's say, infected packages or packages with vulnerabilities, things like that in your code base. There's a lot and then if we zoom out a little bit because you already mentioned you first focus on Azure DevOps. Know it does also similar things to GitHub. You know, you host your code, that type of stuff. I remember back in the day we had a server under our desk and we did SVN and then Team Foundation Server, and then eventually that became Azure DevOps.
So the Azure DevOps thing by itself, is that going away?
[00:04:23] Rob Bos:
Yeah. That's always interesting. And it's always interesting because that is one of the questions we get asked as Microsoft consultants as well. Yeah. And the official answer is, of course, Microsoft has still stated that whatever they release right now, they will continue to support it for ten years, which is true. But also then looking at the other side is if you see where new features are being launched first, that is GitHub all the way. It's just ridiculous how fast GitHub is moving and operating in that that direction.
All of the new shiny things are on the GitHub side. GitHub's phone security for Azure DevOps is a thing, but I think we waited, like, two and a half years before they actually launched that after GitHub's phone security became a thing. Copilots, we've been doing that for four or five years now on the GitHub side. Azure DevOps has just launched their private preview of two or three of the features after years of their so, yes, it is not going away in my opinion. I think there's way too many users still using that on a day to day basis.
But, yeah, if you wanted to have the new shiny features, then you go to GitHub. And, also, there's a perfectly fine setup where you can use both. Even licenses, for example, is relatively trivial to also get a GitHub Enterprise license these days if you already have an Azure DevOps license. So there's plenty of benefits of saying, hey. We're going to stick with Azure DevOps for the things that we really need it for. And that's often, for example, we've have tons of experience in pipelines, and we've created a lot. So why not leave them there, for example?
Azure boards is a big thing. Azure test plans, those things don't have an equivalent on the GitHub side. So it's perfectly fine to stay with those on the platform where it works, and then migrate your source control over to GitHub and start doing Copilot on your source code, for example.
[00:06:10] Barry Luijbregts:
Right. So to get benefits from GitHub, it's already enough to just move your repositories.
[00:06:16] Rob Bos:
That's the the biggest driver we see for migrations these days. Get your source code over because then you get access to additional tooling, both at phone security, for example, that does a little bit more because that has autopilot already enabled on the GitHub side. But also, you get Cloud Agent. We can spin it up autonomously. You get access to the review agent that is there. You get access to the chat from GitHub for Copilot. So there's a lot of functionality that you get additionally available if your source code is hosted on github.com.
And just to to give an indication there, we recently created a training around that. And that training by itself, what are all the features on github.com if you have Copilot, is two hours of just going through all of the features that is there. It's ridiculous. So there's a lot of catch up that Azure DevOps still has to do in that regard. So, yeah, starting with source control migration is usually the easiest partner.
[00:07:13] Barry Luijbregts:
Okay. Cool. So, yeah, Azure DevOps, GitHub, GitHub, the repositories in GitHub then as an enabler for all sorts of features like the cloud agent, GitHub Copilot, Cloud Agent, all sorts of other stuff. And then GitHub Copilot is then just a feature of GitHub, which you can also use without having your repos in there, right? No. Just just on anything really. And that is really cool. So I've been using that now for years, as long as it's been out kinda and seen the evolution. So it's still very new, right? But the capabilities that this thing has is just it's mind blowing still to me to this day. It's running right now here in GitHub Copilot desktop app. I'm having it do something with Opus 4.8, which is so expensive.
But, you know, just let it cook. I'm trying to see if I can get an email from somebody that to say, hey. Stop using all these tokens. It's getting too expensive or something. Yep. That's usually the financial department that will find you. Yeah. We'll see. I haven't I'm trying. Nothing yet. But let's see. I'm doing my best. Yep. So so what's your experience with GitHub Copilot?
[00:08:30] Rob Bos:
Yeah. I I'm at the point where if I join a new project and I'm not allowed to use it, I won't join the project. Okay. This this is now a a minimum requirement that you need to have access to these things. Mhmm. Because my value as an engineer is in the way that I think and in the way that I produce stuff and not necessarily in the way that I hand by create code by hand, I would say. Yeah. And I can think pretty fast, but I can definitely not type that fast. So that's where I really can help me along with those things.
So, yeah, I'm an advocate of at least using it where it makes sense. There's definitely use cases where it doesn't make sense. But, yeah, for the most of the things that I tend to do, I I work a lot of big picture type of projects as well. It's a lot of architecture overview, sometimes a lower level scripting and things like that, and it is just amazing for that. So if you really know how to use it, and that's an interesting thing, then you can get so far with these tools. As long as you have the right guidelines in place and you have realistic expectations of these tools, that's where it really shines.
[00:09:37] Barry Luijbregts:
Yeah. And and that know how to use it. That's kind of a a loaded term. Isn't it just as simple as just typing into the text box and that's it? Yep. That's usually the the the whole bare minimum that is there.
[00:09:49] Rob Bos:
And that's what I see with customers all the time as well. We regularly do customer events. The last time we had an event, we had 30 potential customers there. Some of them that we've helped get on to Ash Devils, for example. And I'm an online consultant, of course. Right? So I start asking questions like, hey. How many of you actually do Copilot already? It's like 8080% or something. How long? Some of them already more than half a year, for example, so you get a feel of where they are. And then you say, how many people actually enabled engineers more than here's a license, have fun.
And then, like, five hands were still up in the air of 30 different organizations. And then you say, okay. And I'll be honest. Is that more than here's a license and here's a YouTube video of half an hour and that's it? And then all of it has one. And then I say, I've helped set up the training curriculum for this, and we are currently at fourteen hours of training. Just to give an indication of there's so much that you learn you need to learn to be able to get the actual not necessarily get the actual results, but get great results out of that at a speed where it makes sense without necessarily using Opus 4.8 for everything and just burning all of the cash that you have there. Oh, come on. That was too easy.
No. That's what we really often see. That people get a license. They get a basic minimum understanding of what it could do, and they start typing in a chat editor. And then often after half an hour, they have a lot of frustration off. I set it to fix the bug, it didn't fix the bug. And then I look at the prompt and go, yeah, you really said it to fix the bug, but you didn't tell where the bug was. What is it? And how does it actually represent itself? Do you have a log file somewhere? No context whatsoever.
And that is usually the biggest eye opener of. You really need to spend some time there. Even how I approach training, for example, we tend to give people a training, and then we wait a week before we do a follow-up training because they need to get hands on time to absorb the the things that we've just shown. And just to give an indication of that, when we started doing Copilot, we were doing a lot of enablement of customers. And I've been giving training at that point for, I wanna say, eleven, twelve months together with a colleague who's been doing the same for other customers as well.
I gave a demo, and then we went for coffee. And during coffee, they came to me and said, I didn't know you could do that with Copilot. I've learned something new. And that was a coworker that has been doing this for almost a year back then. Then we switched roles after the break, and his first demo was to me completely new. I've never thought of it doing it that way. And then you go, wow. And eventually, I looked back and I started to list out all of those moments that I had, for example. And I created some YouTube videos around that, and that became an hour and twenty minutes of nine moments of enlightenment as I called it, where I, as a trainer working on this on a day to day basis, went, oh, but I didn't know you could also do this. And, oh, if I change the way that I approach these tools and I get more value out of that. And looking back, I realized that that nine moments of enlightenment that I had took me six months to internalize that.
Yeah. And then you're talking about an engineer with more than ten years experience, and then still you need half a year to absorb some of these things there before it really clicks and then you can actually go fast. And that's the the common mistake that we see being made here. We approach this like this as a hammer, and we give an a carpenter a new hammer and say, have fun. But we don't tell them it's a pneumatic hammer, and there might be rockets involved, and it really, really goes faster and can also go fast in the wrong direction if you use it the wrong way.
[00:13:28] Barry Luijbregts:
Yeah, and there's a million ways to use it as well. You know, I always explain it as there's the core. That's the text box you type stuff into. You give it some context as in the things you're actually talking about. And then there's all this markdown and tools to control the thing and guide the thing a little bit, which is very confusing as well. Like, markdown, why is it markdown? Yeah. It's
[00:13:55] Rob Bos:
yeah. You gotta love markdown nowadays. But even just with saying, hey. I add files, for example, to the conversation because I think they're interesting. There's so much mishaps that are happening there as well, and especially now with, we're we're getting billed for the actual usage. I still see people just drag entire folders of log files into the context and then say, hey, fix this bug. That will be so helpful if you do one single search for the error and then paste in the five lines of log files that are really interesting.
So having that basic understanding of what an LLM actually does with the code that you send in after you hit enter in a prompt, That is so important. I've seen people that got enablement sessions even from vendors, where they had, like, ninety minutes of demos and everything was shown, and then somebody raised their hand up and said, can you do a prompt against this code base and ask it about, the code complexity in this code base? And then the the person demonstrating it actually typed in that prompt, and we got an answer back. And everybody thought that was the actual code complexity number that was there. So after ninety minutes of explanation, they completely missed the basic things there that LLMs don't do computation. There's nothing really intelligent involved there. It's just trying to predict the next word or the next token that is there.
And that is often skipped over these kinds of things. Whenever we try to do enablements, for example, we got four or five slides that actually explain what an LLM is and what it does, and especially give all the examples where it can show that it doesn't work. And those are still the the benevolent silly things like give me a random number between one and a 100. And then you look at a group full of engineers, you say, do you actually know the the answer to this? And one out of 10 times, nobody has an answer.
And then I I sort of heckled them that they're not really nerds because the answer is 42. That is what an LLM has been trained upon. That's nine out of 10 times that is the answer, especially for all the models. The newer models have some some ways to mitigate that. But even there, you can then show, for example, these days, most of the time, the next answer is 73. So one of the prime numbers that we use as well. So yeah. Those kinds of little things, they're they're key to having a realistic expectation of what you can do with an LLM, And that's often missed. So people think they type into a text box and magic will happen, but, yeah, it's not necessarily magic. You definitely need to check out the results.
[00:16:25] Barry Luijbregts:
Yeah. Definitely. And is that something that you still do as well? Like when you ask a thing to do something, it spits out a bunch of code. Do you still look at the code? Less and less. It depends on what I'm trying to do.
[00:16:39] Rob Bos:
What I've I've grown into and probably something we should have done in the early days anyway, like we're we're like we're approaching coding. We always said, you should have a good harness, for example, around testing. CICD should be in place. Your test should actually be sensible tests. You can look at code coverage and things like that. And that's also still my devils background of I'm looking into those kinds of techniques to have them in place to have more trust in that the code actually does what it needs to do.
And that can go to point to the point of, okay, I wanna summarize this data, write me the code and actually summarize the data, and I need to be able to trust that the sum of the results is actually the actual sum of what is happening there. That's the benefit with code is that you can actually look at it and see, yes, it's actually looping over everything and summarizing it the right way. That is helpful. And then with that, you can start zooming out of, okay, build me the unit test and actually validate this.
Often, you can then get get the question of, yes, what you're now generating the unit test with AI. Can you still trust the unit test there? And I usually counter that with, we've been having the same discussions for the last thirty years. If you don't trust your unit test, throw away mutation testing to the mixing to see if your unit tests actually fail. If you don't wanna write all of the nitty gritty unit tests, maybe write progression tests. And I've been doing that in my career for most of the time anyway, so that's a natural go to of I wanna validate that the outcomes are at least in my realm of expectation, and then level up with some more spelunking usually. I I phrase that off. Let's try and figure things out because it's really interesting to see that if you have your test harness and then ask to a capable model, look at this code, look at the test harness, and tell me where my blind spots are.
That's a whole different way of phrasing the the research that you're trying to do, and often it finds blind spots where you go, oh my god. I couldn't even realize my code could do this. So that's where I tend to do that. So it depends a little bit on am I writing scripts? Because I write a lot of scripts against systems like GitHub, for example, to validate certain things. And there, I tend to go into more of a sort of an investigation validation mode to check, hey. If I have something where I calculate the amount of pull request velocity for a repository, that's something you can still validate.
So at least have sort of a sniff test of does this make sense here so no. That's what I tend to do for scripting. And then another option is I work a lot with web applications, for example. I am really fan of handing over tasks to, for example, Copilot Cloud Agent. And then in the instructions, it's always part of build the code, validate that it works, and show me a screenshot of the before and after. Makes it really easy to visualize. You check if the code is in the sort of realm that you wanted to have set up for.
You can do some more testing on the outside if you need to. So I build in the tools and up to a level that gives me a sense of trust to be able to push this to production for those kinds of things. Another use case that I specifically have, you mentioned it in the beginning, I write a couple of official studio code extensions these days, and those are really, really hard to automatically test. So there's only one thing that you can really do with big changes, run the damn thing and validate it yourself.
And that's also for some of the things that are changing in the UI. I just run it and validate that it works. If it's something where I have a configuration file and a new model is discovered, for example, I need to display that. Yeah. I trust that I have my automation in place that it works. So those are some of the the techniques that you then use for working. Do you actually still need to write every single line of code? For most of the time, I build in so much test harness, for example, that I can trust that it actually does what it needs to do. And, of course, we still do the normal things. Right? You deploy to production sorta in a way that nobody sees it so that you can flip a switch internally, test it out, and validate if it works.
Then put a layer of logging on top of that and then put a layer of incident sort of management on top of, hey. If we have a bug, we actually log the information and feed that back into an LLM system to improve it for next time. And all of a sudden, you get sort of a self healing situation, and then it becomes really interest.
[00:21:08] Barry Luijbregts:
Yeah. That's cool, isn't it? Yeah. So this is all the stuff that we should have been doing already, including creating good specifications. Right? So now we're finding out that our specifications might not be specific. And so we need to go back to those yeah, the basics, right? Specifications, deterministic controls, tests, quality gates, that type of stuff. I like to I'm starting to think about this as, you know we used to. Outsource a lot of software development, so you hire somebody or a team that is in a different country somewhere and to do that you need to write very good specifications and you need to have extremely good tests because the output is nondeterministic.
Sometimes it works, sometimes it doesn't. This is kind of similar to that. Although sometimes the results are better depending
[00:22:01] Rob Bos:
on the models and things that you choose, but you need those controls around it and also then humans to test it, for instance. Exactly. And that's what you see happen now more and more depending on sort of which part of the industry you're in. I've had conversations with customers where they said, we wanna give the engineering team AI, in this case, Copilot, And we have a specific team in mind because they're starting a greenfield project in three months, and they're almost ready to go. I said, cool. Let's start there. And they said, yeah. But now that we're thinking about it, we're not going to give the engineers Copilot yet because our business analysts are not done yet writing all the specs.
And I went, this is your chance to actually bring the business analyst into the team, make them write the the instructions or the the user stories together so that we get better results. I've always been a fan of end to end testing by writing things like Cucumber specs and things like that because that enables you to write natural language of this is my rule. If you have that covered, you can test the dentin from the outside and all of a sudden, a lot of your unit tests go away and you bring in those business folks on board. They write the specs. Often, they write conflicting specs. That's the interesting part.
And how you can find that easier, you can write better specs, and we're almost in the direction of some people are already talking about dark factories where you just push in a spec and code changes come out the other end, and you have no clue what happened in between. Yep. So, yeah, if you if you make all of the the stakeholders, the business analysts, bring them into the team, now you have a very powerful setup of what are we actually implementing. And we've been having that conversation for the last thirty years as well of when do we actually add value to production.
Yeah. We haven't figured out the answer. That's the interesting part.
[00:23:51] Barry Luijbregts:
No. And now instead of an guesstimated man hours, now you actually get a number of it costs this much because we can see it. Yep. So it's this many tokens, this amount of euros, and so that's what it cost costed for this feature. Was it worth it? I don't know. But it becomes more visible now. And it becomes all of a sudden a a conversation point
[00:24:16] Rob Bos:
because it can be very flexible in that sense as well. It used to be we had people hours and that took us to actually make the changes there, and we had license calls and maybe some hardware calls and that was it. And all of a sudden, we have cost in the mix that are very incidental. Your prompt might actually take you twenty minutes, and it cost you 2,000,000 tokens. And if I do the same one, it might cost me 5,000,000 tokens. And figure then out which one of us who had a meaningful session actually contributed value. Super hard to to define.
But now that it actually comes with a monetary cost, all of a sudden, everybody's kicking and screaming of, we need to keep this under control again. It's really interesting.
[00:24:58] Barry Luijbregts:
It is interesting, and that is where are we? June? July? So we've had a month of usage based billing now with GitHub Copilot, where previously it was a premium request units basically for free. So now people are kinda tasting what this thing actually costs to use, and some of them are surprised, of course. I get that. And you you already said it. You built some extensions. You have a very nice one that I use daily and I get all my customers to use as well, and that is called AI Engineering Fluency. I'm looking at it right now.
This is in Versus Code and other places as well. Right? Can you maybe talk about that and why you even created? Yeah. It actually started out as the Copilot token tracker
[00:25:48] Rob Bos:
because I hated PRUs, premium regress units. Because it it was always hiding that there's actual cost involved when you talk to an L. Yeah. And I I have been pushing back to the teams on the GitOps side from day one of. I wanna see that. It comes with computation cost. It comes with monetary cost. It has an environmental impact. I wanna show that information. And somewhere in December last year, they did an update to Versus Code, and all of a sudden, they could show the information from your session. So which tools did it call, for example, what happened, how many tokens were used, and finally I had some information that I could run with.
We were still paying in premium request units, so users only saw, hey, I I went down 10 premium request units this session, and that was it. So I start to work with, okay. We have the files on disk. I have Copilot, and I had some some time available to work on things. And because I was triggered off, I wanna see how far I can bring this forward. So literally started as figuring out all of your sessions on disk and just read the log files on your machine. Look into them and then look at things like chat turns, how long was your session, how many tokens did you actually go through.
So that I could summarize that and then projects of, hey. If I look back at the last thirty days, I can actually see how many tokens you've used. There is some fuzzy calculation available on, hey. A token amounts to this amount of computation. This amount of computation, can actually express that in c o two cost, for example. So how much environmental impact that you actually have there. You can do the same thing for water usage for cooling a data center on average just to give an indication of what happens there.
And if you have c o two, then you can also calculate how many trees you actually need to plant to be able to do something about that c o two impact. That's where it actually started off. I wanna have that kind of information available because there are people that say, hello, good morning every day to Opus four point eight. And that has a big impact on the back end of what is happening. And that's still a thing. I would go so far is that as a user, I would like to schedule certain things and keep environmental impact in mind.
So if I have something that runs every single day, I would like to have the option to choose a data center in Sweden where that runs that runs on hydropower and things like that. But we're not still not there yet. So that's where it started in December of, I wanna have some information to show people that this is more than just asking something, a question, and you get an answer back. Notice actual impact hap And, yeah, of course, in good fashion, from there, everything escalated. We dove deeper. We went broader. I've had some colleagues that spend an innovation day on this. So I think by now, six or seven people have contributed.
It's open source, so people can contribute anyway. And we really started to go broader and a lot deeper in certain things as well. So by now, we support a lot of different AI tools that are available. So not just Copilot, ClotCode, for example, is being picked up as well. The new Mistralvibe, for example, that is currently might still be in beta. I'm not sure. It's being found. So all of those different harnesses that you can use and you name it. Even if it's OpenCode, if it's the pyre harness, we pick up the files from there as well.
From that, we can actually start mapping your sessions. So how long was the session? How many chat turns did you actually have there? How much tools did you call in that session? Session? Are you very specific when you do prompting? Do you actually bring in context files and refs, for example, into your session? If you know all that kind of information, you can start to plot that in a chart to give you a little bit of insights, and then we can also give you very fine grained improvement tips there as well. If we see, for example, that Claude's code continuously automatically summarizes your session, that's an indicator that apparently you don't know what that does and what's the risk of compacting, for example, might be.
So we can at least surface that information and say, hey. We see that this happens all the time. Maybe you wanna be a little bit more specific. We already know that compacting also causes a a cash miss in the next turn, for example. So, hey. We're paying for these things. Caching is the thing, and it's super helpful because that brings the the cost down. So we can actually surface that tip-off. Hey. This is what we see happening. Maybe you wanna look at some information around this and get better in the way that you're using these tools.
So, again, this escalated completely and now has a fluency score in there where we look at different aspects of your usage. And we can see, for example, hey, in the the prompt engineering, you're scoring level three, and in the context engineering, you're scoring at level four, for example. And there we can at least give you some guidance.
[00:30:49] Barry Luijbregts:
Super useful. So I'm I'm looking at it. I am in the fluency score. I am stage four AI strategist.
[00:30:57] Rob Bos:
Awesome.
[00:30:59] Barry Luijbregts:
And I don't know if that's good.
[00:31:01] Rob Bos:
That is the highest level. We take people through different aspects. I'm pulling it up on this side as well to make sure that I have them in the right. And per aspect, we try to find if you're between stage one and four. So it starts with an AI skeptic. You use it, but not necessarily that often. And then you can graduate into an AI explorer, an AI collaborator, stage three, and then AI strategy is a stage four. And at that point, we don't really know which extra tips to give you, so that's a a good thing to be at.
But we do that for, as I said, prompt engineering, context engineering. We look at your tool usage, for example. Have you enabled tools? Do you have enabled tools in your editor? The basic ones are always read file or write file, for example. But there's, of course, so much more that you can install in these editors, it's good to see either, a, you haven't been installed, but also, b, if you're actually using them. And then the last two aspects are customization and workflow integration. So across all of those topics, we look at, okay, this is some of the the metrics that we track.
Where are you in your, basically, maturity of using these tools?
[00:32:10] Barry Luijbregts:
Yeah. And so it also shows usage by editor. So cloud code for instance, also sometimes use. I see that show up cloud desktop, co work, and there's all sorts of places. SQL Server Management Studio of course. And you name it. So do you then need to install and test That's all of the fun part. And
[00:32:31] Rob Bos:
it is. I work for a company that is a Microsoft and GitHub partner, but I'm a consultant, so I'm not necessarily married to a specific product. I wanna have the best tool available that we can use both for me and the people that I train. And the only way to find out about those things is install them all. And that's where the fun stuff first starts to happen. So I have accounts on most of these things available on working certain things. Some of them are free, luckily. But, yeah, I've got them all installed.
[00:33:01] Barry Luijbregts:
That is fun. Yeah. So if you let's be honest. If you compare, let's say, Claude code to GitHub Copilot, different harnesses can have the same models.
[00:33:14] Rob Bos:
What do you think? Which one wins right now? Right? Can change It all depends on where your muscle memory comes from. That is the biggest one. So if you've been using cloth for two years, then probably the answer for that person will be cloth. The interesting part is that and I've had those folks in trainings as well. They looked at it at a certain point in in time, and then they ignored it for one or two years, and then they come back. I've had people in the training where they say, hey. This is the first getting started training on Get a Copilot.
I've looked at it, it was crap. And you have two hours to convince me. What do do as a trainer? But still, after two hours, they came back and said, holy moly. I haven't looked at these tools for two years, and apparently, I missed this, that, and this, and I can also do that. It changes so quickly. And that's often how you see these these things being portrayed where somebody says, yeah, but I can schedule something in Claude and it runs every Monday and it does the magic for me. Mhmm. And I say, Copilot can do that as well.
So the harness itself is becoming more and more fluid as well, and it's no longer a unique selling point. It's more of the experience that you have there and that you want to do. And I recently created an overview of these are all the the sort of the surfaces where you can use Copilot these days. It it's a massive slide off. You you can use this from your editor. You can use this from your CLI. You can run this from your editor and hand it over to the Cloud Agent, and then then it can do something autonomously there. You can trigger this from GitHub mobile and then run the session in Cloud Agent. You can actually go to the web and run a session in Cloud Agent. Cloud Agent actually has automations these days, and you can schedule them in for you.
And it all then leads for me to the Copilot app because that thing is just freaking amazing. Let's see how far they got that thing to work. It was released and built a month ago, I would say. But I've have had access to this since March, it is my go to. It is really energetic setup where everything is really focused on I wanna implement something. So the Copilot app has really been configured to say, whatever you're going to do, you're going to create a pull request because that is where the value of an engineer lives, and it's really geared towards all of those things.
That's my go to these days, because the the the way that they think about it is differently. So Versus Code for me, for example, is really interactive. I wanna see the files that are being changed. I want to see the the harness go through the loop that it has, run the test, then maybe validate it from there. And then if I need a pull request from Versus Code, I need to prompt it, create a pull request, or do it myself. In the CLI, I tend to use it mostly for stuff I have on my machine. Find me this file or hey, if I have an issue on my machine, look at the logs, look at the stack trace, go for it, and then just plung through whatever is in there.
And the Copilot app is really I trust that I have a good setup in my repository. I have my testing in place, for example, and then I can really go faster and just have multiple concurrent sessions and say, just go for it. And that gives me so much more power because I have all of the sessions in the same overview, and I can just look at it from there. Whereas with other editors, depending on what they do, before the CLI, for example, you tend to have a single session and a single shell session. So if you wanna do something concurrently, then you need to create two sessions and hope that the harness actually allows you to create a work stream.
So it's different touch points for different editors, for different types of work that I even have. And it is. I use cloth and cloth code and cloth co work. I use Mistral, Mistral code, and Mistral vibe all intermediately because I try to keep up to date of what the heck they're doing. And the interesting part is that everything starts to group together. They they start to build the same UIs for these things. So you see that Yeah. The harness and the way that they they create those things, yeah, everybody's looking at everybody else, and they pick up the the things that work from them.
So going back to your question, I don't necessarily have a preference. I like the way that GitHub approaches things from a developer perspective. Mhmm. And the tools that are building right now are very good, so that tends to be my go to.
[00:37:26] Barry Luijbregts:
Okay. Yeah. Yeah. I I like it too. Right. It's nice. So the extension AI engineering fluency, do I per se need Versus Code to do it? Is it also in Visual Studio? Versus Code, Visual Studio, JetBrains.
[00:37:43] Rob Bos:
There is a NPM package, so you can run it in a shell or a status line, for example. You can build bring it into Omar Posh if you want to. Can bring it into the Copilot CLI. I think I have a script somewhere to do that for Cloth code, for example, as well. So, yeah, you can bring it in lots of different places.
[00:38:01] Barry Luijbregts:
Nice. And has a GitHub Copilot team not yet
[00:38:05] Rob Bos:
knocked at your door to integrate this into their product somehow? Not specifically. Everybody's very enthusiastic, and they're looking at certain things. And you can see that this kind of information is relevant for a lot of folks. So Mhmm. Versus Code has been building the session debugger, for example, into their setup. They've done the same now with Cache Explorer and things like that. So everybody's learning from each other and then building little things into the product as well. So
[00:38:34] Barry Luijbregts:
Okay. Well, let's see what happens. Maybe maybe they'll buy you out or something. That would be cool. Okay. So we have discussed a lot of stuff already here, and also the extension and how you built that type stuff. Maybe we can finish on this. Where can people find your trainings, like online trainings and things like that to learn more and to get up to speed with GitHub Copilot and other stuff? And, of course, your extension and wherever else.
[00:39:08] Rob Bos:
Oh, do you now have another hour? Yeah. Oh, absolutely. Yeah. You can start, and I'm always available on LinkedIn. I contribute a lot to the kids, go to community, for example. So I tend to share a lot of things over there. So that's your go to. I have a blog called devopsjournal.io. So you can find a lot of things there as well, and then definitely check the about page because that is where all of the links are grouped. And, again, from there, it just escalates in so many different directions. So I'm very active on GitHub. You can follow me there as well.
I have several LinkedIn learning courses on GitHub. GitHub with phone security and GitHub with phone security for Azure DevOps, a responsible use of AI with GitHub Copilot and things like that. I've written two books by now. So GitHub actions and action, if you want to dive deeper into the actions ecosystem and how to work with that, and the GitHub Copilot handbook that came out last year. So those are the the big things there. What we've been doing at my employer, Zebia, we've built the tech.zebia.ns site.
And that is a site that groups information around anything Microsoft, but also for Copilot on all of the updates, product updates, editor updates that are available, and we group a lot of information there. And we actually have a newsletter for that. Newsletter of yesterday was, like, 10 different updates from a single day. So if we wanna stay up to date, that would be a very good starting point because I tend to publish my stuff over to that site as well. So tech.zb and other things.
[00:40:47] Barry Luijbregts:
Okay. Excellent. And all of those things will be in the show notes so that you can just go and click on those and go follow Rob and say hi to him. Awesome. Alright. This was very cool. Maybe we can do it again in a year to see how crazy things are by then. Probably. Yes. Let's do that. Alright. Thank you.
So, hey, man. It's good to see you again. Been too long. It's been a while. Indeed. Yeah. Thank you for having me. Yeah. Well,
[00:00:15] Barry Luijbregts:
we're now in such interesting times and you are an advocate for GitHub Copilot, I think online, and for your customers as well. And I'm using tools that you created as well, and lots of my customers do that as well. And I always tell them to install things that you created. So I would love to talk about that as well. But maybe to start with, could you introduce yourself to the listeners?
[00:00:42] Rob Bos:
Yes. I'm Rob based in The Netherlands. I still call myself a DevOps consultant because that really frames the way that I think and the way that I approach things. So that helps. And I tend to help do that with my customers as well. I tend to help them with whatever do we need to get code into production because that's basically the focus of everything that is there. And from that, that really comes with a devil's mindset. I switched over to the GitHub platform, I wanna say six years ago. It's probably it's even already seven when GitHub Actions became a thing. I mean, it's also been interested in CICD as well because that helps with a lot of the automation parts.
But also with the security around those platforms. And I was doing that on the Azure side or Azure Devo side, I should say. And then all of a sudden, we had good of actions, and that was new. And it has its own supply chain issues, so I dove straight into that. And it's been full time GitHub for me ever since. And still also on the focus of how can we get more out of the platform. So I tend to help customers with everything that that's purchased, basically. So that can be enterprise governance on how to set things up, for example. That can be how do we train engineers to actually understand how things work, either on how do you write your first pull request, for example, up to the point of how do we do supply chain management properly.
And and, of course, you know, the breadth of the ecosystem that GitHub is. So that is quite a lot. For the last year and a half or so, that's really been a a full time dedication almost on GitHub Copilot, of course. It is a tool for a developer in my opinion. But, yeah, everybody saw that we need to do something with AI, and then they call us.
[00:02:27] Barry Luijbregts:
Yeah. So, yeah, GitHub is huge. There's so much in there. Maybe we can stay there for a while before we dive into AI land. Like, GitHub has this thing which called advanced security and there's lots of stuff under there as well. What do you see customers using there? Like, dependent bot and stuff that's also under there. Right? Basically, everything that is there,
[00:02:52] Rob Bos:
and it stems from making your setup more secure. So that can be either your repository itself, making sure that you don't get all sorts of, weird things in there. That can be dependable for dependency scanning, for example. But there's also code scanning involved as well where you scan your own code instead of third party code for, hey, figuring out what is happening, how do certain things work, do I have SQL injection attacks in my own source code, things like that. And two weeks ago, I wanna say, maybe three already, they launched Code Quality.
That's another product owner, the same sort of name. It's not necessarily the phone security. So that becomes a whole suite of enabling things in and around your repository as well. And, again, one of my little things that I really like to to work on as well because there's so much interesting things happening in supply chain and supply chain attacks these days.
[00:03:44] Barry Luijbregts:
Yeah. Yeah. That's yeah. That's very important that you don't have, let's say, infected packages or packages with vulnerabilities, things like that in your code base. There's a lot and then if we zoom out a little bit because you already mentioned you first focus on Azure DevOps. Know it does also similar things to GitHub. You know, you host your code, that type of stuff. I remember back in the day we had a server under our desk and we did SVN and then Team Foundation Server, and then eventually that became Azure DevOps.
So the Azure DevOps thing by itself, is that going away?
[00:04:23] Rob Bos:
Yeah. That's always interesting. And it's always interesting because that is one of the questions we get asked as Microsoft consultants as well. Yeah. And the official answer is, of course, Microsoft has still stated that whatever they release right now, they will continue to support it for ten years, which is true. But also then looking at the other side is if you see where new features are being launched first, that is GitHub all the way. It's just ridiculous how fast GitHub is moving and operating in that that direction.
All of the new shiny things are on the GitHub side. GitHub's phone security for Azure DevOps is a thing, but I think we waited, like, two and a half years before they actually launched that after GitHub's phone security became a thing. Copilots, we've been doing that for four or five years now on the GitHub side. Azure DevOps has just launched their private preview of two or three of the features after years of their so, yes, it is not going away in my opinion. I think there's way too many users still using that on a day to day basis.
But, yeah, if you wanted to have the new shiny features, then you go to GitHub. And, also, there's a perfectly fine setup where you can use both. Even licenses, for example, is relatively trivial to also get a GitHub Enterprise license these days if you already have an Azure DevOps license. So there's plenty of benefits of saying, hey. We're going to stick with Azure DevOps for the things that we really need it for. And that's often, for example, we've have tons of experience in pipelines, and we've created a lot. So why not leave them there, for example?
Azure boards is a big thing. Azure test plans, those things don't have an equivalent on the GitHub side. So it's perfectly fine to stay with those on the platform where it works, and then migrate your source control over to GitHub and start doing Copilot on your source code, for example.
[00:06:10] Barry Luijbregts:
Right. So to get benefits from GitHub, it's already enough to just move your repositories.
[00:06:16] Rob Bos:
That's the the biggest driver we see for migrations these days. Get your source code over because then you get access to additional tooling, both at phone security, for example, that does a little bit more because that has autopilot already enabled on the GitHub side. But also, you get Cloud Agent. We can spin it up autonomously. You get access to the review agent that is there. You get access to the chat from GitHub for Copilot. So there's a lot of functionality that you get additionally available if your source code is hosted on github.com.
And just to to give an indication there, we recently created a training around that. And that training by itself, what are all the features on github.com if you have Copilot, is two hours of just going through all of the features that is there. It's ridiculous. So there's a lot of catch up that Azure DevOps still has to do in that regard. So, yeah, starting with source control migration is usually the easiest partner.
[00:07:13] Barry Luijbregts:
Okay. Cool. So, yeah, Azure DevOps, GitHub, GitHub, the repositories in GitHub then as an enabler for all sorts of features like the cloud agent, GitHub Copilot, Cloud Agent, all sorts of other stuff. And then GitHub Copilot is then just a feature of GitHub, which you can also use without having your repos in there, right? No. Just just on anything really. And that is really cool. So I've been using that now for years, as long as it's been out kinda and seen the evolution. So it's still very new, right? But the capabilities that this thing has is just it's mind blowing still to me to this day. It's running right now here in GitHub Copilot desktop app. I'm having it do something with Opus 4.8, which is so expensive.
But, you know, just let it cook. I'm trying to see if I can get an email from somebody that to say, hey. Stop using all these tokens. It's getting too expensive or something. Yep. That's usually the financial department that will find you. Yeah. We'll see. I haven't I'm trying. Nothing yet. But let's see. I'm doing my best. Yep. So so what's your experience with GitHub Copilot?
[00:08:30] Rob Bos:
Yeah. I I'm at the point where if I join a new project and I'm not allowed to use it, I won't join the project. Okay. This this is now a a minimum requirement that you need to have access to these things. Mhmm. Because my value as an engineer is in the way that I think and in the way that I produce stuff and not necessarily in the way that I hand by create code by hand, I would say. Yeah. And I can think pretty fast, but I can definitely not type that fast. So that's where I really can help me along with those things.
So, yeah, I'm an advocate of at least using it where it makes sense. There's definitely use cases where it doesn't make sense. But, yeah, for the most of the things that I tend to do, I I work a lot of big picture type of projects as well. It's a lot of architecture overview, sometimes a lower level scripting and things like that, and it is just amazing for that. So if you really know how to use it, and that's an interesting thing, then you can get so far with these tools. As long as you have the right guidelines in place and you have realistic expectations of these tools, that's where it really shines.
[00:09:37] Barry Luijbregts:
Yeah. And and that know how to use it. That's kind of a a loaded term. Isn't it just as simple as just typing into the text box and that's it? Yep. That's usually the the the whole bare minimum that is there.
[00:09:49] Rob Bos:
And that's what I see with customers all the time as well. We regularly do customer events. The last time we had an event, we had 30 potential customers there. Some of them that we've helped get on to Ash Devils, for example. And I'm an online consultant, of course. Right? So I start asking questions like, hey. How many of you actually do Copilot already? It's like 8080% or something. How long? Some of them already more than half a year, for example, so you get a feel of where they are. And then you say, how many people actually enabled engineers more than here's a license, have fun.
And then, like, five hands were still up in the air of 30 different organizations. And then you say, okay. And I'll be honest. Is that more than here's a license and here's a YouTube video of half an hour and that's it? And then all of it has one. And then I say, I've helped set up the training curriculum for this, and we are currently at fourteen hours of training. Just to give an indication of there's so much that you learn you need to learn to be able to get the actual not necessarily get the actual results, but get great results out of that at a speed where it makes sense without necessarily using Opus 4.8 for everything and just burning all of the cash that you have there. Oh, come on. That was too easy.
No. That's what we really often see. That people get a license. They get a basic minimum understanding of what it could do, and they start typing in a chat editor. And then often after half an hour, they have a lot of frustration off. I set it to fix the bug, it didn't fix the bug. And then I look at the prompt and go, yeah, you really said it to fix the bug, but you didn't tell where the bug was. What is it? And how does it actually represent itself? Do you have a log file somewhere? No context whatsoever.
And that is usually the biggest eye opener of. You really need to spend some time there. Even how I approach training, for example, we tend to give people a training, and then we wait a week before we do a follow-up training because they need to get hands on time to absorb the the things that we've just shown. And just to give an indication of that, when we started doing Copilot, we were doing a lot of enablement of customers. And I've been giving training at that point for, I wanna say, eleven, twelve months together with a colleague who's been doing the same for other customers as well.
I gave a demo, and then we went for coffee. And during coffee, they came to me and said, I didn't know you could do that with Copilot. I've learned something new. And that was a coworker that has been doing this for almost a year back then. Then we switched roles after the break, and his first demo was to me completely new. I've never thought of it doing it that way. And then you go, wow. And eventually, I looked back and I started to list out all of those moments that I had, for example. And I created some YouTube videos around that, and that became an hour and twenty minutes of nine moments of enlightenment as I called it, where I, as a trainer working on this on a day to day basis, went, oh, but I didn't know you could also do this. And, oh, if I change the way that I approach these tools and I get more value out of that. And looking back, I realized that that nine moments of enlightenment that I had took me six months to internalize that.
Yeah. And then you're talking about an engineer with more than ten years experience, and then still you need half a year to absorb some of these things there before it really clicks and then you can actually go fast. And that's the the common mistake that we see being made here. We approach this like this as a hammer, and we give an a carpenter a new hammer and say, have fun. But we don't tell them it's a pneumatic hammer, and there might be rockets involved, and it really, really goes faster and can also go fast in the wrong direction if you use it the wrong way.
[00:13:28] Barry Luijbregts:
Yeah, and there's a million ways to use it as well. You know, I always explain it as there's the core. That's the text box you type stuff into. You give it some context as in the things you're actually talking about. And then there's all this markdown and tools to control the thing and guide the thing a little bit, which is very confusing as well. Like, markdown, why is it markdown? Yeah. It's
[00:13:55] Rob Bos:
yeah. You gotta love markdown nowadays. But even just with saying, hey. I add files, for example, to the conversation because I think they're interesting. There's so much mishaps that are happening there as well, and especially now with, we're we're getting billed for the actual usage. I still see people just drag entire folders of log files into the context and then say, hey, fix this bug. That will be so helpful if you do one single search for the error and then paste in the five lines of log files that are really interesting.
So having that basic understanding of what an LLM actually does with the code that you send in after you hit enter in a prompt, That is so important. I've seen people that got enablement sessions even from vendors, where they had, like, ninety minutes of demos and everything was shown, and then somebody raised their hand up and said, can you do a prompt against this code base and ask it about, the code complexity in this code base? And then the the person demonstrating it actually typed in that prompt, and we got an answer back. And everybody thought that was the actual code complexity number that was there. So after ninety minutes of explanation, they completely missed the basic things there that LLMs don't do computation. There's nothing really intelligent involved there. It's just trying to predict the next word or the next token that is there.
And that is often skipped over these kinds of things. Whenever we try to do enablements, for example, we got four or five slides that actually explain what an LLM is and what it does, and especially give all the examples where it can show that it doesn't work. And those are still the the benevolent silly things like give me a random number between one and a 100. And then you look at a group full of engineers, you say, do you actually know the the answer to this? And one out of 10 times, nobody has an answer.
And then I I sort of heckled them that they're not really nerds because the answer is 42. That is what an LLM has been trained upon. That's nine out of 10 times that is the answer, especially for all the models. The newer models have some some ways to mitigate that. But even there, you can then show, for example, these days, most of the time, the next answer is 73. So one of the prime numbers that we use as well. So yeah. Those kinds of little things, they're they're key to having a realistic expectation of what you can do with an LLM, And that's often missed. So people think they type into a text box and magic will happen, but, yeah, it's not necessarily magic. You definitely need to check out the results.
[00:16:25] Barry Luijbregts:
Yeah. Definitely. And is that something that you still do as well? Like when you ask a thing to do something, it spits out a bunch of code. Do you still look at the code? Less and less. It depends on what I'm trying to do.
[00:16:39] Rob Bos:
What I've I've grown into and probably something we should have done in the early days anyway, like we're we're like we're approaching coding. We always said, you should have a good harness, for example, around testing. CICD should be in place. Your test should actually be sensible tests. You can look at code coverage and things like that. And that's also still my devils background of I'm looking into those kinds of techniques to have them in place to have more trust in that the code actually does what it needs to do.
And that can go to point to the point of, okay, I wanna summarize this data, write me the code and actually summarize the data, and I need to be able to trust that the sum of the results is actually the actual sum of what is happening there. That's the benefit with code is that you can actually look at it and see, yes, it's actually looping over everything and summarizing it the right way. That is helpful. And then with that, you can start zooming out of, okay, build me the unit test and actually validate this.
Often, you can then get get the question of, yes, what you're now generating the unit test with AI. Can you still trust the unit test there? And I usually counter that with, we've been having the same discussions for the last thirty years. If you don't trust your unit test, throw away mutation testing to the mixing to see if your unit tests actually fail. If you don't wanna write all of the nitty gritty unit tests, maybe write progression tests. And I've been doing that in my career for most of the time anyway, so that's a natural go to of I wanna validate that the outcomes are at least in my realm of expectation, and then level up with some more spelunking usually. I I phrase that off. Let's try and figure things out because it's really interesting to see that if you have your test harness and then ask to a capable model, look at this code, look at the test harness, and tell me where my blind spots are.
That's a whole different way of phrasing the the research that you're trying to do, and often it finds blind spots where you go, oh my god. I couldn't even realize my code could do this. So that's where I tend to do that. So it depends a little bit on am I writing scripts? Because I write a lot of scripts against systems like GitHub, for example, to validate certain things. And there, I tend to go into more of a sort of an investigation validation mode to check, hey. If I have something where I calculate the amount of pull request velocity for a repository, that's something you can still validate.
So at least have sort of a sniff test of does this make sense here so no. That's what I tend to do for scripting. And then another option is I work a lot with web applications, for example. I am really fan of handing over tasks to, for example, Copilot Cloud Agent. And then in the instructions, it's always part of build the code, validate that it works, and show me a screenshot of the before and after. Makes it really easy to visualize. You check if the code is in the sort of realm that you wanted to have set up for.
You can do some more testing on the outside if you need to. So I build in the tools and up to a level that gives me a sense of trust to be able to push this to production for those kinds of things. Another use case that I specifically have, you mentioned it in the beginning, I write a couple of official studio code extensions these days, and those are really, really hard to automatically test. So there's only one thing that you can really do with big changes, run the damn thing and validate it yourself.
And that's also for some of the things that are changing in the UI. I just run it and validate that it works. If it's something where I have a configuration file and a new model is discovered, for example, I need to display that. Yeah. I trust that I have my automation in place that it works. So those are some of the the techniques that you then use for working. Do you actually still need to write every single line of code? For most of the time, I build in so much test harness, for example, that I can trust that it actually does what it needs to do. And, of course, we still do the normal things. Right? You deploy to production sorta in a way that nobody sees it so that you can flip a switch internally, test it out, and validate if it works.
Then put a layer of logging on top of that and then put a layer of incident sort of management on top of, hey. If we have a bug, we actually log the information and feed that back into an LLM system to improve it for next time. And all of a sudden, you get sort of a self healing situation, and then it becomes really interest.
[00:21:08] Barry Luijbregts:
Yeah. That's cool, isn't it? Yeah. So this is all the stuff that we should have been doing already, including creating good specifications. Right? So now we're finding out that our specifications might not be specific. And so we need to go back to those yeah, the basics, right? Specifications, deterministic controls, tests, quality gates, that type of stuff. I like to I'm starting to think about this as, you know we used to. Outsource a lot of software development, so you hire somebody or a team that is in a different country somewhere and to do that you need to write very good specifications and you need to have extremely good tests because the output is nondeterministic.
Sometimes it works, sometimes it doesn't. This is kind of similar to that. Although sometimes the results are better depending
[00:22:01] Rob Bos:
on the models and things that you choose, but you need those controls around it and also then humans to test it, for instance. Exactly. And that's what you see happen now more and more depending on sort of which part of the industry you're in. I've had conversations with customers where they said, we wanna give the engineering team AI, in this case, Copilot, And we have a specific team in mind because they're starting a greenfield project in three months, and they're almost ready to go. I said, cool. Let's start there. And they said, yeah. But now that we're thinking about it, we're not going to give the engineers Copilot yet because our business analysts are not done yet writing all the specs.
And I went, this is your chance to actually bring the business analyst into the team, make them write the the instructions or the the user stories together so that we get better results. I've always been a fan of end to end testing by writing things like Cucumber specs and things like that because that enables you to write natural language of this is my rule. If you have that covered, you can test the dentin from the outside and all of a sudden, a lot of your unit tests go away and you bring in those business folks on board. They write the specs. Often, they write conflicting specs. That's the interesting part.
And how you can find that easier, you can write better specs, and we're almost in the direction of some people are already talking about dark factories where you just push in a spec and code changes come out the other end, and you have no clue what happened in between. Yep. So, yeah, if you if you make all of the the stakeholders, the business analysts, bring them into the team, now you have a very powerful setup of what are we actually implementing. And we've been having that conversation for the last thirty years as well of when do we actually add value to production.
Yeah. We haven't figured out the answer. That's the interesting part.
[00:23:51] Barry Luijbregts:
No. And now instead of an guesstimated man hours, now you actually get a number of it costs this much because we can see it. Yep. So it's this many tokens, this amount of euros, and so that's what it cost costed for this feature. Was it worth it? I don't know. But it becomes more visible now. And it becomes all of a sudden a a conversation point
[00:24:16] Rob Bos:
because it can be very flexible in that sense as well. It used to be we had people hours and that took us to actually make the changes there, and we had license calls and maybe some hardware calls and that was it. And all of a sudden, we have cost in the mix that are very incidental. Your prompt might actually take you twenty minutes, and it cost you 2,000,000 tokens. And if I do the same one, it might cost me 5,000,000 tokens. And figure then out which one of us who had a meaningful session actually contributed value. Super hard to to define.
But now that it actually comes with a monetary cost, all of a sudden, everybody's kicking and screaming of, we need to keep this under control again. It's really interesting.
[00:24:58] Barry Luijbregts:
It is interesting, and that is where are we? June? July? So we've had a month of usage based billing now with GitHub Copilot, where previously it was a premium request units basically for free. So now people are kinda tasting what this thing actually costs to use, and some of them are surprised, of course. I get that. And you you already said it. You built some extensions. You have a very nice one that I use daily and I get all my customers to use as well, and that is called AI Engineering Fluency. I'm looking at it right now.
This is in Versus Code and other places as well. Right? Can you maybe talk about that and why you even created? Yeah. It actually started out as the Copilot token tracker
[00:25:48] Rob Bos:
because I hated PRUs, premium regress units. Because it it was always hiding that there's actual cost involved when you talk to an L. Yeah. And I I have been pushing back to the teams on the GitOps side from day one of. I wanna see that. It comes with computation cost. It comes with monetary cost. It has an environmental impact. I wanna show that information. And somewhere in December last year, they did an update to Versus Code, and all of a sudden, they could show the information from your session. So which tools did it call, for example, what happened, how many tokens were used, and finally I had some information that I could run with.
We were still paying in premium request units, so users only saw, hey, I I went down 10 premium request units this session, and that was it. So I start to work with, okay. We have the files on disk. I have Copilot, and I had some some time available to work on things. And because I was triggered off, I wanna see how far I can bring this forward. So literally started as figuring out all of your sessions on disk and just read the log files on your machine. Look into them and then look at things like chat turns, how long was your session, how many tokens did you actually go through.
So that I could summarize that and then projects of, hey. If I look back at the last thirty days, I can actually see how many tokens you've used. There is some fuzzy calculation available on, hey. A token amounts to this amount of computation. This amount of computation, can actually express that in c o two cost, for example. So how much environmental impact that you actually have there. You can do the same thing for water usage for cooling a data center on average just to give an indication of what happens there.
And if you have c o two, then you can also calculate how many trees you actually need to plant to be able to do something about that c o two impact. That's where it actually started off. I wanna have that kind of information available because there are people that say, hello, good morning every day to Opus four point eight. And that has a big impact on the back end of what is happening. And that's still a thing. I would go so far is that as a user, I would like to schedule certain things and keep environmental impact in mind.
So if I have something that runs every single day, I would like to have the option to choose a data center in Sweden where that runs that runs on hydropower and things like that. But we're not still not there yet. So that's where it started in December of, I wanna have some information to show people that this is more than just asking something, a question, and you get an answer back. Notice actual impact hap And, yeah, of course, in good fashion, from there, everything escalated. We dove deeper. We went broader. I've had some colleagues that spend an innovation day on this. So I think by now, six or seven people have contributed.
It's open source, so people can contribute anyway. And we really started to go broader and a lot deeper in certain things as well. So by now, we support a lot of different AI tools that are available. So not just Copilot, ClotCode, for example, is being picked up as well. The new Mistralvibe, for example, that is currently might still be in beta. I'm not sure. It's being found. So all of those different harnesses that you can use and you name it. Even if it's OpenCode, if it's the pyre harness, we pick up the files from there as well.
From that, we can actually start mapping your sessions. So how long was the session? How many chat turns did you actually have there? How much tools did you call in that session? Session? Are you very specific when you do prompting? Do you actually bring in context files and refs, for example, into your session? If you know all that kind of information, you can start to plot that in a chart to give you a little bit of insights, and then we can also give you very fine grained improvement tips there as well. If we see, for example, that Claude's code continuously automatically summarizes your session, that's an indicator that apparently you don't know what that does and what's the risk of compacting, for example, might be.
So we can at least surface that information and say, hey. We see that this happens all the time. Maybe you wanna be a little bit more specific. We already know that compacting also causes a a cash miss in the next turn, for example. So, hey. We're paying for these things. Caching is the thing, and it's super helpful because that brings the the cost down. So we can actually surface that tip-off. Hey. This is what we see happening. Maybe you wanna look at some information around this and get better in the way that you're using these tools.
So, again, this escalated completely and now has a fluency score in there where we look at different aspects of your usage. And we can see, for example, hey, in the the prompt engineering, you're scoring level three, and in the context engineering, you're scoring at level four, for example. And there we can at least give you some guidance.
[00:30:49] Barry Luijbregts:
Super useful. So I'm I'm looking at it. I am in the fluency score. I am stage four AI strategist.
[00:30:57] Rob Bos:
Awesome.
[00:30:59] Barry Luijbregts:
And I don't know if that's good.
[00:31:01] Rob Bos:
That is the highest level. We take people through different aspects. I'm pulling it up on this side as well to make sure that I have them in the right. And per aspect, we try to find if you're between stage one and four. So it starts with an AI skeptic. You use it, but not necessarily that often. And then you can graduate into an AI explorer, an AI collaborator, stage three, and then AI strategy is a stage four. And at that point, we don't really know which extra tips to give you, so that's a a good thing to be at.
But we do that for, as I said, prompt engineering, context engineering. We look at your tool usage, for example. Have you enabled tools? Do you have enabled tools in your editor? The basic ones are always read file or write file, for example. But there's, of course, so much more that you can install in these editors, it's good to see either, a, you haven't been installed, but also, b, if you're actually using them. And then the last two aspects are customization and workflow integration. So across all of those topics, we look at, okay, this is some of the the metrics that we track.
Where are you in your, basically, maturity of using these tools?
[00:32:10] Barry Luijbregts:
Yeah. And so it also shows usage by editor. So cloud code for instance, also sometimes use. I see that show up cloud desktop, co work, and there's all sorts of places. SQL Server Management Studio of course. And you name it. So do you then need to install and test That's all of the fun part. And
[00:32:31] Rob Bos:
it is. I work for a company that is a Microsoft and GitHub partner, but I'm a consultant, so I'm not necessarily married to a specific product. I wanna have the best tool available that we can use both for me and the people that I train. And the only way to find out about those things is install them all. And that's where the fun stuff first starts to happen. So I have accounts on most of these things available on working certain things. Some of them are free, luckily. But, yeah, I've got them all installed.
[00:33:01] Barry Luijbregts:
That is fun. Yeah. So if you let's be honest. If you compare, let's say, Claude code to GitHub Copilot, different harnesses can have the same models.
[00:33:14] Rob Bos:
What do you think? Which one wins right now? Right? Can change It all depends on where your muscle memory comes from. That is the biggest one. So if you've been using cloth for two years, then probably the answer for that person will be cloth. The interesting part is that and I've had those folks in trainings as well. They looked at it at a certain point in in time, and then they ignored it for one or two years, and then they come back. I've had people in the training where they say, hey. This is the first getting started training on Get a Copilot.
I've looked at it, it was crap. And you have two hours to convince me. What do do as a trainer? But still, after two hours, they came back and said, holy moly. I haven't looked at these tools for two years, and apparently, I missed this, that, and this, and I can also do that. It changes so quickly. And that's often how you see these these things being portrayed where somebody says, yeah, but I can schedule something in Claude and it runs every Monday and it does the magic for me. Mhmm. And I say, Copilot can do that as well.
So the harness itself is becoming more and more fluid as well, and it's no longer a unique selling point. It's more of the experience that you have there and that you want to do. And I recently created an overview of these are all the the sort of the surfaces where you can use Copilot these days. It it's a massive slide off. You you can use this from your editor. You can use this from your CLI. You can run this from your editor and hand it over to the Cloud Agent, and then then it can do something autonomously there. You can trigger this from GitHub mobile and then run the session in Cloud Agent. You can actually go to the web and run a session in Cloud Agent. Cloud Agent actually has automations these days, and you can schedule them in for you.
And it all then leads for me to the Copilot app because that thing is just freaking amazing. Let's see how far they got that thing to work. It was released and built a month ago, I would say. But I've have had access to this since March, it is my go to. It is really energetic setup where everything is really focused on I wanna implement something. So the Copilot app has really been configured to say, whatever you're going to do, you're going to create a pull request because that is where the value of an engineer lives, and it's really geared towards all of those things.
That's my go to these days, because the the the way that they think about it is differently. So Versus Code for me, for example, is really interactive. I wanna see the files that are being changed. I want to see the the harness go through the loop that it has, run the test, then maybe validate it from there. And then if I need a pull request from Versus Code, I need to prompt it, create a pull request, or do it myself. In the CLI, I tend to use it mostly for stuff I have on my machine. Find me this file or hey, if I have an issue on my machine, look at the logs, look at the stack trace, go for it, and then just plung through whatever is in there.
And the Copilot app is really I trust that I have a good setup in my repository. I have my testing in place, for example, and then I can really go faster and just have multiple concurrent sessions and say, just go for it. And that gives me so much more power because I have all of the sessions in the same overview, and I can just look at it from there. Whereas with other editors, depending on what they do, before the CLI, for example, you tend to have a single session and a single shell session. So if you wanna do something concurrently, then you need to create two sessions and hope that the harness actually allows you to create a work stream.
So it's different touch points for different editors, for different types of work that I even have. And it is. I use cloth and cloth code and cloth co work. I use Mistral, Mistral code, and Mistral vibe all intermediately because I try to keep up to date of what the heck they're doing. And the interesting part is that everything starts to group together. They they start to build the same UIs for these things. So you see that Yeah. The harness and the way that they they create those things, yeah, everybody's looking at everybody else, and they pick up the the things that work from them.
So going back to your question, I don't necessarily have a preference. I like the way that GitHub approaches things from a developer perspective. Mhmm. And the tools that are building right now are very good, so that tends to be my go to.
[00:37:26] Barry Luijbregts:
Okay. Yeah. Yeah. I I like it too. Right. It's nice. So the extension AI engineering fluency, do I per se need Versus Code to do it? Is it also in Visual Studio? Versus Code, Visual Studio, JetBrains.
[00:37:43] Rob Bos:
There is a NPM package, so you can run it in a shell or a status line, for example. You can build bring it into Omar Posh if you want to. Can bring it into the Copilot CLI. I think I have a script somewhere to do that for Cloth code, for example, as well. So, yeah, you can bring it in lots of different places.
[00:38:01] Barry Luijbregts:
Nice. And has a GitHub Copilot team not yet
[00:38:05] Rob Bos:
knocked at your door to integrate this into their product somehow? Not specifically. Everybody's very enthusiastic, and they're looking at certain things. And you can see that this kind of information is relevant for a lot of folks. So Mhmm. Versus Code has been building the session debugger, for example, into their setup. They've done the same now with Cache Explorer and things like that. So everybody's learning from each other and then building little things into the product as well. So
[00:38:34] Barry Luijbregts:
Okay. Well, let's see what happens. Maybe maybe they'll buy you out or something. That would be cool. Okay. So we have discussed a lot of stuff already here, and also the extension and how you built that type stuff. Maybe we can finish on this. Where can people find your trainings, like online trainings and things like that to learn more and to get up to speed with GitHub Copilot and other stuff? And, of course, your extension and wherever else.
[00:39:08] Rob Bos:
Oh, do you now have another hour? Yeah. Oh, absolutely. Yeah. You can start, and I'm always available on LinkedIn. I contribute a lot to the kids, go to community, for example. So I tend to share a lot of things over there. So that's your go to. I have a blog called devopsjournal.io. So you can find a lot of things there as well, and then definitely check the about page because that is where all of the links are grouped. And, again, from there, it just escalates in so many different directions. So I'm very active on GitHub. You can follow me there as well.
I have several LinkedIn learning courses on GitHub. GitHub with phone security and GitHub with phone security for Azure DevOps, a responsible use of AI with GitHub Copilot and things like that. I've written two books by now. So GitHub actions and action, if you want to dive deeper into the actions ecosystem and how to work with that, and the GitHub Copilot handbook that came out last year. So those are the the big things there. What we've been doing at my employer, Zebia, we've built the tech.zebia.ns site.
And that is a site that groups information around anything Microsoft, but also for Copilot on all of the updates, product updates, editor updates that are available, and we group a lot of information there. And we actually have a newsletter for that. Newsletter of yesterday was, like, 10 different updates from a single day. So if we wanna stay up to date, that would be a very good starting point because I tend to publish my stuff over to that site as well. So tech.zb and other things.
[00:40:47] Barry Luijbregts:
Okay. Excellent. And all of those things will be in the show notes so that you can just go and click on those and go follow Rob and say hi to him. Awesome. Alright. This was very cool. Maybe we can do it again in a year to see how crazy things are by then. Probably. Yes. Let's do that. Alright. Thank you.
Introductions and Rob's DevOps background
GitHub Advanced Security and supply chain risks
Azure DevOps versus GitHub today
Why moving repositories unlocks GitHub features
Why GitHub Copilot is now essential
Training teams to use AI coding tools well
Trusting AI code with tests and validation
Better specs, business analysts, and AI workflows
Usage based billing and the real cost of AI
Building the AI Engineering Fluency extension
Comparing Copilot, Claude, and other AI tools
Where to find Rob's training and resources
