In this episode, I sat down with Anton Sizikov from GitHub to talk about how AI coding tools have evolved from “smart autocomplete” into full-blown agentic workflows. We unpacked the early days of GitHub Copilot, when the biggest customer concerns were privacy, legal implications, and whether the tool could be trusted at all, and compared that with today’s reality of agents, context engineering, cloud-based workflows, and outcome-driven development. Anton shared how his own usage has changed over time -from manually reviewing every suggestion to orchestrating multiple agents in parallel - and why experienced users often get dramatically more value from these tools because they understand how to shape context, control cost, and build the right guardrails around them.
We also dug into the harder questions engineering teams are wrestling with now: how to measure ROI, how usage-based billing changes developer behavior, and why faster code generation doesn’t automatically mean better business outcomes. Anton explained why there is no universal playbook yet for rolling AI out at scale, why teams need to think more like product owners now that building is cheaper and faster, and why engineers who are still on the fence should start experimenting sooner rather than later. It was a thoughtful conversation about where AI-assisted software development is already delivering value, where it still creates friction, and what engineers need to learn to stay effective as these tools keep changing.
- Anton Sizikov on LinkedIn: https://nl.linkedin.com/in/sizikov
- Anton's Blog: https://blog.cloud-eng.nl/
- GitHub Copilot: https://github.com/features/copilot
- GitHub Models: https://docs.github.com/en/github-models/about-github-models
- Visual Studio Code: https://code.visualstudio.com/
[00:00:04] Anton Sizikov:
Oh, hello, Larry.
[00:00:07] Barry Luijbregts:
Hello, Anton. Hey. Good to meet you. This is the first time we actually see each other. Yeah. Thanks for doing this, man.
[00:00:17] Anton Sizikov:
Oh, my pleasure. That was an interesting topic, so, you know, why not to why not to try it?
[00:00:26] Barry Luijbregts:
Yeah. So maybe for the listeners, can you introduce yourself?
[00:00:31] Anton Sizikov:
Absolutely. What do you do? Who you are? Yeah. My name is Anton. I'm based in The Netherlands in a beautiful city that is called Harlem, right next to Amsterdam. I work for GitHub, has been four years already. I'm a solutions engineer. So I support large enterprise customers across, well, North Of EMEA, so that's been a plus all the Nordics, some UK in the past. Those are GitHub customers, Microsoft customers, and we, well, we work a lot with them, improving their SDLC, like the way they build software, the way they host code. And for the past few years, it was mostly, well, a lot at least about AI, gen AI solutions like GitHub Copilot that we have. So been an interesting journey since we started in the beginning. It was basically like a super simple auto complete, and we were really really busy trying to convince people that that thing actually works. Like, yeah, it's it looks weird, it's a surprise, but indeed computer can write code. It was if you look back right now, it was not the best experience ever because obviously models were very simple, the industry itself didn't really know how to tweak those models and how to move them and how to make sure the results are great.
But we even then, we had a few large customers who agreed and trusted us and then joined the, you know, the joined us on this journey. And now, of course, Copilot is a massive beast. There is so much in there. There is a lot of surfaces where you can engage with it. But the question now is not about can I write code or not? The question is mainly about how can we do that at scale? How can we teach everyone? Are we doing it right? Are we doing it wrong? So all these questions, that's what I'm basically being busy with, helping helping everyone and also trying to learn from them as well because it's a new still for for many for many engineers, it's still somewhat new. So, yeah, it's a that's what I do with GitHub.
[00:02:50] Barry Luijbregts:
That's fun. Yeah. That's lots of fun.
[00:02:53] Anton Sizikov:
Absolutely. Yes.
[00:02:55] Barry Luijbregts:
Do you feel like a like a microsofter?
[00:03:00] Anton Sizikov:
Depends. Depends. We are still somewhat separate, so there is a lot the differences in the tooling that we use, the difference in the way we approach some conversations. We do have a lot of customers that are large Microsoft customers and GitHub customers at the same time, so for them, it doesn't matter. Right? So it's the same, well, what we call an account team. It's a mix of just GitHub and a lot of Microsoft employees there. But we have customers that don't really have a lot to do with Microsoft, so they they they mainly, like, could be AWS shop or GCP shop.
And for them, we are they don't really think of us as Microsoft, but they think of us as GitHub as a separate unit. So, yeah, it depends who's asking. But of course, since the acquisition, there is some sort of transition happening. So we recently unified our office tools. So we've been a lot on the Google Workspace or Google Docs, Gmail, but now we are we've we've shifted over to Outlook, so that helps us better collaborate with our Microsoft colleagues because, well, at least we can share calendars now. That that makes much easier. So you feel more and more connected now. Yeah.
[00:04:24] Barry Luijbregts:
Okay. Are you enjoying that Teams and stuff?
[00:04:29] Anton Sizikov:
Well, I've been it's been an experience. Let's put it this way. So you definitely change your habits. You have to, you know Yeah. UX is slightly different. Like, in the end of the day, just email. So that's kind of same thing. Calendars. Yeah. It's a calendar like, obviously, certain things are a little bit different, but we've got quite a good support for the transition. So we had a lot of engineers actively working with us, resolving, helping with the migration, and I don't know. At this point, like, it's been more like five months or so, maybe a little bit less than that.
So, at this point, it's fine.
[00:05:10] Barry Luijbregts:
Okay. Yeah. I was just just wondering what that's like. Okay. So you mentioned already GitHub Copilot in the very beginning. It's a very different thing than than it is now. I've been using it for years now, which makes me a veteran user. And back in the day we had, you know, the the value proposition of it was very different than now because it couldn't do the same things as now. So what do you think it was back then and how did you convince those customers to use it? Was it basically just a replacement for Stack Overflow or something else?
[00:05:47] Anton Sizikov:
Wow, That's that's a trip down the memory line. Yeah. You know, at least for me, because I do work with a lot with like regulated customers and FSIs and banks and, you know, manufacturing. For for them, it was more the biggest concern was privacy, what happens with data, other any legal implications that what it even means, like if you had something that generates your code, what does it mean in terms of the ownership, in terms of the licensing, and what's not. So mostly, I had to go and show that it works. So that was the first thing to do, of course.
That it works not just on the examples. So, like, you you can literally open any code base and just start typing and you're getting those suggestions in. And they, to the most to like, lot of people were surprised to see that, yep, it does it. Of course, back then we had like what we used to call hallucinations. Right? So it's like, it would invent some APIs that don't exist. It would invent some packages that are outdated or like it wouldn't know about the internal packages that every customer or company has.
But the second piece was almost like, are we even allowed to use it? Like, what happens to the data? So there was a lot of time where I had to spend understanding like all the details, like how the data flows, what do we do with it? I think the biggest question was, does it learn? Like, not really. Right? So those models are like, stat they they they have no memory. There is no yeah. It's there is nothing that you can can send to the model and then it's stuck there. So that was a was a bit of it. I wouldn't I wouldn't we never positioned it as a like a replacement for Stack Overflow. It's initially, it was just very, very smart autocomplete that can complete not just the next, you know, method name or function name, but it was actually building your lines of code, like, at the same time. And the way you steer it was very different to the way you steer it right now. So, like, if you speak about autocomplete, you don't do anything, you just type. Right? And you get like all these Yeah.
Lines offered to you and what we call a next edit suggestion. So just picking the the different part of the file that you're editing and saying like, oh, you're introduced to a property. So probably you want to initially start the go to the constructor and put, you know, the past the value in. So these things improve a lot. But no, it wasn't really a replacement for Stack Overflow because you still had to explore and research, like, what what I'm planning to build. And I think, like, for me, the eye opening moment was, like, once we got the preview internally, I felt like, that's interesting. Like, let let me play with it. And I took a weekend.
I had I have a few personal projects that I run for some home automation, for some, like, photo management tool I have. And I just had a few features in mind that I never had time to to to go back to. And I just sat and been coding for the weekend, and I was like, wow, I'm so much more productive now. Like, I can do so much and it just, you know, after a while you're gonna get used to it, and it, like, it just does it for me. That was such a brilliant experience. It got me really pumped and motivated to go and show.
And if I look back at this right now, I was like, oh my god. There was so many old.
[00:09:39] Barry Luijbregts:
Yeah.
[00:09:40] Anton Sizikov:
Yeah. But still felt magical. Right? Just like Yeah. The first time you use a cloud service or something. Yeah. Yeah. At this point, the magic, I think it's gone. Like, now, you at least I understand it very well. Like, what exactly is happening there? What's what's what's in the on the back end plus. Being a GitHub employee, I well, we we promote inner sourcing a lot, so I can see the code base of those tools. If I need to understand something, can just open the repos, spin up Copilot, and ask it to find draw me some diagrams or even find particular pieces in the code that do Plus, something I think the second experience that helped me a lot was to build my own coding agent, like in the CLI coding agent.
It wasn't very it's not like super advanced or anything, but it just helps you to see what's what's behind, and then it helps you to to drive it when you use any of them. Like, it's not just about Copilot. Like, they're all essentially the same insight, like, the same two nested loops, basically. So yeah. The those things help a lot. Like, the more you understand, the less magic you see there and the more, like, okay, it's a tool. It's a tool I can use. I can I can maybe trust it more? So, yeah, it was different. But back then, it was, yeah, true magic. It was like, wow.
[00:10:56] Barry Luijbregts:
Yeah. Exactly. And now, it is a lot better and different as well. It can do a lot It is very diff. A more tools and buttons and bells and whistles. Mhmm. So how do you use it now?
[00:11:12] Anton Sizikov:
I went through, like, many stages, which to my what I call it a surprise, maybe not to my surprise, but I observed the same patterns when I speak to my customers, especially the long term that could've been with us for several years already. So you start with less trust and more control and you read what the output, then of course we got chat, which was also a bit yeah. Currently, if you look at it right now, it's like it wasn't a great experience. But like after, I think the first breakthrough was of course agentic capabilities. Like when we first implemented agentic capabilities, it was a tool we called GitHub Copilot Workspace.
It's not non existent anymore. Was an experiment again. And it was a coding agent in the cloud. Models were like GPT three five or something goes like really weak model, so they didn't have a lot of context. So it was a a on cloud on the experience and it wasn't great, but it gave the idea that it's things possible. And then after, I think oh, I don't remember. I think it was Cursor who got the proper Cursor and Vinsor who got the proper agent flow because they got better models and they jumped on that ship. We were a little bit late to the game, had to fix a few things first, and then got Copilot agent, like, in the agent mode, we used to call it, in in the ID, in Versus Code, and other IDs.
And that was the first, wasn't still great, like it does the job. But I think somewhere around end of last year or mid last year when the models like Sonnet three three seven, three five were released, then you see the power. And after that, like, I completely stopped coding, like writing code manually, that was on purpose, like, you would probably still want to let the agent finish and then go and, you know, do the manual, like, polishing. And that's what most of the engineers been doing back then. But my goal is to understand the art of possible, like how far I can push it. So I decided like, okay, I'm not writing any code manually anymore and just do with the agent, like learn how to bend it basically to to make sure it does what I want to do. And then I think like by the end of the year when like we we were we went into this really well, not we, we as an industry, not just GitHub, into this mode where the new frontier model is being released every what is it now? Like, every couple weeks or so. Right? Yeah. Yeah. So it's just insane how fast those things develop.
And every single release was like felt like, wow, magic. Cool. It's so much better. I have I don't have to control it as much. It it knows a lot, and it gets better in using the tools and exploring the code base. But I already knew how to to push it because it was my personal goal. Not do not write anything manually, make sure the agent can do it for you. So I believe it was a very useful experience because all those engineers who's been using those tools for a long time, at this point, they're super good at it. Like, they they understand how to fix those little flows, how to provide tech, what we call right now, context engineering. Right? Like how to make sure that you give enough information that is good and useful for the model and as as as little as possible, so you maintain the context.
Right? We know that, yeah, if it's too much in the context, it's not gonna be the great output if it's it's something distracting. So you have to and and and the context windows were very small back then. So you would have to, like, manually even, like, drag in files or something. And now it's not the case anymore. So I think somewhere around the the end of last year, I started moving towards the CLI experience. Still wouldn't say that it's my favorite. I don't yeah. I miss a lot in the CLI, but that was a moment where you realize that I have more trust to the like, I trust the tool now, so I can afford not to look at the every single piece of code that is that the tool generates for me, but rather, like, work in a more like outcome driven way. Like, I want that to happen in my code base, in my application, or even in experiment when which I built.
And only then, like, run the code reviews, run the security tools to make sure that it's fine to to to merge in and and and deploy. And that that was like a a shift. Right? But now with more visual tools like we have in Versus Code, we have Versus Code sessions or agents. It was renamed to agents now, and the GitHub Copilot app. So Copilot app is probably the the one I'm using 99% of the time Mhmm. At at the moment. I love the progress there. And that's that gives me like this combined experience of what I like about CLI, that it's a focus on the process and the outcome.
And what I was missing from the ID perspective, like everything gets integrated, you can easily switch to see the codes, you can, like, there's a button, just open me open it in in Versus Code or another editor and just, you know, run the debugger and test it and, you know, do do like get more into the you know, get your hands dirty. So yeah. So there were like periods and transitions. And the same I see with with the customers, like when we deploy it at scale and then we help them monitor and understand, we see the same. The users that are less experienced, they tend to stick more to typing code, running agents and double checking and verifying and manually editing.
Then there is like a second tier where I would say, or phase, let's be let's call it the phase, where engineers are mostly trying to rely more on on agents. And you can I mean, we don't know what exactly they do? Like, we don't collect prompt and suggestions, so we don't know. But when you look at the patterns, you almost like see from the telemetry that customer show me, oh, this is what we have from GitHub. Like, can you explain us? Can you explain us? And you're gonna start recognizing the patterns. You see that, look, that person is doing this more organic foreclose.
And you can see the outcome. Like if you cross check with other metrics, they produce more code, they produce more pull requests, they produce more releases, they close more tasks. They actually have more time to do fixing the tech debt. So and then the next stage is like running those in parallel. And this is where you need orchestrators, where you need cloud cloud agent when you more rely on the integrations between tools. So, like, the typical pipeline for me would be, I've got a plan, I've got an idea, spin up a few agents to run, make sure they cross check each other, get me a pull request, get the code review done, get the quality controls done.
And on after that, I'll take a look on the code to, like, at the code. Locally, like, I will test that it works. It does the job for me, and all my tests are passing and green, but then I will just scan the code and see what's happening. So there's also like the the the journey is how how big the feature should be. I don't like big features to be implemented at once. I still want to know, you know, what what's happening with my code. So yeah. It's it's shifting. I don't know, like, six months from now, it would could be completely different.
Still not not used to I don't like this idea of, you know, like remote sessions where you check on your phone the the status and your code there. I think I think we need a little bit of time for ourselves away from computers. But I do have agents running in the background on my repos and doing something, and then I come back and check on the results. So I'm kinda partially there, but not yet, and maybe I don't want yet completely to be there. So so you can see it's it's definitely a journey. Yeah. It's it's fun. It's so interesting to observe, like, being in the first, you know, row front seat, see where we go, like, the internal experiments that we run. Some of them are they're not going anywhere, like, we don't we realize that we don't wanna it's not it's not a product. And sometimes it's like, yes, yes, yes, yes. Ship it, please. And we have to wait. It's a bit annoying.
[00:20:04] Barry Luijbregts:
Yeah. And and that's good. I see that you do sometimes deprecate things, which is good. I saw, for instance, GitHub models, which was a preview feature in GitHub. That's now gone away or going away. And that's good. You need to say goodbye to things that aren't working. And to just try things out, see if they are working. So, could the next phase perhaps be human out of the loop? You just feed the thing requirements and you test it?
[00:20:34] Anton Sizikov:
Hope. We'll see. At this point, I gave up on predicting future. So many, everything is moving fast. So just just about the deprecations, I would say, yeah, we we try to deprecate things that are not really going anywhere or doesn't look like a product to us. Models was an interesting beast. I actually like the product, but it was mainly like a gateway to to show the world what what those models are capable of, how to test, how to eval them, how to interact, how to compare. But we're we have it was a front, like a proxy to Asia services anyway. So like if you want to run it in production, yeah, then you need to switch to something like Foundry and then get your models deployed there. And there is a lot of tooling.
So that's also about like, should we build the same tools that exist in another product that all is owned by the same company? But I think it was a good gateway for many well, I don't know numbers unfortunately, but some some customers, some engineers I talked to, yeah, they used it. I did. All my AI experiments, I tend to spin up something on GitHub models and just test it there and then play with them. Actually, thank you for reminding me. I need to update one service that still relies on on models. I I have it Okay. Like, local bot for my like personal automation that I have a Telegram bot that kinda helps me with certain things. And it's updated. Oh, yeah. Good. I'll I'll make a note. Yeah. And tell your agents. Yeah. Yeah. And about the future, I don't know what I see, especially like internally Microsoft and GitHub teams that are very much invested. So we have to talk food, We have to make sure that we we test it all and and see the limit. I think some people say the best perk that we have is the the animated tokens. Right? So we can just spend as much as you as you want and learn because we have to be confident like what we're talking about. We have to try.
So we do have teams that are very much in like into it. So they build this almost what sometimes called like a yeah. Software factories or they they they trying to see that what can we offload. And I think the biggest, at least as of today, the biggest understanding is that you technically can build almost everything. And the neck the challenge is what what exactly you're going to build. Because when you can do almost anything and with a very short time span, because the agent can run overnight and just get you something when you wake up, get it ready for you. So do you then then we always have to ask ourselves, you really need this feature? Do you is it something that you're going to keep in your app, in your in your in your your code base? Like, it's cheap now, so you can throw it away easily.
But still, like, is it is it worth being implemented? And we see that some engineering teams, they became more like agent experienced engine platform engineers. So they build everything that the agent needs to be successful, which is quite fun and interesting. But it shifts, like, they they still ship a lot of features. Like, look at our release page. Look, change look. Like, what is like? Yeah. 10 releases per day. Look at the tooling, like CLI or GitHub Copilot app or Versus Code, they're releasing daily to two times a day, sometimes three times a day.
And they it's not just one bug fix, it's you read the change log, it's like five, seven entries in there. Yeah. It's crazy. Yeah. Yeah. But the the only way they can do it at this space is by building a lot of infrastructure around around it. So that when they have an idea of the feature, they can almost delegate that idea to the agent to a certain degree. You need to plan, of course, you need to provide information, you need to provide maybe like a basic idea like what it should and where it needs to take, you know, like what kind of code it needs to modify, what data is going to be needed for that. Definitely, there is a lot of guardrails in there. But still, once you get that foundation, you can see that the speed increases too much. Sometimes too much. Yeah. So it increases, but sometimes it's just difficult to keep up. So I think the typical example is doc docs. Click.
Yes. So we still need to release docs. We still need to keep them up to date. Partially, it's possible to automate, but nobody wants to read AI generated documentation pages. Yeah. You want to have something that is written by a human with a lot of experience in in in technical writing, and that's where, like, you put a lot of pressure on something that is happening after the development. So your infra, your users actually getting lost sometimes, so I talk to engineers and they say, like, yeah, it's too many features. I don't understand. Like, how do I keep up? It's hard.
So yep. It's a it's human is not going to be taken away, I I I think. But humans are shifting from a bit of like a manual labor when you just sit and type a lot to more abstract or more high level and even like product mindset is needed. So it's it's it is going to be interesting. So for what it will look like in the future, I can't predict, but what I see, I like so far.
[00:26:20] Barry Luijbregts:
Yeah. Yeah. It's a very interesting times. You touched on this a couple times already, which is the value we can get out of these AI tools. Like we're talking about personal productivity here. We get the copilot. Yeah. What I hear from companies and customers a lot, because this stuff is not cheap, it's not for free. Like, is this actually creating value for us and how do we measure that especially? Like I have a customer that's doing a POC with GitHub Copilot and then they literally have a guy do a task without GitHub Copilot, a development task, write down how long it takes, and then one with GitHub Copilot,
[00:27:01] Anton Sizikov:
write down how long it takes. And that's the metric that they use there. Is that the same person building it twice? Yeah. Yeah. That's not a fair experiment. Right? So it's not clean. Well, you know, if if anyone builds like a proper way to measure developer productivity that that it it it you know, it's a good business, definitely. And so far, I didn't see like great examples. Like, yeah, there are plenty of like DoraMetrics, SAFE frameworks, like a lot. But still, there is something that, well, first of all, humans don't like to be measured.
And if you start measuring them, they will play the game and will optimize for that metric. So if you ask them, you know, how many pull requests you create and and you start measuring and ranking them, oh, there will be a lot of pull requests, very small ones. Or, you know, so it's it it it's a difficult game. Yeah. Absolutely. In the past, like in the premium request model, AI was basically free. Like, it was super cheap. Yeah. Now but that was very good for for many engineers to actually learn and see the like how far you can go, what you can do.
With us moving to usage based billing with more other vendors doing the same, you can see this is where the industry is going. So there there is time to actually be mindful about what you built. An example, like, yes, you can run AI agent to give you, I don't know, a summary of to summarize. I don't know. For example, like, in my job, I have to read a lot of change logs, like what are we shipping? You can run an agent and and scrape the file, the the change log website and get your summaries, but you can write a script that would just give you like, you know, a text to like the the header and maybe like a few lines, it's gonna be the same. Like, is it interesting or not? Do I need to read it?
So for sure, we don't have to replace every thing that we do with with AI tools, even though it's possible and fun. We don't have to forget, you know, we still have computers that are very fast and efficient and extra cheap. You can write an algorithm sometimes. But AI can make decisions, can drive more complex flows. Maybe something that you have inside your code base and you do certain things, maybe write a script and point your agent to the script and just do it. It's faster and more reproducible and reliable. So for sure, now when you have to pay for for for the actual consumption, You need to be like, we we have that. We we have conversations coming back again, like how much exactly are we saving?
And it's always difficult because you cannot say that with this tool, you are going to save 25%. Because 25% of what? What are you measuring? What's important to you? For my big, very large manufacturing customer, for them, the frequency of releases is not something they're interested in because they need to build a machine, put the software on it, and ship it to the customer, to their customer. But you do that once a year, and then you need to make sure that that software works. So for them, it's a quality, of tests, amount of lab resource that they, you know, like like what they do.
But for some sort of retail customer or that that build the ship software many times a day and updates and fixes for them, speed to market, time to market, the speed, the frequency of releases, the ability to react on the world that is changing, that is important. So even in in that case, like, you compare those two and for them, the the ROI is measured in different units. So it it is always a challenge. Like, I always start with what exactly do are you trying to improve in terms of yeah. You don't need more code. You need well, as a business, need to earn more money.
Need to provide value to your customers. You need to do something. And what's what's in it for you? It's probably not the time the developer have saved. It's the value they bring back. Maybe they build more features. Maybe they actually your biggest problem is your stability or your older old bugs that never been addressed. Maybe you have, like some banks, have outdated software written in 2090. Yeah. You know, like in eighties or something, in seventies. So they need to update it because nobody understands it and you can just go into it and then make it yeah. It's not gonna bring you new features, but it will fix your future, mitigate the risks that you have.
So it's it's a challenge. It's a challenge how to measure. And mostly the challenge is not that we like the tool is not providing the value, it's about how you use it. Like I was talking to one engineer who's been using like AI productivity agent locally, so it wasn't a copilot to generate code, but it was to check their email, to give them, you know. And there was shocking number, like something like $1,800 per week spent on tokens, which is insane. A bit of work optimization, it went down dramatically like a couple hundreds per month.
And then same value. It's it's it's not the same tool, same value, different cost. Right? So then your productivity didn't change after the optimization, but your your your productivity ROI per per token spent, per dollar spent has increased a lot. So it's hard. Yeah. But it it it there are many dimensions here in how how you measure it. Obviously, it works. Like, there is no doubt. Like, everybody understand that you can do. But now, back to my previous observation is that you need to think about, do you need to do that? Does that bring you anywhere?
And it's not about just features, it's also about the value of that. Is it yeah. You can build a dashboard for something. Do you need it? You can do, you know, a fun thing in your application, but does it bring you anything? You can you you need to be every engineer has to have this almost like product or entrepreneurial mindset. Like, what are we doing here? What what because you have you you can do a lot now. So that's that's that's a challenge.
[00:34:04] Barry Luijbregts:
Yeah. It's a big shift when now you can create anything that you can actually think of right then and there. And so what? What are you going to do with that power basically? Yeah, yeah. So you just touched on ROI, so costs. We we know now GitHub Copilot also does uses based billing like basically all the other vendors as well, right? So we we charge per token here. So how can people optimize what comes out and how much they pay? Because there's lots of little tips and tricks.
[00:34:36] Anton Sizikov:
Oh, absolutely. Well, I actually have like even a full training on that that we give to to all the customers. It's public. But, obviously, I think the example is that productivity tool is the right example. So you can do the achieve the same results with with for a different cost. And that's where I think engineers who've been using tools for a long time, they are in advantage because they have that understanding on how to do more with less capable tools because they had those physical restrictions. Like, you can only put as much into your context window.
You can only you need to be clear and in in in your prompts. You need to be so you need to have that two skill, like one skill is prompting. It's not as relevant anymore as it used to be, but still a good good skill to have. Context engineering is like, what are we going to put into the context? That is super critical because you can get your agent distracted, you can get your agent lost, and then it will recover. So now they will recover, but there will there would be cycles wasted. And I think the other part is make sure that you provide the good tools for your agents, like replace something with a script, so I'm gonna make an API call.
Store data somewhere, like if it's if you're not talking about just coding, like, store it and and review everything that is happening. So I think that's also important. And then that's kind of like a mindset. So you need to understand how the tool works and what's behind it. It's it's a so it's super complicated, but it's very interesting. What's what's behind? What what's under the hood? And after that, yeah, you have like simple solutions that what the model you're going to use. Do you really need that model that was released yesterday to get to provide you a git commit summary. Right? You know? Commit message.
Maybe not. Do you really need to use the cheapest model ever to make something complex and and spend 25 cycles on on on achieving the the result while you can use an expensive model and get it in one shot. Maybe like one, two, three cycles instead. So, you know, it's it's it's a balance, a fine balance, and it's experience a lot. It's very difficult to give recommendations because three, five weeks from now, we'll get new models again. But but still, it's it's important to to to know, to understand, and then to get this, like, what are you giving to the model?
An amount, like, it's the amount of the data you sent and you don't really need a lot. Like my biggest, well, it's not the biggest, but a concern that I have with models that now take like up to 1,000,000 tokens as a context, which is great. But at the same time, it gives you the opportunity to put so much information that that is not relevant for every task. So keeping that in mind is is important. And then there are many other techniques that are being done inside harnesses, so it's maybe because it's not something that engineers, like software engineers that build code with all those agents, they need they don't need to ideally, they don't need to think about all of those aspects. Instead, they need to focus on the software that they're building. And we're not there yet.
We're still in in in in the middle. It's almost like you remember, what is it, like, fifteen ish years ago when Kubernetes started. Yeah. Everybody had to learn it, like all the KubeCTL commands and how you build and deploy. After a while, most of the companies they've been using managed Kubernetes with a layer on top of it, which was built by platform teams. And now, when you build software, just spin up and you repo from a template. You've got, you know, put some configuration in there, and then your buy build pipeline will prepare images, prepare Helm charts, will prepare everything, and it is just going to be shipped. You don't need to know as much of Kubernetes anymore as an engineer to use and deploy there. So I think it's almost the same we'll get eventually where most of those decisions would be taken away. But for now, you need to you need to know what what exactly are you is happening.
So that is important. And then, unfortunately, we have a lot of on the market, those tools that almost like a silver bullet kind of tools. Like, you just put the skill in there and your token consumption is going to All this markdown stuff around the chip. Yeah. But unfortunately, everything that you know, if it's such a simple solution, and it's very competitive market, this market is insane in terms of the competition and everybody needs their share of users. So ask yourself why why wasn't it embedded like they they would have embedded it already into the harness if it was something that really makes the difference in a positive way. So but, yeah, everybody wants it's obvious that the desire here from the engineers is to get rid of that problem. Like, make it solved for me. I need to focus on something else. Still, there is a lot. There are a lot of engineers that want to be almost like knee deep into those details and understand everything.
And they find a lot of joy like I do. But it's unfortunately, or fortunately, not for everybody. Like, somebody wants to build cool mobile apps and to make sure that they're beautiful and users are happy and they don't want to spend time setting up, you know Yeah. The tools. Yeah. The tools and then build extensions and and and then customize them. So definitely, we need to move into the area where it's a it's not a tool that you've spent a lot of time configuring, it's a tool that just works, and then you provide just anything that is needed for to achieve your task.
But for now, we need we we we we need to have that understanding, and we need to come up with best practices, and we need to train each other. So
[00:41:08] Barry Luijbregts:
Yeah. But I I think we're already pretty far along as in Oh, absolutely. We do have it. Yeah. We have a text box. You just type something in there, and it does what you want. Sometimes it's, you know, maybe too expensive. But, yeah, it's also thinking yourself about, is this an automation? Or is this something the LLM should actually do? And is it worth it for the LLM to do this? So co co pilot in this case. That's, you know, we need to retrain ourselves with that as well. And we're all learning this, right, altogether, because it's still very, very new.
[00:41:42] Anton Sizikov:
Yep. That's my favorite question is, like, can you give us the best practices on how to implement it at scale? It's like, well, I have bad There news for is no such thing yet. We'll get there. Eventually, we'll learn what works, what doesn't. But embracing that means that you as a as an organization like engineering leader needs to be also creative and maybe even take a bit of a risk and be ready to back up. You know, step back and like, okay, okay, that doesn't work. We need to solve it in a different way.
I Nobody is alone. We we all have a lot of brilliant minds trying to crack that nut, trying to understand how, what it should look like. Sometimes I said there are some visionaries that are too far ahead and their ideas are not taken yet. Then four or five months later, we think, oh, actually, was right. That's how we need to do it. So I don't know. We'll get there. We'll get there for sure. If there is a problem, we as a human kind, can if we put a lot of effort, enough effort, we solve it. And it is obviously something that we want to solve, so it will be solved. I have a lot of faith in it.
[00:42:56] Barry Luijbregts:
Okay. Cool. So we've been talking for a while already. Maybe we can kinda end on this. So let's say, if there are any engineers out there that are still on the fence with this stuff, maybe they think, you know what, this is not something that I need to use. What would you tell them? How could they get started?
[00:43:18] Anton Sizikov:
Well, first of all, it's a difficult task right now to find such an engineer who never ever ever used anything. I don't like putting that as a, you know, like, triggering FOMO, like Mhmm. Triggering any kind of anxiety about it. I do believe that it's inevitable. Like, you will be using those tools in one shape or another, maybe not something that is advanced and frontier because some compliance or specific type of software you're building, then you will have something less capable, but it's still going to be there, because we see it works, and it does a good job.
So you would have to learn it at at one moment, you know, one way or another. Yeah. It's it's it's a difficult advice. It's like, how do you start? Just start using because so many surfaces where you can get access to it. I understand the the the biggest restriction is that some engineers, they have no access to those tools because of many reasons that, you know, I don't have a good advice for them, like, rather than maybe in your personal time, you know, get a ChatGPT subscription and just play with it. Just see what it can give you back.
Because it once you get, like, a basic understanding, it doesn't matter what tool it is. You know the limits, you know how to drive it, you know when to trust and when not to trust and and verify. That's already a bigger big help. I hope education is catching up. I haven't been talking to like many universities in the the nearest like recently, but I hope that education is going to put a lot of effort into it and we, like schools and universities, are going to teach more of those skills. It's a bit of a challenge that a teacher needs to be good enough have a good enough understanding in order to teach.
So it's yeah. But it's almost like, you know, using a computer. At some point, it was became a must think that you need to be able to know how to use it. Although a lot of people are using just their mobile phones right now and they're happy, but it's still a skill that you need to learn. So those things are going to be more or they're gonna be everywhere, and I I I believe they're going to be less demanding on the skill of the user, but that's a general, like, population. But for engineers, like software engineers, you need to know your tools, you need to know the technology, you need to understand the limits, you need to be able to drive it to to to squeeze the best outcome, results out of it. So, yeah, it's it's definitely something that we all need to to be learning.
I don't know. I don't want to recommend people switching jobs just to get access to those But it's definitely something that we will probably see in the, you know, in the on the market where companies are going to advertise that you are going to get access to those tools. You're going to have a large budget on tokens. You're going to have tasks that are related to AI technologies. We'll probably see that already in there. And and it's been like that in the past, like if you have some advantage because engineers joining companies, they do that for two reasons.
Yep. They want to do something, but they also have to think ahead like, what's next? Like, after that job, what am I going to be in terms of like my employability, my how attractive I am going I'm I'm going to be on the markets in three after three years at this company. And I think the direction is clear right now. So you need to choose wisely.
[00:47:27] Barry Luijbregts:
That's good advice. Okay. Cool. Where can people find you? You have an awesome blog, for instance. Other places as well?
[00:47:38] Anton Sizikov:
At this point, yeah. My blog and my LinkedIn profile are the two things. I do have a X or Twitter account, but it's a bit abandoned now. Mhmm. That's that's another result of all these new technologies. A lot of stuff, you you do it for work. You need to learn a lot, and then it takes time. So, yeah, probably my blog is going to be the best way to to to stay up to date. I I don't post very often, but I when I do, I try to do some good stuff in there. So it's good to be blog at cloudeng.nl. It's yeah. At some point, like, I was very interested into all these cloud technologies.
Yeah. Ops in there, and I wanted I got the domain name, I wanted to build some sort of consultancy business around that. Well, look at me, didn't happen. Yeah. But the domain name is with me. So
[00:48:38] Barry Luijbregts:
Okay. Excellent. So this will be in the show notes for listeners. You can go to the blog. Check it out. There's some good articles on there already, and then you can subscribe there for new stuff when it comes out.
[00:48:51] Anton Sizikov:
Thank you very much, Anton, for hanging out with me. Thank you, Barry. Yeah. Thank you. It was a big fun. All the good questions.
Oh, hello, Larry.
[00:00:07] Barry Luijbregts:
Hello, Anton. Hey. Good to meet you. This is the first time we actually see each other. Yeah. Thanks for doing this, man.
[00:00:17] Anton Sizikov:
Oh, my pleasure. That was an interesting topic, so, you know, why not to why not to try it?
[00:00:26] Barry Luijbregts:
Yeah. So maybe for the listeners, can you introduce yourself?
[00:00:31] Anton Sizikov:
Absolutely. What do you do? Who you are? Yeah. My name is Anton. I'm based in The Netherlands in a beautiful city that is called Harlem, right next to Amsterdam. I work for GitHub, has been four years already. I'm a solutions engineer. So I support large enterprise customers across, well, North Of EMEA, so that's been a plus all the Nordics, some UK in the past. Those are GitHub customers, Microsoft customers, and we, well, we work a lot with them, improving their SDLC, like the way they build software, the way they host code. And for the past few years, it was mostly, well, a lot at least about AI, gen AI solutions like GitHub Copilot that we have. So been an interesting journey since we started in the beginning. It was basically like a super simple auto complete, and we were really really busy trying to convince people that that thing actually works. Like, yeah, it's it looks weird, it's a surprise, but indeed computer can write code. It was if you look back right now, it was not the best experience ever because obviously models were very simple, the industry itself didn't really know how to tweak those models and how to move them and how to make sure the results are great.
But we even then, we had a few large customers who agreed and trusted us and then joined the, you know, the joined us on this journey. And now, of course, Copilot is a massive beast. There is so much in there. There is a lot of surfaces where you can engage with it. But the question now is not about can I write code or not? The question is mainly about how can we do that at scale? How can we teach everyone? Are we doing it right? Are we doing it wrong? So all these questions, that's what I'm basically being busy with, helping helping everyone and also trying to learn from them as well because it's a new still for for many for many engineers, it's still somewhat new. So, yeah, it's a that's what I do with GitHub.
[00:02:50] Barry Luijbregts:
That's fun. Yeah. That's lots of fun.
[00:02:53] Anton Sizikov:
Absolutely. Yes.
[00:02:55] Barry Luijbregts:
Do you feel like a like a microsofter?
[00:03:00] Anton Sizikov:
Depends. Depends. We are still somewhat separate, so there is a lot the differences in the tooling that we use, the difference in the way we approach some conversations. We do have a lot of customers that are large Microsoft customers and GitHub customers at the same time, so for them, it doesn't matter. Right? So it's the same, well, what we call an account team. It's a mix of just GitHub and a lot of Microsoft employees there. But we have customers that don't really have a lot to do with Microsoft, so they they they mainly, like, could be AWS shop or GCP shop.
And for them, we are they don't really think of us as Microsoft, but they think of us as GitHub as a separate unit. So, yeah, it depends who's asking. But of course, since the acquisition, there is some sort of transition happening. So we recently unified our office tools. So we've been a lot on the Google Workspace or Google Docs, Gmail, but now we are we've we've shifted over to Outlook, so that helps us better collaborate with our Microsoft colleagues because, well, at least we can share calendars now. That that makes much easier. So you feel more and more connected now. Yeah.
[00:04:24] Barry Luijbregts:
Okay. Are you enjoying that Teams and stuff?
[00:04:29] Anton Sizikov:
Well, I've been it's been an experience. Let's put it this way. So you definitely change your habits. You have to, you know Yeah. UX is slightly different. Like, in the end of the day, just email. So that's kind of same thing. Calendars. Yeah. It's a calendar like, obviously, certain things are a little bit different, but we've got quite a good support for the transition. So we had a lot of engineers actively working with us, resolving, helping with the migration, and I don't know. At this point, like, it's been more like five months or so, maybe a little bit less than that.
So, at this point, it's fine.
[00:05:10] Barry Luijbregts:
Okay. Yeah. I was just just wondering what that's like. Okay. So you mentioned already GitHub Copilot in the very beginning. It's a very different thing than than it is now. I've been using it for years now, which makes me a veteran user. And back in the day we had, you know, the the value proposition of it was very different than now because it couldn't do the same things as now. So what do you think it was back then and how did you convince those customers to use it? Was it basically just a replacement for Stack Overflow or something else?
[00:05:47] Anton Sizikov:
Wow, That's that's a trip down the memory line. Yeah. You know, at least for me, because I do work with a lot with like regulated customers and FSIs and banks and, you know, manufacturing. For for them, it was more the biggest concern was privacy, what happens with data, other any legal implications that what it even means, like if you had something that generates your code, what does it mean in terms of the ownership, in terms of the licensing, and what's not. So mostly, I had to go and show that it works. So that was the first thing to do, of course.
That it works not just on the examples. So, like, you you can literally open any code base and just start typing and you're getting those suggestions in. And they, to the most to like, lot of people were surprised to see that, yep, it does it. Of course, back then we had like what we used to call hallucinations. Right? So it's like, it would invent some APIs that don't exist. It would invent some packages that are outdated or like it wouldn't know about the internal packages that every customer or company has.
But the second piece was almost like, are we even allowed to use it? Like, what happens to the data? So there was a lot of time where I had to spend understanding like all the details, like how the data flows, what do we do with it? I think the biggest question was, does it learn? Like, not really. Right? So those models are like, stat they they they have no memory. There is no yeah. It's there is nothing that you can can send to the model and then it's stuck there. So that was a was a bit of it. I wouldn't I wouldn't we never positioned it as a like a replacement for Stack Overflow. It's initially, it was just very, very smart autocomplete that can complete not just the next, you know, method name or function name, but it was actually building your lines of code, like, at the same time. And the way you steer it was very different to the way you steer it right now. So, like, if you speak about autocomplete, you don't do anything, you just type. Right? And you get like all these Yeah.
Lines offered to you and what we call a next edit suggestion. So just picking the the different part of the file that you're editing and saying like, oh, you're introduced to a property. So probably you want to initially start the go to the constructor and put, you know, the past the value in. So these things improve a lot. But no, it wasn't really a replacement for Stack Overflow because you still had to explore and research, like, what what I'm planning to build. And I think, like, for me, the eye opening moment was, like, once we got the preview internally, I felt like, that's interesting. Like, let let me play with it. And I took a weekend.
I had I have a few personal projects that I run for some home automation, for some, like, photo management tool I have. And I just had a few features in mind that I never had time to to to go back to. And I just sat and been coding for the weekend, and I was like, wow, I'm so much more productive now. Like, I can do so much and it just, you know, after a while you're gonna get used to it, and it, like, it just does it for me. That was such a brilliant experience. It got me really pumped and motivated to go and show.
And if I look back at this right now, I was like, oh my god. There was so many old.
[00:09:39] Barry Luijbregts:
Yeah.
[00:09:40] Anton Sizikov:
Yeah. But still felt magical. Right? Just like Yeah. The first time you use a cloud service or something. Yeah. Yeah. At this point, the magic, I think it's gone. Like, now, you at least I understand it very well. Like, what exactly is happening there? What's what's what's in the on the back end plus. Being a GitHub employee, I well, we we promote inner sourcing a lot, so I can see the code base of those tools. If I need to understand something, can just open the repos, spin up Copilot, and ask it to find draw me some diagrams or even find particular pieces in the code that do Plus, something I think the second experience that helped me a lot was to build my own coding agent, like in the CLI coding agent.
It wasn't very it's not like super advanced or anything, but it just helps you to see what's what's behind, and then it helps you to to drive it when you use any of them. Like, it's not just about Copilot. Like, they're all essentially the same insight, like, the same two nested loops, basically. So yeah. The those things help a lot. Like, the more you understand, the less magic you see there and the more, like, okay, it's a tool. It's a tool I can use. I can I can maybe trust it more? So, yeah, it was different. But back then, it was, yeah, true magic. It was like, wow.
[00:10:56] Barry Luijbregts:
Yeah. Exactly. And now, it is a lot better and different as well. It can do a lot It is very diff. A more tools and buttons and bells and whistles. Mhmm. So how do you use it now?
[00:11:12] Anton Sizikov:
I went through, like, many stages, which to my what I call it a surprise, maybe not to my surprise, but I observed the same patterns when I speak to my customers, especially the long term that could've been with us for several years already. So you start with less trust and more control and you read what the output, then of course we got chat, which was also a bit yeah. Currently, if you look at it right now, it's like it wasn't a great experience. But like after, I think the first breakthrough was of course agentic capabilities. Like when we first implemented agentic capabilities, it was a tool we called GitHub Copilot Workspace.
It's not non existent anymore. Was an experiment again. And it was a coding agent in the cloud. Models were like GPT three five or something goes like really weak model, so they didn't have a lot of context. So it was a a on cloud on the experience and it wasn't great, but it gave the idea that it's things possible. And then after, I think oh, I don't remember. I think it was Cursor who got the proper Cursor and Vinsor who got the proper agent flow because they got better models and they jumped on that ship. We were a little bit late to the game, had to fix a few things first, and then got Copilot agent, like, in the agent mode, we used to call it, in in the ID, in Versus Code, and other IDs.
And that was the first, wasn't still great, like it does the job. But I think somewhere around end of last year or mid last year when the models like Sonnet three three seven, three five were released, then you see the power. And after that, like, I completely stopped coding, like writing code manually, that was on purpose, like, you would probably still want to let the agent finish and then go and, you know, do the manual, like, polishing. And that's what most of the engineers been doing back then. But my goal is to understand the art of possible, like how far I can push it. So I decided like, okay, I'm not writing any code manually anymore and just do with the agent, like learn how to bend it basically to to make sure it does what I want to do. And then I think like by the end of the year when like we we were we went into this really well, not we, we as an industry, not just GitHub, into this mode where the new frontier model is being released every what is it now? Like, every couple weeks or so. Right? Yeah. Yeah. So it's just insane how fast those things develop.
And every single release was like felt like, wow, magic. Cool. It's so much better. I have I don't have to control it as much. It it knows a lot, and it gets better in using the tools and exploring the code base. But I already knew how to to push it because it was my personal goal. Not do not write anything manually, make sure the agent can do it for you. So I believe it was a very useful experience because all those engineers who's been using those tools for a long time, at this point, they're super good at it. Like, they they understand how to fix those little flows, how to provide tech, what we call right now, context engineering. Right? Like how to make sure that you give enough information that is good and useful for the model and as as as little as possible, so you maintain the context.
Right? We know that, yeah, if it's too much in the context, it's not gonna be the great output if it's it's something distracting. So you have to and and and the context windows were very small back then. So you would have to, like, manually even, like, drag in files or something. And now it's not the case anymore. So I think somewhere around the the end of last year, I started moving towards the CLI experience. Still wouldn't say that it's my favorite. I don't yeah. I miss a lot in the CLI, but that was a moment where you realize that I have more trust to the like, I trust the tool now, so I can afford not to look at the every single piece of code that is that the tool generates for me, but rather, like, work in a more like outcome driven way. Like, I want that to happen in my code base, in my application, or even in experiment when which I built.
And only then, like, run the code reviews, run the security tools to make sure that it's fine to to to merge in and and and deploy. And that that was like a a shift. Right? But now with more visual tools like we have in Versus Code, we have Versus Code sessions or agents. It was renamed to agents now, and the GitHub Copilot app. So Copilot app is probably the the one I'm using 99% of the time Mhmm. At at the moment. I love the progress there. And that's that gives me like this combined experience of what I like about CLI, that it's a focus on the process and the outcome.
And what I was missing from the ID perspective, like everything gets integrated, you can easily switch to see the codes, you can, like, there's a button, just open me open it in in Versus Code or another editor and just, you know, run the debugger and test it and, you know, do do like get more into the you know, get your hands dirty. So yeah. So there were like periods and transitions. And the same I see with with the customers, like when we deploy it at scale and then we help them monitor and understand, we see the same. The users that are less experienced, they tend to stick more to typing code, running agents and double checking and verifying and manually editing.
Then there is like a second tier where I would say, or phase, let's be let's call it the phase, where engineers are mostly trying to rely more on on agents. And you can I mean, we don't know what exactly they do? Like, we don't collect prompt and suggestions, so we don't know. But when you look at the patterns, you almost like see from the telemetry that customer show me, oh, this is what we have from GitHub. Like, can you explain us? Can you explain us? And you're gonna start recognizing the patterns. You see that, look, that person is doing this more organic foreclose.
And you can see the outcome. Like if you cross check with other metrics, they produce more code, they produce more pull requests, they produce more releases, they close more tasks. They actually have more time to do fixing the tech debt. So and then the next stage is like running those in parallel. And this is where you need orchestrators, where you need cloud cloud agent when you more rely on the integrations between tools. So, like, the typical pipeline for me would be, I've got a plan, I've got an idea, spin up a few agents to run, make sure they cross check each other, get me a pull request, get the code review done, get the quality controls done.
And on after that, I'll take a look on the code to, like, at the code. Locally, like, I will test that it works. It does the job for me, and all my tests are passing and green, but then I will just scan the code and see what's happening. So there's also like the the the journey is how how big the feature should be. I don't like big features to be implemented at once. I still want to know, you know, what what's happening with my code. So yeah. It's it's shifting. I don't know, like, six months from now, it would could be completely different.
Still not not used to I don't like this idea of, you know, like remote sessions where you check on your phone the the status and your code there. I think I think we need a little bit of time for ourselves away from computers. But I do have agents running in the background on my repos and doing something, and then I come back and check on the results. So I'm kinda partially there, but not yet, and maybe I don't want yet completely to be there. So so you can see it's it's definitely a journey. Yeah. It's it's fun. It's so interesting to observe, like, being in the first, you know, row front seat, see where we go, like, the internal experiments that we run. Some of them are they're not going anywhere, like, we don't we realize that we don't wanna it's not it's not a product. And sometimes it's like, yes, yes, yes, yes. Ship it, please. And we have to wait. It's a bit annoying.
[00:20:04] Barry Luijbregts:
Yeah. And and that's good. I see that you do sometimes deprecate things, which is good. I saw, for instance, GitHub models, which was a preview feature in GitHub. That's now gone away or going away. And that's good. You need to say goodbye to things that aren't working. And to just try things out, see if they are working. So, could the next phase perhaps be human out of the loop? You just feed the thing requirements and you test it?
[00:20:34] Anton Sizikov:
Hope. We'll see. At this point, I gave up on predicting future. So many, everything is moving fast. So just just about the deprecations, I would say, yeah, we we try to deprecate things that are not really going anywhere or doesn't look like a product to us. Models was an interesting beast. I actually like the product, but it was mainly like a gateway to to show the world what what those models are capable of, how to test, how to eval them, how to interact, how to compare. But we're we have it was a front, like a proxy to Asia services anyway. So like if you want to run it in production, yeah, then you need to switch to something like Foundry and then get your models deployed there. And there is a lot of tooling.
So that's also about like, should we build the same tools that exist in another product that all is owned by the same company? But I think it was a good gateway for many well, I don't know numbers unfortunately, but some some customers, some engineers I talked to, yeah, they used it. I did. All my AI experiments, I tend to spin up something on GitHub models and just test it there and then play with them. Actually, thank you for reminding me. I need to update one service that still relies on on models. I I have it Okay. Like, local bot for my like personal automation that I have a Telegram bot that kinda helps me with certain things. And it's updated. Oh, yeah. Good. I'll I'll make a note. Yeah. And tell your agents. Yeah. Yeah. And about the future, I don't know what I see, especially like internally Microsoft and GitHub teams that are very much invested. So we have to talk food, We have to make sure that we we test it all and and see the limit. I think some people say the best perk that we have is the the animated tokens. Right? So we can just spend as much as you as you want and learn because we have to be confident like what we're talking about. We have to try.
So we do have teams that are very much in like into it. So they build this almost what sometimes called like a yeah. Software factories or they they they trying to see that what can we offload. And I think the biggest, at least as of today, the biggest understanding is that you technically can build almost everything. And the neck the challenge is what what exactly you're going to build. Because when you can do almost anything and with a very short time span, because the agent can run overnight and just get you something when you wake up, get it ready for you. So do you then then we always have to ask ourselves, you really need this feature? Do you is it something that you're going to keep in your app, in your in your in your your code base? Like, it's cheap now, so you can throw it away easily.
But still, like, is it is it worth being implemented? And we see that some engineering teams, they became more like agent experienced engine platform engineers. So they build everything that the agent needs to be successful, which is quite fun and interesting. But it shifts, like, they they still ship a lot of features. Like, look at our release page. Look, change look. Like, what is like? Yeah. 10 releases per day. Look at the tooling, like CLI or GitHub Copilot app or Versus Code, they're releasing daily to two times a day, sometimes three times a day.
And they it's not just one bug fix, it's you read the change log, it's like five, seven entries in there. Yeah. It's crazy. Yeah. Yeah. But the the only way they can do it at this space is by building a lot of infrastructure around around it. So that when they have an idea of the feature, they can almost delegate that idea to the agent to a certain degree. You need to plan, of course, you need to provide information, you need to provide maybe like a basic idea like what it should and where it needs to take, you know, like what kind of code it needs to modify, what data is going to be needed for that. Definitely, there is a lot of guardrails in there. But still, once you get that foundation, you can see that the speed increases too much. Sometimes too much. Yeah. So it increases, but sometimes it's just difficult to keep up. So I think the typical example is doc docs. Click.
Yes. So we still need to release docs. We still need to keep them up to date. Partially, it's possible to automate, but nobody wants to read AI generated documentation pages. Yeah. You want to have something that is written by a human with a lot of experience in in in technical writing, and that's where, like, you put a lot of pressure on something that is happening after the development. So your infra, your users actually getting lost sometimes, so I talk to engineers and they say, like, yeah, it's too many features. I don't understand. Like, how do I keep up? It's hard.
So yep. It's a it's human is not going to be taken away, I I I think. But humans are shifting from a bit of like a manual labor when you just sit and type a lot to more abstract or more high level and even like product mindset is needed. So it's it's it is going to be interesting. So for what it will look like in the future, I can't predict, but what I see, I like so far.
[00:26:20] Barry Luijbregts:
Yeah. Yeah. It's a very interesting times. You touched on this a couple times already, which is the value we can get out of these AI tools. Like we're talking about personal productivity here. We get the copilot. Yeah. What I hear from companies and customers a lot, because this stuff is not cheap, it's not for free. Like, is this actually creating value for us and how do we measure that especially? Like I have a customer that's doing a POC with GitHub Copilot and then they literally have a guy do a task without GitHub Copilot, a development task, write down how long it takes, and then one with GitHub Copilot,
[00:27:01] Anton Sizikov:
write down how long it takes. And that's the metric that they use there. Is that the same person building it twice? Yeah. Yeah. That's not a fair experiment. Right? So it's not clean. Well, you know, if if anyone builds like a proper way to measure developer productivity that that it it it you know, it's a good business, definitely. And so far, I didn't see like great examples. Like, yeah, there are plenty of like DoraMetrics, SAFE frameworks, like a lot. But still, there is something that, well, first of all, humans don't like to be measured.
And if you start measuring them, they will play the game and will optimize for that metric. So if you ask them, you know, how many pull requests you create and and you start measuring and ranking them, oh, there will be a lot of pull requests, very small ones. Or, you know, so it's it it it's a difficult game. Yeah. Absolutely. In the past, like in the premium request model, AI was basically free. Like, it was super cheap. Yeah. Now but that was very good for for many engineers to actually learn and see the like how far you can go, what you can do.
With us moving to usage based billing with more other vendors doing the same, you can see this is where the industry is going. So there there is time to actually be mindful about what you built. An example, like, yes, you can run AI agent to give you, I don't know, a summary of to summarize. I don't know. For example, like, in my job, I have to read a lot of change logs, like what are we shipping? You can run an agent and and scrape the file, the the change log website and get your summaries, but you can write a script that would just give you like, you know, a text to like the the header and maybe like a few lines, it's gonna be the same. Like, is it interesting or not? Do I need to read it?
So for sure, we don't have to replace every thing that we do with with AI tools, even though it's possible and fun. We don't have to forget, you know, we still have computers that are very fast and efficient and extra cheap. You can write an algorithm sometimes. But AI can make decisions, can drive more complex flows. Maybe something that you have inside your code base and you do certain things, maybe write a script and point your agent to the script and just do it. It's faster and more reproducible and reliable. So for sure, now when you have to pay for for for the actual consumption, You need to be like, we we have that. We we have conversations coming back again, like how much exactly are we saving?
And it's always difficult because you cannot say that with this tool, you are going to save 25%. Because 25% of what? What are you measuring? What's important to you? For my big, very large manufacturing customer, for them, the frequency of releases is not something they're interested in because they need to build a machine, put the software on it, and ship it to the customer, to their customer. But you do that once a year, and then you need to make sure that that software works. So for them, it's a quality, of tests, amount of lab resource that they, you know, like like what they do.
But for some sort of retail customer or that that build the ship software many times a day and updates and fixes for them, speed to market, time to market, the speed, the frequency of releases, the ability to react on the world that is changing, that is important. So even in in that case, like, you compare those two and for them, the the ROI is measured in different units. So it it is always a challenge. Like, I always start with what exactly do are you trying to improve in terms of yeah. You don't need more code. You need well, as a business, need to earn more money.
Need to provide value to your customers. You need to do something. And what's what's in it for you? It's probably not the time the developer have saved. It's the value they bring back. Maybe they build more features. Maybe they actually your biggest problem is your stability or your older old bugs that never been addressed. Maybe you have, like some banks, have outdated software written in 2090. Yeah. You know, like in eighties or something, in seventies. So they need to update it because nobody understands it and you can just go into it and then make it yeah. It's not gonna bring you new features, but it will fix your future, mitigate the risks that you have.
So it's it's a challenge. It's a challenge how to measure. And mostly the challenge is not that we like the tool is not providing the value, it's about how you use it. Like I was talking to one engineer who's been using like AI productivity agent locally, so it wasn't a copilot to generate code, but it was to check their email, to give them, you know. And there was shocking number, like something like $1,800 per week spent on tokens, which is insane. A bit of work optimization, it went down dramatically like a couple hundreds per month.
And then same value. It's it's it's not the same tool, same value, different cost. Right? So then your productivity didn't change after the optimization, but your your your productivity ROI per per token spent, per dollar spent has increased a lot. So it's hard. Yeah. But it it it there are many dimensions here in how how you measure it. Obviously, it works. Like, there is no doubt. Like, everybody understand that you can do. But now, back to my previous observation is that you need to think about, do you need to do that? Does that bring you anywhere?
And it's not about just features, it's also about the value of that. Is it yeah. You can build a dashboard for something. Do you need it? You can do, you know, a fun thing in your application, but does it bring you anything? You can you you need to be every engineer has to have this almost like product or entrepreneurial mindset. Like, what are we doing here? What what because you have you you can do a lot now. So that's that's that's a challenge.
[00:34:04] Barry Luijbregts:
Yeah. It's a big shift when now you can create anything that you can actually think of right then and there. And so what? What are you going to do with that power basically? Yeah, yeah. So you just touched on ROI, so costs. We we know now GitHub Copilot also does uses based billing like basically all the other vendors as well, right? So we we charge per token here. So how can people optimize what comes out and how much they pay? Because there's lots of little tips and tricks.
[00:34:36] Anton Sizikov:
Oh, absolutely. Well, I actually have like even a full training on that that we give to to all the customers. It's public. But, obviously, I think the example is that productivity tool is the right example. So you can do the achieve the same results with with for a different cost. And that's where I think engineers who've been using tools for a long time, they are in advantage because they have that understanding on how to do more with less capable tools because they had those physical restrictions. Like, you can only put as much into your context window.
You can only you need to be clear and in in in your prompts. You need to be so you need to have that two skill, like one skill is prompting. It's not as relevant anymore as it used to be, but still a good good skill to have. Context engineering is like, what are we going to put into the context? That is super critical because you can get your agent distracted, you can get your agent lost, and then it will recover. So now they will recover, but there will there would be cycles wasted. And I think the other part is make sure that you provide the good tools for your agents, like replace something with a script, so I'm gonna make an API call.
Store data somewhere, like if it's if you're not talking about just coding, like, store it and and review everything that is happening. So I think that's also important. And then that's kind of like a mindset. So you need to understand how the tool works and what's behind it. It's it's a so it's super complicated, but it's very interesting. What's what's behind? What what's under the hood? And after that, yeah, you have like simple solutions that what the model you're going to use. Do you really need that model that was released yesterday to get to provide you a git commit summary. Right? You know? Commit message.
Maybe not. Do you really need to use the cheapest model ever to make something complex and and spend 25 cycles on on on achieving the the result while you can use an expensive model and get it in one shot. Maybe like one, two, three cycles instead. So, you know, it's it's it's a balance, a fine balance, and it's experience a lot. It's very difficult to give recommendations because three, five weeks from now, we'll get new models again. But but still, it's it's important to to to know, to understand, and then to get this, like, what are you giving to the model?
An amount, like, it's the amount of the data you sent and you don't really need a lot. Like my biggest, well, it's not the biggest, but a concern that I have with models that now take like up to 1,000,000 tokens as a context, which is great. But at the same time, it gives you the opportunity to put so much information that that is not relevant for every task. So keeping that in mind is is important. And then there are many other techniques that are being done inside harnesses, so it's maybe because it's not something that engineers, like software engineers that build code with all those agents, they need they don't need to ideally, they don't need to think about all of those aspects. Instead, they need to focus on the software that they're building. And we're not there yet.
We're still in in in in the middle. It's almost like you remember, what is it, like, fifteen ish years ago when Kubernetes started. Yeah. Everybody had to learn it, like all the KubeCTL commands and how you build and deploy. After a while, most of the companies they've been using managed Kubernetes with a layer on top of it, which was built by platform teams. And now, when you build software, just spin up and you repo from a template. You've got, you know, put some configuration in there, and then your buy build pipeline will prepare images, prepare Helm charts, will prepare everything, and it is just going to be shipped. You don't need to know as much of Kubernetes anymore as an engineer to use and deploy there. So I think it's almost the same we'll get eventually where most of those decisions would be taken away. But for now, you need to you need to know what what exactly are you is happening.
So that is important. And then, unfortunately, we have a lot of on the market, those tools that almost like a silver bullet kind of tools. Like, you just put the skill in there and your token consumption is going to All this markdown stuff around the chip. Yeah. But unfortunately, everything that you know, if it's such a simple solution, and it's very competitive market, this market is insane in terms of the competition and everybody needs their share of users. So ask yourself why why wasn't it embedded like they they would have embedded it already into the harness if it was something that really makes the difference in a positive way. So but, yeah, everybody wants it's obvious that the desire here from the engineers is to get rid of that problem. Like, make it solved for me. I need to focus on something else. Still, there is a lot. There are a lot of engineers that want to be almost like knee deep into those details and understand everything.
And they find a lot of joy like I do. But it's unfortunately, or fortunately, not for everybody. Like, somebody wants to build cool mobile apps and to make sure that they're beautiful and users are happy and they don't want to spend time setting up, you know Yeah. The tools. Yeah. The tools and then build extensions and and and then customize them. So definitely, we need to move into the area where it's a it's not a tool that you've spent a lot of time configuring, it's a tool that just works, and then you provide just anything that is needed for to achieve your task.
But for now, we need we we we we need to have that understanding, and we need to come up with best practices, and we need to train each other. So
[00:41:08] Barry Luijbregts:
Yeah. But I I think we're already pretty far along as in Oh, absolutely. We do have it. Yeah. We have a text box. You just type something in there, and it does what you want. Sometimes it's, you know, maybe too expensive. But, yeah, it's also thinking yourself about, is this an automation? Or is this something the LLM should actually do? And is it worth it for the LLM to do this? So co co pilot in this case. That's, you know, we need to retrain ourselves with that as well. And we're all learning this, right, altogether, because it's still very, very new.
[00:41:42] Anton Sizikov:
Yep. That's my favorite question is, like, can you give us the best practices on how to implement it at scale? It's like, well, I have bad There news for is no such thing yet. We'll get there. Eventually, we'll learn what works, what doesn't. But embracing that means that you as a as an organization like engineering leader needs to be also creative and maybe even take a bit of a risk and be ready to back up. You know, step back and like, okay, okay, that doesn't work. We need to solve it in a different way.
I Nobody is alone. We we all have a lot of brilliant minds trying to crack that nut, trying to understand how, what it should look like. Sometimes I said there are some visionaries that are too far ahead and their ideas are not taken yet. Then four or five months later, we think, oh, actually, was right. That's how we need to do it. So I don't know. We'll get there. We'll get there for sure. If there is a problem, we as a human kind, can if we put a lot of effort, enough effort, we solve it. And it is obviously something that we want to solve, so it will be solved. I have a lot of faith in it.
[00:42:56] Barry Luijbregts:
Okay. Cool. So we've been talking for a while already. Maybe we can kinda end on this. So let's say, if there are any engineers out there that are still on the fence with this stuff, maybe they think, you know what, this is not something that I need to use. What would you tell them? How could they get started?
[00:43:18] Anton Sizikov:
Well, first of all, it's a difficult task right now to find such an engineer who never ever ever used anything. I don't like putting that as a, you know, like, triggering FOMO, like Mhmm. Triggering any kind of anxiety about it. I do believe that it's inevitable. Like, you will be using those tools in one shape or another, maybe not something that is advanced and frontier because some compliance or specific type of software you're building, then you will have something less capable, but it's still going to be there, because we see it works, and it does a good job.
So you would have to learn it at at one moment, you know, one way or another. Yeah. It's it's it's a difficult advice. It's like, how do you start? Just start using because so many surfaces where you can get access to it. I understand the the the biggest restriction is that some engineers, they have no access to those tools because of many reasons that, you know, I don't have a good advice for them, like, rather than maybe in your personal time, you know, get a ChatGPT subscription and just play with it. Just see what it can give you back.
Because it once you get, like, a basic understanding, it doesn't matter what tool it is. You know the limits, you know how to drive it, you know when to trust and when not to trust and and verify. That's already a bigger big help. I hope education is catching up. I haven't been talking to like many universities in the the nearest like recently, but I hope that education is going to put a lot of effort into it and we, like schools and universities, are going to teach more of those skills. It's a bit of a challenge that a teacher needs to be good enough have a good enough understanding in order to teach.
So it's yeah. But it's almost like, you know, using a computer. At some point, it was became a must think that you need to be able to know how to use it. Although a lot of people are using just their mobile phones right now and they're happy, but it's still a skill that you need to learn. So those things are going to be more or they're gonna be everywhere, and I I I believe they're going to be less demanding on the skill of the user, but that's a general, like, population. But for engineers, like software engineers, you need to know your tools, you need to know the technology, you need to understand the limits, you need to be able to drive it to to to squeeze the best outcome, results out of it. So, yeah, it's it's definitely something that we all need to to be learning.
I don't know. I don't want to recommend people switching jobs just to get access to those But it's definitely something that we will probably see in the, you know, in the on the market where companies are going to advertise that you are going to get access to those tools. You're going to have a large budget on tokens. You're going to have tasks that are related to AI technologies. We'll probably see that already in there. And and it's been like that in the past, like if you have some advantage because engineers joining companies, they do that for two reasons.
Yep. They want to do something, but they also have to think ahead like, what's next? Like, after that job, what am I going to be in terms of like my employability, my how attractive I am going I'm I'm going to be on the markets in three after three years at this company. And I think the direction is clear right now. So you need to choose wisely.
[00:47:27] Barry Luijbregts:
That's good advice. Okay. Cool. Where can people find you? You have an awesome blog, for instance. Other places as well?
[00:47:38] Anton Sizikov:
At this point, yeah. My blog and my LinkedIn profile are the two things. I do have a X or Twitter account, but it's a bit abandoned now. Mhmm. That's that's another result of all these new technologies. A lot of stuff, you you do it for work. You need to learn a lot, and then it takes time. So, yeah, probably my blog is going to be the best way to to to stay up to date. I I don't post very often, but I when I do, I try to do some good stuff in there. So it's good to be blog at cloudeng.nl. It's yeah. At some point, like, I was very interested into all these cloud technologies.
Yeah. Ops in there, and I wanted I got the domain name, I wanted to build some sort of consultancy business around that. Well, look at me, didn't happen. Yeah. But the domain name is with me. So
[00:48:38] Barry Luijbregts:
Okay. Excellent. So this will be in the show notes for listeners. You can go to the blog. Check it out. There's some good articles on there already, and then you can subscribe there for new stuff when it comes out.
[00:48:51] Anton Sizikov:
Thank you very much, Anton, for hanging out with me. Thank you, Barry. Yeah. Thank you. It was a big fun. All the good questions.
Meet Anton and his role at GitHub
Does GitHub feel like part of Microsoft?
What early GitHub Copilot was really selling
From autocomplete to coding agents
Can software development go human out of the loop?
How to measure AI value and developer ROI
How to control token costs and use models wisely
Advice for engineers still hesitant about AI tools
Where to follow Anton and his blog
