In this episode, I sat down with Heena Refai, a Cloud Solution Architect at Microsoft, to talk about how enterprise AI has evolved from the big-data and traditional machine learning era into today’s generative AI and agentic systems landscape. Heena reflected on her background in data engineering and machine learning, why strong data foundations still matter even in the age of large pretrained models, and why so many organizations are racing into AI without fully addressing the quality, governance, and structure of the data underneath it.
We then dug deep into Microsoft Foundry: what it is, how the model catalog works, how models become agents through prompts, tools, and enterprise data, and how teams can build, host, evaluate, and govern AI systems at scale. Hina explained Foundry Agent Service, project-based organization, cost controls, Azure API Management as an AI gateway, and the growing need for centralized governance with Microsoft Agent 365. We also touched on developer productivity with GitHub Copilot and why, despite all the advances in AI, fundamentals still matter more than ever.
- Heena Refai on LinkedIn: Heena Refai | LinkedIn
- Microsoft Foundry: https://learn.microsoft.com/en-us/azure/foundry/
- Microsoft Foundry Agent Service: https://learn.microsoft.com/en-us/azure/foundry/agents/overview
- Microsoft Agent Framework: https://learn.microsoft.com/en-us/agent-framework/
- Microsoft Agent 365: https://learn.microsoft.com/en-us/office365/servicedescriptions/microsoft-agent-365/microsoft-agent-365
- Azure API Management: https://azure.microsoft.com/en-us/products/api-management
- Azure Pricing Calculator: https://azure.microsoft.com/en-us/pricing/calculator/
- Azure Container Apps: https://learn.microsoft.com/en-us/azure/container-apps/overview
- Azure AI Search: https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search
- Azure Cosmos DB: https://learn.microsoft.com/en-us/cosmos-db/
- Azure OpenAI Service: https://azure.microsoft.com/en-us/pricing/details/azure-openai/
- Anthropic: https://www.anthropic.com/company
- Claude models in Microsoft Foundry: https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/claude-models
- Claude Fable 5 in Microsoft Foundry Models: https://ai.azure.com/catalog/models/claude-fable-5
- Hugging Face: https://huggingface.co/
- IBM: https://www.ibm.com/us-en
- MongoDB: https://www.mongodb.com/products
- Databricks: https://www.databricks.com/
- SAP: https://www.sap.com/
- ServiceNow: https://www.servicenow.com/
- Microsoft Teams: https://www.microsoft.com/en-us/microsoft-teams/teams-products
- Slack: https://slack.com/
- GitHub Copilot: https://docs.github.com/en/copilot/get-started
- Visual Studio Code: https://code.visualstudio.com/
- AWS: https://aws.amazon.com/what-is-aws/
- Kubernetes: https://kubernetes.io/
[00:00:06] Barry Luijbregts:
So welcome, Hina, and thank you very much for taking the time to talk with me today.
[00:00:12] Heena Refai:
Yeah. Thanks, Barry, for having me on your podcast. I'm very excited to be here. And, yeah, looking forward to it, to to our conversation.
[00:00:21] Barry Luijbregts:
Yeah. I it's it's a good topic. Definitely. So perhaps you can introduce yourself to the listeners.
[00:00:30] Heena Refai:
Yes, of course. So hi everyone. I'm Hina, a Cloud Solution Architect at Microsoft focusing on AI and the use of AI, so applications. I've been with Microsoft for around two years now, and I've been working in IT for around thirteen plus years, all of it in the AI and machine learning space. So, yeah, I'm excited to be here and share my experience on this channel.
[00:00:59] Barry Luijbregts:
Oh, excellent. So you've all you've been working in machine learning for a long time then already. What did that look like, let's call it, ten years ago?
[00:01:09] Heena Refai:
Well so I remember the time that was the whole Hadoop and big data boom at that time. Got it? So I remember being a big data engineer supporting a team of data scientists, building pipelines, know, doing also data engineering work, getting data ready so that they could build models on top of a big insights cluster, I remember at that time, with IBM. So then, yeah, so my role was, at that time, ML engineer was not a formal role, but I was doing somewhat a data engineer and a machine learning engineer, helping data scientists get the data, get the pipelines, do the CICD and DevOps so that the models, the feature engineering that they do, the models that they build, the training, the inferencing retraining happens in a standard way on the platform with the right processes. So that was what I was focusing on there. And it has evolved quite a lot since the onset of generative AI since the past few years.
But traditional machine learning still is quite close to my heart because that's where I started.
[00:02:17] Barry Luijbregts:
Yeah. It it's strange if you think back about it because it was the big data boom, and then, you know, the slogan was, there's like gold in your data and there's so much unused potential in your data. Go use your data. Where where did that go? Do what do you see now with customers? Have they embraced that boom, or is it just fizzled out?
[00:02:37] Heena Refai:
No. To be honest, somehow with this whole wave of AI, data has taken a backseat, and it's surprising to see at enterprises and customers because everyone is so caught up in the hype of AI. We wanna do something with AI. We want to embed it into our systems that they've forgotten to basically look at the underlying data that will support your AI. So the garbage in garbage out is becoming more and more prominent now because if your data is not if you don't have good data, then your AI systems are they're not magic. They're also gonna be as, yeah, as good as the data you feed into it. Right? So I do see a shift from we want to do something with AI, but not looking at if our underlying setup, our underlying basics and foundation is set for scaling the AI. So that's what I see at customers.
[00:03:38] Barry Luijbregts:
Yeah. So they had that data scientists, ML engineers at some point, but they kind of never really did the thing with big data, right? They didn't really clean their data,
[00:03:50] Heena Refai:
a little bit, but not not really. Yeah. Yeah. Not really. Not really. I there were some very good use cases, though. Risk modeling. I think that is still a very important machine learning use case that is being implemented in production at many customers. That is still, like, quite there with machine learning. But now with generative AI, I think that traditional machine learning has taken a backseat, and it's more about finding the latest model on Hugging Face and see what we do with it rather than, you know, going back to the roots and finding the features and doing feature engineering, trying out different algorithms to come up with the specific machine learning model. Yep. That has taken a backseat, I would say.
[00:04:33] Barry Luijbregts:
Yeah. That's I I've never really thought about that from this angle. So, yeah, we used to do machine learning, and then the basis was your own data to find patterns and to do things with that data, find gold, and that required lots of expertise. And then basically, you're training your own model. It's kind of what's happening there. And so I guess maybe now because those models from the hyperscalers are so extremely interesting. It's so fun to talk to them and you get all this magic stuff back. But they do it not on your data per se, or you can do that.
But yet, why do you think that's fizzled out or or now shifted towards what we're doing now? It's kinda strange.
[00:05:13] Heena Refai:
I think it has a lot to do with the hype, but also because data science or being a data scientist, a good data scientist requires a certain set of skills, certain set of learnings, which is not very easy to find, that skill set. It has always been a challenge. I think a couple of years back, there was a saying, right, data scientist is the sexiest job of 2020 even in that period, 2019 to 2020, that big years. Right? And that led to organizations to think that we need to have the specialized people in order to build our own models and do machine learning.
And IT not being the core business of many of the organizations, which are which are not prominently in IT, this was a a difficult task. They did not wanna invest so much in setting up a team and setting up the right foundations, ML ML's life cycle in setting up the entire, yeah, the entire skill set and the people that are needed to build machine learning models or to build the data science muscle in the organization. And with generative AI, that part is quite taken care of. So then they're like, hey. It's easier to just take a pretrained model, maybe augment it with our industry data, do rag, or maybe fine tune it in some cases, and leverage it for the use cases rather than start from scratch.
So that has been a mindset shift that organizations that led to not many organizations completely embracing or having a data science muscle within them.
[00:06:49] Barry Luijbregts:
Yeah. You you don't need to be an expert anymore. You're just you're prompt engineer. Exactly. Strange. Okay. We'll we'll circle back to this probably later on as well. I know you as one of the experts in a product suite or a set or solution, whatever you want to call it, called Microsoft Foundry, previously called AI Foundry. Perhaps it had a different name even before that. I don't know. Can you maybe let's dive into that. What is that even?
[00:07:23] Heena Refai:
Perfect. So so Microsoft Foundry. So I have been working with Foundry so that you said, indeed, it was Azure AI Foundry. It was Azure AI Studio even before that. Mhmm. And it was even before that cognitive services. Right now, the foundry that we see, it has gone through multiple iterations, also a maturity path within the product group or within Microsoft as well. Microsoft Foundry is a complete shop as it is today, is a complete shop for building AI apps and agents. It started out as being this provider of this entire model catalog, 11,000 plus models that you can leverage from multiple providers and you can build apps with.
But now it is also evolving into having evaluation capabilities, tracing capabilities, building that agent harness around agentic AI systems that you can also not just build using the models within Foundry, but also leverage the hosting platform within Foundry with hosted agents and utilize the evaluations that come out of the box, tracing that comes out of the box, fine tuning, and all these capabilities, making it like a completely robust platform for building apps, agents, custom agents as well. So you don't just have to rely purely on Microsoft agent framework or or have any lock in to the platform.
But just giving you that flexibility to build apps agents within Foundry or using just models from Foundry and any frameworks outside. So complete AI tool kit, I would say. That's what Foundry has evolved into.
[00:09:00] Barry Luijbregts:
Yeah. Yeah. And it's huge now. So I remember the days when I used to use cognitive services for super specific things like speech to text or something. Or what else was there? There were several. I even made a course about them. And then they evolved into more generative AI, and you needed to fill out a form and wait, and then maybe you got access to them or something. It was kind quite gated. And now we have all this stuff with all these things you just mentioned, which is a lot of stuff. So I guess maybe the core capability is that you can get your models from a central place. Like so from your Azure subscription, can get GPT, Opus. I don't know what all sorts of models.
[00:09:46] Heena Refai:
Yeah. Indeed. Indeed. So that has that is the core indeed, the core capability from Foundry, the model catalog. The diverse model catalog where you find models from multiple providers. You have Anthropic models, which can be pro first party, which have been announced now recently. They're first party from Azure. So then you don't have to worry about the data processing. You have those guarantees about data processing, data retention from Microsoft. So you have those guarantees by using the models from Anthropic. Right? You also have models from MICE, so Microsoft AI.
We have models from Azure OpenAI, of course. We have many other providers. I think there are 11,000 plus in the catalog. And also, they are categorized into the different use that you wanna use them for, usages, maybe for reasoning or you want to do document processing or you want to do voice agents. So they so it's and it also has leaderboard. It also has ways to benchmark public benchmark benchmarking has been done on public data for those models. So you have a possibility of going and checking, right, which model suits my use case the best and trying it out within your use case. I think that is the core capability within Foundry. And then once you identify the model, you can also then evolve into pending agents and having that entire agent development tool kit around it also embedded in Foundry to leverage. Yeah. Okay.
[00:11:15] Barry Luijbregts:
Okay. So let's let's walk through like I'm gonna use this. Right? So I deploy Foundry in Azure. I pay for it or I don't don't yet. And then I pick my favorite model. Let's call it Fable for instance, and I agree to all the terms. It's fine. I am willing to pay for this Cause I want to use this in my application. I have a, let's call it a dot net application. Now what? What do I do?
[00:11:42] Heena Refai:
So, personally for Fable, that is restricted access. As as you also mentioned, we know why. But that is still possible. So if you get access to the model, you can build your application locally in framework of your choice. You chose the languages.net. For agent orchestration, you want to maybe use Lama Index or you want to use Microsoft Microsoft Agent Framework to build. So agent is nothing but model plus a personality. So that's your system prompt telling your AI system what it is supposed to do. Right? Because a large language model could be used for anything, but you say maybe be a health assistant or be a personal coach Mhmm. Be a content writer for me. Right? So you give a personality with the system front, and you give it arms and legs with the with the way of giving it tools and data.
Right? So that's how your entire so your LNM evolves into being an agent in AI system. So within Foundry, what you can do is once you select your model, you can choose a framework of choice, start building locally, and then you can push publish the agent in Foundry. Either it could be a prompt agent, which is used for simple Q and A kind of use cases or simple chat chatbot agents that you wanna build. You give it tools via the Foundry Catalog, the tool catalog that is there. You can connect to xMCP servers, external tools. You can connect to data sources, AI search, MongoDB, Databricks even, and you can give it action. So that is tools for maybe go and update a record in SAP or create an incident in ServiceNow.
So that is how you can attach those hands, legs, create a complete AI system, and have it hosted on Foundry. That will be a URL that you can then call, an API that you can then call, and your AI system based on the prompt that it receives will perform the actions that it you have defined for it to do.
[00:13:42] Barry Luijbregts:
Okay. So I have this model, right, Fable or whatever. And then you said, then it turns into an agent because then it has a system prompt and all these other things. What if I just wanna use it and prompt it? Like, I wanna send a prompt to Fable and just use what comes back.
[00:13:59] Heena Refai:
That is also possible. You can just play with it in the playground. There's a Foundry playground that is a UI based interface. Just select the model and start chatting with it. If you don't give it a system prompt, like, you don't give it a personality, it's a generic model. It was trained on public data, so it can do anything that's possible in that dataset. If you ask it, give me the weather weather right now. It won't be able to give you the weather today because it does not have that tool to find the weather that is, right now. It has data. It has knowledge only from the data that it was trained on. Right? So what you can do, you can just prompt with it. You can send throw prompts, say, hey. Can you suggest me how I can write this text better? How can you summarize this document for me? All of this you can do. But in order for it to be used for specific use case, it's better or or it is advised to give it a personality via system prompt and attach those actions and tools to make it very specific for the business case that you're trying to solve rather than have it generic. Ah,
[00:15:06] Barry Luijbregts:
okay. And then, yeah, like you said, you can also just and that's kind of a click away, right? You can turn it into an agent within Foundry, give it tools and access to other systems and stuff, and then expose that through API.
[00:15:21] Heena Refai:
Are there other ways to expose it? Yes. You can publish it to Teams. There is a direct integration with M365. You can publish it on Slack. You can publish that agent in a website or embedded in a website. Yeah. You can host it. Even if it is hosted on Foundry, you can have an application that is built outside of Foundry, hosted on Container Apps, web application, you name it. Right? And it just calls the agent or AI system whenever it needs to within Foundry. So that is also possible. So there are multiple ways of exposing this agent AI system to the external VoIP.
[00:16:01] Barry Luijbregts:
Right. Because an agent is kinda an app application that runs. Right? It needs a runtime,
[00:16:07] Heena Refai:
a compute somewhere, and we can do that in Foundry as well. Right? I don't have to do that myself. No. You don't have to. So then about the possibilities. If you do it within Foundry, then natively, it uses Microsoft Agent Framework under the hood as orchestration. It has an agent runtime for, like, for hosted agents, you can actually host on it. It uses its agent runtime. And then you can just have an API that you can call, throw your prompts based on, the personality that you have given it, it will respond.
Or you can build just use the models from Foundry and build your own app externally, hosting it in your platform of choice. It could also be Azure. It could be AWS. It could be on prem. Just leveraging the models from Foundry and then accessing the AI system or the agent outside Foundry via the APIs that or where it is hosted.
[00:16:53] Barry Luijbregts:
Yeah. Yeah. So that is very cool because it has everything kinda built into it that you need, right? So the the compute that you need to to run it is called agent service, right? Foundry agent service. Yes. And as I understand, you can run these clicks together agents like low code agents that just runs there. Or, you know, like you just mentioned, you build it yourself with Microsoft Agent Framework or something else just to and just run it there. Now, this agent service, what type of compute does it use, and can I control that?
[00:17:28] Heena Refai:
So that's a very good question. So under the hood, it uses Azure Container Instances. They are single isolated ones that are from a from a tenant that is managed completely on Microsoft side. So I would say you don't have you cannot configure it completely custom yourself. That has been taken care of by Foundry itself. Under the hood, you have the possibility of deciding how many agents you can host there. The scaling and everything is taken care of by the platform itself. Right? If you really want complete customizations, hey, I wanna decide how many nodes my cluster has, my hosting infrastructure, then I would suggest you could also just deploy your own hosting platform like Azure Container Apps or Kubernetes cluster or web apps application in yeah. Web applications that you can host. Right? Or you can host it externally from Foundry as well. But within Foundry, it is managed. And the reason it is done that way is there are many customers who prefer not getting into the infrastructure management, the versioning, the version updates, the scaling, the HADR requirements from a hosting platform. So they prefer to host it on Foundry, and that's where the agent service comes into play, where you can host it on this managed infrastructure within Foundry.
[00:18:53] Barry Luijbregts:
Right. But I can control some things, right, with capability hosts, for instance, on the Foundry instance or on a project where I can say, you know, for instance, my AI search, so one of the tools that you mentioned, that is this one that I already have, for instance, pointed to this thing. For sure. Yes. So that is the enterprise
[00:19:15] Heena Refai:
integration part of Foundry. So definitely, even if the infrastructure is managed by Microsoft, you have a possibility of injecting it into a network that is within your organization. So you can bring in your own virtual network and have the agents that you build within Foundry hosted within that virtual network. So they will also have a private IP, which is within the range that you have set out within your organization. And then you can have everything with private networking. All the integrations to AI search, for example, can be set up as ingress, egress on that subnet where the agents are hosted. So we're going a bit into networking here. But from enterprise integration perspective, even if the infrastructure is managed by Microsoft, it is injected into your own environment.
So it's not like that the agents are running on Microsoft network. It is running within your enterprise.
[00:20:07] Barry Luijbregts:
Yeah. So so for for details, the compute is still managed by Microsoft. So the the container instances. But AI search, storage, and also Cosmos DB, which it uses to track conversations and things like that. Right? You can bring your own of those threads. Indeed.
[00:20:26] Heena Refai:
So you can bring your own network as well as you can bring your own thread store. So with Cosmos DB, AI search for any data that you is sent into the agents or the agent needs as a knowledge source. That as well you can bring your own resources,
[00:20:43] Barry Luijbregts:
which are which are within your Azure tenant. Yeah. Yeah. And can I mix and match those where I say, you know what? I just wanna bring my own storage and Cosmos and AI search. You guys manage that. Yes. That is also possible.
[00:20:56] Heena Refai:
That is also possible. If you don't provide your own resources, Microsoft will deploy or Foundry will deploy itself the resources that it needs. So these three are what it needs to function. The agent service needs it to function. It will deploy it on Microsoft side. So if you don't provide your own, it will anyways be deployed under the hood on Microsoft side.
[00:21:17] Barry Luijbregts:
Yeah. So so also in a hybrid approach where I say just the just Cosmos DB is the thing I manage and the rest you guys take care of. Yes. That is also possible. Okay. Okay, so that's cool. That's a bit of a nitty gritty thing there with agent service, but at least you can then run your agent there. You also, in the beginning, we had lots of terms, evaluations, you mentioned. Yep. What is that?
[00:21:41] Heena Refai:
So evaluations is based to determine if your agentic AI system is performing the tasks the task that it set out for, so the the personality that you have given it. So you told it to be a health adviser, for example. But it is someone asked it, hey. Can you give me advice about the financial markets in The Netherlands right now? Mhmm. And it responded. Right? So then in this case, it is not adhering to the task you set it out for. So then the task adherence of your AI system, it's queued, and you want to monitor it. You want your AI systems to do exactly what you built them for. And that's where evaluations come in. So you decide certain criteria categories. So you decide task adherence being one.
Groundedness, is it responding based on the data that it has, the knowledge that it has, or is it just responding based on public knowledge? Or is it hallucinating even? So all these criteria as you set out based on the use case which you have built the AI system for, and you can set up an evaluation pipeline where every time it can be offline evaluations. Every time a new version of the AI system goes in, the evaluate evaluation pipeline runs to check if your system is still performing to the threshold that has been set, to the percentage that it has been set for the different categories of evaluators or it could be online evaluation.
When your system is being used, every prompt that is sent and every response that is given out by your system is evaluated to see if it is adhering to the task, if it is grounded, if it is hallucinating, if it is it has any security issues, if it has any errors. Right? So all this can be stepped up within Foundry in an automated way.
[00:23:30] Barry Luijbregts:
That's very cool. So offline evaluations are kinda like unit tests for a model.
[00:23:36] Heena Refai:
Yeah. You can definitely, definitely think about it that way. Yeah. Okay.
[00:23:41] Barry Luijbregts:
So, you know, we want quality, so that's an evaluation. See if this thing because that's so difficult with a nondeterministic thing. Right? What happens when you update to a new version of the model? Is it still the same or who knows? Exactly. Okay. That's good. So there are many other things that people wanna measure and control. One of them is costs, and that's always a very difficult one Because what I see a lot as well is that people use foundries or models come out of foundry and then they put for instance API management as your API management in front of that like as a gateway.
So applications connect to API management and an API management can expose models from, for instance, Foundry and or somewhere else. And then API management has all sorts of rules and things like this many tokens. But in this scenario, or if you just connect directly to Foundry, how do you measure the costs that, for instance, an agent makes or your application makes, right? If I write it myself. How do you control that? Because let's say I deploy this thing to the Internet and people just start using it. Yeah.
Maybe it's popular and, you know, how do I not wake up to a huge bill?
[00:25:02] Heena Refai:
Yeah. So so the course are one of the main use cases for having an AI gateway in front of Foundry. So there is some capabilities within. So if you just have Foundry and you deploy models, you also there's a concept called projects. So within one Foundry, you deploy project for each business or each use case, depending on how your organization is set in. At the project level, you have the possibility of setting rate limits. So, hey, you can use because all model usage within Foundry is based is charged on tokens, tokens per minute. So imagine a token to be four roughly four characters contributes to one token. Right? And characters include blank spaces, letters, numbers, everything.
So the tokens per minute is the criteria in which the costs are calculated. So how Foundry does is at the project level, you can decide, hey, this project can give be given a code of 10,000 per minute and, a 1,000,000 tokens for a month. That is how you set budgets. So then when you bid agents within your project, they consume the code the tokens from these budgets that have been set up. So that is for some token management is possible within Foundry buyer projects. But if you want more granular segregation, like at the agent level for a one use case, I've built an agent and I want the tokens to be measured exclusively for this use case, like you mentioned, a public facing agent.
In that case, you have an AI gateway. AI gateway comes in where you can set up for API rate limits and token limits. So per minute as well as weekly, monthly, or quarterly rate limits that you can set up. So then as soon as it starts hitting those limits, you'll get notifications or the people start getting four two nines. They start getting blocked that, hey, you are crossing your rate limit, so please don't and retry after a certain amount of time. So that is where AI gateway comes in, to have that overall cost governance as one of the topics. There is a lot more to AI gateway with AI governance that you can do. But that gives you that feature of segregating costs and creating budgets and monitoring cost token utilization at API level as well as agent level or project level.
[00:27:29] Barry Luijbregts:
And so you just mentioned Foundry projects, for instance, you can manage tokens. So how many tokens? API management also how many tokens? But how do I know what a token costs? Because an output token for Fable is a very different price from an output token from a different model. So how do you do that?
[00:27:50] Heena Refai:
Yeah, we do have a pricing calculator. So online, it is published, Microsoft pricing calculator, you where you can actually see what each model, how much it costs per token or or tokens. It is also in TPM. So how much is the cost for tokens per minute for certain model? And then you can make an estimate. There's a pricing calculator where you can go and make an estimate of how much roughly my use case would cost if these were the amount of calls it comes in, these were the amount of prompts, prompt tokens coming in, completion tokens coming in coming out.
And, yeah, you can make a rough estimate, and that's how basically the costs are calculated.
[00:28:30] Barry Luijbregts:
Ah, okay. So currently, I need to do this manually. So I need to calculate what it would cost and then set a budget based on then it is x amount of tokens because I'm gonna use Fable for this. Yes. Indeed. Uh-huh. Okay. I'm asking for a friend, so I'm wondering For sure. Is this gonna be because, obviously, you want this to be automatic. Right? I wanna say this project can cost me €300 and this one 5,000, Not tokens because I don't know the tokens. Is that common?
[00:29:02] Heena Refai:
So I think that I don't know, Furt. So short answer, I've not seen that on the roadmap yet to have the cost base segregation. It still is TPM, but the calculation is also pretty straightforward. Right? When you set 10,000 TPM, you also have an idea of how much prompts and completions it can handle and then how many requests also it can handle. Six requests, 10 requests, 20 requests. So then you also have an idea, okay, my app will be used within this time window, how much it can be used. And as soon as it starts hitting limit, you'll start getting four to nine. That's a very popular error Mhmm. And says, hey, rates limit reached. So there is to answer your question directly, there is I I have not seen a cost based segregation coming yet, but that would be a very nice feature to probably discuss with the product group.
[00:29:54] Barry Luijbregts:
Yeah. It would be. Yeah. Yeah. Feedback. Okay. And and so whilst talking about this, you already said, so you have Foundry, which is kind of an instance within Azure, right? And that thing can have projects. Yeah. Which you can then give to, here's your use case or here is a team working on something. So what is now a typical overview architecture within, let's say, you have a larger enterprise, like one central team that manages things, infrastructure team or something. And then we have all these other teams that build things.
So would I, for instance, then deploy one Foundry instance centrally, maybe an API gateway connection there, and then projects. I give a project to team A, project to team B, project to team C. Is that kinda how you do it?
[00:30:46] Heena Refai:
So there is a Microsoft AI reference architecture that is that we have as a starting point that is proposed by Microsoft. I do and in there, the idea is you have a central AI hub with where you have the AI gateway, which manages access to the models, manages access to external systems, MCP servers, other agents, etcetera. You do rate limiting there, cost management, everything. All AI governance capabilities are set up at the gateway. So that is within an AI hub subscription. And then you have agent spoke subscriptions.
And that's where you have Foundry deployed per team, per use case, per business unit within an organization where they start building their apps and agents. So this is how the reference architecture is proposed from Microsoft. But within customers, in reality on the on the ground, I see this being adopted in various ways. Mhmm. Depending on the scale of the organization and how they are set up. There are some customers who who have a central IT team who also want to govern everything centrally. So they not just have the AI gateway centrally, they also have a foundry resource centrally and provide projects to business units to build apps and agents.
That works tools. But there are some organizations who want to completely federate it and adopt the reference architecture as it is prescribed. So then there is an AI governance team, which also owns the AI gateway, also is responsible for triaging the AI use cases to decide which use case would need what tools and capabilities. And then there is the business units, which will have their own subscription or own resource group with Foundry in there who build their apps and agents within it. So and then they have a top governance layer with Agent three sixty five integrated vendor, you know, complete enterprise setup that then adds to adds to also the security monitoring and governance requirements from enterprises.
That is different flavors we see, all adopted from the reference architecture that we have.
[00:32:57] Barry Luijbregts:
Okay. And and maybe like a final topic ish that you just mentioned agent three sixty five. Because I was gonna ask, so how you have all these agents, they are running everywhere. How do you control them and even know that they exist? It's kinda what it is, right? Yes. So an agent sprawl is
[00:33:15] Heena Refai:
is a topic and a a concern that many enterprises are who are adopting agents and building apps and agents with AI are are noticing it and are trying to find solutions for it. So you can build agents in Foundry. You can build agents in Cobalt Studio. You can build custom agents. You can build agents with the m three six extend the m three sixty five agents that are there or even outside. Right? But then how do you control with what is happening where? Where is my data going? Where is it being used? So with agent three sixty five, are we are at Microsoft, they are bringing a consolidated pane, a control pane, right, for agents built anywhere in their enterprise, anywhere within your tenant.
You could monitor what is happening in the agents, where it is hosted, what tools it's accessing, what data it is accessing, also set up guardrails on top of it to to ensure that they access only the data that they should, the systems that they should, they perform only the set of actions that they should. If there are any threat actors acting on AI systems, then alerts are sent out to the SOC team via Defender for AI integration to also take action in real time. Right? So so that is Agent 365 will become the central control center or command center for all agents built in your organizations, be it pro code, custom agents with Foundry, outside Foundry, or more no code agents built via M365 or CoPilot Studio.
Yeah. So that is where Agent three sixty five comes in. Okay.
[00:34:56] Barry Luijbregts:
So much stuff. It's complicated.
[00:35:00] Heena Refai:
And it keeps keeps getting added. More and more things keep coming in every week, every two weeks. So, yeah, it is quite a changing space.
[00:35:10] Barry Luijbregts:
Space. Absolutely. So what's your favorite model today?
[00:35:17] Heena Refai:
So plot four dot eight is is favorite. So I've that's the developer in me talking. Right? So it's I for me, if I have to build something, that's the one I rely on. But also the auto mode. So talking a bit also about GitHub Copilot Mhmm. Which is the environment which is the IDE I use, basically, which is the agent I use for my development within Versus Code. So for me, four dot eight is what gives me the best results. Five dot five also, GPT five .5 also at times, but I've also recently seen the auto more tends to perform really well if I just want to ask some questions, if I want to understand my code better, then even the auto mode performs really well. So, yeah, that space has evolved. The developer productivity space is evolving as well.
[00:36:07] Barry Luijbregts:
Yeah. Absolutely. Yeah. And that's good. So you're not token maxing. You're just actually turning on auto mode. Indeed. Okay. Good. You should check out the latest GPT 5.6. There are three of them. Oh, I haven't I haven't tested those out yet. They're pretty good. So so for reference listener, we're recording this on the July 14. I'm sure by now you guys have GPT six or beyond. We'll see what happens in the future. So and what would you advise your your children to study?
[00:36:40] Heena Refai:
Oh, to my children, I would I would advise the people to and this is this is a bit a tricky answer, but I would advise them to definitely get their basics right and not rely completely on AI because your AI system is as good as the prompter. Right? So you the prompt the person who is sending the prompt needs to have the fundamentals said straight. And then you see the AI AI system as helping you or augmenting to your knowledge rather than, you know, relying fully on it. So to my daughter, I would say, you do need to study. You need do need to get your basics and fundamentals right, and then use AI as a as a assistant rather than, you know, completely relying on it. Yeah.
[00:37:28] Barry Luijbregts:
Yeah. That's gonna be interesting times. We'll see. Yes.
[00:37:31] Heena Refai:
For sure. I already see that happening. Yeah. Yeah. Lands increasing.
[00:37:37] Barry Luijbregts:
Alright. And where can people go and what's the best place to connect with you?
[00:37:42] Heena Refai:
Yeah. I have my LinkedIn profile. It's by the name of Hina Rifai. You can look me up. Yeah. And just drop me a DM. I usually respond within the day. I'm quite active there. So, yeah, that is the place to reach out.
[00:37:59] Barry Luijbregts:
Okay. Well, thank you very much for for teaching me all this stuff today. I learned a lot.
[00:38:04] Heena Refai:
Oh, thank you. And thank you for having me here. I love the conversation. Yeah. And this was a super nice experience.
[00:38:11] Barry Luijbregts:
Okay. Thank you very much, and thank you all for listening.
[00:38:16] Heena Refai:
Thank you. Bye bye.
So welcome, Hina, and thank you very much for taking the time to talk with me today.
[00:00:12] Heena Refai:
Yeah. Thanks, Barry, for having me on your podcast. I'm very excited to be here. And, yeah, looking forward to it, to to our conversation.
[00:00:21] Barry Luijbregts:
Yeah. I it's it's a good topic. Definitely. So perhaps you can introduce yourself to the listeners.
[00:00:30] Heena Refai:
Yes, of course. So hi everyone. I'm Hina, a Cloud Solution Architect at Microsoft focusing on AI and the use of AI, so applications. I've been with Microsoft for around two years now, and I've been working in IT for around thirteen plus years, all of it in the AI and machine learning space. So, yeah, I'm excited to be here and share my experience on this channel.
[00:00:59] Barry Luijbregts:
Oh, excellent. So you've all you've been working in machine learning for a long time then already. What did that look like, let's call it, ten years ago?
[00:01:09] Heena Refai:
Well so I remember the time that was the whole Hadoop and big data boom at that time. Got it? So I remember being a big data engineer supporting a team of data scientists, building pipelines, know, doing also data engineering work, getting data ready so that they could build models on top of a big insights cluster, I remember at that time, with IBM. So then, yeah, so my role was, at that time, ML engineer was not a formal role, but I was doing somewhat a data engineer and a machine learning engineer, helping data scientists get the data, get the pipelines, do the CICD and DevOps so that the models, the feature engineering that they do, the models that they build, the training, the inferencing retraining happens in a standard way on the platform with the right processes. So that was what I was focusing on there. And it has evolved quite a lot since the onset of generative AI since the past few years.
But traditional machine learning still is quite close to my heart because that's where I started.
[00:02:17] Barry Luijbregts:
Yeah. It it's strange if you think back about it because it was the big data boom, and then, you know, the slogan was, there's like gold in your data and there's so much unused potential in your data. Go use your data. Where where did that go? Do what do you see now with customers? Have they embraced that boom, or is it just fizzled out?
[00:02:37] Heena Refai:
No. To be honest, somehow with this whole wave of AI, data has taken a backseat, and it's surprising to see at enterprises and customers because everyone is so caught up in the hype of AI. We wanna do something with AI. We want to embed it into our systems that they've forgotten to basically look at the underlying data that will support your AI. So the garbage in garbage out is becoming more and more prominent now because if your data is not if you don't have good data, then your AI systems are they're not magic. They're also gonna be as, yeah, as good as the data you feed into it. Right? So I do see a shift from we want to do something with AI, but not looking at if our underlying setup, our underlying basics and foundation is set for scaling the AI. So that's what I see at customers.
[00:03:38] Barry Luijbregts:
Yeah. So they had that data scientists, ML engineers at some point, but they kind of never really did the thing with big data, right? They didn't really clean their data,
[00:03:50] Heena Refai:
a little bit, but not not really. Yeah. Yeah. Not really. Not really. I there were some very good use cases, though. Risk modeling. I think that is still a very important machine learning use case that is being implemented in production at many customers. That is still, like, quite there with machine learning. But now with generative AI, I think that traditional machine learning has taken a backseat, and it's more about finding the latest model on Hugging Face and see what we do with it rather than, you know, going back to the roots and finding the features and doing feature engineering, trying out different algorithms to come up with the specific machine learning model. Yep. That has taken a backseat, I would say.
[00:04:33] Barry Luijbregts:
Yeah. That's I I've never really thought about that from this angle. So, yeah, we used to do machine learning, and then the basis was your own data to find patterns and to do things with that data, find gold, and that required lots of expertise. And then basically, you're training your own model. It's kind of what's happening there. And so I guess maybe now because those models from the hyperscalers are so extremely interesting. It's so fun to talk to them and you get all this magic stuff back. But they do it not on your data per se, or you can do that.
But yet, why do you think that's fizzled out or or now shifted towards what we're doing now? It's kinda strange.
[00:05:13] Heena Refai:
I think it has a lot to do with the hype, but also because data science or being a data scientist, a good data scientist requires a certain set of skills, certain set of learnings, which is not very easy to find, that skill set. It has always been a challenge. I think a couple of years back, there was a saying, right, data scientist is the sexiest job of 2020 even in that period, 2019 to 2020, that big years. Right? And that led to organizations to think that we need to have the specialized people in order to build our own models and do machine learning.
And IT not being the core business of many of the organizations, which are which are not prominently in IT, this was a a difficult task. They did not wanna invest so much in setting up a team and setting up the right foundations, ML ML's life cycle in setting up the entire, yeah, the entire skill set and the people that are needed to build machine learning models or to build the data science muscle in the organization. And with generative AI, that part is quite taken care of. So then they're like, hey. It's easier to just take a pretrained model, maybe augment it with our industry data, do rag, or maybe fine tune it in some cases, and leverage it for the use cases rather than start from scratch.
So that has been a mindset shift that organizations that led to not many organizations completely embracing or having a data science muscle within them.
[00:06:49] Barry Luijbregts:
Yeah. You you don't need to be an expert anymore. You're just you're prompt engineer. Exactly. Strange. Okay. We'll we'll circle back to this probably later on as well. I know you as one of the experts in a product suite or a set or solution, whatever you want to call it, called Microsoft Foundry, previously called AI Foundry. Perhaps it had a different name even before that. I don't know. Can you maybe let's dive into that. What is that even?
[00:07:23] Heena Refai:
Perfect. So so Microsoft Foundry. So I have been working with Foundry so that you said, indeed, it was Azure AI Foundry. It was Azure AI Studio even before that. Mhmm. And it was even before that cognitive services. Right now, the foundry that we see, it has gone through multiple iterations, also a maturity path within the product group or within Microsoft as well. Microsoft Foundry is a complete shop as it is today, is a complete shop for building AI apps and agents. It started out as being this provider of this entire model catalog, 11,000 plus models that you can leverage from multiple providers and you can build apps with.
But now it is also evolving into having evaluation capabilities, tracing capabilities, building that agent harness around agentic AI systems that you can also not just build using the models within Foundry, but also leverage the hosting platform within Foundry with hosted agents and utilize the evaluations that come out of the box, tracing that comes out of the box, fine tuning, and all these capabilities, making it like a completely robust platform for building apps, agents, custom agents as well. So you don't just have to rely purely on Microsoft agent framework or or have any lock in to the platform.
But just giving you that flexibility to build apps agents within Foundry or using just models from Foundry and any frameworks outside. So complete AI tool kit, I would say. That's what Foundry has evolved into.
[00:09:00] Barry Luijbregts:
Yeah. Yeah. And it's huge now. So I remember the days when I used to use cognitive services for super specific things like speech to text or something. Or what else was there? There were several. I even made a course about them. And then they evolved into more generative AI, and you needed to fill out a form and wait, and then maybe you got access to them or something. It was kind quite gated. And now we have all this stuff with all these things you just mentioned, which is a lot of stuff. So I guess maybe the core capability is that you can get your models from a central place. Like so from your Azure subscription, can get GPT, Opus. I don't know what all sorts of models.
[00:09:46] Heena Refai:
Yeah. Indeed. Indeed. So that has that is the core indeed, the core capability from Foundry, the model catalog. The diverse model catalog where you find models from multiple providers. You have Anthropic models, which can be pro first party, which have been announced now recently. They're first party from Azure. So then you don't have to worry about the data processing. You have those guarantees about data processing, data retention from Microsoft. So you have those guarantees by using the models from Anthropic. Right? You also have models from MICE, so Microsoft AI.
We have models from Azure OpenAI, of course. We have many other providers. I think there are 11,000 plus in the catalog. And also, they are categorized into the different use that you wanna use them for, usages, maybe for reasoning or you want to do document processing or you want to do voice agents. So they so it's and it also has leaderboard. It also has ways to benchmark public benchmark benchmarking has been done on public data for those models. So you have a possibility of going and checking, right, which model suits my use case the best and trying it out within your use case. I think that is the core capability within Foundry. And then once you identify the model, you can also then evolve into pending agents and having that entire agent development tool kit around it also embedded in Foundry to leverage. Yeah. Okay.
[00:11:15] Barry Luijbregts:
Okay. So let's let's walk through like I'm gonna use this. Right? So I deploy Foundry in Azure. I pay for it or I don't don't yet. And then I pick my favorite model. Let's call it Fable for instance, and I agree to all the terms. It's fine. I am willing to pay for this Cause I want to use this in my application. I have a, let's call it a dot net application. Now what? What do I do?
[00:11:42] Heena Refai:
So, personally for Fable, that is restricted access. As as you also mentioned, we know why. But that is still possible. So if you get access to the model, you can build your application locally in framework of your choice. You chose the languages.net. For agent orchestration, you want to maybe use Lama Index or you want to use Microsoft Microsoft Agent Framework to build. So agent is nothing but model plus a personality. So that's your system prompt telling your AI system what it is supposed to do. Right? Because a large language model could be used for anything, but you say maybe be a health assistant or be a personal coach Mhmm. Be a content writer for me. Right? So you give a personality with the system front, and you give it arms and legs with the with the way of giving it tools and data.
Right? So that's how your entire so your LNM evolves into being an agent in AI system. So within Foundry, what you can do is once you select your model, you can choose a framework of choice, start building locally, and then you can push publish the agent in Foundry. Either it could be a prompt agent, which is used for simple Q and A kind of use cases or simple chat chatbot agents that you wanna build. You give it tools via the Foundry Catalog, the tool catalog that is there. You can connect to xMCP servers, external tools. You can connect to data sources, AI search, MongoDB, Databricks even, and you can give it action. So that is tools for maybe go and update a record in SAP or create an incident in ServiceNow.
So that is how you can attach those hands, legs, create a complete AI system, and have it hosted on Foundry. That will be a URL that you can then call, an API that you can then call, and your AI system based on the prompt that it receives will perform the actions that it you have defined for it to do.
[00:13:42] Barry Luijbregts:
Okay. So I have this model, right, Fable or whatever. And then you said, then it turns into an agent because then it has a system prompt and all these other things. What if I just wanna use it and prompt it? Like, I wanna send a prompt to Fable and just use what comes back.
[00:13:59] Heena Refai:
That is also possible. You can just play with it in the playground. There's a Foundry playground that is a UI based interface. Just select the model and start chatting with it. If you don't give it a system prompt, like, you don't give it a personality, it's a generic model. It was trained on public data, so it can do anything that's possible in that dataset. If you ask it, give me the weather weather right now. It won't be able to give you the weather today because it does not have that tool to find the weather that is, right now. It has data. It has knowledge only from the data that it was trained on. Right? So what you can do, you can just prompt with it. You can send throw prompts, say, hey. Can you suggest me how I can write this text better? How can you summarize this document for me? All of this you can do. But in order for it to be used for specific use case, it's better or or it is advised to give it a personality via system prompt and attach those actions and tools to make it very specific for the business case that you're trying to solve rather than have it generic. Ah,
[00:15:06] Barry Luijbregts:
okay. And then, yeah, like you said, you can also just and that's kind of a click away, right? You can turn it into an agent within Foundry, give it tools and access to other systems and stuff, and then expose that through API.
[00:15:21] Heena Refai:
Are there other ways to expose it? Yes. You can publish it to Teams. There is a direct integration with M365. You can publish it on Slack. You can publish that agent in a website or embedded in a website. Yeah. You can host it. Even if it is hosted on Foundry, you can have an application that is built outside of Foundry, hosted on Container Apps, web application, you name it. Right? And it just calls the agent or AI system whenever it needs to within Foundry. So that is also possible. So there are multiple ways of exposing this agent AI system to the external VoIP.
[00:16:01] Barry Luijbregts:
Right. Because an agent is kinda an app application that runs. Right? It needs a runtime,
[00:16:07] Heena Refai:
a compute somewhere, and we can do that in Foundry as well. Right? I don't have to do that myself. No. You don't have to. So then about the possibilities. If you do it within Foundry, then natively, it uses Microsoft Agent Framework under the hood as orchestration. It has an agent runtime for, like, for hosted agents, you can actually host on it. It uses its agent runtime. And then you can just have an API that you can call, throw your prompts based on, the personality that you have given it, it will respond.
Or you can build just use the models from Foundry and build your own app externally, hosting it in your platform of choice. It could also be Azure. It could be AWS. It could be on prem. Just leveraging the models from Foundry and then accessing the AI system or the agent outside Foundry via the APIs that or where it is hosted.
[00:16:53] Barry Luijbregts:
Yeah. Yeah. So that is very cool because it has everything kinda built into it that you need, right? So the the compute that you need to to run it is called agent service, right? Foundry agent service. Yes. And as I understand, you can run these clicks together agents like low code agents that just runs there. Or, you know, like you just mentioned, you build it yourself with Microsoft Agent Framework or something else just to and just run it there. Now, this agent service, what type of compute does it use, and can I control that?
[00:17:28] Heena Refai:
So that's a very good question. So under the hood, it uses Azure Container Instances. They are single isolated ones that are from a from a tenant that is managed completely on Microsoft side. So I would say you don't have you cannot configure it completely custom yourself. That has been taken care of by Foundry itself. Under the hood, you have the possibility of deciding how many agents you can host there. The scaling and everything is taken care of by the platform itself. Right? If you really want complete customizations, hey, I wanna decide how many nodes my cluster has, my hosting infrastructure, then I would suggest you could also just deploy your own hosting platform like Azure Container Apps or Kubernetes cluster or web apps application in yeah. Web applications that you can host. Right? Or you can host it externally from Foundry as well. But within Foundry, it is managed. And the reason it is done that way is there are many customers who prefer not getting into the infrastructure management, the versioning, the version updates, the scaling, the HADR requirements from a hosting platform. So they prefer to host it on Foundry, and that's where the agent service comes into play, where you can host it on this managed infrastructure within Foundry.
[00:18:53] Barry Luijbregts:
Right. But I can control some things, right, with capability hosts, for instance, on the Foundry instance or on a project where I can say, you know, for instance, my AI search, so one of the tools that you mentioned, that is this one that I already have, for instance, pointed to this thing. For sure. Yes. So that is the enterprise
[00:19:15] Heena Refai:
integration part of Foundry. So definitely, even if the infrastructure is managed by Microsoft, you have a possibility of injecting it into a network that is within your organization. So you can bring in your own virtual network and have the agents that you build within Foundry hosted within that virtual network. So they will also have a private IP, which is within the range that you have set out within your organization. And then you can have everything with private networking. All the integrations to AI search, for example, can be set up as ingress, egress on that subnet where the agents are hosted. So we're going a bit into networking here. But from enterprise integration perspective, even if the infrastructure is managed by Microsoft, it is injected into your own environment.
So it's not like that the agents are running on Microsoft network. It is running within your enterprise.
[00:20:07] Barry Luijbregts:
Yeah. So so for for details, the compute is still managed by Microsoft. So the the container instances. But AI search, storage, and also Cosmos DB, which it uses to track conversations and things like that. Right? You can bring your own of those threads. Indeed.
[00:20:26] Heena Refai:
So you can bring your own network as well as you can bring your own thread store. So with Cosmos DB, AI search for any data that you is sent into the agents or the agent needs as a knowledge source. That as well you can bring your own resources,
[00:20:43] Barry Luijbregts:
which are which are within your Azure tenant. Yeah. Yeah. And can I mix and match those where I say, you know what? I just wanna bring my own storage and Cosmos and AI search. You guys manage that. Yes. That is also possible.
[00:20:56] Heena Refai:
That is also possible. If you don't provide your own resources, Microsoft will deploy or Foundry will deploy itself the resources that it needs. So these three are what it needs to function. The agent service needs it to function. It will deploy it on Microsoft side. So if you don't provide your own, it will anyways be deployed under the hood on Microsoft side.
[00:21:17] Barry Luijbregts:
Yeah. So so also in a hybrid approach where I say just the just Cosmos DB is the thing I manage and the rest you guys take care of. Yes. That is also possible. Okay. Okay, so that's cool. That's a bit of a nitty gritty thing there with agent service, but at least you can then run your agent there. You also, in the beginning, we had lots of terms, evaluations, you mentioned. Yep. What is that?
[00:21:41] Heena Refai:
So evaluations is based to determine if your agentic AI system is performing the tasks the task that it set out for, so the the personality that you have given it. So you told it to be a health adviser, for example. But it is someone asked it, hey. Can you give me advice about the financial markets in The Netherlands right now? Mhmm. And it responded. Right? So then in this case, it is not adhering to the task you set it out for. So then the task adherence of your AI system, it's queued, and you want to monitor it. You want your AI systems to do exactly what you built them for. And that's where evaluations come in. So you decide certain criteria categories. So you decide task adherence being one.
Groundedness, is it responding based on the data that it has, the knowledge that it has, or is it just responding based on public knowledge? Or is it hallucinating even? So all these criteria as you set out based on the use case which you have built the AI system for, and you can set up an evaluation pipeline where every time it can be offline evaluations. Every time a new version of the AI system goes in, the evaluate evaluation pipeline runs to check if your system is still performing to the threshold that has been set, to the percentage that it has been set for the different categories of evaluators or it could be online evaluation.
When your system is being used, every prompt that is sent and every response that is given out by your system is evaluated to see if it is adhering to the task, if it is grounded, if it is hallucinating, if it is it has any security issues, if it has any errors. Right? So all this can be stepped up within Foundry in an automated way.
[00:23:30] Barry Luijbregts:
That's very cool. So offline evaluations are kinda like unit tests for a model.
[00:23:36] Heena Refai:
Yeah. You can definitely, definitely think about it that way. Yeah. Okay.
[00:23:41] Barry Luijbregts:
So, you know, we want quality, so that's an evaluation. See if this thing because that's so difficult with a nondeterministic thing. Right? What happens when you update to a new version of the model? Is it still the same or who knows? Exactly. Okay. That's good. So there are many other things that people wanna measure and control. One of them is costs, and that's always a very difficult one Because what I see a lot as well is that people use foundries or models come out of foundry and then they put for instance API management as your API management in front of that like as a gateway.
So applications connect to API management and an API management can expose models from, for instance, Foundry and or somewhere else. And then API management has all sorts of rules and things like this many tokens. But in this scenario, or if you just connect directly to Foundry, how do you measure the costs that, for instance, an agent makes or your application makes, right? If I write it myself. How do you control that? Because let's say I deploy this thing to the Internet and people just start using it. Yeah.
Maybe it's popular and, you know, how do I not wake up to a huge bill?
[00:25:02] Heena Refai:
Yeah. So so the course are one of the main use cases for having an AI gateway in front of Foundry. So there is some capabilities within. So if you just have Foundry and you deploy models, you also there's a concept called projects. So within one Foundry, you deploy project for each business or each use case, depending on how your organization is set in. At the project level, you have the possibility of setting rate limits. So, hey, you can use because all model usage within Foundry is based is charged on tokens, tokens per minute. So imagine a token to be four roughly four characters contributes to one token. Right? And characters include blank spaces, letters, numbers, everything.
So the tokens per minute is the criteria in which the costs are calculated. So how Foundry does is at the project level, you can decide, hey, this project can give be given a code of 10,000 per minute and, a 1,000,000 tokens for a month. That is how you set budgets. So then when you bid agents within your project, they consume the code the tokens from these budgets that have been set up. So that is for some token management is possible within Foundry buyer projects. But if you want more granular segregation, like at the agent level for a one use case, I've built an agent and I want the tokens to be measured exclusively for this use case, like you mentioned, a public facing agent.
In that case, you have an AI gateway. AI gateway comes in where you can set up for API rate limits and token limits. So per minute as well as weekly, monthly, or quarterly rate limits that you can set up. So then as soon as it starts hitting those limits, you'll get notifications or the people start getting four two nines. They start getting blocked that, hey, you are crossing your rate limit, so please don't and retry after a certain amount of time. So that is where AI gateway comes in, to have that overall cost governance as one of the topics. There is a lot more to AI gateway with AI governance that you can do. But that gives you that feature of segregating costs and creating budgets and monitoring cost token utilization at API level as well as agent level or project level.
[00:27:29] Barry Luijbregts:
And so you just mentioned Foundry projects, for instance, you can manage tokens. So how many tokens? API management also how many tokens? But how do I know what a token costs? Because an output token for Fable is a very different price from an output token from a different model. So how do you do that?
[00:27:50] Heena Refai:
Yeah, we do have a pricing calculator. So online, it is published, Microsoft pricing calculator, you where you can actually see what each model, how much it costs per token or or tokens. It is also in TPM. So how much is the cost for tokens per minute for certain model? And then you can make an estimate. There's a pricing calculator where you can go and make an estimate of how much roughly my use case would cost if these were the amount of calls it comes in, these were the amount of prompts, prompt tokens coming in, completion tokens coming in coming out.
And, yeah, you can make a rough estimate, and that's how basically the costs are calculated.
[00:28:30] Barry Luijbregts:
Ah, okay. So currently, I need to do this manually. So I need to calculate what it would cost and then set a budget based on then it is x amount of tokens because I'm gonna use Fable for this. Yes. Indeed. Uh-huh. Okay. I'm asking for a friend, so I'm wondering For sure. Is this gonna be because, obviously, you want this to be automatic. Right? I wanna say this project can cost me €300 and this one 5,000, Not tokens because I don't know the tokens. Is that common?
[00:29:02] Heena Refai:
So I think that I don't know, Furt. So short answer, I've not seen that on the roadmap yet to have the cost base segregation. It still is TPM, but the calculation is also pretty straightforward. Right? When you set 10,000 TPM, you also have an idea of how much prompts and completions it can handle and then how many requests also it can handle. Six requests, 10 requests, 20 requests. So then you also have an idea, okay, my app will be used within this time window, how much it can be used. And as soon as it starts hitting limit, you'll start getting four to nine. That's a very popular error Mhmm. And says, hey, rates limit reached. So there is to answer your question directly, there is I I have not seen a cost based segregation coming yet, but that would be a very nice feature to probably discuss with the product group.
[00:29:54] Barry Luijbregts:
Yeah. It would be. Yeah. Yeah. Feedback. Okay. And and so whilst talking about this, you already said, so you have Foundry, which is kind of an instance within Azure, right? And that thing can have projects. Yeah. Which you can then give to, here's your use case or here is a team working on something. So what is now a typical overview architecture within, let's say, you have a larger enterprise, like one central team that manages things, infrastructure team or something. And then we have all these other teams that build things.
So would I, for instance, then deploy one Foundry instance centrally, maybe an API gateway connection there, and then projects. I give a project to team A, project to team B, project to team C. Is that kinda how you do it?
[00:30:46] Heena Refai:
So there is a Microsoft AI reference architecture that is that we have as a starting point that is proposed by Microsoft. I do and in there, the idea is you have a central AI hub with where you have the AI gateway, which manages access to the models, manages access to external systems, MCP servers, other agents, etcetera. You do rate limiting there, cost management, everything. All AI governance capabilities are set up at the gateway. So that is within an AI hub subscription. And then you have agent spoke subscriptions.
And that's where you have Foundry deployed per team, per use case, per business unit within an organization where they start building their apps and agents. So this is how the reference architecture is proposed from Microsoft. But within customers, in reality on the on the ground, I see this being adopted in various ways. Mhmm. Depending on the scale of the organization and how they are set up. There are some customers who who have a central IT team who also want to govern everything centrally. So they not just have the AI gateway centrally, they also have a foundry resource centrally and provide projects to business units to build apps and agents.
That works tools. But there are some organizations who want to completely federate it and adopt the reference architecture as it is prescribed. So then there is an AI governance team, which also owns the AI gateway, also is responsible for triaging the AI use cases to decide which use case would need what tools and capabilities. And then there is the business units, which will have their own subscription or own resource group with Foundry in there who build their apps and agents within it. So and then they have a top governance layer with Agent three sixty five integrated vendor, you know, complete enterprise setup that then adds to adds to also the security monitoring and governance requirements from enterprises.
That is different flavors we see, all adopted from the reference architecture that we have.
[00:32:57] Barry Luijbregts:
Okay. And and maybe like a final topic ish that you just mentioned agent three sixty five. Because I was gonna ask, so how you have all these agents, they are running everywhere. How do you control them and even know that they exist? It's kinda what it is, right? Yes. So an agent sprawl is
[00:33:15] Heena Refai:
is a topic and a a concern that many enterprises are who are adopting agents and building apps and agents with AI are are noticing it and are trying to find solutions for it. So you can build agents in Foundry. You can build agents in Cobalt Studio. You can build custom agents. You can build agents with the m three six extend the m three sixty five agents that are there or even outside. Right? But then how do you control with what is happening where? Where is my data going? Where is it being used? So with agent three sixty five, are we are at Microsoft, they are bringing a consolidated pane, a control pane, right, for agents built anywhere in their enterprise, anywhere within your tenant.
You could monitor what is happening in the agents, where it is hosted, what tools it's accessing, what data it is accessing, also set up guardrails on top of it to to ensure that they access only the data that they should, the systems that they should, they perform only the set of actions that they should. If there are any threat actors acting on AI systems, then alerts are sent out to the SOC team via Defender for AI integration to also take action in real time. Right? So so that is Agent 365 will become the central control center or command center for all agents built in your organizations, be it pro code, custom agents with Foundry, outside Foundry, or more no code agents built via M365 or CoPilot Studio.
Yeah. So that is where Agent three sixty five comes in. Okay.
[00:34:56] Barry Luijbregts:
So much stuff. It's complicated.
[00:35:00] Heena Refai:
And it keeps keeps getting added. More and more things keep coming in every week, every two weeks. So, yeah, it is quite a changing space.
[00:35:10] Barry Luijbregts:
Space. Absolutely. So what's your favorite model today?
[00:35:17] Heena Refai:
So plot four dot eight is is favorite. So I've that's the developer in me talking. Right? So it's I for me, if I have to build something, that's the one I rely on. But also the auto mode. So talking a bit also about GitHub Copilot Mhmm. Which is the environment which is the IDE I use, basically, which is the agent I use for my development within Versus Code. So for me, four dot eight is what gives me the best results. Five dot five also, GPT five .5 also at times, but I've also recently seen the auto more tends to perform really well if I just want to ask some questions, if I want to understand my code better, then even the auto mode performs really well. So, yeah, that space has evolved. The developer productivity space is evolving as well.
[00:36:07] Barry Luijbregts:
Yeah. Absolutely. Yeah. And that's good. So you're not token maxing. You're just actually turning on auto mode. Indeed. Okay. Good. You should check out the latest GPT 5.6. There are three of them. Oh, I haven't I haven't tested those out yet. They're pretty good. So so for reference listener, we're recording this on the July 14. I'm sure by now you guys have GPT six or beyond. We'll see what happens in the future. So and what would you advise your your children to study?
[00:36:40] Heena Refai:
Oh, to my children, I would I would advise the people to and this is this is a bit a tricky answer, but I would advise them to definitely get their basics right and not rely completely on AI because your AI system is as good as the prompter. Right? So you the prompt the person who is sending the prompt needs to have the fundamentals said straight. And then you see the AI AI system as helping you or augmenting to your knowledge rather than, you know, relying fully on it. So to my daughter, I would say, you do need to study. You need do need to get your basics and fundamentals right, and then use AI as a as a assistant rather than, you know, completely relying on it. Yeah.
[00:37:28] Barry Luijbregts:
Yeah. That's gonna be interesting times. We'll see. Yes.
[00:37:31] Heena Refai:
For sure. I already see that happening. Yeah. Yeah. Lands increasing.
[00:37:37] Barry Luijbregts:
Alright. And where can people go and what's the best place to connect with you?
[00:37:42] Heena Refai:
Yeah. I have my LinkedIn profile. It's by the name of Hina Rifai. You can look me up. Yeah. And just drop me a DM. I usually respond within the day. I'm quite active there. So, yeah, that is the place to reach out.
[00:37:59] Barry Luijbregts:
Okay. Well, thank you very much for for teaching me all this stuff today. I learned a lot.
[00:38:04] Heena Refai:
Oh, thank you. And thank you for having me here. I love the conversation. Yeah. And this was a super nice experience.
[00:38:11] Barry Luijbregts:
Okay. Thank you very much, and thank you all for listening.
[00:38:16] Heena Refai:
Thank you. Bye bye.
Hina's background in AI and machine learning
From big data pipelines to modern ML practice
Why strong data foundations still matter for AI
How generative AI changed the data science mindset
What Microsoft Foundry is and how it evolved
Turning models into agents with tools and data
Hosting agents with Foundry agent service
Evaluations, quality checks, and hallucination control
Managing token budgets, pricing, and AI gateways
Enterprise architecture, governance, and agent sprawl
Favorite models and advice on learning with AI
Where to connect with Hina
