If you're like me, you burn through your tokens very quickly and if you're developing AI features, well, you need to protect yourself from that. You need some guardrails. And one of the ways that you can do that is by logging everything. I mean, everything and we do a pretty good job of logging things, but we could do a little bit better. And I say a little bit better because, you know, there's so many things that could

And I say a little bit better because, you know, there's so many things that could make it better. Like, for example, if we look at our ticket triage controller, so not only do we create the AI run, but then we also manually create the AI usage, it would be nice if we just logged the usage period, anytime that we interacted with the AI, be it for prompting or streaming or generating embeddings, you know, things like that, it would be

prompting or streaming or generating embeddings, you know, things like that, it would be great if we could just, if that happens, we log it and then we're done. We can. And that's what we're going to do. We are going to modify our migrations. And the reason why we're just going to modify these migrations as opposed to creating new ones is because I'm just going to wipe everything out of the database just so

creating new ones is because I'm just going to wipe everything out of the database just so that we have some fresh data to look at. And, you know, there's nothing that's going to be, you know, well, is this old data or is this new data? We're just going to start completely over. So for our AI runs table, we are going to add a new column. It's a string column, and it's going to be called invocation ID.

So for our AI runs table, we are going to add a new column. It's a string column, and it's going to be called invocation ID. Now this can be no. And we also want to index this because this is now going to become the way that we link a run with a usage, because any time that we run, you know, any AI thing, we're going to get this invocation ID, which is going to allow us to correlate between the two. And that's going to be important because what we are going to do is use events,

two. And that's going to be important because what we are going to do is use events, events fire all of the time. And so whenever an event fires for when we prompt or when we stream or when we generate embeddings or anytime we do anything, we aren't necessarily going to have. The AI run ID, but we will have the invocation ID. So that's what we're doing. We are going to add this invocation ID column to the AI runs, as well as the AI

So that's what we're doing. We are going to add this invocation ID column to the AI runs, as well as the AI usages, which means we're going to get rid of the foreign key for the AI runs table. And we're just going to add the invocation there. It really doesn't matter as far as the order is concerned. And that's going to at least get us started so that we can migrate our database , wiping everything out, starting from scratch. And we are good there.

everything out, starting from scratch. And we are good there. So the next thing that we need to do is, you know, set up the ability to listen for events. And there are a lot of events that occur anytime we prompt, anytime we stream, generate embeddings, anytime we chat, you know, all of these things have events that we can listen for. And then we can log them. Since there's so many that we can do, we're just going to pick one and run with

And then we can log them. Since there's so many that we can do, we're just going to pick one and run with it. And it's going to be prompt because chances are the majority of the time we are going to be prompting things. I know that, you know, we've streamed and things like that, but we're just going to do this once so that we can see the process and go from there just to save time and be simple.

this once so that we can see the process and go from there just to save time and be simple. So we need something that is going to handle that event. So let's make a class, which we'll call listeners record AI usage. And if we open up record AI usage, all we need is a method called handle, we're going to handle the prompted events are actually it's called agent prompted event. The first thing that we are going to do, so we are going to get the usage and we get that from the event response, hopefully we have a response so that we can get

we get that from the event response, hopefully we have a response so that we can get the usage there. Because if we don't have a usage, then there's nothing left to do, we're just going to return and we're going to be done. But then to, you know, this is going to execute for every prompt. We don't want to write to the database for the same prompt. So we will check to see if AI usage where the invocation ID is the same as the

We don't want to write to the database for the same prompt. So we will check to see if AI usage where the invocation ID is the same as the event invocation ID. And if that exists, then too, we want to return because if that record is there , we don't want a duplicate. While I'm thinking about it, we need to open up the AI run model and the AI usage model because I will forget to do this.

usage model because I will forget to do this. We need to add the invocation ID to the fillable array because we all know what 's going to happen if we don't. It will be no. And then we'll scratch our heads as to why that's the case and then we'll beat ourselves up trying to figure out what's wrong and then lo and behold, it's because we did something

up trying to figure out what's wrong and then lo and behold, it's because we did something stupid. We didn't add that to the fillable array. So if we have a usage already, then we don't want to do anything at all. Otherwise, we will create a new record and this is where we can go from our ticket triage controller. Well, we could just copy that because that's essentially what we want. There are, of course, going to be some differences such as we don't have the

Well, we could just copy that because that's essentially what we want. There are, of course, going to be some differences such as we don't have the running. No more. So we'll take that out. Usage prompt tokens is going to be fine. Something I've found out cost USD no models don't give us that information doesn't total tokens that's not really there either. So what we are going to have to do as far as that is concerned is come up with

tokens that's not really there either. So what we are going to have to do as far as that is concerned is come up with prompt tokens and we're going to set that to our prompt tokens there. Then we will have the completed or the completion tokens, which will be that expression there. And then for prompt tokens, we will, of course, have prompt tokens for completion tokens. We will have completion tokens. And then for total tokens, we will take both the prompt tokens plus the

We will have completion tokens. And then for total tokens, we will take both the prompt tokens plus the completion tokens. So there we have our totals. And then we need the invocation ID set to the event invocation ID. So inside of the ticket triage controller, we can get rid of where we write that usage to the database because that's going to be automatically handled for us. But we do need to update our ticket here so that the invocation ID will come from the response

But we do need to update our ticket here so that the invocation ID will come from the response object. So if we have a response object, we will have an invocation ID and it is a property. It is not a key in the array. So with that in place, all we have to do is wire up the event. So let's do that by making a new provider. We'll just call it event service provider. And inside of here, we don't need the register or the boot method, we'll just

We'll just call it event service provider. And inside of here, we don't need the register or the boot method, we'll just have the protected listen array so that we will listen for the agent prompted event. And we have the record AI usage class that will handle that event. So with that in place, now we don't have to set up the event service provider since we used artisan to create that, it's automatically added to the providers file. There it is. I love that feature that is so nice and it saves time.

There it is. I love that feature that is so nice and it saves time. So with that done, everything should be in place so that whenever we prompt for our ticket triage agent, it should be logged in the database. So let's pick a ticket, any ticket doesn't matter, let's triage and then we will check the database. Here we can see the data in the AI runs table. Let's scroll all the way over the invocation ID is null.

Here we can see the data in the AI runs table. Let's scroll all the way over the invocation ID is null. If we look at the usages, the invocation ID is not null. So what we should do then is update later on so that here when we have succeeded, then we will add the invocation ID. If we fail, we will add the invocation ID or not, but I guess, you know, really , it doesn't matter for a failure, but for a success, we definitely want that. So let's try it again.

matter for a failure, but for a success, we definitely want that. So let's try it again. Let's pick another record. Let's triage and inside of the database, we can see that now we have that invocation ID, but the AI usages has the same invocation ID. And if we take a look at the usages, now we can see that we have prompt tokens, completion tokens, and then the total tokens. Now of course, we would want to do this for every event, just so that it doesn

tokens, and then the total tokens. Now of course, we would want to do this for every event, just so that it doesn 't matter if it's prompting or streaming or whatever. We would be capturing that information so that we would have at least some accurate and up-to-date information about what we were using. Because the guardrail that we need to set up is to see how many tokens we have used within a day.

used within a day. So let's open up our AI config file. And it would be nice if we had this just kind of built in, but we don't. We can easily add it though. What I want is a daily allowance for the amount of tokens that we could use. So we could do something like daily team token budget. And ideally it would be an environment variable that we could call daily team token yada yada yada.

token yada yada yada. But let's just do this. We could set it to, you know, 50,000 so that now that we have a total amount of tokens that we can use to calculate how many tokens we've used within a day, we can actually use this to check and a great place to do that would be inside of some middleware. So let's use artisan to make some middleware and we'll call it enforce AI budget.

So let's use artisan to make some middleware and we'll call it enforce AI budget. And all we really want to do is calculate how many tokens have been used for the day. And if we reach that threshold, well then no more AI for you. So let's first of all, let's see if we get the agent prompt, but that's not really what we need. In fact, we don't need any of that. Instead, what we need is a request and the very first thing that we will need

In fact, we don't need any of that. Instead, what we need is a request and the very first thing that we will need to do is get the team that the current user is in. So we'll get the team ID from the request user current team ID. And if we don't have a team ID, then we will just return next passing in the request. Then the next thing that we want to do is get how many tokens that we have used today. So we can call this tokens today, we will get that from our AI usage and we

today. So we can call this tokens today, we will get that from our AI usage and we want where the team ID is the team ID that we retrieved, where the dates and we're going to use the created at is now to date string. And then we just want to sum the total tokens so that then it just becomes a very simple comparison. If tokens today is greater than or equal to config AI, and I forget what I

comparison. If tokens today is greater than or equal to config AI, and I forget what I called it, daily team token budget, then we want to fail. So we will return response. Let's do adjacent structure that has a message of daily AI token budget reached . And then we can use for 29 as the status code. But then of course, we need to return next passing in request. That way we don't break the middleware chain.

But then of course, we need to return next passing in request. That way we don't break the middleware chain. So from there, it's just a matter of opening up app dot PHP. Is this bootstrap? Nope. Let's config. We need bootstrap app dot PHP and inside of the with middleware method, we want middleware alias. And then we will say AI budget for app HTTP middleware and for AI budget.

alias. And then we will say AI budget for app HTTP middleware and for AI budget. I'm going to leave it. Know that that's wrong, but it's there, so it's going to stay there. So that then, of course, we can just go to our routes. And if we wanted to include the middleware here, we could just say AI budget. And then all of our AI routes are being for the lack of a better term protected by our budget middleware. Now of course, the next thing is tiering.

budget middleware. Now of course, the next thing is tiering. And this is something that we've talked about already. If we open up our ticket to triage, you know, we have tiered this using the cheapest model. So the basic idea is to think about what tier of a model that you want an agent to use. Keep it simple jobs cheap and then save them more expensive models where they are actually needed.

are actually needed. Now, of course, all of these things are very important because AI isn't a one time purchase. It's a usage based service, and that means every feature that you ship has a cost curve. So you need to make cost a first class requirement.