You know, it's not necessary, but in a real application, you know, tracking our AI interactions would be very useful. So let's implement that. To do this, let's create a couple of models. The first we'll just call AI run. The second is going to be AI usage so that we will essentially log two things. So first we will log any time that we interact with AI. That is what the AI run table is going to be for. So it doesn't matter what it is.

That is what the AI run table is going to be for. So it doesn't matter what it is. If we are sending off a prompt to AI, we're going to store it in AI run. Now, if there is a successful response, if we have, you know, actual usage of the AI, then we're going to log that in the AI usage table. And of course, there's going to be a relationship between the two. A usage is going to have a run, but not every run is going to have a usage. So let's start by opening up the create AI runs table migration as well as the AI usages

So let's start by opening up the create AI runs table migration as well as the AI usages table migration. And we are going to start with AI runs. So of course, we want an ID, but we also want a foreign ID for the team. Because of course, a user is going to execute the AI prompt, but that user is a member of the team. So we want this constrained and we want it to cascade on delete. The next thing that we want is something similar, except that instead of team

So we want this constrained and we want it to cascade on delete. The next thing that we want is something similar, except that instead of team ID, it will be user ID. And anytime that we run a command, it is going to be in relation to a ticket ID . So we need a foreign key as well. However, instead of cascade on delete, let's do no on delete. But then we need to know certain things about whatever it is that we were trying to run,

But then we need to know certain things about whatever it is that we were trying to run, such as what kind of feature are we doing? Like for example, our triage agent would have a feature key of, let's say, tri age or something along those lines. In the next episode, we are going to implement some chat or conversation abilities in which case it will have a different feature key. This way we can look at this data, however we need to, if we need to filter it

case it will have a different feature key. This way we can look at this data, however we need to, if we need to filter it based upon whatever feature, then you know, we can do that. So next, we need the status and we also need to know the provider because some of our agents could use one type of provider, other agents could use another being able to know which agent was used for a particular run would be very beneficial. And if we know the provider, it would be useful to know the model that we used

agent was used for a particular run would be very beneficial. And if we know the provider, it would be useful to know the model that we used from that provider. We could also have a hash of the input, just something that is unique for each prompt. And let's also have some other metadata such as when the AI command, if you will, was started at, then we could have a finished at, but you know, it may fail or there could be something

at, then we could have a finished at, but you know, it may fail or there could be something else that goes wrong, or we just want to update this when it's actually complete. And then finally, we want an error message in case if there was an error, and that is of course going to be nullable as well, we'll stick with our time stamps. And that is our definition for the AI runs table. So once again, every time that we run something from an AI, then we're going to log it here.

So once again, every time that we run something from an AI, then we're going to log it here. But of course, if it's successful, if we get a response, we want to log that as well. So for our usages, we will first of all have a foreign ID to the AI runs table, because as I said, every usage will have a run, and we want to cascade on delete, because if we don't have a usage or rather, if we don't have an AI run, then we don't have a usage

don't have a usage or rather, if we don't have an AI run, then we don't have a usage for it. So we might as well just delete it. Next, we want to know how many tokens the response has. So we will have an unsigned integer called prompt tokens. We'll have the default of zero. Let's also have the completed tokens, which we will once again default to zero. Then let's also have a total tokens. Then it would be useful to have the cost of this usage so that we will have

Then let's also have a total tokens. Then it would be useful to have the cost of this usage so that we will have cost USD. You can of course use whatever currency that you want. That can be null and then our timestamps. So let's go ahead and let's migrate database so that I don't forget to do that later. Then we can turn our attention to the models. So let's open up AI run, AI usage, and we'll start with the fillable and casts array.

So let's open up AI run, AI usage, and we'll start with the fillable and casts array. I'm just going to paste it in, because it's of course the attributes that we just described, and then we want to cast the started at and finished at as date times, but then we need to set up our relationships. So we will have our usage, which returns a has one relationship so that we will return this has one AI usage, but then we also need to know what ticket this AI, this

return this has one AI usage, but then we also need to know what ticket this AI, this is the wrong file. This needs to be inside of AI run. So we have our fillable in our casts, we have our relationship to usage, and then we need the belongs to relationship for the tickets. So we will return belongs to the ticket class. So now we can turn our attention to our usage.

So we will return belongs to the ticket class. So now we can turn our attention to our usage. I'm going to paste in the fillable again once again. It's just attributes we defined, but we need to set up the public function run relationship, which is a belongs to relationship. And we will return that belongs to relationship to the AI run class column is AI run ID. So now we just need to open up our ticket triage controller. And anytime that we try to triage, we will log the running of the AI.

So now we just need to open up our ticket triage controller. And anytime that we try to triage, we will log the running of the AI. I don't know what to call that. I don't want to say run AI, but that's the name of it. So maybe we stick with that so that we log running the AI prompt, and then we log the usage of running that prompt. And we can do that before and after we actually run our prompt. So before we run the prompt, we will run AI run and we will create that new record.

So before we run the prompt, we will run AI run and we will create that new record. We set the team ID to the ticket to team ID, the user to the logged in user, the ticket, the ticket ID, the feature is going to be ticket triage. Status is going to be running providers open AI model will set to know and then everything else so that then we will attempt to run our prompt. And after we run it, we want to log the usage, but really we want to do that after we update

And after we run it, we want to log the usage, but really we want to do that after we update after we set the summary and the messages so that here we will update the run, we'll set the status to succeeded, then we will set finished at to now. And if our response has a usage, so we can do this if property exists and we want to check if the response has usage or and this is probably redundant, but it also wouldn't hurt to check if the response usage is set, then we want to update or not

wouldn't hurt to check if the response usage is set, then we want to update or not update. We want to create usage information, the usage property is part of the SDK. So anytime that we send off a prompt, anytime that we interact with the AI, we 're going to get a response and that response could have a usage property is going to contain all of the information that we need about the token usage, basically everything that we are going

the information that we need about the token usage, basically everything that we are going to store in the database. So we're going to check if we have usage and you know, that's just to redundant . We don't need that. If usage is set, then we are going to get the usage so that we can easily store that in the database and it's going to be very simple. We get the run ID, we get the prompt tokens from the usage, the completion

the database and it's going to be very simple. We get the run ID, we get the prompt tokens from the usage, the completion token from the usage, the total as well as the cost, if it's there, and that is really all that we need to do as far as the usage is concerned, we do need a using statement for that. Did we pull in the AI run model, no, we didn't. So let's do that as well. So that now really the only thing we need to do is update the run if there was

So let's do that as well. So that now really the only thing we need to do is update the run if there was some kind of error. So we will update run to where we will set the status to failed. We can also update the finished at to now. And most importantly, we need the error message, which we will get from our exception. But if that's case, we log it, then we're just going to re throw. So with that in place, we should be able to triage a ticket that information

But if that's case, we log it, then we're just going to re throw. So with that in place, we should be able to triage a ticket that information should be stored in the database. And if we take a look at the AI runs table, we have our record ticket triage, it succeeded, provider was open AI, but of course, the more interesting information is the usage. We can see how many are prompt tokens where the completion tokens and in this particular

We can see how many are prompt tokens where the completion tokens and in this particular case, we don't have total tokens, nor do we have the cost. So bear in mind, the SDK gives us the ability to at least have access to that information. But the AI provider and the model have to also provide that information to the SDK. So we might not always get all of the information that we need, or rather, all of the information that we would want to log.

of the information that we would want to log. But you know, that's okay, there's no standard that these AI providers are going to follow. And of course, it would be nice if there was one, and maybe one day there will. But for right now, this is going to be fine. So that means in the next episode, when we implement our ticket assistant agent , then we will be able to log those interactions as well.

we will be able to log those interactions as well.