If you're AI tests hit real providers, then your CI is going to be slow. It's going to be flaky, and most importantly, it's going to be expensive. So let's look at how we can essentially lock down our tests. We'll use some f akes and we will also be sure to prevent our agents from doing anything with a provider. So the first thing that we are going to test is our ticket triage. There's a couple of tests that I want to do. The first is going to test the response. The second is going to test the prompt. And let's start with the response. So we'll just say triage returns

to test the prompt. And let's start with the response. So we'll just say triage returns correct response. I don't know what else to call that naming things is difficult. It always is. So the first thing that we want to do is create a fake ticket triage agent. And we can do that by calling the fake method. This is something that every agent has. If we create an agent class, we can create a fake of that agent class. That way we test and we don't hit our

class, we can create a fake of that agent class. That way we test and we don't hit our provider. It's an awesome feature that well, it's just awesome. So what we want to do here is create a fake ticket triage, but we also want to supply a fake response, because we aren't going to be hitting our provider. So what do we do? We create a new structured agent response, because that is exactly what our ticket triage returns. It returns structured data. So we have

because that is exactly what our ticket triage returns. It returns structured data. So we have a structured agent response. If we have a structured agent response, then it kind of goes without saying that we have other types of responses that deal with streams and other things like that. So our structured agent response needs to return everything that a normal agent response is going to include, such as an invocation ID. It doesn't matter what it is. We can make up whatever

to include, such as an invocation ID. It doesn't matter what it is. We can make up whatever we want. I'm going to use invocation test. Then we need to define the structured data that is going to be returned with this response. And we need, well, a variety of things. The first is going to be the priority, which we can set to three. Then we have the department, which I guess we'll do billing, because this will be a ticket for refunds, basically. So next we have the

billing, because this will be a ticket for refunds, basically. So next we have the sentiment, we want some tags, which for this, we'll just have refund. Then finally, we will have the summary. And once again, you know, this data doesn't really matter. This is all part of a fake response. So we can provide whatever we want. And then we could test with that. But the summary is customer asks about a refund. So as far as our structured data is concerned, that's it.

summary is customer asks about a refund. So as far as our structured data is concerned, that's it. The next thing we need is the text, which once again, does matter, we'll just say structured output. And then we need the usage data, which we can create a new usage object. Then we can just pass in some values. And then we also need some meta information, which we will new up meta, we'll pass in open AI and GPT test. Once again, all of this is just fake information, we can put

pass in open AI and GPT test. Once again, all of this is just fake information, we can put whatever we want. I'm going to stop saying that because I've said it a lot. Now, of course, there are a few use statements that we need to pull in, such as the ticket triage, we need the usage. And we also need the meta. So we have our fake ticket agent, we are returning fake structured data. So now we just need a ticket so that we have an actual ticket that we can tri

structured data. So now we just need a ticket so that we have an actual ticket that we can tri age. So we're going to use the ticket factory to create. And the subject will just be refund requests. Doesn't have to be anything special. And then we want to get to the response from our agent. So we will create a new ticket triage. I really hate that name. I am so, so sorry. And we're going to make that so that we can prompt and we will include the subject and the

sorry. And we're going to make that so that we can prompt and we will include the subject and the ticket subject. So that our fake response is not going to be in this response variable. So then it 's just a matter of our actual assertions. So we expect that the response priority to be three. And then we expect the response department to be billing. And there we go. We are ready to test that. So the fake static method is going to return or really it doesn't return. It essentially swaps the

that. So the fake static method is going to return or really it doesn't return. It essentially swaps the real provider for a fake gateway, which is going to make the test completely local. And it gives us the ability to actually test, you know, structured data. But let's go ahead and run this PHP artisan test tests feature is a feature or features? I don't test enough. So I don't know. There it is. It is feature. Okay. So feature AI. And let's see what we get. Well, we see that one

There it is. It is feature. Okay. So feature AI. And let's see what we get. Well, we see that one failed, which one failed the prompt test. That's okay. But the response test did pass. Now, by calling fake, yes, we are swapping the real provider with a fake gateway. But it never hurts to have, you know, some kind of seat belts just to protect you a little bit more. I say a little bit more seat belts are okay. So let's let's say that fake here is the seat

more. I say a little bit more seat belts are okay. So let's let's say that fake here is the seat belts. So we get in the car, we wear the seat belts. That is our primary safety feature. We want some airbags though. And after we call fake, we can call the prevent stray prompts method. And this is our airbag. It basically makes our fake gateway strict, because if any prompts are actually sent without a fake response, then the test fails immediately. And the great thing is it's not just

sent without a fake response, then the test fails immediately. And the great thing is it's not just about prompts. We also have similar tools for testing other kinds of agents, like embeddings. We can also use it for tools like files and stores and things like that. So it's great. We're good . So this is our first test. Let's copy it, because we want to implement the other test. So let 's paste this in, because anytime that we want to test our ticket triage with the fake gateway,

's paste this in, because anytime that we want to test our ticket triage with the fake gateway, we have to supply a structured object. So this just keeps us from having to type everything again . We also have to have a ticket that we would want to use with this test. But here's the main difference. I don't want the response. I don't care about the response. I just need to make sure that the ticket triage was prompted. So we have this assert prompted method. And as you can see, Int

the ticket triage was prompted. So we have this assert prompted method. And as you can see, Int elliSense is popping up several other methods as well. But we are interested in the assert prompted so that we will get the prompt. And we just want to ensure that the prompt contains the text that we used for the prompt, which was refund request. So if we go back and run our tests again, both of those should pass. One of them did not. And yeah, because we didn't actually prompt, did we? I

those should pass. One of them did not. And yeah, because we didn't actually prompt, did we? I deleted that one line. So we do care about, well, we still don't care about the response. We do care about if we prompt. I guess that that's the thing. So now that test should pass. And we have two passing tests. And we don't have to worry about those tests hitting our provider, which is great. It 's saving us money. So now I want to test the embeddings behavior. I want to ensure that the embedd

's saving us money. So now I want to test the embeddings behavior. I want to ensure that the embedd ings are generated. And to do so, we are going to use a similar strategy. So let's open up that embeddings generated test. Once again, we don't care about those things. But we do want our embeddings. And just like with our agent, we are going to use the fake method. And we need to provide a fake vector. And it doesn't matter what these values are, as long as they are valid

provide a fake vector. And it doesn't matter what these values are, as long as they are valid vector values. So by using the fake method, we are swapping to the fake gateway. Let's change the description so that it fakes embeddings generation. And so now that we have these fake embeddings, we can embed that doesn't sound great. So we will use embeddings for and it doesn't matter what it's for, because all we care about is if we can generate them, and we

doesn't matter what it's for, because all we care about is if we can generate them, and we want the first, so that we can expect the embedding to be array. And then we also wanted to have count of three. So once again, if we run our tests, we should now have three passing tests. But once again, you know, we have our seat belt, it would be great to also have our air bags. So we will call prevent stray embeddings, making it so that our tests will automatically fail,

bags. So we will call prevent stray embeddings, making it so that our tests will automatically fail, if for whatever reason the fake gateway is bypassed. So let's do one more test. We've test prompts responses, embeddings, let's create a test for testing file and storage ops test. For our description, fakes provider file and store operations. And you'll never guess what we're going to do. Files fake stores fake, therefore switching both of these to use the fake

going to do. Files fake stores fake, therefore switching both of these to use the fake gateway. So we can create a file using files, put we want document from path. And let's see if we can just pass some path here, let's pull in the document. So that gives us a file, we want a store to store the file. So we'll call this test store so that we can then use the store to add the file. And we'll set it for team ID of one. So our expectation is going to be file ID is not to be empty.

And we'll set it for team ID of one. So our expectation is going to be file ID is not to be empty. But once again, let's put in the airbags so that we prevent stray operations for the files and for the stores, we prevent stray operations. So if we were to run our tests now, we have four passing tests. So the key takeaway is that, well, the easy thing to say is that everything is testable. You don't need API keys. The fakes and using fake responses make your tests

testable. You don't need API keys. The fakes and using fake responses make your tests deterministic. But you can also test behavior, just like we did with testing the prompts. And while using the fakes are going to protect you from actually testing against the provider, you can always use the fail safe to prevent stray prompts and stray operations. That makes your tests safe. And you don't have to worry about hitting a provider.

tests safe. And you don't have to worry about hitting a provider.