All right, welcome to a new chapter. We're now gonna move on to semantic search, but I'll warn you, I think at least initially this can be a little overwhelming because there's a lot of new concepts and terms and keywords to be aware of. But don't worry, I'm gonna simplify this as best as I can. Alright, so first up, what is semantic search? Well, think of it like this. It allows you to, to fetch results that are conceptually related to your query,

It allows you to fetch results that are conceptually related to your query, even if your actual query doesn't show up in the document itself. And here's what I mean. I'll give you an example. Let's say you search for dog. All right? Well, traditionally with a traditional approach, it's gonna scan your collection and find any references to dog, right? And if dog shows up, it gets included in the result set.

and find any references to dog, right? And if dog shows up, it gets included in the result set. However, if one of these documents refers to how to take care of your pets, well, very possibly you would want that, uh, document included as well. But it's simply not gonna happen, right? Dog does not equal pet, so it's not included, right? That's a shame. Here's another example. Let's say you search for car. Well, once again, it's gonna fetch any documents

Let's say you search for car. Well, once again, it's gonna fetch any documents that reference a car, but I don't know. What if a document and, and remember document is record in your database. Uh, and that's the parallel anyways. What if one of these records references automobile? Well, that probably is a match, right? I'm searching for car. If there's a reference to automobile, that should be included.

I'm searching for car. If there's a reference to automobile, that should be included. But again, traditionally it's not because car does not equal automobile. So wouldn't it be cool though if there was a way to match the vibe of what I'm searching for rather than the exact sequence of characters, right? The general vibe of my query is just things related to cars. And yes, that would include automobiles. Uh, yes, that would include anything else you can think of.

And yes, that would include automobiles. Uh, yes, that would include anything else you can think of that would be related to that term. So we can allow for that through semantic search. And I'm gonna show you how. Now to allow for this, we effectively need to convert the relevant portion of our document into which known as an embedding. An embedding is just a numeric representation of that document. Or I bet you've seen this before.

of that document. Or I bet you've seen this before. It's an array of floating point numbers, a very long array of floating point numbers. All right? And here's the cool thing. You don't need to know what those numbers do, trust me, you don't. But what you do need to know is once again, that embedding is a numeric equivalent or representation of the documents.

that embedding is a numeric equivalent or representation of the documents. All right? So this is where it gets kind of cool. What if we also converted your search query dog into an embedding as well? Well, now we have an embedding for the query and we have an embedding for the documents. Could we then write a mathematical equation to compare how close they are to one another? And yes, we can. This is known as cosine similarity.

how close they are to one another? And yes, we can. This is known as cosine similarity. So the way it works is it's almost like geography. If they are conceptually related, they will be closer to one another. And, and by the way, uh, machine learning and LLMs will construct these embeddings in such a way so that uh, if they are conceptually related, they are closer to one another. Okay? If they are not conceptually related,

they are closer to one another. Okay? If they are not conceptually related, if they have totally different vibes, if maybe, uh, your query is love, but all your documents are about PHP, maybe that distance is going to be quite high. All right? So again, that's something to be aware of with cosine similarity. A smaller number means closer and closer means closer together.

A smaller number means closer and closer means closer together. More related. Alright, let's write some code. Let's begin with config/scouts. And here is the schema for our course. And yeah, if you think about it, we would need an embedding for this document and it would probably make sense to include the name and description, uh, along with that. Okay, so we have two ways to handle this. The first option is the manual approach. Mm-hmm.

Okay, so we have two ways to handle this. The first option is the manual approach. Mm-hmm. You would maybe have an artisan command, you would loop over your courses and for each one you would concatenate to the course name and the description, and you would fire off an API request to something like open ai and you would say, Hey, here's a string. Convert this into the embedding. All right? OpenAI will do that. It'll return it as a response.

Convert this into the embedding. All right? OpenAI will do that. It'll return it as a response. You then save it to your courses table or pivot table or just sync it up with type sense. Okay? So now on the type sense end, for each document you would have the course name, the course description, and the course embedding, which again is a numeric representation of the name and description.

which again is a numeric representation of the name and description. Alright? So if you took that approach, you would need to set up this field. We would call it embedding the type, remember we said was an array of floating point numbers. So let's do floats. Um, you need to decide could there be situations where a course does not have an embedding? Very possibly. So I might set this to true.

where a course does not have an embedding? Very possibly. So I might set this to true. And then finally, we should specify the number of dimensions. Just think of that sort of like the, uh, the number of items within that embedding array. Uh, of course the more items you have, well the more targeted you can be, the more context you have, but also it's just much more data.

the more context you have, but also it's just much more data. So you'll need to research this on your own to figure out what the correct number might be. Maybe it's 256, maybe you want 1536 as you have here. That's what I use. But yeah, just research that a little bit on your own. Alright, so yeah, this is what you would do. You now have, uh, a new field, uh, within your schema. You would generate those embeddings

You now have, uh, a new field, uh, within your schema. You would generate those embeddings and then you would sync those up. However, there's a second option that is much easier type sense can just do the workforce, which I really like. We can set up a local model and we can instruct it that hey, we need to make sure that for this schema we always have an embedding that consists of the concatenated name and description. And yet you don't have to do any of

of the concatenated name and description. And yet you don't have to do any of that work at all, which is really cool. And that's the approach we're gonna take. Okay, let's get started. So to start from scratch, once again, we would create a new field where the name is embedding. The type is an array of float values. Uh, but this is where we branch off a little bit. Check this out, we'll say embed

Uh, but this is where we branch off a little bit. Check this out, we'll say embed and where is this coming from? Well, what fields are we going to reach for? Well, like I said, we're gonna use name and description. And when I have more than one, like I said, it's just going to be concatenated. Okay? So we're creating an embedding from or using the name and the description. Next I need to specify the model config.

or using the name and the description. Next I need to specify the model config because as you can imagine, there's lots of different models and they're constructed in different ways and some are more efficient than others. It really just depends. And you have to do, uh, a decent amount of research to figure out what model you want. If you use OpenAI for everything, of course you can use a third party uh, model,

If you use OpenAI for everything, of course you can use a third party uh, model, but you can also use a local one that gets downloaded by typesense. And that's what we're gonna use here. Okay? So have a look here and you can see a long list of supported models. Like I said, nobody expects you to know all of these. Just do some research to figure out what your use case is and which one might be the best fit.

Just do some research to figure out what your use case is and which one might be the best fit. Uh, but yeah, maybe we'll use this one at the top for brevity. So we can just copy that and that is our model name. Alternatively, if you're using open ai, then you might wanna reach for the text-embedding-ada-002 model. That's the common one at the time of this recording. So if you wanted to take that approach, yeah,

That's the common one at the time of this recording. So if you wanted to take that approach, yeah, you would specify the model name and that would be textEmbedding three, uh, small. And then you would also need to generate an API key, uh, through your OpenAI dashboard and then reference that, uh, maybe something like this, openAiKey and that should do the trick. But like I said, we're just gonna use a model that gets automatically downloaded, uh, by typesense.

But like I said, we're just gonna use a model that gets automatically downloaded, uh, by typesense. So I can do typesense/Model that we referenced. And that should do it, believe it or not. Okay, so now what have we specified here? We specified that whenever a new document is created or updated an embedding is going to be generated and it's going to use the name and description fields concatenated together. Alright, so now I think we're all set to go.

and description fields concatenated together. Alright, so now I think we're all set to go. Let's try this out in the terminal. So we've changed our schema, right? So we should run php artisan scout:flush on app/Models/Course. Right next we should reimport, but now check this out. If I were to run php artisan tinker, let's say app\Models\Course::factory()->create(['course_name' => 'all about type sense']), you get the typos for free.

and the course name will be, uh, all about type sense, you get the typos for free. Okay? So now think about it, that took just a tiny bit longer, right? And that's because as part of that we also generated the embedding and it got synchronized with type sense. So check this out. We're gonna run a curl command. Let's see if I can find it. Uh, pretty close, not quite though.

Let's see if I can find it. Uh, pretty close, not quite though. We want to hit the typesense port 8108/collection/courses. And then we're gonna search through, through the documents. So let's set the query to * for now. And yeah, here are the results. Here's the, the insides of that index. Uh, so sure enough we can see a record where the course name is all about typesense.

Uh, so sure enough we can see a record where the course name is all about type sense. And check it out. You have a massive, massive, massive, massive, massive, uh, numeric representation of the course. Name and description. How cool is that? And it was done automatically for you. Look, I'm still going here. There we go. We finally get to the embedding. Okay? So now think about it. We can perform a search, uh, we can accept that query from the user.

So now think about it. We can perform a search, uh, we can accept that query from the User. We convert the query into an embedding and then we compare them. And again, we don't have to do it ourselves. We can just have type sense do it for us, which is the best part. All right, let's get going. All right, back to my editor. Let's return to our routes file. And you're gonna love this, let's say return course.

Let's return to our routes file. And you're gonna love this, let's say return course and let's do it one step at a time. We're gonna search for type sense that was included in the course name and we will fetch the results. Okay? So no semantic search going on here, just traditional search, uh, gets compiled into a collection of relevant courses and we return it as json. So if I look in the browser, oh, we get nothing.

of relevant courses and we return it as json. So if I look in the browser, oh, we get nothing. And you know what? I bet that is, let's go back to our Scout file. Ah, yes. Um, when we were playing around, I must have set the default query. Buy two description, let's query by name or name and description if you want. Okay, back to Chrome. Give it a refresh. And sure enough, this is basic search.

Okay, back to Chrome. Give it a refresh. And sure enough, this is basic search. We know how to do this. It uh, it returns this result, which is great. Okay? But yeah. Now if I were to look for maybe search engine, something like that, well come back to Chrome type search does not equal search engine. So it doesn't get included in the results. So now let's switch over to a semantic search. And here's the best thing. All I need to do is override

So now let's switch over to a semantic search. And here's the best thing. All I need to do is override how we are querying. So instead of querying by the course name, we're gonna query by the embedding. And here again, here's the best part, notice that at no point did I convert search engine into an embedding myself. That's going to happen automatically by typesense. Okay? So if I were to come back one more time

That's going to happen automatically by type sense. Okay? So if I were to come back one more time and give it a refresh fresh, whoa, it's ridiculous. I mean, I've done this for a while and it still kind of blows my mind that this all just works. It's, it, it's, it's ridiculous. It's so cool. We search for search engine, okay? Behind the scenes search engine gets converted into an embedding and then we check the nearest neighbor, we check the cosign similarity against all

embedding and then we check the nearest neighbor, we check the cosine similarity against all of the embeddings for each of the courses. And the ones that are closer are closer matches, right? Or better matches, but it's not entirely clear to me, right? So for example, if we, let's do this, let's just read from the request, all right? And let's once again send through Taylor Otwell's favorite book and notice we look for twilight,

through Taylor Atwell's favorite book and notice we look for twilight, or he's searching for twilight for a second read, and yet it still returns type sense. Okay? And this can be a little confusing, I guarantee this is going to be confusing. Why do I search for twilight? But I see type sense. Well, there's a couple reasons. First up type sense is the only record that we have in our index currently.

First up type sense is the only record that we have in our index currently. And second, it doesn't know like which items to be included or which items to not be included. It just knows how close they are to one another. So I'm gonna make this a little more clear by changing this from get, which gets us a collection of courses and I'm gonna switch it to raw, which gives us the raw response, alright? And now there's two things I want you to be aware of.

which gives us the raw response, alright? And now there's two things I want you to be aware of. First up, notice that for each course or each document and the response is also the embedding. And I wanna remind you just how massive this is. And this is just one course embedding. If I have a hundred results, then I would have this times a hundred and it's just not necessary. I'm not gonna do anything with it. Mm-hmm.

and it's just not necessary. I'm not gonna do anything with it. Mm-hmm. So as a general good practice, exclude the embedding from your search results and you do that via your search brands. You could say exclude fields and I will reference the embedding. There we go. And now it's gone. Uh, yeah, there's no point in paying the cost of that payload if you're not gonna do anything with it.

Uh, yeah, there's no point in paying the cost of that payload if you're not gonna do anything with it. Okay? So next up point your attention here, vector distance. Again, this is the cosign similarity. This is the distance between your search query and the course embedding here. And it's gonna be somewhere around zero to two. Here it is, uh, 0.95. So once again, the smaller, the lower that number, the lower the distance, the closer the match.

So once again, the smaller, the lower that number, the lower the distance, the closer the match. So let's do this. Let's create a couple more records. php artisan tinker, and I will do another one. We'll say Laravel techniques. All right, we'll do another one that is the love doctor and we'll do another one. Uh, something totally unrelated. Um, dogs and cat training or something like that. Okay? So now we have a bunch of courses.

Um, dogs and cat training or something like that. Okay? So now we have a bunch of courses. However, here's one thing to be aware of. The description is included as part of the schema, right? And because we have gibberish here, our score may not be quite accurate, right? Because maybe some of this is getting reused. So with that in mind, here's what I think I would actually like to do. Back to config/scout.

what I think I would actually like to do. Back to config/scout. And let's say the embedding is just going to consist of the name here, okay? That's the only thing I care about currently. Just remember in real life, you probably would want that description included. Alright? So once again, we flush and then we import, alright, back to chrome. We give it a refresh and now we have a bunch more results.

and then we import, alright, back to chrome. We give it a refresh and now we have a bunch more results. So we are currently searching for type sense and sure enough, that's going to be the top results. Notice the vector distance is now, um, much smaller again because we're not including that gibberish text. Uh, but if we close this out, here's the next result. And the vector distance is 0.88 is 0.9294. And notice it goes up.

0.9294. And notice it goes up. So again, the smaller the distance, the closer the match, which is why this one of course is going to be the best match. Uh, let's instead search for dog and of course that one's going to be included, but we're doing semantic search here. So we could also do pets and that one gets included. How about teaching pets? Yep.

So we could also do pets and that one gets included. How about teaching pets? Yep. And notice, even though again, my search query is not included here, it's checking the general vibe, uh, the concept of what you're searching for. And of course this is going to be a match. And notice the vector distance is quite low, whereas if I scroll down, the least match is gonna be something very technical, uh,

whereas if I scroll down, the least match is gonna be something very technical, uh, like Laravel techniques, alright, in that desert. So I get it. This is so much to take in at least initially. Suddenly you're learning about semantic search and vectors and co-sign similarity and numeric representation and vector distance. It's just a lot, right? But hopefully you now understand the basic approach. If you ever want to perform a search based upon the general

understand the basic approach. If you ever want to perform a search based upon the general vibe of the user's query rather than the exact sequence of characters they type in, then you wanna reach for semantic search. And again, you do that by converting their query into an embedding and then you compare that against the embeddings that are stored within, uh, in this case the courses table and the ones that are closer are just naturally

that are stored within, uh, in this case the courses table and the ones that are closer are just naturally going to be better matches. So here's where I'm gonna leave you all with. Notice, there's no real logic that determines whether or not one is included or one is excluded. Instead it's just ordered by the vector distance. So notice if I were to search for gibberish, right, I would still see matching records. And again, that's because it's still a match.

I would still see matching records. And again, that's because it's still a match. It's just a match with a very large vector distance. So that is something to be aware of. Okay? I will see you in the next video.