Issue go without saying that our applications are completely dependent upon the system and the system resources such as CPU and memory. But between those two things, memory can be the most problematic. And, and really it's not the memory's fault, the memory's just sitting there. The problem is sometimes we build memory leaks into our applications.

The problem is sometimes we build memory leaks into our applications. Of course we don't mean to, but it happens and we need to know how to fix it. So here we have a job to generate receipts for the orders, and this runs nightly, but unfortunately the guys are having to restart this process multiple times. And the problem is obvious that it, it's using too much memory.

And the problem is obvious that it, it's using too much memory. It, it has, well, it needs more memory than what we can throw at it. And all we are doing here is fetching all of the orders because we want to generate receipts for those orders. And then we iterate over the orders so that we can generate the content for the receipt. Then we get the file name, we store the receipt, and that's essentially it.

Then we get the file name, we store the receipt, and that's essentially it. You know, we have some other methods, but those are for generating the content. So I have a command that we can use to run this. It's called demo memory leak because it has a memory leak. So we'll run this and we can see that the memory was set to a certain amount. And that's just for the sake of this demo, because at some point in time we have to error out

And that's just for the sake of this demo, because at some point in time we have to error out because we've used too much memory. But one thing to notice is that the memory keeps building, you know, it doesn't decrease, it doesn't stay constant. It starts at 100 megabytes. And then as we process through, we end up at 108 megabytes and we don't even get all the way through. The 5,000 orders that we have, we error out just at the end.

The 5,000 orders that we have, we error out just at the end. And that's frustrating. So let's identify where the memory is being consumed and it's never released. And we start here where we fetch all of the orders. That's what we are doing. We are loading all of them into memory at once. And we're not just, you know, pulling in the orders, but we are also getting the items, the products,

And we're not just, you know, pulling in the orders, but we are also getting the items, the products, the user relationships. So it ends up being thousands of eloquent model objects just sitting in ram. So this single line can use a ton of memory. And in fact, whenever we run this, we see that it starts at 100 meg. So that's a lot of meg for in, uh,

that it starts at 100 meg. So that's a lot of meg for in, uh, just processing these orders. But that isn't the only problem because here we start to iterate over those orders, but we are still holding onto those orders in memory. We have that entire collection. So even after we process an order, you know, you would think that we process an order, we're done with it. That's outta memory. No, we are still holding onto

that we process an order, we're done with it. That's outta memory. No, we are still holding onto that order and we're not letting go of it. And this is a very common problem because essentially what we end up doing is processing data as if memory is infinite. We have this infinite amount of memory and we'll just process whatever we need to. That's never a good way of processing data because memory is finite.

That's never a good way of processing data because memory is finite. Even if you are running on a dedicated machine, it's important to still, keep in mind that memory is extremely important because if you run onto memory, the application fails. Now, Laravel gives us two tools that we can use to solve this particular problem. The first is lazy. The second is chunk. They essentially do the same thing.

The first is lazy. The second is chunk. They essentially do the same thing. The the primary difference is just how we use them inside of our code. But the idea is that we get to process our orders in certain sized chunks. So if we wanted to process 50 orders at a time, we can do that or 100, or maybe we want to do 250. There is no magic number here. It takes a little trial

There is no magic number here. It takes a little trial and error to see what works best for the process that we are currently working with. Now the, the thing about the lazy method is that we can use a fluent style of programming so that after we get our chunk of records, then we can iterate over each of those. And yeah, that's that. That's gonna be it. So what we have to do then is just take, well,

And yeah, that's that. That's gonna be it. So what we have to do then is just take, well, we don't need that log message there. We need to take everything inside of this for each loop, and we are going to cut that out and paste it inside of the callback that we passed to the each method there. But I wanna line everything up so that it looks pretty and we can get rid of that for each loop. So that is going to make a dramatic difference.

and we can get rid of that for each loop. So that is going to make a dramatic difference. It is going to affect the time that it takes to process all of those, but memory well, we'll see. So instead of starting at 100, we started at 32 megabytes and it grew a little bit, but we can see that now we are not idling, but it's at 58 megabytes and we aren't going over that. So we processed all 5,000 orders

and we aren't going over that. So we processed all 5,000 orders with 58 megabytes of memory. That's, well, that's almost half of what we did before. And of course, chunk size is going to make, you know, a little difference in that. If we had a chunk size of 500, well, let's see what happens there. If we run this again, yes, the memory is going to grow a little bit

If we run this again, yes, the memory is going to grow a little bit because that's just part of processing data, but it looks like we are topping out at 54 megabytes, which is just a little smaller than what we had before. And okay, it just jumped to 62. But still, I mean, we are dramatically more efficient from a memory perspective than we were before because now we are actually processing smaller chunks of records as opposed to working with,

than we were before because now we are actually processing smaller chunks of records as opposed to working with, you know, the whole thing. Now, as I said, you know, we could use the chunk method here instead, which is practically going to be the same thing, except that the syntax that we use is gonna be a little bit different. So we'll call chunk and then you know, our chunk size, and then we have a callback function that executes

So we'll call chunk and then you know, our chunk size, and then we have a callback function that executes with the chunks of orders that we need to work with. So here we can go back to that for each loop if we wanted. Let's just copy all of that. Well, we need to comment all that out. So let's comment all of that out. Then we will paste a tier inside of the chunk method and we're gonna see the same result. Or maybe not the exact same result,

and we're gonna see the same result. Or maybe not the exact same result, but it's going to be the same except there's an error. And O of course need to iterate over it, But it's gonna work now 42 meg. 52 MEG 54. So roughly the same because this looks like the same kind of behavior. Then it's gonna jump to over 60, maybe it'll be 62.

because this looks like the same kind of behavior. Then it's gonna jump to over 60, maybe it'll be 62. Was it 62 last time? I don't remember. But you get the idea. We have the tools built in to work with a large sets of data efficiently. Oh, 54 meg, that's even better. Now, of course, I feel like that I need to point out that this is a discussion primarily targeting queues because you know, a typical web request, the server gets a request, there's a little processing,

because you know, a typical web request, the server gets a request, there's a little processing, it gives the response, and then everything about that request is gone. So we can be a little sloppier with the resources that we have, but don't be sloppy. We need to be good stewards of the resources that are given to us within our application. Now, cues on the other hand, you know, cues are there for processing large amounts of data

Now, cues on the other hand, you know, cues are there for processing large amounts of data or running long running processes. Those are where we need to really be concerned about the resources that we have, especially memory. But you know, the, the fix is very simple. Just use lazy or chunk process your data in batches and that's going to solve a very large part of the problem. Of course, do your own testing for your own processes,

and that's going to solve a very large part of the problem. Of course, do your own testing for your own processes, find the right batch size and you won't have an issue.