Memory Limit Problem0:29
issues. Let me show you a couple of examples. I'll just use the routes file here in Laravel. Let's say I'm going to build up a range of 1 million items. So 1, 2, 3, 4, 5, 6. And then we'll just return done here. So if I switch over, and it actually works here surprisingly, but that's because locally I have my limits set very high. Let's add one more zero though, and if I give it a run, there we go. We hit the memory limit.
Let's add one more zero though, and if I give it a run, there we go. We hit the memory limit. So what's happening here is we're trying to build up this array in memory, and we have so many items here, php basically collapsed and said, I can't do no more. It's like Drago at the end of Rocky IV. He finally gets knocked down, he's trying to get up, and he just can't. He reached his limit, and the same is true here. Okay, so in situations like this where you're dealing with massive data sets, and as part of this, you can think of situations where you're iterating over all of the items in a massive database table, you'll encounter the same memory issues.
Using Generators Instead1:19
of this, you can think of situations where you're iterating over all of the items in a massive database table, you'll encounter the same memory issues. So instead, we can reach for a generator, and with that approach, rather than building up this array in memory all at once, we can instead keep it as minimal as possible. It'll only build up as much as it needs to know the current value, what the next one is, things like that. Now in the case of range here, we could rewrite it. This is a super common example. We could say, begin at something, and then where to end, and you know, we would want to do some basic guarding.
We could say, begin at something, and then where to end, and you know, we would want to do some basic guarding. I'm going to do a very trivial implementation of this, where we just say, start at the beginning, and as long as the currentItem is less than what our limit is, then increment. And now here, rather than returning anything or writing to an array, we don't want to build that up in memory. Instead, I'm simply going to yield the currentValue. So now, when I call this function, it's not going to return the array, it's going to return a generator. Take a look at this.
Iterating a Generator2:21
a generator. Take a look at this. I'm simply going to dump and make a call to customRange. All right, let's switch back, refresh, and sure enough, I don't have an array, I have a generator instance. And as you might have guessed, the generator class implements the iterator contract. So that means I could do something like this. For each customRange as i, then do something with i. In this case, I'll dump it, and then we'll actually stop it so it doesn't go too long. We'll say if i is less than, you know, 10,000.
In this case, I'll dump it, and then we'll actually stop it so it doesn't go too long. We'll say if $i is less than, you know, 10,000. As long as that's the case, then dump it, otherwise return done. Okay, so if I come back to Chrome, I give this a refresh, sure enough, it's going to dump all of those items down to 1,000. So the key thing to understand here is with this approach, we never build up that $items array in memory. There's no array that has, what is that, 10 million items in it? That's never going to happen with this approach. All right, let's keep playing around.
That's never going to happen with this approach. All right, let's keep playing around. I'll do another example here. Let's say I'm just going to yield one, and then do it a couple more times. So this shows you can yield as many times as you need to within a function. So now if I call the test, and once again, if I dump it, it takes a while to wrap your head around this, but you're not getting an array in response. So don't expect it. You're going to get a generator that you can then iterate over using something like forEach. foreach testAsItem, then dump the item.
You're going to get a generator that you can then iterate over using something like forEach. forEach testAsItem, then dump the item. So if I switch back, refresh, there we go. Once again, we used a generator to iterate over this small data set. Again, remember, the main advantage and use case is when you're dealing with very large data sets that would otherwise exceed your memory limit. Now what else? Things to be aware of. We'll create our generator here. You can't interact with it as if it was an array.
Generator Limitations4:15
We'll create our generator here. You can't interact with it as if it was an array. So if I were to dump the first item there, that's not going to work. So if I give it a refresh, you can't use a generator as an array. So instead, you'd say, well, give me the current item, because remember, the generator we get back, that implements the Iterator interface. So we can get the current item, or we could say dump that, then move the pointer to the next item. And if you get the current there, it'll be 2 in this case, 1, 2, and then to get the third one.
And if you get the current there, it'll be 2 in this case, 1, 2, and then to get the third one. So it's a little bit different there. And of course, if you need to, in certain cases, you can call the iteratorToArray function. So as you can imagine, this will cast the iterator back to an array. Just be careful with that, though, because if you have a massive list again, you've reintroduced your memory problem. Here's another example. Let me boot up php artisan tinker. I have an example here, a Project model.
Generators in Laravel Factories5:10
Let me boot up php artisan tinker. I have an example here, a Project model. Let's say, for whatever reason, you want a million items in there. All right. 1, 2, 3, 4, 5, 6. I always have to count it out. Let's say you want to build up a million new Projects in the database. Well, as you can imagine, if I run this, it's going to do its best, and then it blows up. AllowedMemorySize is exhausted. There's no room left.
AllowedMemorySize is exhausted. There's no room left. And this is because, behind the scenes, Laravel uses the range function. If you want to see it, where would it be? The factory builder. Let's call create. It calls a make function. Let's see. So amount is the time, so that would be this right here. Let's look down.
So amount is the time, so that would be this right here. Let's look down. Yeah. So you can see it's doing an array_map over the total number of times you specified. So it's basically saying, whatever that is, a million times, make a new project. So it's building up this array in memory, and it's not up for the job. It blows up. So what you could do instead is just wire something up yourself. We'll say for i equals zero, and i is less than times, so you'll give that to us. This is just for an example.
We'll say for i equals zero, and i is less than times, so you'll give that to us. This is just for an example. Then i plus plus, and here is where I will yield a new Project, and I'm just going to hard-code it. So this line here is important. It shows that you can basically yield anything you want, a number, a string, a function. You can even yield from another generator. There's all sorts of things you can do here. But also, when I call this make function, we're not going to hit this code yet. So let's try it out.
But also, when I call this make function, we're not going to hit this code yet. So let's try it out. I'll just say, make a hundred, and then return. Done. And you'll notice, real quick, if I switch to my database, I don't have any projects here. But if I come to Chrome and we give this a refresh, I'm still not going to have any projects here. That's because this code is not yet being executed. Instead, I have a generator that I can iterate over to yield this call.
That's because this code is not yet being executed. Instead, I have a generator that I can iterate over to yield this call. So for example, if I save this to factory, and I say, well, give me the current one, that's actually going to yield this first time. So if I run it, come back to SQL Pro, refresh, now I have an item. I'll delete that. Let's do another one. We will do the first item, then I'll go to the next spot, and then we will do the second item. All right.
item. All right. We'll run it, and this time I should have two items in the database. So what we can see is, I can iterate over this as much as I want. So for each, we'll call it a hundred times, and here, we'll dump the $result. So $result will be equal to the result of this create call. So come back to Chrome, refresh, and you'll see we get a hundred of those projects. And we'll also see a hundred new items within the database there. So again, notice that at no point are we building up these records in memory. So let's try one last thing.
So again, notice that at no point are we building up these records in memory. So let's try one last thing. I'll just comment that out. We'll do one, two, three, four, five, six. Let's create a million records using a generator. Come back to Chrome. Now it's going to take a while, but notice we're not running into memory limit issues. So if I go back to SQL Pro, we're probably not going to wait for this. But notice, there's now 32,000 records, 36, 38, 40, 48. It'll keep going without hitting the memory limit.
Lazy Collections and Cursor8:29
But notice, there's now 32,000 records, 36, 38, 40, 48. It'll keep going without hitting the memory limit. Anyways, I'll come back and close that out. We don't care. Finally, as of Laravel 6, your collections can be lazy as well. Now we'll cover this in the What's New in Laravel 6 series, but real quick, you'll see that there is a lazy method that'll return a lazy collection. So you'll see here, this does, in fact, use a generator. So you'll see things like, how about collapse? There we go.
So you'll see things like, how about collapse? There we go. yield. So this method collapses a collection of items down to one array, but you'll see on your regular collection, when you collapse, it's deferring to an array helper method. This will all be done in memory, whereas with this approach, it's using a generator. Finally, on your query builder, you'll see a cursor method that will also return a lazy collection. So this would allow you to translate an Eloquent query to a lazy collection. So for example, I could say, app()->project->cursor(), and now you'll see I have a lazy collection.
So this would allow you to translate a Eloquent query to a lazy collection. So for example, I could say, app()->project()->cursor(), and now you'll see I have a lazy collection. So if I say, give me the first item, it'll still effectively work the same. A few things will be different, of course. For example, you don't have the array access that you normally would if you did this approach. But of course, with this approach, because we have so many records, I can't even perform the query. Anyways, that's just an aside, but in a nutshell, you now understand generators and php.
