We need a way to overcome some of the limitations with text-based searching. In fairness, using regular expressions is probably good enough in most cases. In fact, the first few shifts were exclusively regular expressions and string-based replacement. But to handle situations where code is commented or there's context we need to be aware of, we're going to need to actually analyze the code. Now every language has a set of rules, and this rule is called a grammar. In fact, we can find the specification for this grammar in one of PHP's repos. So for example, a comment can either be a single-line comment or a delimited comment. And a single-line comment starts with a literal // or a #. We can see all sorts of definitions for the different types of things we know within PHP.

And a single-line comment starts with a literal // or a #. We can see all sorts of definitions for the different types of things we know within PHP. For example, a variable name starts with a $ sign and a name. With this grammar, we can build things like state machines to know what's next. In fact, that's how PHP lint detects syntax errors. For example, it knows that it expected a ; next. So to parse PHP, we could actually use something like token_get_all. This is a built-in function that takes a string of code and gives us an array of tokens. And those tokens are predefined by PHP, such as the open tag or an echo. This is good, but it's a little flat.

And those tokens are predefined by php, such as the open tag or an echo. This is good, but it's a little flat. It doesn't give us a lot of dimension about what the code actually is. It doesn't give us that context that we need. To get that, we can use an actual parser, something that actually parses the code and can give us a higher-level context. And the best library for that is NGIC's PHP parser. This can parse PHP code going all the way back to PHP 5. And what it gives us is something called an abstract syntax tree, or an AST. And if we look at one of their examples, when it parses code, it actually gives us a multidimensional

And what it gives us is something called an abstract syntax tree, or an AST. And if we look at one of their examples, when it parses code, it actually gives us a multidimensional array of higher-level information, like this was a function call, and here were its arguments. This is much more powerful. In fact, any of the static analysis tools like phpStan, SOM, Rector, and of course Shift use NGIC's parser. So I've gone ahead and added this to the project as a NGIC parser class, and I've actually taken the one from Shift's code base. So let's look through this quickly. We pass in a finder, which I'll talk about here in a second.

So let's look through this quickly. We pass in a finder, which I'll talk about here in a second. And the only method in here is parse. It takes a string of code, and we set up what's called a lexer. In this case, I'm enabling some options, such as parsing comments, giving me the line numbers, as well as the file position. The rest of this code is pulled from their wiki. We basically create one of their parsers for php7. This is technically php7 and above, and then we parse that code into an AST. Now I've added another snippet that I've also pulled from the wiki that does some name resolution.

This is technically php 7 and above, and then we parse that code into an AST. Now I've added another snippet that I've also pulled from the wiki that does some name resolution. This allows me to get the fully qualified name of any references in code. And finally, we build up that node finder, and we call the search method on our finder to get any instances of the code that we care about, and we return those processed. This gives us back the meta information we care about, much like that matches array when we were using regular expressions. This parser class is pretty standard. There's not much you're going to change. It's mostly just enabling the options that you care about for the lexer.

There's not much you're going to change. It's mostly just enabling the options that you care about for the lexer. What you'll be building a lot of are these finders, and I've created one called debugCalls. It has two methods, the search method, which returns true or false based on if it's a bit of code we care about, and that process method, which returns an array of the metadata that we want. Right now I'm just returning the line and the offset for that snippet of code. We probably need to return more, but let's take a look at what we actually need to parse. In our case, we care about function calls to vardump or varexport, export, dd, as well as print_r.

In our case, we care about function calls to var_dump or var_export, export, dd, as well as print_r. Of course, the whole reason we're using parser is we want to filter special calls. This is something like var_export, and when we pass it, that second argument of true. Let's start with this top piece. What I like to do in search is basically always return true, and then I want to add a bunch of guard clauses to get rid of the things that I don't care about. Right away we know that if the node that we pass in is not an instance of a function call, in this case, func_call, then we would return false. This is just a reference to the class NickXParser that represents a function call.

in this case, func call, then we would return false. This is just a reference to the class NickXParser that represents a function call. So if that node's not a function call, then we know we don't care about it. So that handles this portion. Now we would want to check to see if that function call is actually one of the functions we care about. So if we look at the function call object, it has a name and arguments. So the name is probably what we're going to check. Now knowing a little bit about NickXParser, we only care about the name portion. If it's an expression of some kind, that's something pretty dynamic and not something

Now knowing a little bit about NickXParser, we only care about the name portion. If it's an expression of some kind, that's something pretty dynamic and not something we're going to be able to handle. What that would look like would be something like foo equals var_dump, and then we call foo with whatever. This is an example where the name portion of the function call is an expression. We're not going to worry about those types of calls right now. So we could go ahead and write another instance of check where nodeName has to be an instance of name. If it's not, then we would want to go ahead and return false because we just can't handle

of name. If it's not, then we would want to go ahead and return false because we just can't handle those types of dynamic calls. Alright, now we're finally getting down to the meat of it. If the name for that node to lowercase is not in the array of the names we care about, which would be these, then we could ignore it. And that should handle everything for this first piece that we care about. I'm just going to put a little to-do on this, and let's jump down to process. So we have our lineNumber, we have our offset. We probably also care about what the actual function was that we were calling in that.

So we have our line number, we have our offset. We probably also care about what the actual function was that we were calling in that instance. So just like we did in the search method, we could get instanceName to lower string. Alright, let's jump back to our debugCalls.php script. Let's go ahead and create one of our new finders. We'll say nickicParser, and we'll pass it our new debugCallsFinder. Now I'm still going to keep this regular expression. It's a good way to weed out files we don't care about. The parser can actually be pretty resource intensive, especially for large PHP files.

It's a good way to weed out files we don't care about. The parser can actually be pretty resource intensive, especially for large PHP files. So if we don't even find a string representation of any of these functions we care about, then let's just skip it. So I'm going to update this code to say if it's not found, then continue. And let's move this code down outside of our new logic. Okay, what we can do now is we can call our finder, parse with the contents of the file, and save the instances that it finds. And for now, let's just print_r those instances. Alright, let's call that script and see what things look like.

And for now, let's just print_r those instances. Alright, let's call that script and see what things look like. Great, okay, it's a little noisy, but we're actually getting an array of the different lines, the offsets, and the function call, in this case print_r, for that particular file. So let's clean it up to actually give us a better output, much like our linter. So if we go back to displayError, let's update this now to take those instances. If we go down to displayError, it's still an array. So let's change this back to lineNumber, and we'll add in the calls, line, and start. So that should give us the lineNumber, followed by that colon, a space, and then now we can

So let's change this back to lineNumber, and we'll add in the calls, line, and start. So that should give us the lineNumber, followed by that colon, a space, and then now we can say something like contains call to, and let's output, instead of a unique thing of these calls, the calls function name. And let's finish that off with that closing back tick and a line break. Now calls is actually an array. I want to loop over the individual calls for each one of these lines. So let's change this to call, and we'll wrap this in a foreach. So foreach calls as call, drop that in, bring that back up, and I think with that line break already in the foreach loop, we should be back to the way things were.

So foreach calls as call, drop that in, bring that back up, and I think with that line break already in the foreach loop, we should be back to the way things were. So let's scroll up here, get rid of this print_r just to reduce the noise, and let's clear this out and run things again. Awesome, this is exactly what we wanted. Our linter on line 10 has print_r, our var_export has one on line 7. Ooh, that's kind of interesting. Debug calls has nothing. Oh, this is a good example where the string search found references in the file, but it's not actually a code call.

Oh, this is a good example where the string search found references in the file, but it's not actually a code call. So let's add a little guard clause for that. So we'll say, if empty instances, then also continue. All right, let's take a look one more time. Great, our debug calls out of there. So now let's address that var_export, because this file only has a call that is returning that instead of outputting it. So going back to our finder, and up here, let's fill in this toDo. Over to the above, we can say, if in_array nodeName strtolower(string) is either var_export

So going back to our finder, and up here, let's fill in this to do. Over to the above, we can say, if in array node name toLower string is either var_export or print_r, then we're going to want to do some additional checks. The next check we could do is to say that the node itself has to have a certain number of arguments, in this case, 2. So we can count the args and say it's equal to 2. Now I think I know what we need to do next, but at this point, what I'd like to do is actually just print out what that node looks like so I can get a little more information on what kind of checks to do next. So let's just var_dump the node args 1, because we know that's the argument that we're going.

on what kind of checks to do next. So let's just var_dump the nodeArgs[1], because we know that's the argument that we're going to want to inspect. I don't really know what true looks like in Nickic parser land, so let's find out. I'm going to clear this out, run this again, and we can see here we have true, true in parts, it's a name, okay, we know how to deal with those, it's a constant fetch is what that's considered. Interesting, I'm not familiar with one of those. Alright, this is going to get a little long, so I'm going to go ahead and break this down a line.

Okay, it didn't seem to be filtering the way I wanted. Let's go take a look. args, what's args? args is an arg, okay, I'm referencing the name of the argument, I need to be referencing the value. Alright, let's fix that up. So if args value is an instance of const fetch, and if args value name is true, then we will return false. Alright, trying one more time. Hey, look at that, there we go.

Alright, trying one more time. Hey, look at that, there we go. We see that we no longer have var_export. In fact, we have less things coming out in all and even our print_r looks better. Let's look at this all one to get both of those examples. We'll see that lines 10 and line 13 are the ones where I'm passing that additional option. So if we go back here, we can see in our report that line 10 and line 13 are no longer there. And just for giggles, let's change this to false. And line 10 is back in our list. That's pretty awesome.