I'm a compiler geek, with a slant towards scripting languages.
I work on the Javascript team at Mozilla. Before that I did a PhD in compiler optimizations, static analyses and scripting languages.
I’ve worked on phc since 2005, and been its maintainer since 2007. I wrote the optimizer, and nearly everything performance related.
I had mixed reactions upon hearing about the release of HPHP [1], the new PHP compiler from Facebook. There are a few aspects to this so I’ll start with the technical stuff. I always love the social aspect, so skip to the bottom if you like whining and tears.
I don’t know the answer to this. I haven’t seen anyone even mention PHP’s references, which are incredibly gnarly for a static analyser [2]. HPHP might just ignore it, which isn’t necessarily a bad idea. I’m wary of ignoring edge cases, as they tend to interact in horrible ways, but I guess Facebook already run all their code off it, so it can’t be that bad.
In general, I’ve found that ignoring the edge cases is bad when compiling PHP. There are a million of them, and they all interact. They interact worst of all in static analysis, because you have to consider all possible paths. Its the sort of thing where if you nail it 100%, then you have something amazing and widely applicable, so that’s what I was aiming for with my PhD. I suspect HPHP doesn’t consider all paths, and makes all sorts of hacky assumptions. [3] This is probably a really good idea. I did the opposite in the optimizer, and the result is instead immature and slow.
Facebook said HPHP reduces by half their number of servers. PHP’s libraries are already written in C, which gives it the appearance of being fast, even though the interpreter is dog-slow. This implies that HPHP-compiled PHP code is much more than twice as fast as the PHP interpreter. This probably means HPHP is way faster than phc‘s compiled code as well.
I saw some criticism for not building a JIT on LLVM. But:
PHP is not like other dynamic languages. Duck-typing is possible, but most of the community best practices come from Java (along with the class system), and so its not used that much. Monkey-patching — switching out classes and methods from objects at run-time — isn’t possible, except with the hackiest of hacky unsupported extensions. Dynamicism in PHP tends to involve templates instead, like with Smarty. If you want to analyse it, then you just need to run it a few times, get all the templates to be instantiated, and compile all the generated PHP code. I’ve started calling this "deployment-time analysis", since in server-side, you probably know all the code you’re going to compile at deployment-time. So a compiler is a perfectly reasonable approach for PHP, and a JIT is probably not needed.
People seem to want to know if HPHP is widely useful outside of Facebook, and some people are saying "no". I disagree strongly. In order for HPHP to be useful, you need to have a PHP application which is suffering due to PHP interpreter performance. That matches Facebook perfectly, and they’ve always been the canonical example I use to explain why PHP compilers are interesting. But you don’t have to be Facebook size or scale to have performance problems.
I’ve heard the argument "you don’t need a compiler, since PHP is rarely the bottleneck" for many years. I think its complete bollox. But I wrote a compiler for PHP, so I would say that.
Unless your PHP server is sitting there idling (which is probably the case for many PHP servers out there), then you could make use of a PHP compiler. For small timers, all components of your application are going to be sitting on the same box, contending for the same resources. Even if you assume the DB is the bottleneck, the resources the interpreter consumes could be more profitably spent on the DB.
The PHP interpreter is also quite memory hungry, as interpreters go. Any PHP value in your program uses 68 bytes of overhead [6]. An array of a million values takes over 68 MB. If HPHP is able to convert your million value array to native C types, it will only take 4MB. I’m sure your caches could make good use of that savings.
However, optimization isn’t only about speed. The main value is that they give you freedom in how you code.
There is a meme in the scripting language communities that {PHP,Ruby,Python,Perl,etc} are "fast-enough". If you need it to go faster, then you should take your hot-loop and rewrite it in C. HPHP will free you from such concerns.
You should consider also that PHP is considered relatively fast. Its not — the interpreter is dog-slow — but programs written in PHP are typically not that slow. This is because most of PHP’s huge standard library is written in C, with a thin layer of PHP over it. Anytime you call a string function, your PHP C string is passed into the C library, the pointers are manipulated and the bits are twiddled, and then it’s handed back to your code. Its a bit like driving in America: it takes a few minutes to get on the freeway, but once you’re on it you’re there in no time.
This is not necessarily a good thing:
Believe me, if your entire application just ran PHP interpreted code, it would not be fast at all. But people who write PHP functions and libraries don’t want to write C. They like PHP, are productive in it, and any time spent arsing around in C is wasted when there’s a website falling apart and a long list of features due yesterday. HPHP will free you from such concerns.
Compilers also provide other niceties. You don’t have to unroll your own loops, or move constant expressions out of loop headers. I don’t know if HPHP supports these, but I’m sure it could.
Allowing your existing code to go faster is hardly the point though. Really, the point is that you can do more in less time. Suppose you’ve decided that your application needs to response to the user in 500ms. The DB takes 200ms, the request takes 200ms, the framework takes 50ms and your code only has 50ms to run [7]. That’s quite a constraint. This leads to people using PHP as a simple templating layer, instead of as the Turing-complete langauge it is. I expect we’ll hear a lot more about HPHP, simply because of how freeing it is to the user.
So even if you only have a small VPS, instead of massive server farms like Facebook, you’re likely to find a use for HPHP. I’m sure shared hosts will set it all up soon for their users, and everyone will be happy.
A better question is, how widely applicable is a compiler which doesn’t support all of PHP’s dynamic constructs? Funnily enough I did a bit of research on this. We chose the opposite tactic for phc: trying to stay 100% compatible with Zend, all the extensions (even the 3rd-party, unpublished, top-secret, proprietary ones), etc. You would expect we would first research whether this was useful? Well no. We built it, and then I did some analysis to check whether it was useful. You can read it in depth [8]. Basically, I downloaded 700 packages off sourceforge, and wrote some phc plugins to check for evals and dynamic includes (a dynamic include uses a variable as its parameter, instead of a constant or literal).
Result: 40% of PHP packages use dynamic constructs. Now this isn’t quite as scientific as it should be. Lots of those programs were old, and styles have changed. eval is discouraged these days, but it’s probably still used, if only to get around the weaknesses of the PHP parser. In particular, this doesn’t imply that HPHP is somehow fatally flawed for not supporting dynamic constructs. It just means it might not be so useful to you, the common PHP programmer.
An easy way to get round dynamic includes is to just consider the PHP files in the directory structure. There was a good research paper by Wassermann on supporting that, but I find it very hard going, so you probably will too. Still, a naive approach is to just stick a switch statement in, and compile everything that makes sense. This is how you would deal with things like WordPress plugins. It does mean that if you change your plugins, you’re just going to have to recompile. If you’re using a compiler, I doubt you would find that a problem.
But not everyone is happy with this new compiler, such as, well, me.
Lets start with a quick whine.
You would think they could at least invite me to the party. Those bastards.
More seriously, I actually was annoyed at all the news reports about HPHP, principally because they were largely bullshit. And I knew it, and I couldn’t say more because I told Facebook I wouldn’t. There is very little more irritating than idiots being wrong on the internet, and the news stories brought out thousands of them! Reddit and Hacker News were literally covered with stories about HPHP. Hundreds of trolls emerged from under their bridges, not knowing the difference between a bytecode-based interpreter, a caching PHP accelerator, and a native compiler (which is fine, until they start saying they’re all the same). Think about it now still makes me angry.
I’m also slightly annoyed that people all of a sudden care about PHP compilers. I worked on one for 4 years and I could not convince anyone to give a shit. But now that its got the Facebook logo on it, all of a sudden PHP compilers are the greatest thing ever. Bah.
One saving grace is that they didn’t patent it. I have an email in my inbox from one of the HPHP developers saying he couldn’t talk to me about the compiler because they might patent it. That’s pretty shitty. Thank god they open-sourced it instead [9]. It sounds like it was a bit touch and go for a while.
Which bring us to the question of whether they should have used phc?
Obviously it would be great if they had used phc, and I’m not privy to the reasons they didn’t [10]. The design decisions we made in phc were aimed at maximum compatabilty, and the performance suffered [11] as a result. The optimizer was designed to solve these problems, and I believe it would have, but it is not mature enough now, and was still a twinkle in my eye when HPHP started two years ago.
Facebook was solving their performance problems, not building a PHP compiler for general use. If they were doing the latter, it would be much easier to criticise their approach, but for now I can’t say I would have advised them otherwise. On the other hand, they probably didn’t need to build their own parser – its a tricky problem and phc‘s parser and front-end are excellent. Had they gone another way, then they could probably have started to use phc‘s optimizer [12], which while immature and slow to compile, is pretty state-of-the-art and has great potential (if I do say so myself).
A better approach would probably have been to hire all the programmers who worked on PHP compilers, to get that expertise in house. They did try to hire me, but only recently. I’m honestly surprised that they haven’t tried to hire the Shannon Weyrick, who is currently working on rphp, his second PHP compiler [13].
When they annouced HPHP, I would have said it was phc‘s death toll. The original phc authors, Edsko and John, have moved on to other projects, and I’ve run it mostly solo for about two years. But I havent worked on phc in about 6 months, and my hatred of PHP makes it unlikely I will again. My requests for new contributors to step up has fallen on deaf ears, and my summer intern hasn’t decided to take over either.
Since no-one wants to take on the compiler, the new competition from Facebook should probably kill it, right? Maybe not. Over the last week, traffic to the phc website has increased by five times [14]. Facebook has unleashed some sort of latent interest in PHP compilers that I haven’t been able to extract from people. So perhaps this might be the rebirth of phc, not its death.
And phc is better than HPHP in some ways. HPHP is almost certainly faster because they didn’t have to deal with eval, dynamic stuff, and because they don’t use the Zend libraries. But phc was specifically designed to work with the Zend libraries, with eval, with everything. So it’s probably a better fit for most projects than HPHP.
If you want to take over phc, then join the mailing lists, download the source code, read the death notice and contribution page, and email me for commit access.
phc will likely live on anyway. The front-end is pretty slick: Facebook ran it over their million lines of code and only had one or two problems. It gives a lovely AST to allow all sorts of code transformation tools, has a nice plugin interface (for C++ lovers) and an XML interface (for the rest of you), and will spit your code out largely as you put it into it. Its certainly the most mature and well tested part of the whole project.
The optimizer is pretty slick as well, but in a different way. I’m pretty sure its the most advanced static analyser for PHP, and it’s waiting to be put to good use. That said, it’s damn slow, and not mature (read: pretty buggy), and itself doesn’t support eval and dynamic includes (surprise!!). The optimizer is waiting for some love — I could imagine it making a pretty nice "automatically find out what types your function may be passed" kind of linter.
Otherwise, phc will only live on as part of the Roadsend Raven compiler. I understand that they’re going to take the optimizer and the parser from phc, and that will be really interesting.
Finally, what does it mean for me? Well, I’ve left that ship already. I’ve hated PHP for a long time, and have no desire to go back to it. I’m doing a startup now, but when I go back to regular employment I will be looking for another scripting language run-time. There are plenty to choose from, in particlar Unladen swallow and TraceMonkey. Mozilla looks like an amazing place to work, so I think I’ve worked out my backup plan.
I’m inventing a new sport today, which I call “malicious code reviews”. I spent a few hours reading some really very bad code, and in retaliation against its author(s), I’m going to code review it [1]. The code comes from PHP version 5.2.8, the latest stable release. This particular file is Zend/zend_operators.h [2]. You might want to open it in a new window, or in a popup, so that you can follow along.
#if 0&&HAVE_BCMATH #include "ext/bcmath/libbcmath/src/bcmath.h" #endif
I’ve skipped a few minor problems to go straight to the laughably poor, the #if 0. Funny story though: I saw a “code review” in PHP recently which chastised the addition of a #if 0. I initially thought that, finally, someone is actually stepping up to stop the rot within the PHP engine. Sadly, they instead complained that according to the rules of the PHP project, an #if 0 must also have the author’s name added to it. The mind boggles.
There is a limited amount of reasonable code, which I’ll skip, followed by the is_numeric_string() function [4], only one of the finest examples of poor code I’ve ever seen. I linked to it above, so I recommend that you actually read along as I go. This will be no fun unless you can actually see it.
It starts off, surprisingly, with the most thorough comment I have yet to see in PHP. It is merely average in terms of what you might read in a gcc source file, but here it is a shiny gold nugget floating in a murky brown sea. However, it degrades fairly rapidly. You might notice this giant function is in a header, and that it is declared to be static inline. This is a prelude for what’s to come.
The function starts with some whitespace skipping [5]:
while (*str == ' ' || *str == '\t' || *str == '\n' || *str == '\r' || *str == '\v' || *str == '\f') { str++; length--; }
Just two lines above the function they have two macros, called ZEND_IS_DIGIT and ZEND_IS_XDIGIT. Could they not have added ZEND_IS_WHITESPACE? A pity, but a tiny flaw compared to the gaping maw of despair that follows a little bit later. The code continues fine: check for a digit, check if its hex (with comments, very good) until we come to this line:
for (type = IS_LONG; !(digits >= MAX_LENGTH_OF_LONG && (dval || allow_errors == 1)); digits++, ptr++)
I’d like somebody to come forward and explain why
type = IS_LONG
is in the loop initialization statement. And why the loop condition is so unreadable. And why the elements of the loop header are not related in any way at all!!! But this is just the start. The next line is a doozy:
check_digits:
Do you feel the fear? I feel the fear. Its a label. That means that somewhere in this function, there is a goto. Not that there’s anything wrong with gotos. Sure, if misused they can lead to unreadable, spaghetti co– OH MY GOOD GOD. I’ve found some gotos, but they go to a different label. Two labels. And the first one is in a for-loop! Don’t panic, maybe its readable. Maybe the second one is also in the for-loop. Please? Pretty please?
Fuck. Fuck fuck fuck. I’d like to suggest you try to work it out yourself, but you’d probably prefer not to. If you haven’t looked at the code yet, now is the time. Don’t worry if you don’t know C — with code like this, knowing the language is not the advantage you’d expect.
The first label, check_digits, is in a for-loop. That for-loop has two gotos (and one continue and one break, just to make things more readable), which both go to the other label process_double. process_double is outside the loop, deeply nested in a completely separate series of if-else statements. They have also given up on comments by now). After checking a few more conditions, only then do you jump back into the previous for-loop!!!! Oh wait. No, that’s not right. They’re in different paths. There actually added control-flow edges from an else-body to within a for-loop in the if-body. Just wow.
I’d like to say that this horrendous function is over. After all, this is just the first function I’ve come across. But while there is only simple (but uncommented) code remaining in the function, I have a final nit to pick.
if (ptr != str + length) { if (!allow_errors) { return 0; } if (allow_errors == -1) { zend_error(E_NOTICE, "A non well formed numeric value encountered"); } }
There is a check to see if allow_errors is -1, even though the comment only mentioned two possible values for allow_errors. So what does it mean for it to be -1? We’re saved from figuring it out because the check can’t even trigger. If allow_errors was non-zero, the function would have returned already.
Now, you might consider this a minor nit, and its easy to see that it wasn’t fixed when you consider how deeply it was nested [6]. But this sort of thing is the rule, not the exception in PHP sources. Broken windows built on top of other broken windows.
#define Z_DVAL(zval) (zval).value.dval #define Z_STRVAL(zval) (zval).value.str.val #define Z_STRLEN(zval) (zval).value.str.len
except one
#define Z_BVAL(zval) ((zend_bool)(zval).value.lval)
where an enterprising soul mustn’t have thought hard before committing. PHP has long been lambasted for its inconsistency; it turns out that this inconsistency is not limited to just its libraries, syntax or semantics.
#if HAVE_SETLOCALE && defined(ZEND_WIN32) && !defined(ZTS) \\ && defined(_MSC_VER) && (_MSC_VER >= 1400) /* This is performance improvement of tolower() on Windows * and VC2005 * GIves 10-18% on bench.php */
Unfortunately, this file is not an isolated incident. The entire Zend/ directory — the core of the whole PHP implementation — is a filthy mess. While the is_numeric_string() function might be the worst code I have ever seen, most of the Zend/ files contain a lot of hideous code: poor organized, badly written, badly documented, unreadable messes.
I’m aware that as a ‘code review’, this is actually pretty poor, primarily due to its lack of constructive criticism. So this is more of a detailed, flame-bait-y rant about the quality of code in this file, which I will use to assure people that the rest of the code I’ve seen in the PHP project is of similar caliber. And none of that is really very constructive.
As it happens, I am preparing a much more constructive piece about why the code in PHP is so bad, how it got this way, and how to fix it. But first I’d like to demonstrate how poor the code currently is, and it’s difficult to do this without some bile and vitriol.
I didn’t want to bog down the intro with my method, but it should probably be here anyway.
If you know Zend internals, spot the difference:
SEPARATE_ZVAL_IF_NOT_REF(ppzv);
and
if (!(*ppzv)->is_ref) SEPARATE_ZVAL(ppzv);
