OK OpenTable, I'll go ahead and debug your WAF issue for you

I learned about restaurant reservation apps while on a trip to San Francisco. My wife and I were walking around town and decided to pop into the first restaurant that looked interesting. We tried a couple places that were literally (and I mean it literally) empty. The experience kept going like:

Me: Table for two please.

Host or hostess: Do you have a reservation on [app name]?

Me: Uh, no.

Host or hostess: We can't seat you then.

Me: Literally every table here is empty.

Host or hostess: Yes but if someone with a reservation on [app name] shows up and there isn't a free table for them we'll receive bad feedback.

Eventually one restaurant let us in if we agreed to be fast. Not one reservation showed-up the entire time we were there. It was just the two of us in this fully-staffed restaurant with no other diners.

Being from Chicago, this was an odd experience. In Chicago a "reservation" means that if you're lucky you'll be added to the front of the wait list when you arrive. There is no situation, except maybe some high end places I'm not allowed in, where a restaurant is leaving a table empty. A real, live person will always win over a phantom reservation that may or may not show. "Bad feedback" is hardly intimidating here.

I use apps like OpenTable maybe every 6 months. It's usually if I'm going somewhere new because it's a good way to browse around. Maybe the city I'm going to works more like San Francisco than Chicago? (Unlikely)

The other day I tried to login from my desktop and was greeted with a 503 (access forbidden) error:

503 Access Denied

OK, that's pretty straightforward right? They're using Akamai as a content delivery network (CDN) and also web application firewall (WAF). This type of error almost always means the WAF is blocking my IP address. I don't think I did anything to cause that but maybe my neighbor tried hacking OpenTable.

Their support chat is on a different subdomain that was reachable. Our conversation went approximately like:

Me: Yeah, hey there. I'm getting a 503 on OpenTable.com, it looks like you're blocking my IP address.

Support person: Yes, this happens all the time. Let me reset your password.

Me: Don't do that, it's not my pa...

Support person: Alright I sent you a password reset link.

Me: Alright. Now I'm getting a 503 on your password reset page.

Support person: Yes, this happens all the time. Can you try incognito mode?

Me: Same problem. (Note: I tried this before contacting them)

Support person: Can you try a different browser?

Me: Look, that's not going to work either. I'll try Firefox now and.. wut?

In my hubris I assumed I knew the problem - IP address blocking. With Firefox working I now had a fun mystery to investigate.

No errors in Firefox

So their WAF is blocking my request based on something other than IP address. I first tried overriding the user-agent in Firefox to match my Chrome user-agent. Firefox still worked. So it's not like I'm on the latest version of Chrome and their WAF rules are only allowing up to the previous version. I decided to compare the browser signatures (note - this is still with the user-agent override in place):

Comparing the browser signatures

These are both private/incognito windows so no plugins are loaded. These detected things should be as plain as possible.

I started by adding all those sec-whatever headers to Firefox. None of them triggered the problem.

I was running out of things to try. I saw the Accept-Language headers were slightly different. Sure, but, c'mon, that can't matter at all can it?

Accept-Language header

That's weird. Setting Accept-Language to en-US,en;q=0.9 causes the WAF in front of OpenTable to return a 503. This is not default behavior and is something OpenTable would have configured on their own. Oh, to translate.. Accept-Language=en-US,en;q=0.9 is the browser's way of telling the web server "I would prefer if you returned US English to me but anything that is 90% English adjacent is fine too."

Wait, really? I still can't believe this.

Wait, really?

Yes, really. I tried many variations and en-US,en;q=0.9 was the only one that could re-create it. Perhaps that plus another combination of headers causes this. It feels like they saw some bad behavior and clicked a button labeled roughly block requests similar to this.

Also if en-US,en;q=0.9 is not a valid Accept-Language setting then the proper HTTP response is 406 (not acceptable). That again makes me suspect they are trying to block some pattern they don't understand.

I tried some things that really should be invalid Accept-Language combinations and they were all fine.

Trying other languages

To reiterate, this error is clearly being returned by their Akamai instance. This is evident in both their server header and IP address:

After some more trial and error

So the request is never reaching the OpenTable web site. They have configured something in their Akamai WAF to block this particular header. The error code returned means they think you are doing something bad vs something wrong but innocent.

Alright, well good times. Based on the conversation with their support rep this is a common problem and not one they think is urgent to fix. I hope this was helpful to at least one person also perplexed by this problem.



Related