HEADS UP: API temporarily down
Lawton Campbell.8517:
Just disabled all of the APIs.
Not entirely sure of the reason, but the Javascript contexts for the API frontend servers are constantly crashing. Due to the holiday I won’t have the bandwidth to debug the problem until tomorrow evening, so I’ve temporarily taken the APIs down in the meanwhile. I should be able to figure out what’s going wrong tomorrow and probably get everything back up by Sunday.
I’ll post updates to this thread if something goes horribly wrong.
Sorry about the downtime
EDIT: Going to be down until Monday at least; going to add explicit rate-limiting to the API. I’ll have some more details later, still need to run some numbers.
aandiarie.7195:
It happens. Thank you for giving us the info and etc.
It could be a lot worse. At least this time Jormag didn’t send a blizzard to cut off the power.
CJay.5013:
Thank you for letting us know! Looking forward to seeing it back up again.
Freezy.4952:
Thank you for your diligence in this matter.
Looking forward to checking my bank ‘on the go’ again soon :P
Lenaja.5463:
Thanks for the info. I was wondering why my app suddenly did not show dailies anymore.
One small suggestion. If the service is off, maybe you could send a 503 header service unavailable so we can better handle this case in the future.
Reincarnated.1754:
Thanks for the Heads Up.
Christian.1408:
Thanks for the heads up!
Is there any way to determine the API’s current status? Something like https://api.guildwars2.com/v2/status ? That would be really nice.
Yamagawa.5941:
Dunno about official ways to determine if its up or not.
For your smoke-test pleasure try
https://api.guildwars2.com/v2/commerce/prices
or
https://api.guildwars2.com/v2/items
Both should return lists of numeric item IDs, and currently are returning ‘not-found’
Leablo.2651:
Does this have any relation to the recent issues with the in-game TP where requests seem to stall and have to be repeated? That has been happening for some weeks now.
Lawton Campbell.8517:
One small suggestion. If the service is off, maybe you could send a 503 header service unavailable so we can better handle this case in the future.
I should adjust that. Been turning off the APIs by disabling all endpoints via config; I should add an explicit off switch that sends the proper response.
Thanks for the heads up!
Is there any way to determine the API’s current status? Something like https://api.guildwars2.com/v2/status ? That would be really nice.
The root (https://api.guildwars2.com) kind of works like this? When the v1/v2 endpoints are disabled they’re removed from the list. But using an explicit status code to indicate “the world is on fire” is probably easier on the client side.
Does this have any relation to the recent issues with the in-game TP where requests seem to stall and have to be repeated? That has been happening for some weeks now.
AFAIK the API failure was totally self-contained and shouldn’t have affected other systems.
lynnae.4095:
looks like, at least certain, endpoints have been up and down today. How big is the fire? ie would you appreciate it if people held off work on new features today that would hit the endpoints that seem to be up (or were up last time my services ran)
Lawton Campbell.8517:
looks like, at least certain, endpoints have been up and down today. How big is the fire? ie would you appreciate it if people held off work on new features today that would hit the endpoints that seem to be up (or were up last time my services ran)
They should be entirely down, except around 24h ago when I briefly turned them on to get some metrics out.
Which endpoints seem to be up? I’m terrified.
lynnae.4095:
The trading post prices endpoint is up and down, can we PM here? I can send you the times logs of when I’ve been able to get trading post data:
https://api.guildwars2.com/v2/commerce/prices
and the associated:
https://api.guildwars2.com/v2/commerce/prices?ids=
Most recently about 30 min ago 19:14 UTC
I’m checking the data that’s been returned now, take me a couple of minutes as today I’ve been moving databases.
Data looks legit from a cursory inspection, as in it hasn’t been the same record being returned each time
Lawton Campbell.8517:
Regarding the sporiadic up nature, queicherius provided a graph and that correlates strongly with a minor config change I made (to turn the APIs back on for the office IP addresses). I’m not sure why/????/how there’s a correlation there, but I’ve undid it (so the behavior should go back to the solid red bar of “down”.
This week is going to be a lot of fun.
(EDIT: boo the forums strip emoji).
lynnae.4095:
hmm, here’s the list of times I’ve gotten a response since Saturday morning (hourly checks):
everything fine through 05:09 then nothing. Then sporadic successes:
2017/03/18 23:09:15
2017/03/19 06:09:15
2017/03/19 08:09:15
2017/03/19 10:09:15
2017/03/19 19:14:31 (new server, new time, one day I’ll fix that, but probably not)
I can’t see from here if those all correlate, with that much green I feel unlucky about the big break in hits today
Lawton Campbell.8517:
Which timezone? I made the config change at 3/18/2017 7:06 PM PDT (UTC-7); and it normally takes a few minutes to propagate out.
lynnae.4095:
That’s all UTC
(and have some extra words so the forums let me post. Hi Mom!)
Bjyoric Orcstone.1728:
I found that Nginx is great as a rate limiting reverse proxy. Just lookup “rate limiting with nginx” to get started. Extra nice that you can partition the rate limiting. So >5reqs/sec I slowed a little for a couple seconds but >20reqs/sec got throttled to 4/s for a big chunk of time.
Good luck, I feel your pain. This stuff always happens at 2AM on a holiday.
Elayne.9514:
Is the API still down or is the problem local for me?
Renske.6178:
Still down for me as well so I doubt it’s local!
Darkashara.4871:
Still down here too definitly not local
Leablo.2651:
Going to be down until Monday at least
Key phrase being “at least”. It’s Monday now but as of this writing in PDT it’s roughly 6 AM. Give the man a chance to work.
palduno.9372:
I use Overlay timer for wvw, since the reset no more wvw maps, its out of work. I know that Overlay isnt an Anet product, my question is: is an existing link between the Api shut down and the unworking wvw maps?.
Sry for my poor english.
Itsjon.2869:
I use Overlay timer for wvw, since the reset no more wvw maps, its out of work. I know that Overlay isnt an Anet product, my question is: is an existing link between the Api shut down and the unworking wvw maps?.
Sry for my poor english.
Yes the existing links to api are shut down right now. That is why your overlay is not working.
morrolan.9608:
3pm Monday and still down.
Chaba.5410:
3pm Monday and still down.
:feelsbadman:
Lawton Campbell.8517:
Distributed systems are hard
Tomorrow might be a better day.
Illconceived Was Na.9781:
Distributed systems are hard
Tomorrow might be a better day.
Thanks for the update.
(Get some rest; the API can wait, even if many of us are going through withdrawal.)
Bjyoric Orcstone.1728:
I bet the biggest number of hits on the API comes from the wiki – trading post prices.
Simple test if API is up: https://api.guildwars2.com/v2/build
Currently returns {"error":“not found”}
ShadowCatz.8437:
Distributed systems are hard
Tomorrow might be a better day.
How many days are “tomorrow”?
Hactarus.8563:
Still not enable on Quaggans. I’ll wait another day.
$ curl -i https://api.guildwars2.com
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: application/json; charset=utf-8
Server: Quaggans
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
X-Powered-By: ARR/2.5
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Date: Tue, 21 Mar 2017 08:42:34 GMT
[]
Nekres.1038:
Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?
gegula.8249:
any info when this will be up? i need my tradepost stuff ;(
Orstep.1532:
Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?
Please consider this. Static endpoints such as /recipes, /recipes/search, /items, /achievements etc are all down, and they don’t have to be.
Ruggs.8420:
Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?
Beyond setting reasonable cache headers that’s really the client’s job. I know I maintain a local cache for my API consumers.
Orstep.1532:
Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?
Beyond setting reasonable cache headers that’s really the client’s job. I know I maintain a local cache for my API consumers.
True.
It still bugs me that the API went down the morning after I implemented a caching bit into my script. All I would need right now to continue working on my school assignment is being able to run my script just once. (Fyi, I only need the static endpoints for my tool to function)
EAGLE.3429:
Could there please have a rough Eta as to how long the API site will be down for? I understand that if ya give an ETA and things dont turn out the way ya expect it may take longer. Any info would be appreciated.
Lawton Campbell.8517:
Do some sort of caching, that, when API is down, can still dispense all the resources that have been frozen on take down?
Please consider this. Static endpoints such as /recipes, /recipes/search, /items, /achievements etc are all down, and they don’t have to be.
Those endpoints aren’t actually static. Greatly increased request volume to those endpoints is what caused these crashes, unfortunately.
any info when this will be up? i need my tradepost stuff ;(
I’m trying to get it back up today; I still need to get someone to review my changes to make sure I don’t cascade the failure to other systems if I turn everything back on.
Could there please have a rough Eta as to how long the API site will be down for? I understand that if ya give an ETA and things dont turn out the way ya expect it may take longer. Any info would be appreciated.
ETA was yesterday, except during testing there was a small bug in my implementation that caused it to use the wrong instance of the centralized rate-limiting service (i.e. it used the one that everything else uses instead of the API-specific one). So I punted on the deploy until we’re sure that the changes aren’t going to overwhelm anything.
EAGLE.3429:
Fair enough. Thank for the update. I fully understand that in programming things happen that are beyond your control. Id rather wait a bit longer for a good release then have a crappy release. Are you thinking tomorrow hopefully?
Lawton Campbell.8517:
I’m really hoping for today; pretty sure I’ve got all the kinks worked out.
Behellagh.1468:
All the things crossed.
EAGLE.3429:
If there is anything that we the community do to help ya out please ask. You may be surprised how many people would help out.
krethas.4287:
#hugops to you and a sympathetic nod re: distsys being hard.
(also, glad to see rate limiting coming. I saw the frequency of requests some sites were making just for my one account and the ops person in me went AAAAAAAAAA)
Lawton Campbell.8517:
Okay, the API is tentatively back up with a limit of 60 reqs/minute while I double-check some things.
Will bump that up to the full 600 reqs/min in an hour or so when I’m confident that nothing has caught fire (which might happen, but doesn’t look likely at this point).
EAGLE.3429:
Thank you. My verify bot is already recieving data from it on my end.
Corax.7381:
Woo! You’re da real MVP.
Lawton Campbell.8517:
And the ratelimit is bumped up to 600 reqs/minute.
As far as I can see, everything is mostly humming along okay.
Smoaking.6125:
You’re the best! Thanks for all your hard work, Lawton.
Tengauge.2165:
^This, very much this. I truly appreciate the hard work you put into the API, and even show it off to IT coworkers when I need to explain to them why the whole world needs well documented APIs for everything. Your efforts are strongly appreciated by all of us geeks in the community.
Renske.6178:
Awesome!! Thank you for you hard work. I was usng it for a school project so im very glad its back up. =D