Odd Price endpoint behaviour
Ravenhearte.5913:
Hey people,
So, when querying the price endpoint for ids, I get back a list of 22581 ids. When I page in the endpoint, I end up with 22557 objects. I checked page 112, and it had 181 objects, which seemed odd, I then consoled out the total objects in each page as it was read in, page 76 returns 195 objects, page 78 : 181. Is there a reason for these missing 24 objects? Shouldn’t all valid ids be represented in the returned objects? Basically, is this a bug, or a “feature”?
SingerOfTheFall.3546:
Which IDs did not return any info? You can try to get information about them via the items endpoint, and check what items are those exactly. As a wild guess, those might be the items that are potentially sell-able, but have never been obtained by the players (e.g. items that are not available in the game yet).
Ravenhearte.5913:
It was my understanding that only items that have been found in the game are added to the endpoints. Specifically, I was under the impression that only items that have been listed for sale enter the commerce endpoints. So that solution seems improbable. I’m about to head to bed, but if no one has an answer tomorrow, I’ll write up an algorithm to find the specific ids that are missing.
Lawton Campbell.8517:
So… /v2/items and /v2/commerce are backed by completely different databases. I guess technically /v2/items isn’t backed by a database, it’s backed by the .dat file. /v2/items also has a manually-maintained whitelist in it so it doesn’t spill the beans on everything; that whitelist doesn’t talk to the /v2/commerce database.
It’s possible that there are just items missing from the whitelist, or it’s possible that there’s just some weird stuff in the /v2/commerce database. There was a historical bug wherein users could place buy orders for arbitrary item ids regardless of their validity; I think that was all cleaned up but there might be some remaining detritus?
In any case, the inconsistency is a known fun bit that we’ll probably never fix.
Ravenhearte.5913:
That kind of sucks. I was using the count of the ids returned to determine when I had finished paging in the objects, which works fine for the item and recipe endpoints. Back to the drawing board I guess.
Lawton Campbell.8517:
That kind of sucks. I was using the count of the ids returned to determine when I had finished paging in the objects, which works fine for the item and recipe endpoints. Back to the drawing board I guess.
We might have it fixed at some point, but pagination should still work using the X-Page-Total response header. It’ll always have the same number of pages, you just might not get the full count for each page (the broken objects will be omitted and you’ll get a 206 status code instead of 200). Alternatively, the Link header has an RFC5988-formatted datum that contains a URL to the next page — when there’s no ‘next’ link you’re at the end.
Totally agree that it’s a massive annoyance though, it just happens to be in a lower-level component that I’m not really familiar with so I can’t go in and fix it myself 
Ravenhearte.5913:
I’m actually getting 200 codes on all pages, but I can compare X-RESULT-COUNT against what I am expecting to find pages with discrepancies. The trick would seem to be to figure out which ids are missing on a per page basis, as I’m running the loading in coroutines and can’t be guaranteed the order the pages are finishing. If I could do that I could delete them from the master idList and then be back to being able to compare that count to total objects loaded so I know when all coroutines have finished. As a hack, I guess I could create an independent count variable that starts with the count from idList, then I could simply subtract the calculated discrepancy amounts, but that’s so hack-ish. Would rather find a more elegant solution.
Ravenhearte.5913:
For those curious, ids affected are:
31113 31120 31124 31127 31130 31131 31132 31133 31135 31136 31137 31139 31140 31144 31167 31168 31169 31175 31178 29952 29960 29964 29974 29978
I dealt with them by searching for the first objects id in the master idList, grabbing a range of 200, and then taking that sublist and removing all ids that successfully returned objects, I then deleted the ids that remained from the master idList.
Lawton Campbell.8517:
Cool, thanks for looking those up. As a temporary workaround, I’m going to blacklist those ids from appearing in the id list. Won’t fix the root of the problem but it’ll be something.
Lawton Campbell.8517:
For those curious, ids affected are:
31113 31120 31124 31127 31130 31131 31132 31133 31135 31136 31137 31139 31140 31144 31167 31168 31169 31175 31178 29952 29960 29964 29974 29978
These ids should now all be blacklisted. They won’t be returned from /v2/commerce/listings anymore.
Chokapik.3741:
I just found a similar issue in v2/skins.
This page returns a total of 3520 skin ids.
However, when trying to request data for all of these ids, I end up with 3519 objects.
The skin causing the issue seems to be id=2984.
https://api.guildwars2.com/v2/skins/2984
Edit: I also wrote a quick jsfiddle that could be handy when we have these kinds of problems. By changing the URL at the top of the script, you can check the other endpoints as well (I checked some of them, haven’t found any issues so far).