Update HLTB for Steam to v1.1.1

[jcdoll]

Updates the HLTB for Steam plugin submodule to v1.1.1.

Minor update to make it easier for users to contribute name fixes.

Changes

• Changed name fix order to substitute -> sanitize -> substitute (supports raw Steam names with symbols)
• Fixed cache to always refetch misses so name fixes take effect immediately
• Added name fixes: Borderlands GOTY, Civilization VI, Batman Arkham Asylum GOTY, Napoleon Total War

[shdwmtr]

There has to be a better solution to hard-coding names. I haven't looked into it at all, but why couldn't you just search by appid instead of the game name?

[jcdoll]

Yeah, I looked into it when doing the initial dev and nothing exists.

HLTB provides no official API. Wrappers in other languages are name based lookup only, e.g.
https://pypi.org/project/howlongtobeatpy/

My Lua backend is based on porting over the python wrapper for the most part.

The funny thing is that HLTB has the Steam IDs internally, but you can only obtain them after finding the game first by name.

So one approach would be to scrape it, add results to your own db, and then query that by SteamID. Pretty fragile and adds hosting needs which I'm not interested in.
https://codeberg.org/Crashdummy/HowLongToBeatApi

The simplest option would be to find a reliable third party that scrapes and hosts a DB for querying.

I'm not super worried about the scaling though. Two thoughts:

1. ~99% of games work fine with no corrections, so I don't expect the intersection of (games that people play & needs corrections) to be that large. Probably hundreds but not thousands. I've made the instructions for adding corrections very DIY friendly now, so folks can add updates and I can release a new version once every few weeks until it trickles off.

2. If the long tail persists, I can use the name correction data to improve the "name simplifier" code that I already added to the plugin. HLTB search uses exact matching, so you basically need to strip down the input name so that it matches more things and then find a match.

[shdwmtr]

You could keep your current algorithm, and when you think you found the game, check its HLTB id against their price API: https://howlongtobeat.com/api/price-checks/${id}. From there, you can see if the Steam appid matches the appid in the response, and if not retry.

Of course there is more overhead here, but it would be a lot more accurate and likely would never require more than 2 tries.

[jcdoll]

Check SteamID for validation - 100% agreed, I already do that.

https://github.com/jcdoll/hltb-millennium-plugin/blob/92eaa96e38a0b6e65b42b0d16f712ddd26cacd8d/backend/hltb_match.lua#L68

Btw IsThereAnyDeal API has some HLTB data but it is for internal use only. I've reached out to their team but it is going to remain for internal (first-party) use only for the foreseeable future.

[shdwmtr]

You can actually query your entire Steam library at once too, with appid's provided.

Endpoint: https://howlongtobeat.com/api/steam/getSteamImportData
Post data:

{ 
  "steamUserId": "your profile id, ex: ShadowMonster70",
  "steamOmitData":0
}

I don't know everything you need, but this seems like a solid solution.
This doesn't provide all the stats you might want, but at least it perfectly maps appid to HLTB id.

[jcdoll]

Oh that's an interesting find. Certainly spits out a lot of data. I'll look into it.

Sample cmd:

curl -X POST "https://howlongtobeat.com/api/steam/getSteamImportData" -H "Content-Type: application/json" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" -H "Referer: https://howlongtobeat.com/" -d "{\"steamUserId\":\"mulard\",\"steamOmitData\":0}"

[jcdoll]

Ah I guess the limitation will be that your profile needs to be public.

Where did you find that endpoint btw?

[shdwmtr]

Ah yeah, true; that's subpar. I found it on the website:

https://howlongtobeat.com/steam

[jcdoll]

Got it.

I'll play around with that new endpoint. If the current user profile is public we could just use that I believe. And then for non-public profiles, which I want to support, that endpoint could help to automate the name correction lookup stuff.

All definitely in the future work bucket, needs some experimentation.

[jcdoll]

I'm converting to draft because I'm working to add the pre-fetch step using that new HLTB endpoint you found.

New approach:
public profile users -> get bulk (steam_id, hltb_id) from that step = reliable fetching
Private profile users -> still do name matching

Based on that endpoint I was able to query the libraries of the largest Steam users and add additional name fixes via a script. And it wasn't actually that many. ~250 total based on the whale libraries. So I think we will be in good shape after this.

Probably will get something out tonight before work starts again for the week.

[jcdoll]

Closing this one and opening 1.2.0 because it's a significant enough change