Recreate HTTPSConnection after errors

[dgilmanAIDENTIFIED]

This is a fix to #556

If a network error happens when self.__conn.request is called the underlying HTTPSConnection is left in a bad state. If you're trying to retry errors the next request raises a CannotSendRequest when you get to the next self.__conn.request call.

This PR keeps track of successful self.__conn.getrequest() calls and resets the connection object if a previous request was not successful.

[ridersofrohan]

This would also probably fix #378

[dgilmanAIDENTIFIED]

We've been using this in production for a year now successfully so I hope this can be merged as it fixes what was for us a frequent issue. I am going to leave this PR open in hopes the Recurly team can take it over and merge the fix. However, I am no longer going to maintain it as we are no longer Recurly customers.

[davidemerritt]

@bhelx - will this be implemented - I just ran into a similar problem.

[bhelx]

It seems like to recreate the entire connection every call might be wasteful for the problem. However i'm not sure what the result of this is. Does it also recreate the underlying connection or just the connection object? Is there a more precise way to detect when the connection needs to be reset?

[dgilmanAIDENTIFIED]

needs_reset is toggled back to False after a successful Response() (see below). So this path is only ever triggered when you had an exception that prevented the toggling.

[bhelx]

seems simple enough!

[dgilmanAIDENTIFIED]

Another way of looking at it: if you were creating a new Recurly object for each API call you avoided this bug but you made a lot of redundant connection objects and Recurly instances. If you didn't make a new Recurly object each time your Recurly instance would occasionally break when the unhandled exception was raised below. But with this change you're reusing a connection object if there is no exception raised and re-creating one when there is an exception / it's in the broken state.

[bhelx]

@davidemerritt i haven't worked for Recurly since 2020. So pinging @douglasmiller for this issue.

[bhelx]

Just a note: resetting the connection every call seems kind of extreme. Perhaps you can just catch the exception CannotSendRequest, recreate the connection, then retry the request.

But there could be some deeper problem worth looking into.

[davidemerritt]

thanks @bhelx!

@douglasmiller - I would expect that the client = recurly.Client(settings.RECURLY_API_KEY) client setup would handle connection pooling and be persistent. If it starts breaking at a certain point in production then we have no way to re-initialize this since it is created as a singleton. Would be great for this to be addressed. Thank you