HTTP /metrics doesn't respond after network failure

[man1207]

rtpengine version the issue has been seen with

12.5.1.36

Used distribution and its version

Ubuntu 24.04.2 LTS

Linux kernel version used

6.8.0-41-generic

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

After network failure http-endpoint '/metrics' responds in regular manner

Unexpected behaviour you saw

After network failure http-endpoint '/metrics' doesn't respond, on localhost too.

Steps to reproduce the problem

1. Run rtpengine
2. Make temporary network failure
3. Try make curl http://ip:port/metrics from another host / or localhost

Additional program output to the terminal or logs illustrating the issue

Anything else?

No response

[rfuchs]

What does "Make temporary network failure" mean exactly?

[man1207]

Metrics are gathered by VictoriaMetrics from host with rtpengine opened http port.
When network failed, ie tunnel is break accidentially or link is down, tcp connections are stucked and rtpengine http socket stop working. VictoriaMetrics can't gather metrics ever after network is up.
If I ssh to localhost and try run curl http://ip:port/metrics - it is stucked too. No respond from exporter.

[rfuchs]

So remove IP address or set link down? Cannot reproduce with that.

[sipwise-lab-trunk] root@spce:~# rg listen.*http /etc/rtpengine/rtpengine.conf
5:listen-http = 0.0.0.0:2225
[sipwise-lab-trunk] root@spce:~# ip a ls
...
3: neth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
...
    inet 192.168.1.170/24 brd 192.168.1.255 scope global dynamic neth1
...
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
# HELP rtpengine_sessions Owned sessions
# TYPE rtpengine_sessions gauge
...
[sipwise-lab-trunk] root@spce:~# ip a del 192.168.1.170/24 dev neth1 
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
^C
[sipwise-lab-trunk] root@spce:~# 130 # ip a add 192.168.1.170/24 dev neth1 
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
# HELP rtpengine_sessions Owned sessions
# TYPE rtpengine_sessions gauge
...
[sipwise-lab-trunk] root@spce:~# ip a ls

...
3: neth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
...
    inet 192.168.1.170/24 scope global neth1
...
[sipwise-lab-trunk] root@spce:~# ip link set neth1 down 
[sipwise-lab-trunk] root@spce:~# ip a ls
...
3: neth1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
...
    inet 192.168.1.170/24 scope global neth1
...
[sipwise-lab-trunk] root@spce:~# ip link set neth1 up
[sipwise-lab-trunk] root@spce:~# ip a ls
...
3: neth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
...
    inet 192.168.1.170/24 scope global neth1
...
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
# HELP rtpengine_sessions Owned sessions
# TYPE rtpengine_sessions gauge
...
...
...
[sipwise-lab-trunk] root@spce:~# rg listen.*http /etc/rtpengine/rtpengine.conf
5:listen-http = 192.168.1.170:2225
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
# HELP rtpengine_sessions Owned sessions
# TYPE rtpengine_sessions gauge
...
[sipwise-lab-trunk] root@spce:~# ip a del 192.168.1.170/24 dev neth1 
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
^C
[sipwise-lab-trunk] root@spce:~# 130 # ip a add 192.168.1.170/24 dev neth1 
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
# HELP rtpengine_sessions Owned sessions
# TYPE rtpengine_sessions gauge
...
[sipwise-lab-trunk] root@spce:~# ip link set neth1 down 
[sipwise-lab-trunk] root@spce:~# ip link set neth1 up
[sipwise-lab-trunk] root@spce:~# curl http://192.168.1.170:2225/metrics
# HELP rtpengine_sessions Owned sessions
# TYPE rtpengine_sessions gauge
...

[man1207]

Further I'll be back with correct list of actions to reproduce issue.

[man1207]

Additional info.

After network failure rtpegnine stucked on 9906 and I observed it 3 days. After 3 days I got some output from host.

open file descriptors

# ls /proc/$(pidof rtpengine)/fd | wc -l
29

lsof port 9906

# lsof -p $(pidof rtpengine) | grep 9906
rtpengine 1049482 rtpengine   13u     IPv4           61483189      0t0        TCP hostname:9906 (LISTEN)
rtpengine 1049482 rtpengine   20u     IPv4           81225168      0t0        TCP hostname:9906->scrapper:58481 (CLOSE_WAIT)

CLOSE-WAIT connections on port 9906

# ss -tan sport = :9906 | grep CLOSE-WAIT | wc -l
2992

[man1207]

@rfuchs I have reproduced the bug.
Here is a list of steps:

1. Make continuos requests onto port 9906 from another host.

while true; do curl -m 0.8 -s http://hostname:9906/metrics >/dev/null || true sleep 0.2; done

2. Run watch on rtpengine host

watch -n 1 'ss -tan sport = :9906 | grep CLOSE-WAIT | wc -l'

3. Block port 9906 by iptables on rtpengine host

iptables -I INPUT -p tcp --dport 9906 -j DROP

4. Wait for some time (I waited for ~20-60 seconds)
5. Unblock port 9906 by iptables on rtpengine host

iptables -D INPUT -p tcp --dport 9906 -j DROP

6. Try request port 9906 again

curl -v http://hostname:9906/metrics

My output was

• Host hostname:9906 was resolved.
• IPv6: (none)
• IPv4: 10.1.1.1
• Trying 10.1.1.1:9906...
• Connected to hostname (10.1.1.1) port 9906

GET /metrics HTTP/1.1
Host: hostname:9906
User-Agent: curl/8.5.0
Accept: /

7. Look at watch launched on step 2

My output was (~10-15 min left)

Every 1.0s: ss -tan sport = :9906 | grep CLOSE-WAIT | wc -l
98

The solution is to restart rtpengine only. After restart rtpengine starts accept new connections on port 9906

[rfuchs]

1480574 is probably the fix

[rfuchs]

The relevant commits have been backported into the mr12.5 branch. Feel free to test with a build from that branch if you can.

Backports to mr12.5.1 and hotfix release are pending.

[man1207]

It seems to be fixed after have been updated to 13.5.1
Problem is not reproduced anymore
Thx!