HAProxy reloads same time as Data Plane API

[tony-malek]

Hello,

I tried for a while to resolve this, but have not been able to unfortunately. I've also not had much joy from reaching out to the wider HAProxy community for support and was told I should report this as a potential bug. Here is some of the technical information -

I am running HAProxy version:
HAProxy version 2.4.24-0ubuntu0.22.04.1 2023/10/31

And Data Plane API version:

HAProxy Data Plane API v3.0.3 f1d4189a
Build date: 2024-10-10T13:42:11Z

One of the reasons we wanted to switch to using the 'AWS EC2 service discovery' with the Data Plane API is so we could minimise our reloads.

But any time I add a new server to one of our Auto scaling groups, HAProxy will reload which is not what should happen according to the below documentation:

https://www.haproxy.com/documentation/haproxy-data-plane-api/advanced-features/aws-ec2-service-discovery/

It mentions:

"When you register more instances of the same service, the API fills in disabled server slots. You can thus scale up or down without a reload, in most cases."

This is not what is happening for me though. Any scaling event (a new server or terminating one) will generate this in /var/log/dataplaneapi.log

time="2024-12-04T14:41:44Z" level=info msg="Reloaded Data Plane API"

And a corresponding reload that happened to HAProxy (this is from syslog with exact same timestamp),

Dec 4 14:41:44 i-04568442dd67098c5-haproxy-prod systemd[1]: Reloaded HAProxy Load Balancer.

This what my '/etc/haproxy/dataplaneapi.yml' file looks like:

config_version: 2
name: i-04568442dd67098c5-haproxy-prod.eu1.traveltek
dataplaneapi:  host: 0.0.0.0
  port: 5555
  advertised:
    api_address: ""
    api_port: 0
  scheme:
  - http
  userlist:
    userlist: dataplaneapi
  transaction:
    transaction_dir: /var/lib/dataplaneapi/transactions
    backups_number: 10
    backups_dir: /var/lib/dataplaneapi/backups
  resources:
    maps_dir: /etc/haproxy/maps
    ssl_certs_dir: /etc/haproxy/ssl    general_storage_dir: /etc/haproxy/general
    spoe_dir: /etc/haproxy/spoe
haproxy:
  config_file: /etc/haproxy/haproxy.cfg  haproxy_bin: /usr/sbin/haproxy
  reload:
    reload_delay: 5
    service_name: haproxy
    reload_strategy: systemd
log_targets:
- log_to: file
  log_file: /var/log/dataplaneapi.log
  log_level: info
  log_types:
  - access
  - app

and this is the Service Discovery file -

[
  {
    "description": "Production Environment",
    "enabled": true,
    "id": "51429dc6-4f2e-4b70-bf90-e35e5a7eeaaa",
    "ipv4_address": "private",
    "name": "ttek-prod",
    "region": "eu-west-1",
    "retry_timeout": 10,
    "server_slots_base": 10,
    "server_slots_growth_type": "exponential"
  }
]

After I set up the Data plane API, it modified my original /etc/haproxy/haproxy.cfg file and put a comment at the top saying it was now managing it. I of course have the relevant tags added to my ASG for this to work, and the backend will automatically be created. In other words adding the below tags:

Results in the backend automatically being created in HAProxy configuration file like below:

If I add a 2nd server, the IP is populated into the HAProxy configuration file as well and the Data Plane API and HAProxy service both reload as soon as it discovers the new instance. I am not able to figure out why it does this. I believed that the idle slots were available and ready to be used and wouldn't require a reload.

Is the HAProxy config. file not meant to be rewrote every time a new server is added to the ASG with the relevant tags?

Please let me know if I can provide any further information. Thank you.

[tony-malek]

Is anyone able to help me with this?

[mjuraga]

Hi @tony-malek is something else changing your file for you? Data Plane API uses inotify to reread the config file and reload some of it's configuration, and the part where the reload of HAProxy happens is weird. It seems something externally other than Data Plane API is managing this HAProxy file.