AWS AppConfig agent error “connection refused”

AWS AppConfig service it’s useful for feature flag functionality, you can access it directly via API but this is not the suggested method, for production workload it’s a best practice to use the provided agent. If you are using AppConfig on Kubernetes or EKS you should add the appconfig-agent to your deployment by adding:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
  labels:
    app: my-application-label
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-application-label
  template:
    metadata:
      labels:
        app: my-application-label
    spec:
      containers:
      - name: my-app
        image: my-repo/my-image
        imagePullPolicy: IfNotPresent
      - name: appconfig-agent
        image: public.ecr.aws/aws-appconfig/aws-appconfig-agent:2.x
        ports:
        - name: http
          containerPort: 2772
          protocol: TCP
        env:
        - name: SERVICE_REGION
          value: region
        imagePullPolicy: IfNotPresent

This method will work but in some edge cases you could “randomly” get an exception like this:

cURL error 7: Failed to connect to localhost port 2772 after 0 ms: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:2772/applications/APPLICATION_NAME/environments/ENVIRONMENT_NAME/configurations/CONFIGURATION_NAME

If you take a look at the logs you could notice that the AppConfig agent has been explicitly shut down:

[appconfig agent] INFO shutdown complete (actual duration: 50ms)
[appconfig agent] INFO received terminated signal, shutting down
[appconfig agent] INFO shutting down in 50ms
[appconfig agent] INFO stopping server on localhost:2772

digging into the logs you could notice that the master container is still working for some seconds after the appconfig-agent has been shut down, that’s the problem! appconfig-agent is very fast to shut down, if your primary container is still working when appconfig has been shut down, your primary container will not be able to connect to the agent and you will get the error.

How to make sure that appconfig-agent is always active in a deployment? the new Sidecar Container feature, added in the recent 1.29 Kubernetes release, is a perfect fit: the container in the sidecar (appconfig-agent) will be the first to start and the last to stop, your primary container will always find the sidecar ready.

Modify the deployment this way:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
  labels:
    app: my-application-label
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-application-label
  template:
    metadata:
      labels:
        app: my-application-label
    spec:
      containers:
      - name: my-app
        image: my-repo/my-image
        imagePullPolicy: IfNotPresent
      initContainers: 
      - name: appconfig-agent
        image: public.ecr.aws/aws-appconfig/aws-appconfig-agent:2.x
        restartPolicy: Always
        ports:
        - name: http
          containerPort: 2772
          protocol: TCP
        env:
        - name: SERVICE_REGION
          value: region
        imagePullPolicy: IfNotPresent