kubernetesingress控制器traefik获取真实客户源IP

一.现象

公司kubenetes生产环境使用的ingress控制器是traefik，默认是通过deployment部署的，现在研发上反馈不能获取客户的真实源IP地址，通过x_forward_for获取的IP地址都是kubernetes集群内部的IP地址。

二.解决思路

通过查找traefik的官方文档

Traefik EntryPoints Documentation - Traefik

traefik传输客户源地址到后端需要配置Forwarded Headers参数，用于x_forwarded_for保存客户源IP地址

具体配置如下

 yaml格式，用于启用的时候指定配置文件

1.  
   ## Static configuration
2.  
   entryPoints:
3.  
   web:
4.  
   address: ":80"
5.  
   forwardedHeaders:
6.  
   insecure: true

TOML格式，用于traefik启动的时候指定配置文件，这个文件在kubernetes中可以以comfigmap存在

1.  
   ## Static configuration
2.  
   [entryPoints]
3.  
   [entryPoints.web]
4.  
   address = ":80"
5.  
    
6.  
   [entryPoints.web.forwardedHeaders]
7.  
   insecure = true

命名行格式

1.  
   --entryPoints.web.address=:80
2.  
   --entryPoints.web.forwardedHeaders.insecure

三.问题处理

1.公司现有环境配置

1.  
   ---
2.  
   apiVersion: apps/v1
3.  
   kind: Deployment
4.  
   metadata:
5.  
   annotations:
6.  
   meta.helm.sh/release-name: traefik
7.  
   meta.helm.sh/release-namespace: default
8.  
   labels:
9.  
   app.kubernetes.io/instance: traefik
10.  
    app.kubernetes.io/managed-by: Helm
11.  
    app.kubernetes.io/name: traefik
12.  
    helm.sh/chart: traefik-9.11.0
13.  
    name: traefik
14.  
    namespace: default
15.  
    resourceVersion: '505763774'
16.  
    spec:
17.  
    progressDeadlineSeconds: 600
18.  
    replicas: 6
19.  
    revisionHistoryLimit: 10
20.  
    selector:
21.  
    matchLabels:
22.  
    app.kubernetes.io/instance: traefik
23.  
    app.kubernetes.io/name: traefik
24.  
    strategy:
25.  
    rollingUpdate:
26.  
    maxSurge: 1
27.  
    maxUnavailable: 1
28.  
    type: RollingUpdate
29.  
    template:
30.  
    metadata:
31.  
    creationTimestamp: null
32.  
    labels:
33.  
    app.kubernetes.io/instance: traefik
34.  
    app.kubernetes.io/managed-by: Helm
35.  
    app.kubernetes.io/name: traefik
36.  
    helm.sh/chart: traefik-9.11.0
37.  
    spec:
38.  
    containers:
39.  
    - args:
40.  
    - '--global.checknewversion'
41.  
    - '--global.sendanonymoususage'
42.  
    - '--entryPoints.traefik.address=:9000/tcp'
43.  
    - '--entryPoints.web.address=:8000/tcp'
44.  
    - '--entryPoints.websecure.address=:8443/tcp'
45.  
    - '--api.dashboard=true'
46.  
    - '--ping=true'
47.  
    - '--providers.kubernetescrd'
48.  
    - '--providers.kubernetesingress'
49.  
    image: 'traefik:2.3.3'
50.  
    imagePullPolicy: IfNotPresent
51.  
    livenessProbe:
52.  
    failureThreshold: 3
53.  
    httpGet:
54.  
    path: /ping
55.  
    port: 9000
56.  
    scheme: HTTP
57.  
    initialDelaySeconds: 10
58.  
    periodSeconds: 10
59.  
    successThreshold: 1
60.  
    timeoutSeconds: 2
61.  
    name: traefik
62.  
    ports:
63.  
    - containerPort: 9000
64.  
    name: traefik
65.  
    protocol: TCP
66.  
    - containerPort: 8000
67.  
    name: web
68.  
    protocol: TCP
69.  
    - containerPort: 8443
70.  
    name: websecure
71.  
    protocol: TCP
72.  
    readinessProbe:
73.  
    failureThreshold: 1
74.  
    httpGet:
75.  
    path: /ping
76.  
    port: 9000
77.  
    scheme: HTTP
78.  
    initialDelaySeconds: 10
79.  
    periodSeconds: 10
80.  
    successThreshold: 1
81.  
    timeoutSeconds: 2
82.  
    resources: {}
83.  
    securityContext:
84.  
    capabilities:
85.  
    drop:
86.  
    - ALL
87.  
    readOnlyRootFilesystem: true
88.  
    runAsGroup: 65532
89.  
    runAsNonRoot: true
90.  
    runAsUser: 65532
91.  
    terminationMessagePath: /dev/termination-log
92.  
    terminationMessagePolicy: File
93.  
    volumeMounts:
94.  
    - mountPath: /data
95.  
    name: data
96.  
    - mountPath: /tmp
97.  
    name: tmp
98.  
    dnsPolicy: ClusterFirst
99.  
    restartPolicy: Always
100.  
     schedulerName: default-scheduler
101.  
     securityContext:
102.  
     fsGroup: 65532
103.  
     serviceAccount: traefik
104.  
     serviceAccountName: traefik
105.  
     terminationGracePeriodSeconds: 60
106.  
     volumes:
107.  
     - emptyDir: {}
108.  
     name: data
109.  
     - emptyDir: {}
110.  
     name: tmp
111.  
     status:
112.  
     availableReplicas: 6
113.  
     conditions:
114.  
     - lastTransitionTime: '2022-10-10T07:58:50Z'
115.  
     lastUpdateTime: '2022-10-10T07:58:50Z'
116.  
     message: Deployment has minimum availability.
117.  
     reason: MinimumReplicasAvailable
118.  
     status: 'True'
119.  
     type: Available
120.  
     - lastTransitionTime: '2020-11-25T22:53:59Z'
121.  
     lastUpdateTime: '2022-11-17T10:44:40Z'
122.  
     message: ReplicaSet "traefik-54bf67c74d" has successfully progressed.
123.  
     reason: NewReplicaSetAvailable
124.  
     status: 'True'
125.  
     type: Progressing
126.  
     observedGeneration: 13
127.  
     readyReplicas: 6
128.  
     replicas: 6
129.  
     updatedReplicas: 6
130.  
      
131.  
     ---
132.  
     apiVersion: v1
133.  
     kind: Service
134.  
     metadata:
135.  
     annotations:
136.  
     meta.helm.sh/release-name: traefik
137.  
     meta.helm.sh/release-namespace: default
138.  
     labels:
139.  
     app.kubernetes.io/instance: traefik
140.  
     app.kubernetes.io/managed-by: Helm
141.  
     app.kubernetes.io/name: traefik
142.  
     helm.sh/chart: traefik-9.11.0
143.  
     name: traefik
144.  
     namespace: default
145.  
     resourceVersion: '505762379'
146.  
     spec:
147.  
     clusterIP: 10.96.252.108
148.  
     externalTrafficPolicy: Cluster
149.  
     ports:
150.  
     - name: web
151.  
     nodePort: 30079
152.  
     port: 80
153.  
     protocol: TCP
154.  
     targetPort: web
155.  
     selector:
156.  
     app.kubernetes.io/instance: traefik
157.  
     app.kubernetes.io/name: traefik
158.  
     sessionAffinity: None
159.  
     type: NodePort
160.  
     status:
161.  
     loadBalancer: {}
162.  
      

发现trafik启动的参数里面没有指定配置文件，也没有以comfigmap形式存在，所以只能考虑通过添加启动参数的方式（即命令行方式）解决问题

2.解决问题

在deployment部署中的traefik启动参数中添加 - '--entrypoints.web.forwardedHeaders.insecure'和- '--entrypoints.websecure.forwardedHeaders.insecure'启动参数，其中标红的web和websecure根据--entryPoints.web.address=:8000/tcp和--entryPoints.websecure.address=:8443/tcp要保持一致，有可能不一样。

修改后的derloyment的yaml文件如下：

1.  
   ---
2.  
   apiVersion: apps/v1
3.  
   kind: Deployment
4.  
   metadata:
5.  
   annotations:
6.  
   meta.helm.sh/release-name: traefik
7.  
   meta.helm.sh/release-namespace: default
8.  
   labels:
9.  
   app.kubernetes.io/instance: traefik
10.  
    app.kubernetes.io/managed-by: Helm
11.  
    app.kubernetes.io/name: traefik
12.  
    helm.sh/chart: traefik-9.11.0
13.  
    name: traefik
14.  
    namespace: default
15.  
    resourceVersion: '505763774'
16.  
    spec:
17.  
    progressDeadlineSeconds: 600
18.  
    replicas: 6
19.  
    revisionHistoryLimit: 10
20.  
    selector:
21.  
    matchLabels:
22.  
    app.kubernetes.io/instance: traefik
23.  
    app.kubernetes.io/name: traefik
24.  
    strategy:
25.  
    rollingUpdate:
26.  
    maxSurge: 1
27.  
    maxUnavailable: 1
28.  
    type: RollingUpdate
29.  
    template:
30.  
    metadata:
31.  
    creationTimestamp: null
32.  
    labels:
33.  
    app.kubernetes.io/instance: traefik
34.  
    app.kubernetes.io/managed-by: Helm
35.  
    app.kubernetes.io/name: traefik
36.  
    helm.sh/chart: traefik-9.11.0
37.  
    spec:
38.  
    containers:
39.  
    - args:
40.  
    - '--global.checknewversion'
41.  
    - '--global.sendanonymoususage'
42.  
    - '--entryPoints.traefik.address=:9000/tcp'
43.  
    - '--entryPoints.web.address=:8000/tcp'
44.  
    - '--entryPoints.websecure.address=:8443/tcp'
45.  
    - '--api.dashboard=true'
46.  
    - '--ping=true'
47.  
    - '--providers.kubernetescrd'
48.  
    - '--providers.kubernetesingress'
49.  
    - '--entrypoints.web.forwardedHeaders.insecure'
50.  
    - '--entrypoints.websecure.forwardedHeaders.insecure'
51.  
    image: 'traefik:2.3.3'
52.  
    imagePullPolicy: IfNotPresent
53.  
    livenessProbe:
54.  
    failureThreshold: 3
55.  
    httpGet:
56.  
    path: /ping
57.  
    port: 9000
58.  
    scheme: HTTP
59.  
    initialDelaySeconds: 10
60.  
    periodSeconds: 10
61.  
    successThreshold: 1
62.  
    timeoutSeconds: 2
63.  
    name: traefik
64.  
    ports:
65.  
    - containerPort: 9000
66.  
    name: traefik
67.  
    protocol: TCP
68.  
    - containerPort: 8000
69.  
    name: web
70.  
    protocol: TCP
71.  
    - containerPort: 8443
72.  
    name: websecure
73.  
    protocol: TCP
74.  
    readinessProbe:
75.  
    failureThreshold: 1
76.  
    httpGet:
77.  
    path: /ping
78.  
    port: 9000
79.  
    scheme: HTTP
80.  
    initialDelaySeconds: 10
81.  
    periodSeconds: 10
82.  
    successThreshold: 1
83.  
    timeoutSeconds: 2
84.  
    resources: {}
85.  
    securityContext:
86.  
    capabilities:
87.  
    drop:
88.  
    - ALL
89.  
    readOnlyRootFilesystem: true
90.  
    runAsGroup: 65532
91.  
    runAsNonRoot: true
92.  
    runAsUser: 65532
93.  
    terminationMessagePath: /dev/termination-log
94.  
    terminationMessagePolicy: File
95.  
    volumeMounts:
96.  
    - mountPath: /data
97.  
    name: data
98.  
    - mountPath: /tmp
99.  
    name: tmp
100.  
     dnsPolicy: ClusterFirst
101.  
     restartPolicy: Always
102.  
     schedulerName: default-scheduler
103.  
     securityContext:
104.  
     fsGroup: 65532
105.  
     serviceAccount: traefik
106.  
     serviceAccountName: traefik
107.  
     terminationGracePeriodSeconds: 60
108.  
     volumes:
109.  
     - emptyDir: {}
110.  
     name: data
111.  
     - emptyDir: {}
112.  
     name: tmp
113.  
     status:
114.  
     availableReplicas: 6
115.  
     conditions:
116.  
     - lastTransitionTime: '2022-10-10T07:58:50Z'
117.  
     lastUpdateTime: '2022-10-10T07:58:50Z'
118.  
     message: Deployment has minimum availability.
119.  
     reason: MinimumReplicasAvailable
120.  
     status: 'True'
121.  
     type: Available
122.  
     - lastTransitionTime: '2020-11-25T22:53:59Z'
123.  
     lastUpdateTime: '2022-11-17T10:44:40Z'
124.  
     message: ReplicaSet "traefik-54bf67c74d" has successfully progressed.
125.  
     reason: NewReplicaSetAvailable
126.  
     status: 'True'
127.  
     type: Progressing
128.  
     observedGeneration: 13
129.  
     readyReplicas: 6
130.  
     replicas: 6
131.  
     updatedReplicas: 6
132.  
      
133.  
     ---
134.  
     apiVersion: v1
135.  
     kind: Service
136.  
     metadata:
137.  
     annotations:
138.  
     meta.helm.sh/release-name: traefik
139.  
     meta.helm.sh/release-namespace: default
140.  
     labels:
141.  
     app.kubernetes.io/instance: traefik
142.  
     app.kubernetes.io/managed-by: Helm
143.  
     app.kubernetes.io/name: traefik
144.  
     helm.sh/chart: traefik-9.11.0
145.  
     name: traefik
146.  
     namespace: default
147.  
     resourceVersion: '505762379'
148.  
     spec:
149.  
     clusterIP: 10.96.252.108
150.  
     externalTrafficPolicy: Cluster
151.  
     ports:
152.  
     - name: web
153.  
     nodePort: 30079
154.  
     port: 80
155.  
     protocol: TCP
156.  
     targetPort: web
157.  
     selector:
158.  
     app.kubernetes.io/instance: traefik
159.  
     app.kubernetes.io/name: traefik
160.  
     sessionAffinity: None
161.  
     type: NodePort
162.  
     status:
163.  
     loadBalancer: {}

3.验证测试

通过和研发和业务验证测试，通过x_forwarded_for获取的客户端源Ip地址格式如下

 其中第一个公网IP地址是客户的真实IP地址，其他几个是经过的k8s节点和pod的IP地址，至此解决业务上获取真实客户IP地址的需求。

这篇好文章是转载于：学新通技术网