Auto-Gen Rules

Pods are one of the most common object types in Kubernetes and as such are the focus of most types of validation rules. But creation of Pods directly is almost never done as it is considered an anti-pattern. Instead, Kubernetes has many higher-level controllers that directly or indirectly manage Pods, namely the Deployment, DaemonSet, StatefulSet, Job, and CronJob resources. Third-party custom resources may also “wrap” or leverage the Pod template as part of their resources as well. Writing policy that targets Pods but must be written for every one of these controllers would be tedious and inefficient. Kyverno solves this issue by supporting automatic generation of policy rules for higher-level controllers from a rule written exclusively for a Pod. For rules which match on Pods in addition to other kinds, auto-generation is not activated.

For example, when creating a validation policy like below which checks that all images come from an internal, trusted registry, the policy applies to all resources capable of generating Pods.

 1apiVersion : kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: restrict-image-registries
 5spec:
 6  validationFailureAction: Enforce
 7  rules:
 8  - name: validate-registries
 9    match:
10      any:
11      - resources:
12          kinds:
13          - Pod
14    validate:
15      message: "Images may only come from our internal enterprise registry."
16      pattern:
17        spec:
18          containers:
19          - image: "registry.domain.com/*"

Once the policy is created, these other resources can be shown in auto-generated rules which Kyverno adds to the policy under the status object.

 1status:
 2  autogen:
 3    rules:
 4    - exclude:
 5        resources: {}
 6      generate:
 7        clone: {}
 8        cloneList: {}
 9      match:
10        any:
11        - resources:
12            kinds:
13            - DaemonSet
14            - Deployment
15            - Job
16            - StatefulSet
17            - ReplicaSet
18            - ReplicationController
19        resources: {}
20      mutate: {}
21      name: autogen-validate-registries
22      validate:
23        message: Images may only come from our internal enterprise registry.
24        pattern:
25          spec:
26            template:
27              spec:
28                containers:
29                - image: registry.domain.com/*
30    - exclude:
31        resources: {}
32      generate:
33        clone: {}
34        cloneList: {}
35      match:
36        any:
37        - resources:
38            kinds:
39            - CronJob
40        resources: {}
41      mutate: {}
42      name: autogen-cronjob-validate-registries
43      validate:
44        message: Images may only come from our internal enterprise registry.
45        pattern:
46          spec:
47            jobTemplate:
48              spec:
49                template:
50                  spec:
51                    containers:
52                    - image: registry.domain.com/*

Rule auto-generation behavior is controlled by the policy annotation pod-policies.kyverno.io/autogen-controllers. You can change the value of the annotation to customize the target Pod controllers for the auto-generated rules. For example, Kyverno generates rules for a Deployment and Job if the annotation is defined as pod-policies.kyverno.io/autogen-controllers=Deployment,Job. To disable auto-generating rules for Pod controllers entirely, set it to the value none.

In addition to standard Kubernetes Pod controllers, auto-gen also supports custom resources where a Pod template has been embedded. For example, the OpenKruise custom resource CloneSet embeds a Pod template and may produce auto-gen rules when CloneSet is included in the value of the pod-policies.kyverno.io/autogen-controllers annotation.

Kyverno skips generating Pod controller rules whenever the following resources fields/objects are specified in a match or exclude block as these filters may not be applicable to Pod controllers:

• names
• selector
• annotations

Additionally, Kyverno only auto-generates rules when the resource kind specified in a combination of match and exclude is no more than Pod. Mutate rules which match on Pod and use a JSON patch are also excluded from rule auto-generation as noted here.

When disabling auto-generation rules for select Pod controllers, or when auto-generation does not apply, Kyverno still applies policy matching on Pods to those spawned by those controllers. To exempt these Pods, use preconditions with an expression similar to the below which may allow Pods created by a Job controller to pass.

1- key: Job
2  operator: AnyNotIn
3  value: "{{ request.object.metadata.ownerReferences[].kind }}"

Exclusion by Metadata

In some cases, it may be desirable to use an exclude block applied to Pods that uses either labels or annotations. For example, the following match and exclude statement may be written, the purpose of which would be to match any Pods except those that have the annotation policy.test/require-requests-limits=skip.

 1rules:
 2  - name: validate-resources
 3    match:
 4      any:
 5      - resources:
 6          kinds:
 7            - Pod
 8    exclude:
 9      any:
10      - resources:
11          annotations:
12            policy.test/require-requests-limits: skip

When Kyverno sees these types of fields as mentioned above it skips auto-generation for the rule. The next choice may be to use preconditions to achieve the same effect but by writing an expression that looks at request.object.metadata.*. As part of auto-generation, Kyverno will see any variables from AdmissionReview such as that beginning with request.object and translate it for each of the applicable Pod controllers. The result may be that the auto-generated rule for, as an example, Deployments will get translated to request.object.spec.template.metadata.* which references the metadata object inside the Pod template and not the metadata object of the Deployment itself. To work around this and have preconditions which are not translated for these metadata use cases, double quote the object portion of the variable as shown below.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: require-requests-limits
 5spec:
 6  validationFailureAction: Enforce
 7  background: true
 8  rules:
 9    - name: validate-resources
10      match:
11        any:
12        - resources:
13            kinds:
14              - Pod
15      preconditions:
16        all:
17        - key: "{{ request.\"object\".metadata.annotations.\"policy.test.io/require-requests-limits\" || '' }}"
18          operator: NotEquals
19          value: skip
20      validate:
21        message: "CPU and memory resource requests and limits are required."
22        pattern:
23          spec:
24            containers:
25              - resources:
26                  requests:
27                    memory: "?*"
28                    cpu: "?*"
29                  limits:
30                    memory: "?*"

The result will have the same effect as the first snippet which uses an exclude block and have the benefit of auto-generation coverage.

Similar to the automatic translation of expressions beginning with request.object.metadata.*, Kyverno also auto-generates rules for Pod controllers when a pattern specifies the same structure. For example, the disallow default namespace policy is a validate rule which uses an overlay pattern to ensure that neither a Pod nor any of its controllers can use the default Namespace.

1pattern:
2  metadata:
3    namespace: "!default"

With auto-gen set to its default, this will get translated in the case of Deployments and others to the following below which is not the desire when expressing such a rule as the namespace field is not defined under the Pod template when using a Pod controller. This auto-generated pattern will therefore cause all applicable Pod controllers to be in violation of the translated pattern.

1pattern:
2  spec:
3    template:
4      metadata:
5        namespace: "!default"

In such cases, auto-gen should be disabled as described above and one or more rules written to explicitly control the matching resources and the patterns/expressions used against them.