Generate Rules
A generate rule can be used to create new Kubernetes resources in response to some other event including things like resource creation, update, or delete, or even by creating or updating a policy itself. This is useful to create supporting resources, such as new RoleBindings or NetworkPolicies for a Namespace or perform other automation tasks that may either require other tools or be scripted.
Some common use cases for generate rules include:
- Namespace provisioner / Namespace-as-a-Service
- Create resources like a NetworkPolicy, ResourceQuota, and RoleBinding when a new Namespace is created
- Clone and synchronize Secrets and ConfigMaps
- Copy and then synchronize changes from one or more Secrets or ConfigMaps to Namespaces across the cluster
- Retroactive creation of NetworkPolicies
- In brownfield clusters, introduce a policy which will install a NetworkPolicy in all applicable Namespaces from a definition stored in a policy
- Custom eventing system
- Create Kubernetes Events and associate them with any object and have any message based upon an eligible admission review
Generate rules come in two flavors. They can either apply to admission events that occur across the cluster (ex., creation of a new Namespace), or they can apply to preexisting resources in the cluster (ex., an existing Namespace). Those which apply to admission events are considered standard generate rules while those which apply to preexisting resources are known as “generate existing” rules and are covered below.
Generate rules support match
and exclude
blocks and many of the other common Kyverno policy constructs such as preconditions, context variables, and more.
Kyverno can keep generated resources in sync to prevent tampering by use of a synchronize
property. When synchronize
is set to true
, the generated resource is kept in-sync with the source resource. Synchronization is beneficial in that modifications to the generated resource may be reverted, and changes to the source resource will be propagated. In addition to these effects, synchronization will ensure that the matching resource responsible for the triggering of the generation behavior is watched for changes. Should those changes result in a false match (including deletion), then it will result in the generated resource being removed to ensure the desired state is always maintained.
When using a generate rule, the source resource can either be an existing resource in the cluster, or a new resource defined in the rule itself. When the source is an existing resource in the cluster such as a ConfigMap or Secret, for example, the clone
object is used. See the Clone Source section for more details. When the source is defined directly in the rule, the data
object is used. See the Data Source section for more details. These are mutually exclusive and only one may be specified per rule.
Because Kyverno can generate any type of Kubernetes resource, including custom resources, in some cases it may be necessary to grant the Kyverno background controller’s ServiceAccount additional permissions. To enable Kyverno to generate these other types, see the section on customizing permissions. Kyverno will assist you in these situations by validating and informing you if the background controller does not have the level of permissions required at the time the rule or policy is installed.
Data Source
The source of a generated resource may be defined in the Kyverno policy/rule directly. This is useful in that the full contents of the source can be templated making the resource Kyverno generates highly dynamic and variable depending on the circumstances. To do this, define the generate.data
object to store the contents of the resource to be created. Variable templating is supported for all fields in the data
object. With synchronization enabled, later modification of the contents of that data
object will cause Kyverno to update all downstream (generated) resources with the changes.
The following table shows the behavior of deletion and modification events on components of a generate rule with a data source declaration. “Downstream” refers to the generated resource(s). “Trigger” refers to the resource responsible for triggering the generate rule as defined in a combination of match
and exclude
blocks. Note that when using a data source with sync enabled, deletion of the rule/policy responsible for a resource’s generation will cause immediate deletion of any/all downstream resources.
Action | Sync Effect | NoSync Effect |
---|---|---|
Delete Downstream | Downstream recreated | Downstream deleted |
Delete Rule/Policy | Downstream deleted | Downstream retained |
Delete Trigger | Downstream deleted | None |
Modify Downstream | Downstream reverted | Downstream modified |
Modify Rule/Policy | Downstream synced | Downstream unmodified |
Modify Trigger | Downstream deleted | None |
Data Examples
This policy sets the Zookeeper and Kafka connection strings for all Namespaces based upon a ConfigMap defined within the rule itself.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: zk-kafka-address
5spec:
6 rules:
7 - name: k-kafka-address
8 match:
9 any:
10 - resources:
11 kinds:
12 - Namespace
13 exclude:
14 any:
15 - resources:
16 namespaces:
17 - kube-system
18 - default
19 - kube-public
20 - kyverno
21 generate:
22 synchronize: true
23 apiVersion: v1
24 kind: ConfigMap
25 name: zk-kafka-address
26 # generate the resource in the new namespace
27 namespace: "{{request.object.metadata.name}}"
28 data:
29 kind: ConfigMap
30 metadata:
31 labels:
32 somekey: somevalue
33 data:
34 ZK_ADDRESS: "192.168.10.10:2181,192.168.10.11:2181,192.168.10.12:2181"
35 KAFKA_ADDRESS: "192.168.10.13:9092,192.168.10.14:9092,192.168.10.15:9092"
In this example, new Namespaces will receive a NetworkPolicy that denies all inbound and outbound traffic. Similar to the first example, the generate.data
object is used to define, as an overlay pattern, the spec
for the NetworkPolicy resource.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: default
5spec:
6 rules:
7 - name: deny-all-traffic
8 match:
9 any:
10 - resources:
11 kinds:
12 - Namespace
13 exclude:
14 any:
15 - resources:
16 namespaces:
17 - kube-system
18 - default
19 - kube-public
20 - kyverno
21 generate:
22 kind: NetworkPolicy
23 apiVersion: networking.k8s.io/v1
24 name: deny-all-traffic
25 namespace: "{{request.object.metadata.name}}"
26 data:
27 spec:
28 # select all pods in the namespace
29 podSelector: {}
30 policyTypes:
31 - Ingress
32 - Egress
For other examples of generate rules, see the policy library.
Note
The fieldspec.generateExistingOnPolicyUpdate
is no longer required for “classic” generate rules, is deprecated, and will be removed in an upcoming version.Clone Source
When a generate policy should take the source from a resource which already exists in the cluster, a clone
object is used instead of a data
object. When triggered, the generate policy will clone from the resource name and location defined in the rule to create the new resource. Use of the clone
object implies no modification during the path from source to destination and Kyverno is not able to modify its contents (aside from metadata used for processing and tracking).
Tip
In situations where it may be required to slightly modify the cloned resource, for example to add labels or annotations, an additional mutate rule may be added to the policy so that Kyverno modifies the resource in flight.The following table shows the behavior of deletion and modification events on components of a generate rule with a clone source declaration. “Downstream” refers to the generated resource(s). “Trigger” refers to the resource responsible for triggering the generate rule as defined in a combination of match
and exclude
blocks. “Source” refers to the clone source. Note that when using a clone source with sync enabled, deletion of the rule/policy responsible for a resource’s generation or deletion of the clone source will NOT cause deletion of any downstream resources. This behavior differs when compared to data declarations.
Action | Sync Effect | NoSync Effect |
---|---|---|
Delete Downstream | Downstream recreated | Downstream deleted |
Delete Rule/Policy | Downstream retained | Downstream retained |
Delete Source | Downstream deleted | Downstream retained |
Delete Trigger | Downstream deleted | None |
Modify Downstream | Downstream reverted | Downstream modified |
Modify Rule/Policy | Downstream unmodified | Downstream unmodified |
Modify Source | Downstream synced | Downstream unmodified |
Modify Trigger | Downstream deleted | None |
Clone Examples
In this policy, designed to clone and keep downstream Secrets in-sync with the source, the source of the data is an existing Secret resource named regcred
which is stored in the default
Namespace. Notice how the generate
rule here instead uses the generate.clone
object when the origin data exists within Kubernetes. With synchronization enabled, any modifications to the regcred
source Secret in the default
Namespace will cause all downstream generated resources to be updated.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: sync-secrets
5spec:
6 rules:
7 - name: sync-image-pull-secret
8 match:
9 any:
10 - resources:
11 kinds:
12 - Namespace
13 generate:
14 apiVersion: v1
15 kind: Secret
16 name: regcred
17 namespace: "{{request.object.metadata.name}}"
18 synchronize: true
19 clone:
20 namespace: default
21 name: regcred
For other examples of generate rules, see the policy library.
Cloning Multiple Resources
Kyverno has the ability to clone multiple resources in a single rule definition for use cases where several resources must be cloned from a source Namespace to a destination Namespace. By using the generate.cloneList
object, multiple kinds from the same Namespace may be specified. Use of an optional selector
can scope down the source of the clones to only those having the matching label(s). The below policy clones Secrets and ConfigMaps from the staging
Namespace which carry the label allowedToBeCloned="true"
.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: sync-secret-with-multi-clone
5spec:
6 rules:
7 - name: sync-secret
8 match:
9 any:
10 - resources:
11 kinds:
12 - Namespace
13 exclude:
14 any:
15 - resources:
16 namespaces:
17 - kube-system
18 - default
19 - kube-public
20 - kyverno
21 generate:
22 namespace: "{{request.object.metadata.name}}"
23 synchronize: true
24 cloneList:
25 namespace: staging
26 kinds:
27 - v1/Secret
28 - v1/ConfigMap
29 selector:
30 matchLabels:
31 allowedToBeCloned: "true"
Generating Bindings
In order for Kyverno to generate a new RoleBinding or ClusterRoleBinding resource, its ServiceAccount must first be bound to the same Role or ClusterRole which you’re attempting to generate. If this is not done, Kubernetes blocks the request because it sees a possible privilege escalation attempt from the Kyverno background controller’s ServiceAccount. This particularity is not specific to Kyverno but rather how Kubernetes RBAC is designed to work.
For example, if you wish to write a generate
rule which creates a new RoleBinding resource granting some user the admin
role over a new Namespace, the Kyverno background controller’s ServiceAccount must have a ClusterRoleBinding in place for that same admin
role.
Create a new ClusterRoleBinding for the Kyverno background controller’s ServiceAccount
1apiVersion: rbac.authorization.k8s.io/v1
2kind: ClusterRoleBinding
3metadata:
4 name: kyverno:generate-admin
5roleRef:
6 apiGroup: rbac.authorization.k8s.io
7 kind: ClusterRole
8 name: admin
9subjects:
10- kind: ServiceAccount
11 name: kyverno-background-controller
12 namespace: kyverno
Now, create a generate
rule as you normally would which assigns a test user named steven
to the admin
ClusterRole for a new Namespace. The built-in ClusterRole named admin
in this rule must match the ClusterRole granted to the Kyverno background controller’s ServiceAccount in the previous ClusterRoleBinding.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: steven-rolebinding
5spec:
6 rules:
7 - name: steven-rolebinding
8 match:
9 any:
10 - resources:
11 kinds:
12 - Namespace
13 generate:
14 kind: RoleBinding
15 apiVersion: rbac.authorization.k8s.io/v1
16 name: steven-rolebinding
17 namespace: "{{request.object.metadata.name}}"
18 data:
19 subjects:
20 - kind: User
21 name: steven
22 apiGroup: rbac.authorization.k8s.io
23 roleRef:
24 kind: ClusterRole
25 name: admin
26 apiGroup: rbac.authorization.k8s.io
When a new Namespace is created, Kyverno will generate a new RoleBinding called steven-rolebinding
which grants the user steven
the admin
ClusterRole over said new Namespace.
Linking trigger with downstream
In some cases, a triggering (source) resource and generated (downstream) resource need to share the same life cycle. That is, when the triggering resource is deleted so too should the generated resource. This is valuable because some resources are only needed in the presence of another, for example a Service of type LoadBalancer
necessitating the need for a specific network policy in some CNI plug-ins.
When a generate rule has synchronization enabled (synchronize: true
), deletion of the triggering resource will automatically cause deletion of the downstream (generated) resource. In addition to deletion, if the triggering resource is altered in a way such that it no longer matches the definition in the rule, that too will cause removal of the downstream resource. In cases where synchronization needs to be disabled, if the trigger and downstream are both Namespaced resources and in the same Namespace, the ownerReference technique can be used.
It is possible to set the ownerReferences
field in the generated resource which, when pointed to the trigger, will cause deletion of the trigger to instruct Kubernetes to garbage collect the downstream. With the below example, when the generated ConfigMap specifies the metadata.ownerReferences[]
object and defines the following fields including uid
, which references the triggering Service resource, an owner-dependent relationship is formed. Later, if the Service is deleted, the ConfigMap will be as well. See the Kubernetes documentation for more details including an important caveat around the scoping of these references. Specifically, Namespaced resources cannot be the owners of cluster-scoped resources, and cross-namespace references are also disallowed.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: demo-ownerref
5spec:
6 background: false
7 rules:
8 - name: demo-ownerref-svc-cm
9 match:
10 any:
11 - resources:
12 kinds:
13 - Service
14 generate:
15 kind: ConfigMap
16 apiVersion: v1
17 name: "{{request.object.metadata.name}}-gen-cm"
18 namespace: "{{request.namespace}}"
19 synchronize: false
20 data:
21 metadata:
22 ownerReferences:
23 - apiVersion: v1
24 kind: Service
25 name: "{{request.object.metadata.name}}"
26 uid: "{{request.object.metadata.uid}}"
27 data:
28 foo: bar
Generate for Existing resources
Use of a generate
rule is common when creating net new resources from the point after which the policy was created. For example, a Kyverno generate
policy is created so that all future Namespaces can receive a standard set of Kubernetes resources. However, it is also possible to generate resources based on existing resources. This can be extremely useful especially for Namespaces when deploying Kyverno to an existing cluster where you wish policy to apply retroactively.
Kyverno supports generation for existing resources. Generate existing policies are applied when the policy is created and in the background which creates target resources based on the match statement within the policy. They may also optionally be configured to apply upon updates to the policy itself. By defining the spec.generateExisting
set to true
, a generate rule will take effect for existing resources which have the same match characteristics.
Note that the benefits of using a “generate existing” rule is only the moment the policy is installed. Once the initial generation effects have been produced, the rule functions like a “standard” generate rule from that point forward. Generate existing rules are therefore primarily useful for one-time use cases when retroactive policy should be applied.
Generate Existing Examples
By default, policy will not be applied to existing trigger resources when it is installed. This behavior can be configured via generateExisting
attribute. Only if you set generateExisting
to true
will Kyverno generate the target resource in existing triggers on policy CREATE and UPDATE events.
In this example policy, which triggers based on the resource kind Namespace
a new NetworkPolicy will be generated in all new or existing Namespaces.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: generate-resources
5spec:
6 generateExisting: true
7 rules:
8 - name: generate-existing-networkpolicy
9 match:
10 any:
11 - resources:
12 kinds:
13 - Namespace
14 generate:
15 kind: NetworkPolicy
16 apiVersion: networking.k8s.io/v1
17 name: default-deny
18 namespace: "{{request.object.metadata.name}}"
19 synchronize: true
20 data:
21 metadata:
22 labels:
23 created-by: kyverno
24 spec:
25 podSelector: {}
26 policyTypes:
27 - Ingress
28 - Egress
Similarly, this ClusterPolicy will create a PodDisruptionBudget
resource for existing or new Deployments. Note that use of this policy may require granting of additional permissions as explained above. See the documentation here.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: create-default-pdb
5spec:
6 generateExisting: true
7 rules:
8 - name: create-default-pdb
9 match:
10 any:
11 - resources:
12 kinds:
13 - Deployment
14 exclude:
15 resources:
16 namespaces:
17 - local-path-storage
18 generate:
19 apiVersion: policy/v1
20 kind: PodDisruptionBudget
21 name: "{{request.object.metadata.name}}-default-pdb"
22 namespace: "{{request.object.metadata.namespace}}"
23 synchronize: true
24 data:
25 spec:
26 minAvailable: 1
27 selector:
28 matchLabels:
29 "{{request.object.metadata.labels}}"
Note
The fieldspec.generateExistingOnPolicyUpdate
has been replaced by spec.generateExisting
. The former is no longer required, is deprecated, and will be removed in an upcoming version.How It Works
Kyverno will create an intermediate object called a UpdateRequest
which is used to queue work items for the final resource generation. To get the details and status of a generated resource, check the details of the UpdateRequest
. The following will give the list of UpdateRequests
.
1kubectl get updaterequests -A
A UpdateRequest
status can have one of four values:
Completed
: theUpdateRequest
controller created resources defined in the policyFailed
: theUpdateRequest
controller failed to process the rulesPending
: the request is yet to be processed or the resource has not been createdSkip
: marked when triggering the generate policy by adding a label/annotation to the existing resource, while the selector is not defined in the policy itself.
Kyverno processes generate rules in a combination of the admission controller and the background controller. For further details of the internals of how these work and how high availability and scale are handled, refer to the High Availability page.
Troubleshooting
Troubleshooting of problems with generate rules often comes down to only a few things:
- Policies no longer work after an upgrade when using the scale to zero method. If possible, delete and attempt to reinstall all generate policies after an upgrade to 1.10 so they may be revalidated. Many fields allowed in previous versions of Kyverno are disallowed going forward.
- An intermediary UpdateRequest failed to be applied. Although Kyverno checks that the necessary permissions are present at the time a policy is created, either this isn’t happening or there is some other reason why the UpdateRequest cannot be reconciled. See the How It Works section above.
- The intended trigger did not cause a resource to be generated. Check that Kyverno is not excluding the username or group in its ConfigMap, and check that the resource filter is not discarding those requests. See the Configuring Kyverno guide for details on both.