|
| 1 | +--- |
| 2 | +title: Storage Versions |
| 3 | +content_type: concept |
| 4 | +weight: 110 |
| 5 | +--- |
| 6 | + |
| 7 | +<!-- overview --> |
| 8 | +The Kubernetes API server stores objects, relying on an etcd-compatible backing |
| 9 | +store (often, the backing storage is etcd itself). Each object is serialized |
| 10 | +using a particular version of that API type; for example, the v1 representation |
| 11 | +of a ConfigMap. Kubernetes uses the term _storage version_ to describe how an |
| 12 | +object is stored in your cluster. |
| 13 | + |
| 14 | +The Kubernetes API also relies on automatic conversion; for example, if you have |
| 15 | +a HorizontalPodAutoscaler, then you can interact with that |
| 16 | +HorizontalPodAutoscaler using any mix of the v1 and v2 versions of the |
| 17 | +HorizontalPodAutoscaler API. Kubernetes is responsible for converting each API |
| 18 | +call so that clients do not see what version is actually serialized. |
| 19 | + |
| 20 | +For cluster administrators, object storage version is an important concept to |
| 21 | +understand since it is what links the API representation of the object to the |
| 22 | +actual encoding in the storage backend. This can be important for when the |
| 23 | +underlying binary encodings of the object matter, such as for encryption at |
| 24 | +rest, or API deprecation. |
| 25 | + |
| 26 | +The same API may have multiple storage versions that the API Server can then |
| 27 | +convert to an object schema. A single object that is part of that resource must |
| 28 | +only have one storage version at any time. This means that the API Server is |
| 29 | +aware of the binary encodings of the objects and is able to convert between all |
| 30 | +the stored versions to the API Representation of the object dynamically. |
| 31 | + |
| 32 | +The version of an object is separate from the storage version entirely. For |
| 33 | +example, a `v1alpha1` and `v1beta1` API Object for the same Resource will be |
| 34 | +encoded the same in storage as long as the storage version has not been updated |
| 35 | +between the two objects. |
| 36 | + |
| 37 | +<!-- body --> |
| 38 | + |
| 39 | +## Storage version to resource mapping |
| 40 | + |
| 41 | +Every resource will have 1 active storage version at any point in time, meaning |
| 42 | +that any write to an object will store the object at that storage version. The |
| 43 | +storage version can be updated however, making it so that objects can be stored |
| 44 | +at differing versions. One object will only be stored at one storage version at |
| 45 | +any time. |
| 46 | + |
| 47 | +Reads from the API Server will convert the stored data to the API representation |
| 48 | +of the object. This makes it so that old storage versions can sit indefinitely |
| 49 | +as long as no updates occur to the object. Writes, on the other hand, will |
| 50 | +convert the stored object to the new representation upon update. |
| 51 | + |
| 52 | +## Storage versions for custom resources {#CustomResourceDefinition-storage-version} |
| 53 | + |
| 54 | +[Custom |
| 55 | +resources](/docs/concepts/extend-kubernetes/api-extension/custom-resources/#storage) are |
| 56 | +defined dynamically, and as such differ from built in Kubernetes types with |
| 57 | +their storage version. Builtin objects generally have their storage encoding |
| 58 | +defined separately from their API types, where the stored object acts as a hub |
| 59 | +and the specific version of the resource does not matter apart from being a |
| 60 | +field in the object schema. |
| 61 | + |
| 62 | +However, for custom resources, a certain version of the resource must be set as |
| 63 | +the storage version. The schema defined by that specific version of the custom |
| 64 | +resource will be used as the encoding of the resource in the storage layer. See |
| 65 | +the [advanced CRD |
| 66 | +featureset](/docs/concepts/extend-kubernetes/api-extension/custom-resources/#advanced-features-and-flexibility) |
| 67 | +for more detailed information on the API setup and versioning. |
| 68 | + |
| 69 | +For example see this CustomResourceDefinition for _crontabs_: |
| 70 | + |
| 71 | +```yaml |
| 72 | +apiVersion: apiextensions.k8s.io/v1 |
| 73 | +kind: CustomResourceDefinition |
| 74 | +metadata: |
| 75 | + name: crontabs.example.com |
| 76 | +spec: |
| 77 | + group: example.com |
| 78 | + # list of versions supported by this CustomResourceDefinition |
| 79 | + versions: |
| 80 | + - name: v1beta1 |
| 81 | + # Each version can be enabled/disabled by Served flag. |
| 82 | + served: true |
| 83 | + # One and only one version must be marked as the storage version. |
| 84 | + storage: true |
| 85 | + schema: |
| 86 | + openAPIV3Schema: |
| 87 | + type: object |
| 88 | + properties: |
| 89 | + host: |
| 90 | + type: string |
| 91 | + port: |
| 92 | + type: string |
| 93 | + - name: v1 |
| 94 | + served: true |
| 95 | + storage: false |
| 96 | + schema: |
| 97 | + openAPIV3Schema: |
| 98 | + type: object |
| 99 | + properties: |
| 100 | + host: |
| 101 | + type: string |
| 102 | + port: |
| 103 | + type: string |
| 104 | + time: |
| 105 | + type: string |
| 106 | + conversion: |
| 107 | + strategy: None |
| 108 | + scope: Namespaced |
| 109 | + names: |
| 110 | + plural: crontabs |
| 111 | + singular: crontab |
| 112 | + kind: CronTab |
| 113 | + shortNames: |
| 114 | + - ct |
| 115 | +``` |
| 116 | +
|
| 117 | +The `v1beta1` API definition is used as the storage version, meaning that any |
| 118 | +updates or creation of `crontabs` will be stored with the object schema of the |
| 119 | +`v1beta1` api. In this case it actually would mean that the `v1` API object |
| 120 | +would never be able to store the `time` field since it is not part of the |
| 121 | +storage definition. This schema is used in the storage layer as the binary |
| 122 | +encoding of the object itself. Trying to set two versions as the stored version |
| 123 | +at the same time is considered invalid, since that would mean that two data |
| 124 | +schemes would be considered valid ways to store the objects at the same time. |
| 125 | + |
| 126 | +Upon modification of the version that is used for storage, that version of the |
| 127 | +API will be used to store any new or update CRs. Watching or getting the object |
| 128 | +will have the object be in use but will just convert the object from the old |
| 129 | +storage version and not affect the object. Only updating or creating will have |
| 130 | +an effect and use the newly defined storage version. |
| 131 | + |
| 132 | +## How storage versions are relevant to encryption at rest |
| 133 | + |
| 134 | +There are tools to [encrypt the at rest |
| 135 | +storage](/docs/tasks/administer-cluster/kms-provider/) of a cluster, especially |
| 136 | +for cluster secrets. This adds an additional layer of protection for data |
| 137 | +exfiltration since the actual stored data in the cluster is encrypted. This |
| 138 | +means that the API Server is actually decrypting the data as it retrieves them |
| 139 | +from storage. the data from storage. The APIServer must have the key for that |
| 140 | +storage version in order to decode the object properly. |
| 141 | + |
| 142 | +The storage version in this case is more than just the binary encoding of the |
| 143 | +object. As long as what is stored can be somehow converted into the API object, |
| 144 | +it can be used as a storage version. |
| 145 | + |
| 146 | +## Migrating to a different storage version |
| 147 | + |
| 148 | +Multiple storage versions for a single resource can pose problems for cluster |
| 149 | +administrators. A cluster administrator may not remove old versions of an API |
| 150 | +for CRDs which may be unsupported until they are sure that all objects are no |
| 151 | +longer using the storege version associated with it. With a large number of |
| 152 | +objects and an opaque view into which ones are new and which ones still are |
| 153 | +backed by old storage versions, it makes it difficult to tell when a version can |
| 154 | +be safely removed. If a version is removed prematurely, it can mean being unable |
| 155 | +to read the object entirely. |
| 156 | + |
| 157 | +Another important issue is the use of encryption keys as defined in the section |
| 158 | +above. Since a resource must be actively in use to update the storage version, |
| 159 | +when a key rotation is done, both the old encryption key and the new encryption |
| 160 | +key must remain in use until the administrator is sure all objects have been |
| 161 | +written to at least once. This poses both security risks and usability issues, |
| 162 | +since a key cannot be fully removed from use until then. |
| 163 | + |
| 164 | +See [storage version |
| 165 | +migration](/docs/tasks/manage-kubernetes-objects/storage-version-migration) on |
| 166 | +examples of how to run a migration to ensure that all objects are using a newer |
| 167 | +storage version without manual intervention. |
0 commit comments