patch容易出現字段沖突
近期在使用client-go對某個k8s原生資源進行patch操作時，出現了字段沖突導致的patch失敗問題，具體是patch嘗試修改資源的某個字段的類型，比如將readiness probe的類型從tcp修改為httpGet，patch時希望修改probe類型但被認為是一種追加動作，導致apiserver端驗證錯誤不允許為一種類型的probe指定多個handler：
當然，處理方式可以在patch數據中為要刪除的readiness tcp probe加一個刪除標記，這樣patch請求到達apiserver的時候就可以被正確處理達到替換的目的：
"spec": {
"containers":[
{
"name":"xxx",
"readinessProbe":{
"exec":nil, // delete
"httpGet":{ // add
}
}
}
}]
}
給我帶來的疑惑是使用kubectl apply時為什么就沒這個問題呢？
kubectl apply使用3-way patch
kubectl apply命令會在要apply的資源對象上添加last-apply-configuration，表示最近一次通過kubectl apply更新的資源清單，如果某個資源一直都是通過apply來更新，那么ast-apply-configuration與對象一致
對於k8s原生的資源如deployment、pod等，kubectl apply時通過3-way patch生成strategicpatch類型的patch數據，其中：
注意如果是crd資源，用的應該是jsonmergepatch.CreateThreeWayJSONMergePatch
# staging/src/k8s.io/kubectl/pkg/cmd/apply/apply.go
// 根據original、modified、current三方數據生成最終patch請求的數據
if openapiPatch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, lookupPatchMeta, p.Overwrite); err != nil {
fmt.Fprintf(errOut, "warning: error calculating patch from openapi spec: %v\n", err)
} else {
patchType = types.StrategicMergePatchType
patch = openapiPatch
}
current是集群中當前的資源數據：
// info.Get通過RestClient請求api獲取對象
if err := info.Get(); err != nil {
// err是not found error，說明是首次創建
if !errors.IsNotFound(err) {
return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:\n%s\nfrom server for:", info.String()), info.Source, err)
}
// Create the resource if it doesn't exist
// First, update the annotation used by kubectl apply
// 如果集群中當前的對象沒有last-apply-configuration注解，那么先用這個對象本身生存anno並更新到集群
if err := util.CreateApplyAnnotation(info.Object, unstructured.UnstructuredJSONScheme); err != nil {
return cmdutil.AddSourceToErr("creating", info.Source, err)
}
}
modified是此次需要apply放入數據（比如-f指定的文件內容）:
// Get the modified configuration of the object. Embed the result
// as an annotation in the modified configuration, so that it will appear
// in the patch sent to the server.
// 可以看看這個方法具體的實現，會把自身encode之后放到自己的last-apply-configuration之中（覆蓋可能已經存在的這個anno）
modified, err := util.GetModifiedConfiguration(info.Object, true, unstructured.UnstructuredJSONScheme)
original是集群中當前資源的LastAppliedConfigAnnotation數據：
// Retrieve the original configuration of the object from the annotation.
original, err := util.GetOriginalConfiguration(obj)
// GetOriginalConfiguration retrieves the original configuration of the object
// from the annotation, or nil if no annotation was found.
func GetOriginalConfiguration(obj runtime.Object) ([]byte, error) {
annots, err := metadataAccessor.Annotations(obj)
if err != nil {
return nil, err
}
if annots == nil {
return nil, nil
}
// 直接取的annotation
original, ok := annots[v1.LastAppliedConfigAnnotation]
if !ok {
return nil, nil
}
return []byte(original), nil
}
有了這三方數據之后，strategicpatch.CreateThreeWayMergePatch方法就會產生最終要patch的數據
- 根據集群中當前資源數據currentMap和此次要修改的數據Modified計算出那些字段是新增的，計算增量時忽略哪些要被刪除的字段
- 因為集群中的對象可能被修改過（人為或者某些組件）且這些修改不會更新last-apply-configuration anno，所以這里apply計算哪些字段是新增的時，就需要以集群當前狀態和此次的apply數據modified來決定
- 根據集群中當前資源的original（last-apply-configuration anno）數據和此次要修改的數據Modified計算出哪些字段是要刪除的（設置為"-"），忽略增加的字段
- kubectl apply認為沖突的字段應該通過相鄰的兩次apply操作來計算
// CreateThreeWayMergePatch reconciles a modified configuration with an original configuration,
// while preserving any changes or deletions made to the original configuration in the interim,
// and not overridden by the current configuration. All three documents must be passed to the
// method as json encoded content. It will return a strategic merge patch, or an error if any
// of the documents is invalid, or if there are any preconditions that fail against the modified
// configuration, or, if overwrite is false and there are conflicts between the modified and current
// configurations. Conflicts are defined as keys changed differently from original to modified
// than from original to current. In other words, a conflict occurs if modified changes any key
// in a way that is different from how it is changed in current (e.g., deleting it, changing its
// value). We also propagate values fields that do not exist in original but are explicitly
// defined in modified.
func CreateThreeWayMergePatch(original, modified, current []byte, schema LookupPatchMeta, overwrite bool, fns ...mergepatch.PreconditionFunc) ([]byte, error) {
// 三方數據都反序列化為unstracture通用結構
originalMap := map[string]interface{}{}
if len(original) > 0 {
if err := json.Unmarshal(original, &originalMap); err != nil {
return nil, mergepatch.ErrBadJSONDoc
}
}
modifiedMap := map[string]interface{}{}
if len(modified) > 0 {
if err := json.Unmarshal(modified, &modifiedMap); err != nil {
return nil, mergepatch.ErrBadJSONDoc
}
}
currentMap := map[string]interface{}{}
if len(current) > 0 {
if err := json.Unmarshal(current, ¤tMap); err != nil {
return nil, mergepatch.ErrBadJSONDoc
}
}
// The patch is the difference from current to modified without deletions, plus deletions
// from original to modified. To find it, we compute deletions, which are the deletions from
// original to modified, and delta, which is the difference from current to modified without
// deletions, and then apply delta to deletions as a patch, which should be strictly additive.
deltaMapDiffOptions := DiffOptions{
IgnoreDeletions: true,
SetElementOrder: true,
}
// DiffOptions中IgnoreDeletions設置為true，根據集群中當前資源數據currentMap和此次要修改的數據計算出那些字段是新增的，
// 計算增量時先忽略那些要被刪除的
deltaMap, err := diffMaps(currentMap, modifiedMap, schema, deltaMapDiffOptions)
if err != nil {
return nil, err
}
deletionsMapDiffOptions := DiffOptions{
SetElementOrder: true,
IgnoreChangesAndAdditions: true,
}
// DiffOptions中IgnoreDeletions默認值為false，根據集群中當前資源的last-apply數據和此次要修改的數據計算出那些字段是要
// 刪除的，這里忽略增量的數據
// 當有字段沖突時，這里會把original即上一次apply中的該字段標記為刪除，deletionsMap中的值為nil
deletionsMap, err := diffMaps(originalMap, modifiedMap, schema, deletionsMapDiffOptions)
if err != nil {
return nil, err
}
mergeOptions := MergeOptions{}
// 將deletionsMap和deltaMap做一次合並，生成最終要patch的數據
patchMap, err := mergeMap(deletionsMap, deltaMap, schema, mergeOptions)
if err != nil {
return nil, err
}
return json.Marshal(patchMap)
}
func diffMaps(original, modified map[string]interface{}, schema LookupPatchMeta, diffOptions DiffOptions) (map[string]interface{}, error) {
// 記錄結果
patch := map[string]interface{}{}
// Compare each value in the modified map against the value in the original map
// 遍歷originalMap這個unstrctureMap的每一個key
for key, modifiedValue := range modified {
originalValue, ok := original[key]
if !ok {
// Key was added, so add to patch
// 如果value不存在於originalMap，但是存在於modifiedMap，並且IgnoreChangesAndAdditions為false
if !diffOptions.IgnoreChangesAndAdditions {
// 結果添加modifiedMap中的這個kv
patch[key] = modifiedValue
}
continue
}
// original和modified中都有value，就看value是不是同一種類型
if reflect.TypeOf(originalValue) != reflect.TypeOf(modifiedValue) {
// Types have changed, so add to patch
// 類型一樣並且IgnoreChangesAndAdditions為false，那么結果添加modifiedMap中的這個kv
if !diffOptions.IgnoreChangesAndAdditions {
patch[key] = modifiedValue
}
continue
}
// Types are the same, so compare values
// original和modified中都有value，就看value是同一種類型
// 那么根據具體的類型，調用handleMapDiff或handleSliceDiff處理
switch originalValueTyped := originalValue.(type) {
// value的類型是一個復合結構
case map[string]interface{}:
modifiedValueTyped := modifiedValue.(map[string]interface{})
err = handleMapDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
// value的類型是一個slice切片結構
case []interface{}:
modifiedValueTyped := modifiedValue.([]interface{})
err = handleSliceDiff(key, originalValueTyped, modifiedValueTyped, patch, schema, diffOptions)
default:
// 既不是map也不是slice，那么直接用modifiedValue替換originalValue
replacePatchFieldIfNotEqual(key, originalValue, modifiedValue, patch, diffOptions)
}
if err != nil {
return nil, err
}
}
// 如果ignoreDeletions為false，那么遍歷originalMap的每一個key，如果modefiedMap中不存在value，那么在最終的結果中
// 標記該key為需要刪除
updatePatchIfMissing(original, modified, patch, diffOptions)
return patch, nil
}
從上面的分析可以看出，kubect在apply時通過3-way patch的方式，可以計算出哪些字段是要新增的，哪些字段是要被刪除的，以避免沖突的出現，如果original中的數據（last-apply）與modifed不能正確計算出要被刪除的字段，也會出現apply失敗的問題，比如資源通過kubectl create創建則沒有last-apply-configuration注解，這個時候如果修改字段的值類型，即使通過kubectl apply也會失敗。