內(nèi)容導(dǎo)讀：
背景
上文通過(guò)下面的配置就實(shí)現(xiàn)了驗(yàn)收測(cè)試和壓力測(cè)試，對(duì)此有以下疑問(wèn)：
- metadata定義腳本和類型，說(shuō)明接口能執(zhí)行shell，那它是怎么實(shí)現(xiàn)的？
- type未設(shè)置是怎樣的執(zhí)行邏輯？type有哪些值，各有什么作用？
本文將通過(guò)源碼來(lái)解答以上問(wèn)題
源碼粗讀
本文采用粗讀源碼方式，因?yàn)閣ebhook是一個(gè)功能點(diǎn)，不算Flagger核心流程。
源碼下載
源碼：https://github.com/fluxcd/flagger
git clone https://github.com/fluxcd/flagger.git
定位webhooks接口定義代碼
接口定義信息：Kind為Canary，webhooks位于spec.analysis.webhooks
Canary代碼位置：pkg/apis/flagger/v1beta1/canary.go:44
接著點(diǎn)擊CanarySpec、CanaryAnalysis就找到了webhooks屬性定義。
webhooks對(duì)應(yīng)的結(jié)構(gòu)體為CanaryWebhook：
// CanaryWebhook holds the reference to external checks used for canary analysis
type CanaryWebhook struct {
// Type of this webhook
Type HookType `json:"type"`
// Name of this webhook
Name string `json:"name"`
// URL address of this webhook
URL string `json:"url"`
// false會(huì)觸發(fā)告警，目前支持confirm-rollout、confirm-traffic-increase、confirm-promotion階段
MuteAlert bool `json:"muteAlert"`
// Request timeout for this webhook
Timeout string `json:"timeout,omitempty"`
// Metadata (key-value pairs) for this webhook
// +optional
Metadata *map[string]string `json:"metadata,omitempty"`
// Number of retries for this webhook
// +optional
Retries int `json:"retries,omitempty"`
}
定位CanaryWebhook相關(guān)代碼
只需關(guān)注兩個(gè)函數(shù)：
-
CallEventWebhook：位于pkg/controller/webhook.go:106 -
CallWebhook：位于pkg/controller/webhook.go:87
說(shuō)明：
events.go為系統(tǒng)的創(chuàng)建Webhook（用戶創(chuàng)建的由spec.analysis.webhooks定義），最終是調(diào)用CallEventWebhook
解讀CallEventWebhook和CallWebhook
比較兩個(gè)函數(shù)差異點(diǎn)和共同點(diǎn)如下圖：
最終調(diào)用callWebhook：發(fā)起httpPOST請(qǐng)求的常規(guī)代碼。
小結(jié)：?jiǎn)栴}1結(jié)論
Flagger Webhook會(huì)向目標(biāo)地址發(fā)送方法的http POST請(qǐng)求，發(fā)送數(shù)據(jù)結(jié)構(gòu)如下：
// CanaryWebhookPayload holds the deployment info and metadata sent to webhooks
type CanaryWebhookPayload struct {
// Name of the canary
Name string `json:"name"`
// Namespace of the canary
Namespace string `json:"namespace"`
// Phase of the canary analysis
Phase CanaryPhase `json:"phase"`
// Hash from the TrackedConfigs and LastAppliedSpec of the Canary.
// Can be used to identify a Canary for a specific configuration of the
// deployed resources.
Checksum string `json:"checksum"`
// Metadata (key-value pairs) for this webhook
Metadata map[string]string `json:"metadata,omitempty"`
}
以上就解答了問(wèn)題1：Flagger僅發(fā)送http請(qǐng)求，具體邏輯由目標(biāo)接口實(shí)現(xiàn)。
分析loadtester接口實(shí)現(xiàn)代碼
我們用到了http://flagger-loadtester.test/
loadtester服務(wù)啟動(dòng)入口
找到接口http://flagger-loadtester.test/處理邏輯
最后按metadata.type執(zhí)行任務(wù)。
解讀Webhook type
用到Webhook type的相關(guān)代碼如下
一共有8個(gè)可選值，7個(gè)階段
各值含義如下：
- confirm-rollout：在擴(kuò)展金絲雀部署之前執(zhí)行，可用于手動(dòng)批準(zhǔn)。Canary 將暫停，直到 webhook 返回成功的 HTTP 狀態(tài)代碼。
- pre-rollout：在將流量路由到金絲雀之前執(zhí)行。如果預(yù)部署鉤子失敗，則金絲雀前進(jìn)將暫停，并且如果失敗數(shù)量達(dá)到閾值，金絲雀將回滾。
- rollout：在指標(biāo)檢查之前的每次迭代分析過(guò)程中執(zhí)行。如果 rollout 調(diào)用失敗，則金絲雀進(jìn)度將暫停并最終回滾。
- confirm-traffic-increase：在金絲雀的權(quán)重增加之前執(zhí)行。金絲雀前進(jìn)將暫停，直到該鉤子返回 HTTP 200。
- confirm-promotion：在升級(jí)步驟之前執(zhí)行。金絲雀升級(jí)將暫停，直到掛鉤返回 HTTP 200。升級(jí)暫停時(shí)，F(xiàn)lagger 將繼續(xù)運(yùn)行指標(biāo)檢查和推出掛鉤。
- post-rollout：在金絲雀升級(jí)或回滾后執(zhí)行。如果發(fā)布后 webhook 失敗，則會(huì)記錄錯(cuò)誤。
- rollback：當(dāng)金絲雀部署處于“正在進(jìn)行”或“等待”狀態(tài)時(shí)，會(huì)執(zhí)行回滾鉤子。這提供了在分析期間或等待確認(rèn)時(shí)回滾的能力。如果回滾鉤子返回成功的 HTTP 狀態(tài)代碼，F(xiàn)lagger 將停止分析并將金絲雀發(fā)布標(biāo)記為失敗。
- event：每次 Flagger 發(fā)出 Kubernetes 事件時(shí)都會(huì)執(zhí)行事件掛鉤。配置后，F(xiàn)lagger 在金絲雀部署期間執(zhí)行的每個(gè)操作都將通過(guò) HTTP POST 請(qǐng)求以 JSON 形式發(fā)送。
說(shuō)明：前面7個(gè)值是Canary對(duì)應(yīng)的七個(gè)階段，event表示Canary創(chuàng)建了k8s事件就會(huì)觸發(fā)。
Webhook type定義如下：
confirm-rollout 源碼解讀
- 類型為
ConfirmRolloutHook - 反向引用找到使用
ConfirmRolloutHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runConfirmRolloutHooks(canary *flaggerv1.Canary, canaryController canary.Controller) bool {
for _, webhook := range canary.GetAnalysis().Webhooks {
if webhook.Type == flaggerv1.ConfirmRolloutHook {
err := CallWebhook(*canary, canary.Status.Phase, webhook)
if err != nil {
return false
}
}
}
return true
}
- 反向引用找到使用
runConfirmRolloutHooks代碼（僅保留關(guān)鍵代碼）
func (c *Controller) checkCanaryStatus(canary *flaggerv1.Canary, canaryController canary.Controller, scalerReconciler canary.ScalerReconciler, shouldAdvance bool) bool {
c.recorder.SetStatus(canary, canary.Status.Phase)
if canary.Status.Phase == flaggerv1.CanaryPhaseProgressing ||
canary.Status.Phase == flaggerv1.CanaryPhaseWaitingPromotion ||
canary.Status.Phase == flaggerv1.CanaryPhasePromoting ||
canary.Status.Phase == flaggerv1.CanaryPhaseFinalising {
return true
}
var err error
canary, err = c.flaggerClient.FlaggerV1beta1().Canaries(canary.Namespace).Get(context.TODO(), canary.Name, metav1.GetOptions{})
if err != nil {
// 按ns和name獲取，err表示Canary不存在
return false
}
if shouldAdvance {
// 調(diào)用runConfirmRolloutHooks
if isApproved := c.runConfirmRolloutHooks(canary, canaryController); !isApproved {
// 接口返回false，即未通過(guò)審批
return false
}
canaryPhaseProgressing := canary.DeepCopy()
// 審批通過(guò)后，canary狀態(tài)為CanaryPhaseProgressing（將由上面的邏輯直接返回true）
canaryPhaseProgressing.Status.Phase = flaggerv1.CanaryPhaseProgressing
if err := canaryController.SyncStatus(canary, flaggerv1.CanaryStatus{Phase: flaggerv1.CanaryPhaseProgressing}); err != nil {
return false
}
return false
}
return false
}
- 值為true的條件：shouldAdvance為true的條件：canary狀態(tài)為（Progressing、Waiting、WaitingPromotion、Promoting、Finalising）、worklod有變化、worklod依賴資源（ConfigMap+Secret）有變化。查看
shouldAdvance代碼：
func (c *Controller) shouldAdvance(canary *flaggerv1.Canary, canaryController canary.Controller) (bool, error) {
if canary.Status.Phase == flaggerv1.CanaryPhaseProgressing ||
canary.Status.Phase == flaggerv1.CanaryPhaseWaiting ||
canary.Status.Phase == flaggerv1.CanaryPhaseWaitingPromotion ||
canary.Status.Phase == flaggerv1.CanaryPhasePromoting ||
canary.Status.Phase == flaggerv1.CanaryPhaseFinalising {
return true, nil
}
// Make sure to sync lastAppliedSpec even if the canary is in a failed state.
if canary.Status.Phase == flaggerv1.CanaryPhaseFailed {
return false, err
}
}
newTarget, err := canaryController.HasTargetChanged(canary)
if err != nil {
return false, err
}
if newTarget {
return newTarget, nil
}
newCfg, err := canaryController.HaveDependenciesChanged(canary)
if err != nil {
return false, err
}
return newCfg, nil
}
pre-rollout 源碼解讀
- 類型為
PreRolloutHook - 反向引用找到使用
PreRolloutHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runPreRolloutHooks(canary *flaggerv1.Canary) bool {
for _, webhook := range canary.GetAnalysis().Webhooks {
if webhook.Type == flaggerv1.PreRolloutHook {
err := CallWebhook(*canary, flaggerv1.CanaryPhaseProgressing, webhook)
if err != nil {
return false
} else {
c.recordEventInfof(canary, "Pre-rollout check %s passed", webhook.Name)
}
}
}
return true
}
- 反向引用找到使用
runPreRolloutHooks代碼（僅保留關(guān)鍵代碼）
// 灰度流量為0 且 canary的迭代數(shù)為0（0表示對(duì)業(yè)務(wù)有效，像AB測(cè)試和藍(lán)綠測(cè)試迭代數(shù)非0） 且 非影子/鏡像流量
if canaryWeight == 0 && cd.Status.Iterations == 0 &&
!(cd.GetAnalysis().Mirror && mirrored) {
c.recordEventInfof(cd, "Starting canary analysis for %s.%s", cd.Spec.TargetRef.Name, cd.Namespace)
// run pre-rollout web hooks
if ok := c.runPreRolloutHooks(cd); !ok {
if err := canaryController.SetStatusFailedChecks(cd, cd.Status.FailedChecks+1); err != nil {
c.recordEventWarningf(cd, "%v", err)
}
return
}
} else {
// rollout執(zhí)行代碼
if ok := c.runAnalysis(cd); !ok {
if err := canaryController.SetStatusFailedChecks(cd, cd.Status.FailedChecks+1); err != nil {
c.recordEventWarningf(cd, "%v", err)
}
return
}
}
rollout（默認(rèn)值） 源碼解讀
- 類型為
RolloutHook - 反向引用找到使用
RolloutHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runAnalysis(canary *flaggerv1.Canary) bool {
// run external checks
for _, webhook := range canary.GetAnalysis().Webhooks {
// type為空也走此邏輯
if webhook.Type == "" || webhook.Type == flaggerv1.RolloutHook {
err := CallWebhook(*canary, flaggerv1.CanaryPhaseProgressing, webhook)
if err != nil {
return false
}
}
}
return true
}
- 執(zhí)行
runAnalysis代碼位于pre-rollout執(zhí)行代碼處，執(zhí)行條件為pre-rollout的相反條件
// 灰度流量為0 且 canary的迭代數(shù)為0（0表示對(duì)業(yè)務(wù)有效，像AB測(cè)試和藍(lán)綠測(cè)試迭代數(shù)非0） 且 非影子/鏡像流量
if canaryWeight == 0 && cd.Status.Iterations == 0 &&
!(cd.GetAnalysis().Mirror && mirrored) {
c.recordEventInfof(cd, "Starting canary analysis for %s.%s", cd.Spec.TargetRef.Name, cd.Namespace)
// run pre-rollout web hooks
if ok := c.runPreRolloutHooks(cd); !ok {
if err := canaryController.SetStatusFailedChecks(cd, cd.Status.FailedChecks+1); err != nil {
c.recordEventWarningf(cd, "%v", err)
}
return
}
} else {
if ok := c.runAnalysis(cd); !ok {
if err := canaryController.SetStatusFailedChecks(cd, cd.Status.FailedChecks+1); err != nil {
c.recordEventWarningf(cd, "%v", err)
}
return
}
}
confirm-traffic-increase 源碼解讀
- 類型為
ConfirmTrafficIncreaseHook - 反向引用找到使用
ConfirmTrafficIncreaseHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runConfirmTrafficIncreaseHooks(canary *flaggerv1.Canary) bool {
for _, webhook := range canary.GetAnalysis().Webhooks {
if webhook.Type == flaggerv1.ConfirmTrafficIncreaseHook {
err := CallWebhook(*canary, flaggerv1.CanaryPhaseProgressing, webhook)
if err != nil {
return false
}
}
}
return true
}
- 反向引用找到使用
runConfirmTrafficIncreaseHooks代碼（僅保留關(guān)鍵代碼）
// 計(jì)算下次要增加流量
if c.nextStepWeight(cd, canaryWeight) > 0 {
if !mirrored &&
(cd.Status.Phase != flaggerv1.CanaryPhasePromoting &&
cd.Status.Phase != flaggerv1.CanaryPhaseWaitingPromotion &&
cd.Status.Phase != flaggerv1.CanaryPhaseFinalising) {
if promote := c.runConfirmTrafficIncreaseHooks(cd); !promote {
return
}
}
c.runCanary(cd, canaryController, meshRouter, mirrored, canaryWeight, primaryWeight, maxWeight)
}
confirm-promotion 源碼解讀
- 類型為
ConfirmPromotionHook - 反向引用找到使用
ConfirmPromotionHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runConfirmPromotionHooks(canary *flaggerv1.Canary, canaryController canary.Controller) bool {
for _, webhook := range canary.GetAnalysis().Webhooks {
if webhook.Type == flaggerv1.ConfirmPromotionHook {
err := CallWebhook(*canary, flaggerv1.CanaryPhaseProgressing, webhook)
if err != nil {
return false
} else {
c.recordEventInfof(canary, "Confirm-promotion check %s passed", webhook.Name)
}
}
}
return true
}
- 反向引用找到使用
runConfirmPromotionHooks代碼（僅保留關(guān)鍵代碼）
if c.nextStepWeight(cd, canaryWeight) > 0 {
// run hook only if traffic is not mirrored
if !mirrored &&
(cd.Status.Phase != flaggerv1.CanaryPhasePromoting &&
cd.Status.Phase != flaggerv1.CanaryPhaseWaitingPromotion &&
cd.Status.Phase != flaggerv1.CanaryPhaseFinalising) {
if promote := c.runConfirmTrafficIncreaseHooks(cd); !promote {
return
}
}
c.runCanary(cd, canaryController, meshRouter, mirrored, canaryWeight, primaryWeight, maxWeight)
}
func (c *Controller) runCanary(canary *flaggerv1.Canary, canaryController canary.Controller,
meshRouter router.Interface, mirrored bool, canaryWeight int, primaryWeight int, maxWeight int) {
// 灰度流量定義的最大灰度流量(analysis.maxWeight)：下一步將把流量全部切換到primary
if canaryWeight >= maxWeight {
// check promotion gate
if promote := c.runConfirmPromotionHooks(canary, canaryController); !promote {
return
}
}
}
post-rollout 源碼解讀
- 類型為
PostRolloutHook - 反向引用找到使用
PostRolloutHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runPostRolloutHooks(canary *flaggerv1.Canary, phase flaggerv1.CanaryPhase) bool {
for _, webhook := range canary.GetAnalysis().Webhooks {
if webhook.Type == flaggerv1.PostRolloutHook {
err := CallWebhook(*canary, phase, webhook)
if err != nil {
c.recordEventWarningf(canary, "Post-rollout hook %s failed %v", webhook.Name, err)
return false
} else {
c.recordEventInfof(canary, "Post-rollout check %s passed", webhook.Name)
}
}
}
return true
}
- 反向引用找到使用
runPostRolloutHooks代碼（僅保留關(guān)鍵代碼）
// scale canary to zero if promotion has finished
if cd.Status.Phase == flaggerv1.CanaryPhaseFinalising {
if scalerReconciler != nil {
if err := scalerReconciler.PauseTargetScaler(cd); err != nil {
c.recordEventWarningf(cd, "%v", err)
return
}
}
if err := canaryController.ScaleToZero(cd); err != nil {
c.recordEventWarningf(cd, "%v", err)
return
}
// set status to succeeded
if err := canaryController.SetStatusPhase(cd, flaggerv1.CanaryPhaseSucceeded); err != nil {
c.recordEventWarningf(cd, "%v", err)
return
}
c.recorder.SetStatus(cd, flaggerv1.CanaryPhaseSucceeded)
// Canary狀態(tài)為成功觸發(fā)
c.runPostRolloutHooks(cd, flaggerv1.CanaryPhaseSucceeded)
c.recordEventInfof(cd, "Promotion completed! Scaling down %s.%s", cd.Spec.TargetRef.Name, cd.Namespace)
c.alert(cd, "Canary analysis completed successfully, promotion finished.",
false, flaggerv1.SeverityInfo)
return
}
rollback 源碼解讀
- 類型為
RollbackHook - 反向引用找到使用
RollbackHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) runRollbackHooks(canary *flaggerv1.Canary, phase flaggerv1.CanaryPhase) bool {
for _, webhook := range canary.GetAnalysis().Webhooks {
if webhook.Type == flaggerv1.RollbackHook {
err := CallWebhook(*canary, phase, webhook)
if err != nil {
c.recordEventInfof(canary, "Rollback hook %s not signaling a rollback", webhook.Name)
} else {
c.recordEventWarningf(canary, "Rollback check %s passed", webhook.Name)
return true
}
}
}
return false
}
- 反向引用找到使用
runRollbackHooks代碼（僅保留關(guān)鍵代碼）
if cd.Status.Phase == flaggerv1.CanaryPhaseProgressing ||
cd.Status.Phase == flaggerv1.CanaryPhaseWaiting ||
cd.Status.Phase == flaggerv1.CanaryPhaseWaitingPromotion {
if ok := c.runRollbackHooks(cd, cd.Status.Phase); ok {
c.recordEventWarningf(cd, "Rolling back %s.%s manual webhook invoked", cd.Name, cd.Namespace)
c.alert(cd, "Rolling back manual webhook invoked", false, flaggerv1.SeverityWarn)
// 真正回滾邏輯
c.rollback(cd, canaryController, meshRouter, scalerReconciler)
return
}
}
event 源碼解讀
- 類型為
EventHook - 反向引用找到使用
EventHook代碼（僅保留關(guān)鍵代碼）
func (c *Controller) sendEventToWebhook(r *flaggerv1.Canary, eventType, template string, args []interface{}) {
webhookOverride := false
for _, canaryWebhook := range r.GetAnalysis().Webhooks {
if canaryWebhook.Type == flaggerv1.EventHook {
webhookOverride = true
err := CallEventWebhook(r, canaryWebhook, fmt.Sprintf(template, args...), eventType)
if err != nil {
c.logger.With("canary", fmt.Sprintf("%s.%s", r.Name, r.Namespace)).Errorf("error sending event to webhook: %s", err)
}
}
}
// c.eventWebhook來(lái)源于環(huán)境變量"EVENT_WEBHOOK_URL"
if c.eventWebhook != "" && !webhookOverride {
hook := flaggerv1.CanaryWebhook{
Name: "events",
URL: c.eventWebhook,
}
err := CallEventWebhook(r, hook, fmt.Sprintf(template, args...), eventType)
if err != nil {
c.logger.With("canary", fmt.Sprintf("%s.%s", r.Name, r.Namespace)).Errorf("error sending event to webhook: %s", err)
}
}
}
- 反向引用找到使用
sendEventToWebhook代碼（僅保留關(guān)鍵代碼）：Canary產(chǎn)生的所有k8s Event都會(huì)執(zhí)行sendEventToWebhook
func (c *Controller) recordEventInfof(r *flaggerv1.Canary, template string, args ...interface{}) {
c.logger.With("canary", fmt.Sprintf("%s.%s", r.Name, r.Namespace)).Infof(template, args...)
// 記錄event到k8s
c.eventRecorder.Event(r, corev1.EventTypeNormal, "Synced", fmt.Sprintf(template, args...))
c.sendEventToWebhook(r, corev1.EventTypeNormal, template, args)
}
func (c *Controller) recordEventErrorf(r *flaggerv1.Canary, template string, args ...interface{}) {
c.logger.With("canary", fmt.Sprintf("%s.%s", r.Name, r.Namespace)).Errorf(template, args...)
c.eventRecorder.Event(r, corev1.EventTypeWarning, "Synced", fmt.Sprintf(template, args...))
c.sendEventToWebhook(r, corev1.EventTypeWarning, template, args)
}
func (c *Controller) recordEventWarningf(r *flaggerv1.Canary, template string, args ...interface{}) {
c.logger.With("canary", fmt.Sprintf("%s.%s", r.Name, r.Namespace)).Infof(template, args...)
c.eventRecorder.Event(r, corev1.EventTypeWarning, "Synced", fmt.Sprintf(template, args...))
c.sendEventToWebhook(r, corev1.EventTypeWarning, template, args)
}
附錄
Goland本地啟動(dòng)服務(wù)
參考文檔：https://docs.flagger.app/dev/dev-guide#manual-testing
增加啟動(dòng)參數(shù)：-kubeconfig=/Users/admin/.kube/config -log-level=info -mesh-provider=istio -metrics-server=http://prom.istio.cn:9090
啟動(dòng)配置
執(zhí)行日志：
調(diào)試效果：
結(jié)語(yǔ)
本文以Webhook疑問(wèn)為出發(fā)點(diǎn)，通過(guò)粗讀源碼全面解讀了Webhook相關(guān)知識(shí)，同時(shí)附上了Goland本地調(diào)試方法。
文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-847702.html
請(qǐng)用微信掃碼關(guān)注下?? ，持續(xù)更新云原生DevOps最佳實(shí)踐。文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-847702.html
到了這里，關(guān)于自動(dòng)化金絲雀部署：Flagger全面解讀webhook（含源碼）的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
