碼上歡樂
相關內容    簡體    繁體

drone的pipeline原理與代碼分析

本文轉載自   查看原文   2019-01-22 17:11   1871    pipeline/ jenkins/ 雲原生開發/ drone

最近的一個項目，需要實現一個工作任務流（task pipeline），基於之前CICD的經驗，jenkins pipeline和drone的pipeline進入候選。

drone是基於go的cicd解決方案，github上有1.6萬+star，本文簡單對比了其和jenkins的區別，重點介紹了drone的pipeline原理，並簡單分析了代碼。

jenkins 與 drone

對比項 jenkins drone
pipeline定義 編寫jenkinsfile 編寫流程yml
運行方式 在一個pod里運行 每一步驟起對應的container，通過掛載volume實現數據共享
運行環境 物理機或者容器環境，包括K8S docker容器環境
開發語言 java golang

drone pipeline好處是相對更輕量級，yml定義也相對簡潔清晰，按照功能來划分容器，可以方便的實現task的復用，而jenkins則是完全打包到一個鏡像，會造成單個鏡像體積過大，比如jenkins的單個鏡像超過2G。

drone的pipeline，是基於https://github.com/cncd/pipeline 實現的，這里簡單分析下其原理。

編譯和執行 drone pipeline

要了解一個程序的原理，先從輸入輸出講起。

先安裝：

go get -u github.com/cncd/pipeline
go install github.com/cncd/pipeline/pipec

然后測試

cd $GOPATH/github.com/cncd/pipeline/samples/sample_1
# ll
total 28
drwxr-xr-x  2 root root 4096 Jan 22 11:44 ./
drwxr-xr-x 13 root root 4096 Jan 22 11:02 ../
-rw-r--r--  1 root root  549 Jan 22 11:02 .env
-rw-r--r--  1 root root 6804 Jan 22 16:30 pipeline.json
-rw-r--r--  1 root root  229 Jan 22 11:02 pipeline.yml
-rw-r--r--  1 root root  138 Jan 22 11:02 README.md
  • pipeline.yml 定義文件
  • pipeline.json 編譯后的配置文件
  • .env 環境變量

先來查看pipeline.yml 定義

workspace:
  base: /go
  path: src/github.com/drone/envsubst

clone:
  git:
    image: plugins/git
    depth: 50

pipeline:
  build:
    image: golang:1.7
    commands:
      - go get -t ./...
      - go build
      - go test -v

上面的yml定義了：

  • 工作目錄workspace
  • 初始化工作，git clone倉庫,倉庫地址在.env里定義
  • 然后是定義pipeline，
    • pipeline下面是step數組，這里只有一個build
    • 使用golang:1.7鏡像
    • 構建命令在commands數組里定義

通過pipec compilecompile配置文件：

# pipec compile
Successfully compiled pipeline.yml to pipeline.json

查看編譯后的pipeline.json

{
  "pipeline": [
    {
      "name": "pipeline_clone_0",
      "alias": "git",
      "steps": [
        {
          "name": "pipeline_clone_0",
          "alias": "git",
          "image": "plugins/git:latest",
          "working_dir": "/go/src/github.com/drone/envsubst",
          "environment": {
            "CI": "drone",
            "CI_BUILD_CREATED": "1486119586",
            "CI_BUILD_EVENT": "push",
            "CI_BUILD_NUMBER": "6",
            "CI_BUILD_STARTED": "1486119585",
            "CI_COMMIT_AUTHOR": "bradrydzewski",
            "CI_COMMIT_AUTHOR_NAME": "bradrydzewski",
            "CI_COMMIT_BRANCH": "master",
            "CI_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "CI_COMMIT_REF": "refs/heads/master",
            "CI_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "CI_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "CI_REPO": "drone/envsubst",
            "CI_REPO_LINK": "https://github.com/drone/envsubst",
            "CI_REPO_NAME": "drone/envsubst",
            "CI_REPO_REMOTE": "https://github.com/drone/envsubst.git",
            "CI_SYSTEM": "pipec",
            "CI_SYSTEM_ARCH": "linux/amd64",
            "CI_SYSTEM_LINK": "https://github.com/cncd/pipec",
            "CI_SYSTEM_NAME": "pipec",
            "CI_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "DRONE": "true",
            "DRONE_ARCH": "linux/amd64",
            "DRONE_BRANCH": "master",
            "DRONE_BUILD_CREATED": "1486119586",
            "DRONE_BUILD_EVENT": "push",
            "DRONE_BUILD_LINK": "https://github.com/cncd/pipec/drone/envsubst/6",
            "DRONE_BUILD_NUMBER": "6",
            "DRONE_BUILD_STARTED": "1486119585",
            "DRONE_COMMIT": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_COMMIT_AUTHOR": "bradrydzewski",
            "DRONE_COMMIT_BRANCH": "master",
            "DRONE_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "DRONE_COMMIT_REF": "refs/heads/master",
            "DRONE_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_JOB_STARTED": "1486119585",
            "DRONE_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "DRONE_REPO": "drone/envsubst",
            "DRONE_REPO_LINK": "https://github.com/drone/envsubst",
            "DRONE_REPO_NAME": "envsubst",
            "DRONE_REPO_OWNER": "drone",
            "DRONE_REPO_SCM": "git",
            "DRONE_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "PLUGIN_DEPTH": "50"
          },
          "volumes": [
            "pipeline_default:/go"
          ],
          "networks": [
            {
              "name": "pipeline_default",
              "aliases": [
                "git"
              ]
            }
          ],
          "on_success": true,
          "auth_config": {}
        }
      ]
    },
    {
      "name": "pipeline_stage_0",
      "alias": "build",
      "steps": [
        {
          "name": "pipeline_step_0",
          "alias": "build",
          "image": "golang:1.7",
          "working_dir": "/go/src/github.com/drone/envsubst",
          "environment": {
            "CI": "drone",
            "CI_BUILD_CREATED": "1486119586",
            "CI_BUILD_EVENT": "push",
            "CI_BUILD_NUMBER": "6",
            "CI_BUILD_STARTED": "1486119585",
            "CI_COMMIT_AUTHOR": "bradrydzewski",
            "CI_COMMIT_AUTHOR_NAME": "bradrydzewski",
            "CI_COMMIT_BRANCH": "master",
            "CI_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "CI_COMMIT_REF": "refs/heads/master",
            "CI_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "CI_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "CI_REPO": "drone/envsubst",
            "CI_REPO_LINK": "https://github.com/drone/envsubst",
            "CI_REPO_NAME": "drone/envsubst",
            "CI_REPO_REMOTE": "https://github.com/drone/envsubst.git",
            "CI_SCRIPT": "CmlmIFsgLW4gIiRDSV9ORVRSQ19NQUNISU5FIiBdOyB0aGVuCmNhdCA8PEVPRiA+ICRIT01FLy5uZXRyYwptYWNoaW5lICRDSV9ORVRSQ19NQUNISU5FCmxvZ2luICRDSV9ORVRSQ19VU0VSTkFNRQpwYXNzd29yZCAkQ0lfTkVUUkNfUEFTU1dPUkQKRU9GCmNobW9kIDA2MDAgJEhPTUUvLm5ldHJjCmZpCnVuc2V0IENJX05FVFJDX1VTRVJOQU1FCnVuc2V0IENJX05FVFJDX1BBU1NXT1JECnVuc2V0IENJX1NDUklQVAp1bnNldCBEUk9ORV9ORVRSQ19VU0VSTkFNRQp1bnNldCBEUk9ORV9ORVRSQ19QQVNTV09SRAoKZWNobyArICJnbyBnZXQgLXQgLi8uLi4iCmdvIGdldCAtdCAuLy4uLgoKZWNobyArICJnbyBidWlsZCIKZ28gYnVpbGQKCmVjaG8gKyAiZ28gdGVzdCAtdiIKZ28gdGVzdCAtdgoK",
            "CI_SYSTEM": "pipec",
            "CI_SYSTEM_ARCH": "linux/amd64",
            "CI_SYSTEM_LINK": "https://github.com/cncd/pipec",
            "CI_SYSTEM_NAME": "pipec",
            "CI_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "DRONE": "true",
            "DRONE_ARCH": "linux/amd64",
            "DRONE_BRANCH": "master",
            "DRONE_BUILD_CREATED": "1486119586",
            "DRONE_BUILD_EVENT": "push",
            "DRONE_BUILD_LINK": "https://github.com/cncd/pipec/drone/envsubst/6",
            "DRONE_BUILD_NUMBER": "6",
            "DRONE_BUILD_STARTED": "1486119585",
            "DRONE_COMMIT": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_COMMIT_AUTHOR": "bradrydzewski",
            "DRONE_COMMIT_BRANCH": "master",
            "DRONE_COMMIT_MESSAGE": "added a few more test cases for escaping behavior",
            "DRONE_COMMIT_REF": "refs/heads/master",
            "DRONE_COMMIT_SHA": "d0876d3176965f9552a611cbd56e24a9264355e6",
            "DRONE_JOB_STARTED": "1486119585",
            "DRONE_REMOTE_URL": "https://github.com/drone/envsubst.git",
            "DRONE_REPO": "drone/envsubst",
            "DRONE_REPO_LINK": "https://github.com/drone/envsubst",
            "DRONE_REPO_NAME": "envsubst",
            "DRONE_REPO_OWNER": "drone",
            "DRONE_REPO_SCM": "git",
            "DRONE_WORKSPACE": "/go/src/github.com/drone/envsubst",
            "HOME": "/root",
            "SHELL": "/bin/sh"
          },
          "entrypoint": [
            "/bin/sh",
            "-c"
          ],
          "command": [
            "echo $CI_SCRIPT | base64 -d | /bin/sh -e"
          ],
          "volumes": [
            "pipeline_default:/go"
          ],
          "networks": [
            {
              "name": "pipeline_default",
              "aliases": [
                "build"
              ]
            }
          ],
          "on_success": true,
          "auth_config": {}
        }
      ]
    }
  ],
  "networks": [
    {
      "name": "pipeline_default",
      "driver": "bridge"
    }
  ],
  "volumes": [
    {
      "name": "pipeline_default",
      "driver": "local"
    }
  ],
  "secrets": null
}

簡單分析結構：

  • pipeline 定義了執行的stage，每個stage有一個或者多個step
  • networks、volumes、secrets 分別定義網絡、存儲和secrets
    • 通過network，實現container互通
    • 通過volumes實現數據共享

最后執行，通過pipec exec：

# pipec exec
proc "pipeline_clone_0" started
+ git init
Initialized empty Git repository in /go/src/github.com/drone/envsubst/.git/
+ git remote add origin https://github.com/drone/envsubst.git
+ git fetch --no-tags --depth=50 origin +refs/heads/master:
From https://github.com/drone/envsubst
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
+ git reset --hard -q d0876d3176965f9552a611cbd56e24a9264355e6
+ git submodule update --init --recursive
proc "pipeline_clone_0" exited with status 0
proc "pipeline_step_0" started
+ go get -t ./...
+ go build
+ go test -v
=== RUN   TestExpand
--- PASS: TestExpand (0.00s)
=== RUN   TestFuzz
--- PASS: TestFuzz (0.01s)
=== RUN   Test_len
--- PASS: Test_len (0.00s)
=== RUN   Test_lower
--- PASS: Test_lower (0.00s)
=== RUN   Test_lowerFirst
--- PASS: Test_lowerFirst (0.00s)
=== RUN   Test_upper
--- PASS: Test_upper (0.00s)
=== RUN   Test_upperFirst
--- PASS: Test_upperFirst (0.00s)
=== RUN   Test_default
--- PASS: Test_default (0.00s)
PASS
ok  	github.com/drone/envsubst	0.009s
proc "pipeline_step_0" exited with status 0

pipeline 原理分析

編譯過程

可以形象的理解為 .env+pipeline.yml --> pipeline.json

編譯過程不復雜，主要是解析pipeline.yml為Config:

Config struct {
		Cache     libcompose.Stringorslice
		Platform  string
		Branches  Constraint
		Workspace Workspace
		Clone     Containers
		Pipeline  Containers
		Services  Containers
		Networks  Networks
		Volumes   Volumes
		Labels    libcompose.SliceorMap
	}

然后轉換為json對應的config：

Config struct {
		Stages   []*Stage   `json:"pipeline"` // pipeline stages
		Networks []*Network `json:"networks"` // network definitions
		Volumes  []*Volume  `json:"volumes"`  // volume definitions
		Secrets  []*Secret  `json:"secrets"`  // secret definitions
	}

該部分主要代碼在pipeline/frontend里

執行過程

我們主要關注執行過程，主要代碼在pipeline/backend里。

首先是讀取配置文件為backend.Config

config, err := pipeline.Parse(reader)
	if err != nil {
		return err
	}

然后創建執行環境，目前的代碼僅docker可用，k8s是空代碼。

var engine backend.Engine
	if c.Bool("kubernetes") {
		engine = kubernetes.New(
			c.String("kubernetes-namepsace"),
			c.String("kubernetes-endpoint"),
			c.String("kubernetes-token"),
		)
	} else {
		engine, err = docker.NewEnv()
		if err != nil {
			return err
		}
	}

接着開始執行

	ctx, cancel := context.WithTimeout(context.Background(), c.Duration("timeout"))
	defer cancel()
	ctx = interrupt.WithContext(ctx)

	return pipeline.New(config,
		pipeline.WithContext(ctx),
		pipeline.WithLogger(defaultLogger),
		pipeline.WithTracer(defaultTracer),
		pipeline.WithEngine(engine),
	).Run()

其中pipeline.NEW創建了Runtime對象;

type Runtime struct {
	err     error  // 錯誤信息
	spec    *backend.Config  // 配置信息
	engine  backend.Engine  // docker engine
	started int64 // 開始時間

	ctx    context.Context
	tracer Tracer
	logger Logger
}

其中Engine，操作容器的interface，目前僅docker可用。

// Engine defines a container orchestration backend and is used
// to create and manage container resources.
type Engine interface {
	// Setup the pipeline environment.
	Setup(context.Context, *Config) error
	// Start the pipeline step.
	Exec(context.Context, *Step) error
	// Kill the pipeline step.
	Kill(context.Context, *Step) error
	// Wait for the pipeline step to complete and returns
	// the completion results.
	Wait(context.Context, *Step) (*State, error)
	// Tail the pipeline step logs.
	Tail(context.Context, *Step) (io.ReadCloser, error)
	// Destroy the pipeline environment.
	Destroy(context.Context, *Config) error
}

關注Run：

// Run starts the runtime and waits for it to complete.
func (r *Runtime) Run() error {
    
	// 延遲函數，用於銷毀docker env
	defer func() {
		r.engine.Destroy(r.ctx, r.spec)
	}()
	
	// 初始化docker engine
	r.started = time.Now().Unix()
	if err := r.engine.Setup(r.ctx, r.spec); err != nil {
		return err
	}
   
   // 依次運行stage
	for _, stage := range r.spec.Stages {
		select {
		case <-r.ctx.Done():
			return ErrCancel
	    // 執行
		case err := <-r.execAll(stage.Steps):
			if err != nil {
				r.err = err
			}
		}
	}

	return r.err
}

重點在於使用errgroup.Group通過協程方式運行step：


// 執行所有steps
func (r *Runtime) execAll(procs []*backend.Step) <-chan error {
	var g errgroup.Group
	done := make(chan error)
    // 遍歷執行step
	for _, proc := range procs {
	   // 協程 exec
		proc := proc
		g.Go(func() error {
			return r.exec(proc)
		})
	}

	go func() {
		done <- g.Wait()
		close(done)
	}()
	return done
}

// 執行單個step
func (r *Runtime) exec(proc *backend.Step) error {
	switch {
	case r.err != nil && proc.OnFailure == false:
		return nil
	case r.err == nil && proc.OnSuccess == false:
		return nil
	}
	
	// trace日志
	if r.tracer != nil {
		state := new(State)
		state.Pipeline.Time = r.started
		state.Pipeline.Error = r.err
		state.Pipeline.Step = proc
		state.Process = new(backend.State) // empty
		if err := r.tracer.Trace(state); err == ErrSkip {
			return nil
		} else if err != nil {
			return err
		}
	}
   
   // docker engine執行
	if err := r.engine.Exec(r.ctx, proc); err != nil {
		return err
	}
   
   // 記錄日志信息
	if r.logger != nil {
		rc, err := r.engine.Tail(r.ctx, proc)
		if err != nil {
			return err
		}

		go func() {
			r.logger.Log(proc, multipart.New(rc))
			rc.Close()
		}()
	}

	if proc.Detached {
		return nil
	}

   // 等待docker engine執行完成
	wait, err := r.engine.Wait(r.ctx, proc)
	if err != nil {
		return err
	}

	if r.tracer != nil {
		state := new(State)
		state.Pipeline.Time = r.started
		state.Pipeline.Error = r.err
		state.Pipeline.Step = proc
		state.Process = wait
		if err := r.tracer.Trace(state); err != nil {
			return err
		}
	}
   
	if wait.OOMKilled {
		return &OomError{
			Name: proc.Name,
			Code: wait.ExitCode,
		}
	} else if wait.ExitCode != 0 {
		return &ExitError{
			Name: proc.Name,
			Code: wait.ExitCode,
		}
	}
	return nil
}

作者：Jadepeng
出處：jqpeng的技術記事本--http://www.cnblogs.com/xiaoqi
您的支持是對博主最大的鼓勵，感謝您的認真閱讀。
本文版權歸作者所有，歡迎轉載，但未經作者同意必須保留此段聲明，且在文章頁面明顯位置給出原文連接，否則保留追究法律責任的權利。


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 Redis Pipeline原理分析 Redis Pipeline原理分析 XSS原理及代碼分析 Laravel Pipeline原理及使用 [代碼上線]-pipeline Tomcat源碼分析 （四）----- Pipeline和Valve Jenkins Pipeline 編譯后端 Java 代碼 PCA主成分分析 原理講解 python代碼實現 淺談代碼段加密原理(防止靜態分析) mininet-wifi無線自組網實現原理及代碼分析
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM