Software developer notes

Not every company uses CI, especially when the company is small. In this post I would like to share my humble experience of automation and how it has improved processes in my work.

There are plenty of tasks which may block me and my сolleagues. To better illustrate the problem here are typical scenarios:

1. Suppose you have some idea how to fix the bug. Testing this guess may imply rebuilding your project with additional settings, patches and testing. This process takes about 10 minutes on my workstation and our code base. Sometimes it's too long especially when you have other priority tasks.
2. Not every developer can have opportunity or tuned environment to build the final product. For example, some tools for protecting or signing of binaries are tightly coupled with hardware id for security reasons (replacement hardware will require additional time).
3. Sometimes manager and qa engineer can ask the product with latest changes. If it combines previous issues then you will spend more time.
4. In the case of rebuilding "the world" it is helpful to have automation server, because it's usually time consuming operation. It gets complicated if you are working with a cross platform product. For me it took about a hour.
5. In the presence of linters, static analyzers, unit and integration tests there is need to periodically run all of them in order to check product's health. Without automation I strongly guarantee such checks will be rare. It is handy and boring work to run all checks on any code changes, nobody has so much time. I witnessed unit tests being broken for a couple of months!

To solve these problems I started to use Jenkins, launched on separate physical machine (to exclude any impact to other hosts). It's not the only automation server, you can use paid TeamCity or conditionally free github actions. But idea is the same everywhere.

Below are tasks performed by Jenkins (they are depicted very roughly so you can easily adapt them under other ci tools):

1. Release job performs building, testing and other preparations to generate final product (but it is deployed manually). We trigger this task when release branch contains uptodate features and business is ready to "Release".
2. Night job is like previous one, but its purpose is to produce night builds with full checks on current development branch. It's primarly used by qa engineers (or developers to view all statistics) to ensure current state of product.
3. Request job is core one with set of parameters to indicate whether we should perform all tests, whether we should protect and sign binaries etc. It's usually triggered by previous jobs but it's also launched periodically on any code changes or by user manually. Ideally this job must be separated to explicitly have common template. But this limitation caused by Jenkins itself. You will see later I used declarative pipelines and there is no easy way to reuse them. On other system this limitation can be absent.
4. Notify job is the simplest one. It encapsulates "direct" notification for developers. For example it can be telegram message with brief descrpition of build's status like count of passed tests, commit id etc.

From user's point of view automation looks simpler:

1. Whenever code changes Jenkins runs lightweight Request job with needed checks and analysis to detect any regression as fast as possible
2. Whenever someone wants to see current state of the product with more detailed information, it's enough to get night build
3. Whenever business is ready Release job will be triggered manualy to perform all needed step to create to the product (for simplicity we perform deploy manually, but it can be automated too).

In overall it greatly reduses the load of software developers and at the same time it adds some kind of parallelism. To better illustrate the solution, I depicted samples of jobs below (Jenkins is highly customizable system, and in order to run samples some plugins should be installed).

/*	
	Night job which triggers RequestJob at midnight.
	It also saves full log and artifacts.
 */
pipeline {
	agent any
	triggers {
		/*	Running job at midnight */
		cron('0 0 * * *')
	}
	options {
		/*	Artifacts time to live */
		buildDiscarder(logRotator(artifactDaysToKeepStr: '10', 
			artifactNumToKeepStr: '', 
			daysToKeepStr: '', 
			numToKeepStr: '50'))
	}
	stages {
		stage('DownstreamJobRun') {
			steps {
				script {
					/*	Triggering request job  */
					triggeredBuild = build job: 'RequestJob', 
						parameters: 
							[string(name: 'USE_TESTS', value: 'All'), 
							string(name: 'NOTES', value: 'Night job')]
							
					/*  Copying full log of request job  */
					println triggeredBuild.getRawBuild().getLog()
					
					/*  Copying artifacts  */
					copyArtifacts(projectName: 'RequestJob', 
						selector: specific("${triggeredBuild.getNumber()}"), 
						target : "JobArtifacts")
						
					/*	Archiving artifacts  */
					archiveArtifacts artifacts: 'JobArtifacts/**/*'
				}
			}
		}
	}
	post {
		cleanup {
			/*  Triggering notification job */
			build job: 'NotificationJob',   parameters: 
				[string(name: 'TITLE', value: "$JOB_NAME"),  
				string(name: 'STATUS', value: "${currentBuild.currentResult}"),
				string(name: 'LINK', value: "$BUILD_URL")    ], wait: false
				
			/*	Workspace cleaning  */
			cleanWs(cleanWhenNotBuilt: true,
				cleanWhenFailure: true,
				cleanWhenAborted: true,
				cleanWhenSuccess: true,
				cleanWhenUnstable: true,
				deleteDirs: true,
				disableDeferredWipeout: true,
				notFailBuild: true,
				patterns: [[pattern: '**/*', type: 'INCLUDE']])
		}
	}
}

/*
	Example of core template of job with many parameters in 
	order to perform complex tasks. It automatically notifies when 
	it completes.
*/
pipeline {
	agent any
	triggers { 
		pollSCM('H/59 * * * *') 
	}
	options {
		/*	Artifacts time to live */
		buildDiscarder(logRotator(artifactDaysToKeepStr: '10', 
			artifactNumToKeepStr: '', 
			daysToKeepStr: '', 
			numToKeepStr: '50'))
	}
	parameters {
		choice(name: 'USE_TESTS', 
			choices: ['Fast', 'All', 'None'], 
			description: 'Controls the level of binaries testing')
		choice(name: 'PROTECTION_CHOICE', 
			choices: ['NoProtection', 'UseProtection'], 
			description: 'Enabled/disables protection')
		choice(name: 'PACKAGING_CHOICE', 
			choices: ['None', 'NoSign', 'UseSign'], 
			description: 'Pick something')
	}
	stages {
		stage('Build') {
			steps {
				echo 'Building...'
				/*	Creation of binaries  */
			}
		}
		stage('Test') {
			when { 
				expression { 
					return params.USE_TESTS != 'None'
				} 
			}
			steps {
				echo 'Testing...'
				/*	Run some tests  */
			}
		}
		stage('Protect') {
			when { 
				expression {
					return params.PROTECTION_CHOICE != 'NoProtection' || 
						params.USE_TESTS == 'All'
				} 
			}
			steps {
				echo 'Protecting....'
				/*  Perform some protecting  */
			}
		}
		stage('Package') {
			when { 
				expression {
					return params.PACKAGING_CHOICE != 'None' || 
						params.USE_TESTS == 'All'
				} 
			}
			steps {
				echo 'Packaging....'
				/*	Perform some packaging */
			}
		}
	}
	post {
		success {
			archiveArtifacts artifacts: 'Build/*.exe, Build/*.dll, Build/*.pdb'
		}
		
		/*	Workspace cleaning */
		cleanup {
			build job: 'NotificationJob',   parameters: [string(name: 'TITLE', 
				value: "Some message"),  
				string(name: 'STATUS', value: "${currentBuild.currentResult}"),
				string(name: 'LINK', value: "$BUILD_URL")    ], wait: false
				cleanWs(cleanWhenNotBuilt: true,
				cleanWhenFailure: true,
				cleanWhenAborted: true,
				cleanWhenSuccess: true,
				cleanWhenUnstable: true,
				deleteDirs: true,
				disableDeferredWipeout: true,
				notFailBuild: true,
				patterns: [[pattern: '**/*', type: 'INCLUDE']])
		}
	}
}

/*	Example of job for telegram notification. 
	In order to use it create needed credentials
*/
pipeline {
	agent any
	/*	Text parameters for notification */
	parameters {
		string(name: 'TITLE', 
			description: 'Title of notification')
		string(name: 'STATUS', 
			description: 'Status of notification')
		string(name: 'LINK', 
			description: 'Link address for details')
	}
	stages {
		stage('Notification') {
			steps {
				echo 'Executing notification job'
				/*	Creation message and actual telegram notification */
				withCredentials([string(credentialsId: 'BotToken', 
					variable: 'TOKEN'), string(credentialsId: 'GroupID', 
					variable: 'CHAT_ID')]) {
					script {
						def msg = "
${params.TITLE}
\n${params.STATUS}\nDetails"
						httpRequest (consoleLogResponseBody: true,
							contentType: 'APPLICATION_FORM',
							httpMode: 'POST',
							url: "https://api.telegram.org/bot${TOKEN}/sendMessage",
							requestBody : "parse_mode=html&text=${msg}&chat_id=${CHAT_ID}",
							validResponseCodes: '200')
					}
				}
			}
		}
	}
}