Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt
This commit is contained in:
Frostebite
GitHub
@@ -1,12 +1,14 @@
|
||||
import { customAlphabet } from 'nanoid';
|
||||
import * as core from '@actions/core';
|
||||
import AndroidVersioning from './android-versioning';
|
||||
import CloudRunnerConstants from './cloud-runner/services/cloud-runner-constants';
|
||||
import CloudRunnerNamespace from './cloud-runner/services/cloud-runner-namespace';
|
||||
import CloudRunnerBuildGuid from './cloud-runner/services/cloud-runner-guid';
|
||||
import Input from './input';
|
||||
import Platform from './platform';
|
||||
import UnityVersioning from './unity-versioning';
|
||||
import Versioning from './versioning';
|
||||
import { GitRepoReader } from './input-readers/git-repo';
|
||||
import { GithubCliReader } from './input-readers/github-cli';
|
||||
import { Cli } from './cli/cli';
|
||||
|
||||
class BuildParameters {
|
||||
public editorVersion!: string;
|
||||
@@ -33,15 +35,22 @@ class BuildParameters {
|
||||
public cloudRunnerCluster!: string;
|
||||
public awsBaseStackName!: string;
|
||||
public gitPrivateToken!: string;
|
||||
public remoteBuildCluster!: string;
|
||||
public awsStackName!: string;
|
||||
public kubeConfig!: string;
|
||||
public githubToken!: string;
|
||||
public cloudRunnerMemory!: string;
|
||||
public cloudRunnerCpu!: string;
|
||||
public kubeVolumeSize!: string;
|
||||
public kubeVolume!: string;
|
||||
public kubeStorageClass!: string;
|
||||
public chownFilesTo!: string;
|
||||
public customJobHooks!: string;
|
||||
public cachePushOverrideCommand!: string;
|
||||
public cachePullOverrideCommand!: string;
|
||||
public readInputFromOverrideList!: string;
|
||||
public readInputOverrideCommand!: string;
|
||||
public checkDependencyHealthOverride!: string;
|
||||
public startDependenciesOverride!: string;
|
||||
public cacheKey!: string;
|
||||
|
||||
public postBuildSteps!: string;
|
||||
public preBuildSteps!: string;
|
||||
@@ -52,6 +61,10 @@ class BuildParameters {
|
||||
public gitSha!: string;
|
||||
public logId!: string;
|
||||
public buildGuid!: string;
|
||||
public cloudRunnerBranch!: string;
|
||||
public cloudRunnerIntegrationTests!: boolean;
|
||||
public cloudRunnerBuilderPlatform!: string | undefined;
|
||||
public isCliMode!: boolean;
|
||||
|
||||
static async create(): Promise<BuildParameters> {
|
||||
const buildFile = this.parseBuildFile(Input.buildName, Input.targetPlatform, Input.androidAppBundle);
|
||||
@@ -63,7 +76,7 @@ class BuildParameters {
|
||||
// Todo - Don't use process.env directly, that's what the input model class is for.
|
||||
// ---
|
||||
let unitySerial = '';
|
||||
if (!process.env.UNITY_SERIAL) {
|
||||
if (!process.env.UNITY_SERIAL && Input.githubInputEnabled && Cli.options === undefined) {
|
||||
//No serial was present so it is a personal license that we need to convert
|
||||
if (!process.env.UNITY_LICENSE) {
|
||||
throw new Error(`Missing Unity License File and no Serial was found. If this
|
||||
@@ -75,8 +88,6 @@ class BuildParameters {
|
||||
} else {
|
||||
unitySerial = process.env.UNITY_SERIAL!;
|
||||
}
|
||||
core.setSecret(unitySerial);
|
||||
// ---
|
||||
|
||||
return {
|
||||
editorVersion,
|
||||
@@ -101,12 +112,12 @@ class BuildParameters {
|
||||
androidSdkManagerParameters,
|
||||
customParameters: Input.customParameters,
|
||||
sshAgent: Input.sshAgent,
|
||||
gitPrivateToken: await Input.gitPrivateToken(),
|
||||
gitPrivateToken: Input.gitPrivateToken || (await GithubCliReader.GetGitHubAuthToken()),
|
||||
chownFilesTo: Input.chownFilesTo,
|
||||
cloudRunnerCluster: Input.cloudRunnerCluster,
|
||||
cloudRunnerBuilderPlatform: Input.cloudRunnerBuilderPlatform,
|
||||
awsBaseStackName: Input.awsBaseStackName,
|
||||
kubeConfig: Input.kubeConfig,
|
||||
githubToken: await Input.githubToken(),
|
||||
cloudRunnerMemory: Input.cloudRunnerMemory,
|
||||
cloudRunnerCpu: Input.cloudRunnerCpu,
|
||||
kubeVolumeSize: Input.kubeVolumeSize,
|
||||
@@ -115,14 +126,24 @@ class BuildParameters {
|
||||
preBuildSteps: Input.preBuildSteps,
|
||||
customJob: Input.customJob,
|
||||
runNumber: Input.runNumber,
|
||||
branch: await Input.branch(),
|
||||
// Todo - move this out of UserInput and into some class that determines additional information (as needed)
|
||||
githubRepo: await Input.githubRepo(),
|
||||
remoteBuildCluster: Input.cloudRunnerCluster,
|
||||
branch: Input.branch.replace('/head', '') || (await GitRepoReader.GetBranch()),
|
||||
cloudRunnerBranch: Input.cloudRunnerBranch.split('/').reverse()[0],
|
||||
cloudRunnerIntegrationTests: Input.cloudRunnerTests,
|
||||
githubRepo: Input.githubRepo || (await GitRepoReader.GetRemote()) || 'game-ci/unity-builder',
|
||||
isCliMode: Cli.isCliMode,
|
||||
awsStackName: Input.awsBaseStackName,
|
||||
gitSha: Input.gitSha,
|
||||
logId: customAlphabet(CloudRunnerConstants.alphabet, 9)(),
|
||||
buildGuid: CloudRunnerNamespace.generateBuildName(Input.runNumber, Input.targetPlatform),
|
||||
buildGuid: CloudRunnerBuildGuid.generateGuid(Input.runNumber, Input.targetPlatform),
|
||||
customJobHooks: Input.customJobHooks(),
|
||||
cachePullOverrideCommand: Input.cachePullOverrideCommand(),
|
||||
cachePushOverrideCommand: Input.cachePushOverrideCommand(),
|
||||
readInputOverrideCommand: Input.readInputOverrideCommand(),
|
||||
readInputFromOverrideList: Input.readInputFromOverrideList(),
|
||||
kubeStorageClass: Input.kubeStorageClass,
|
||||
checkDependencyHealthOverride: Input.checkDependencyHealthOverride,
|
||||
startDependenciesOverride: Input.startDependenciesOverride,
|
||||
cacheKey: Input.cacheKey,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user