Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt
This commit is contained in:
Frostebite
GitHub
@@ -1,7 +1,7 @@
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { GitRepoReader } from './input-readers/git-repo';
|
||||
import { GithubCliReader } from './input-readers/github-cli';
|
||||
import { Cli } from './cli/cli';
|
||||
import CloudRunnerQueryOverride from './cloud-runner/services/cloud-runner-query-override';
|
||||
import Platform from './platform';
|
||||
|
||||
const core = require('@actions/core');
|
||||
@@ -14,64 +14,69 @@ const core = require('@actions/core');
|
||||
* Todo: rename to UserInput and remove anything that is not direct input from the user / ci workflow
|
||||
*/
|
||||
class Input {
|
||||
public static cliOptions;
|
||||
public static githubInputEnabled: boolean = true;
|
||||
|
||||
// also enabled debug logging for cloud runner
|
||||
static get cloudRunnerTests(): boolean {
|
||||
return Input.getInput(`cloudRunnerTests`) || Input.getInput(`CloudRunnerTests`) || false;
|
||||
}
|
||||
public static getInput(query) {
|
||||
if (Input.githubInputEnabled) {
|
||||
const coreInput = core.getInput(query);
|
||||
if (coreInput && coreInput !== '') {
|
||||
return coreInput;
|
||||
}
|
||||
}
|
||||
const alternativeQuery = Input.ToEnvVarFormat(query);
|
||||
|
||||
private static getInput(query) {
|
||||
const coreInput = core.getInput(query);
|
||||
if (Input.githubInputEnabled && coreInput && coreInput !== '') {
|
||||
return coreInput;
|
||||
// query input sources
|
||||
if (Cli.query(query, alternativeQuery)) {
|
||||
return Cli.query(query, alternativeQuery);
|
||||
}
|
||||
|
||||
return Input.cliOptions !== undefined && Input.cliOptions[query] !== undefined
|
||||
? Input.cliOptions[query]
|
||||
: process.env[query] !== undefined
|
||||
? process.env[query]
|
||||
: process.env[Input.ToEnvVarFormat(query)]
|
||||
? process.env[Input.ToEnvVarFormat(query)]
|
||||
: '';
|
||||
if (CloudRunnerQueryOverride.query(query, alternativeQuery)) {
|
||||
return CloudRunnerQueryOverride.query(query, alternativeQuery);
|
||||
}
|
||||
|
||||
if (process.env[query] !== undefined) {
|
||||
return process.env[query];
|
||||
}
|
||||
|
||||
if (alternativeQuery !== query && process.env[alternativeQuery] !== undefined) {
|
||||
return process.env[alternativeQuery];
|
||||
}
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
static get region(): string {
|
||||
return Input.getInput('region') || 'eu-west-2';
|
||||
}
|
||||
|
||||
static async githubRepo() {
|
||||
return (
|
||||
Input.getInput('GITHUB_REPOSITORY') ||
|
||||
Input.getInput('GITHUB_REPO') ||
|
||||
// todo - move this to some class specific for determining additional information
|
||||
(await GitRepoReader.GetRemote()) ||
|
||||
'game-ci/unity-builder'
|
||||
);
|
||||
static get githubRepo() {
|
||||
return Input.getInput('GITHUB_REPOSITORY') || Input.getInput('GITHUB_REPO') || undefined;
|
||||
}
|
||||
|
||||
static async branch() {
|
||||
if (await GitRepoReader.GetBranch()) {
|
||||
// todo - move this to some class specific for determining additional information
|
||||
return await GitRepoReader.GetBranch();
|
||||
} else if (Input.getInput(`GITHUB_REF`)) {
|
||||
return Input.getInput(`GITHUB_REF`).replace('refs/', '').replace(`head/`, '');
|
||||
static get branch() {
|
||||
if (Input.getInput(`GITHUB_REF`)) {
|
||||
return Input.getInput(`GITHUB_REF`).replace('refs/', '').replace(`head/`, '').replace(`heads/`, '');
|
||||
} else if (Input.getInput('branch')) {
|
||||
return Input.getInput('branch');
|
||||
} else {
|
||||
return 'main';
|
||||
return '';
|
||||
}
|
||||
}
|
||||
static get cloudRunnerBuilderPlatform() {
|
||||
const input = Input.getInput('cloudRunnerBuilderPlatform');
|
||||
if (input) {
|
||||
return input;
|
||||
}
|
||||
if (Input.cloudRunnerCluster !== 'local') {
|
||||
return 'linux';
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
static get gitSha() {
|
||||
if (Input.getInput(`GITHUB_SHA`)) {
|
||||
return Input.getInput(`GITHUB_SHA`);
|
||||
} else if (Input.getInput(`GitSHA`)) {
|
||||
return Input.getInput(`GitSHA`);
|
||||
} else if (GitRepoReader.GetSha()) {
|
||||
// todo - move this to some class specific for determining additional information
|
||||
return GitRepoReader.GetSha();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -88,7 +93,7 @@ class Input {
|
||||
}
|
||||
|
||||
static get customImage() {
|
||||
return Input.getInput('customImage');
|
||||
return Input.getInput('customImage') || '';
|
||||
}
|
||||
|
||||
static get projectPath() {
|
||||
@@ -165,8 +170,36 @@ class Input {
|
||||
return Input.getInput('sshAgent') || '';
|
||||
}
|
||||
|
||||
static async gitPrivateToken() {
|
||||
return Input.getInput('gitPrivateToken') || (await Input.githubToken());
|
||||
static get gitPrivateToken() {
|
||||
return core.getInput('gitPrivateToken') || false;
|
||||
}
|
||||
|
||||
static get customJob() {
|
||||
return Input.getInput('customJob') || '';
|
||||
}
|
||||
|
||||
static customJobHooks() {
|
||||
return Input.getInput('customJobHooks') || '';
|
||||
}
|
||||
|
||||
static cachePushOverrideCommand() {
|
||||
return Input.getInput('cachePushOverrideCommand') || '';
|
||||
}
|
||||
|
||||
static cachePullOverrideCommand() {
|
||||
return Input.getInput('cachePullOverrideCommand') || '';
|
||||
}
|
||||
|
||||
static readInputFromOverrideList() {
|
||||
return Input.getInput('readInputFromOverrideList') || '';
|
||||
}
|
||||
|
||||
static readInputOverrideCommand() {
|
||||
return Input.getInput('readInputOverrideCommand') || '';
|
||||
}
|
||||
|
||||
static get cloudRunnerBranch() {
|
||||
return Input.getInput('cloudRunnerBranch') || 'cloud-runner-develop';
|
||||
}
|
||||
|
||||
static get chownFilesTo() {
|
||||
@@ -187,15 +220,14 @@ class Input {
|
||||
return Input.getInput('preBuildSteps') || '';
|
||||
}
|
||||
|
||||
static get customJob() {
|
||||
return Input.getInput('customJob') || '';
|
||||
}
|
||||
|
||||
static get awsBaseStackName() {
|
||||
return Input.getInput('awsBaseStackName') || 'game-ci';
|
||||
}
|
||||
|
||||
static get cloudRunnerCluster() {
|
||||
if (Cli.isCliMode) {
|
||||
return Input.getInput('cloudRunnerCluster') || 'aws';
|
||||
}
|
||||
return Input.getInput('cloudRunnerCluster') || 'local';
|
||||
}
|
||||
|
||||
@@ -207,11 +239,6 @@ class Input {
|
||||
return Input.getInput('cloudRunnerMemory') || '750M';
|
||||
}
|
||||
|
||||
static async githubToken() {
|
||||
// Todo - move GitHubCLI out of the simple input class. It is in fact not input from the user.
|
||||
return Input.getInput('githubToken') || (await GithubCliReader.GetGitHubAuthToken()) || '';
|
||||
}
|
||||
|
||||
static get kubeConfig() {
|
||||
return Input.getInput('kubeConfig') || '';
|
||||
}
|
||||
@@ -224,7 +251,30 @@ class Input {
|
||||
return Input.getInput('kubeVolumeSize') || '5Gi';
|
||||
}
|
||||
|
||||
static get kubeStorageClass(): string {
|
||||
return Input.getInput('kubeStorageClass') || '';
|
||||
}
|
||||
|
||||
static get checkDependencyHealthOverride(): string {
|
||||
return Input.getInput('checkDependencyHealthOverride') || '';
|
||||
}
|
||||
|
||||
static get startDependenciesOverride(): string {
|
||||
return Input.getInput('startDependenciesOverride') || '';
|
||||
}
|
||||
|
||||
static get cacheKey(): string {
|
||||
return Input.getInput('cacheKey') || Input.branch;
|
||||
}
|
||||
|
||||
static get cloudRunnerTests(): boolean {
|
||||
return Input.getInput(`cloudRunnerTests`) || false;
|
||||
}
|
||||
|
||||
public static ToEnvVarFormat(input: string) {
|
||||
if (input.toUpperCase() === input) {
|
||||
return input;
|
||||
}
|
||||
return input
|
||||
.replace(/([A-Z])/g, ' $1')
|
||||
.trim()
|
||||
|
||||
Reference in New Issue
Block a user