Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)
* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt
This commit is contained in:
Frostebite
GitHub
@@ -1,12 +1,12 @@
|
||||
import CloudRunnerLogger from '../services/cloud-runner-logger';
|
||||
import { TaskParameterSerializer } from '../services/task-parameter-serializer';
|
||||
import { CloudRunnerState } from '../state/cloud-runner-state';
|
||||
import { CloudRunnerStepState } from '../state/cloud-runner-step-state';
|
||||
import { BuildStep } from '../steps/build-step';
|
||||
import { SetupStep } from '../steps/setup-step';
|
||||
import { CloudRunnerFolders } from '../services/cloud-runner-folders';
|
||||
import { CloudRunnerStepState } from '../cloud-runner-step-state';
|
||||
import { CustomWorkflow } from './custom-workflow';
|
||||
import { WorkflowInterface } from './workflow-interface';
|
||||
import * as core from '@actions/core';
|
||||
import { CloudRunnerBuildCommandProcessor } from '../services/cloud-runner-build-command-process';
|
||||
import path from 'path';
|
||||
import CloudRunner from '../cloud-runner';
|
||||
|
||||
export class BuildAutomationWorkflow implements WorkflowInterface {
|
||||
async run(cloudRunnerStepState: CloudRunnerStepState) {
|
||||
@@ -21,41 +21,36 @@ export class BuildAutomationWorkflow implements WorkflowInterface {
|
||||
try {
|
||||
CloudRunnerLogger.log(`Cloud Runner is running standard build automation`);
|
||||
|
||||
core.startGroup('pre build steps');
|
||||
if (!CloudRunner.buildParameters.isCliMode) core.startGroup('pre build steps');
|
||||
let output = '';
|
||||
if (CloudRunnerState.buildParams.preBuildSteps !== '') {
|
||||
output += await CustomWorkflow.runCustomJob(CloudRunnerState.buildParams.preBuildSteps);
|
||||
if (CloudRunner.buildParameters.preBuildSteps !== '') {
|
||||
output += await CustomWorkflow.runCustomJob(CloudRunner.buildParameters.preBuildSteps);
|
||||
}
|
||||
core.endGroup();
|
||||
if (!CloudRunner.buildParameters.isCliMode) core.endGroup();
|
||||
CloudRunnerLogger.logWithTime('Configurable pre build step(s) time');
|
||||
|
||||
core.startGroup('setup');
|
||||
output += await new SetupStep().run(
|
||||
new CloudRunnerStepState(
|
||||
'alpine/git',
|
||||
TaskParameterSerializer.readBuildEnvironmentVariables(),
|
||||
CloudRunnerState.defaultSecrets,
|
||||
),
|
||||
);
|
||||
core.endGroup();
|
||||
CloudRunnerLogger.logWithTime('Download repository step time');
|
||||
if (!CloudRunner.buildParameters.isCliMode) core.startGroup('build');
|
||||
CloudRunnerLogger.log(baseImage.toString());
|
||||
CloudRunnerLogger.logLine(` `);
|
||||
CloudRunnerLogger.logLine('Starting build automation job');
|
||||
|
||||
core.startGroup('build');
|
||||
output += await new BuildStep().run(
|
||||
new CloudRunnerStepState(
|
||||
baseImage,
|
||||
TaskParameterSerializer.readBuildEnvironmentVariables(),
|
||||
CloudRunnerState.defaultSecrets,
|
||||
),
|
||||
output += await CloudRunner.Provider.runTask(
|
||||
CloudRunner.buildParameters.buildGuid,
|
||||
baseImage.toString(),
|
||||
BuildAutomationWorkflow.BuildWorkflow,
|
||||
`/${CloudRunnerFolders.buildVolumeFolder}`,
|
||||
`/${CloudRunnerFolders.buildVolumeFolder}/`,
|
||||
CloudRunner.cloudRunnerEnvironmentVariables,
|
||||
CloudRunner.defaultSecrets,
|
||||
);
|
||||
core.endGroup();
|
||||
if (!CloudRunner.buildParameters.isCliMode) core.endGroup();
|
||||
CloudRunnerLogger.logWithTime('Build time');
|
||||
|
||||
core.startGroup('post build steps');
|
||||
if (CloudRunnerState.buildParams.postBuildSteps !== '') {
|
||||
output += await CustomWorkflow.runCustomJob(CloudRunnerState.buildParams.postBuildSteps);
|
||||
if (!CloudRunner.buildParameters.isCliMode) core.startGroup('post build steps');
|
||||
if (CloudRunner.buildParameters.postBuildSteps !== '') {
|
||||
output += await CustomWorkflow.runCustomJob(CloudRunner.buildParameters.postBuildSteps);
|
||||
}
|
||||
core.endGroup();
|
||||
if (!CloudRunner.buildParameters.isCliMode) core.endGroup();
|
||||
CloudRunnerLogger.logWithTime('Configurable post build step(s) time');
|
||||
|
||||
CloudRunnerLogger.log(`Cloud Runner finished running standard build automation`);
|
||||
@@ -65,4 +60,62 @@ export class BuildAutomationWorkflow implements WorkflowInterface {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private static get BuildWorkflow() {
|
||||
const setupHooks = CloudRunnerBuildCommandProcessor.getHooks(CloudRunner.buildParameters.customJobHooks).filter(
|
||||
(x) => x.step.includes(`setup`),
|
||||
);
|
||||
const buildHooks = CloudRunnerBuildCommandProcessor.getHooks(CloudRunner.buildParameters.customJobHooks).filter(
|
||||
(x) => x.step.includes(`build`),
|
||||
);
|
||||
const builderPath = path.join(CloudRunnerFolders.builderPathAbsolute, 'dist', `index.js`).replace(/\\/g, `/`);
|
||||
return `apt-get update > /dev/null
|
||||
apt-get install -y zip tree npm git-lfs jq unzip git > /dev/null
|
||||
npm install -g n > /dev/null
|
||||
n stable > /dev/null
|
||||
${setupHooks.filter((x) => x.hook.includes(`before`)).map((x) => x.commands) || ' '}
|
||||
export GITHUB_WORKSPACE="${CloudRunnerFolders.repoPathAbsolute.replace(/\\/g, `/`)}"
|
||||
${BuildAutomationWorkflow.setupCommands(builderPath)}
|
||||
${setupHooks.filter((x) => x.hook.includes(`after`)).map((x) => x.commands) || ' '}
|
||||
${buildHooks.filter((x) => x.hook.includes(`before`)).map((x) => x.commands) || ' '}
|
||||
${BuildAutomationWorkflow.BuildCommands(builderPath, CloudRunner.buildParameters.buildGuid)}
|
||||
${buildHooks.filter((x) => x.hook.includes(`after`)).map((x) => x.commands) || ' '}`;
|
||||
}
|
||||
|
||||
private static setupCommands(builderPath) {
|
||||
return `export GIT_DISCOVERY_ACROSS_FILESYSTEM=1
|
||||
echo "game ci cloud runner clone"
|
||||
mkdir -p ${CloudRunnerFolders.builderPathAbsolute.replace(/\\/g, `/`)}
|
||||
git clone -q -b ${CloudRunner.buildParameters.cloudRunnerBranch} ${
|
||||
CloudRunnerFolders.unityBuilderRepoUrl
|
||||
} "${CloudRunnerFolders.builderPathAbsolute.replace(/\\/g, `/`)}"
|
||||
chmod +x ${builderPath}
|
||||
echo "game ci cloud runner bootstrap"
|
||||
node ${builderPath} -m remote-cli`;
|
||||
}
|
||||
|
||||
private static BuildCommands(builderPath, guid) {
|
||||
const linuxCacheFolder = CloudRunnerFolders.cacheFolderFull.replace(/\\/g, `/`);
|
||||
const distFolder = path.join(CloudRunnerFolders.builderPathAbsolute, 'dist');
|
||||
const ubuntuPlatformsFolder = path.join(CloudRunnerFolders.builderPathAbsolute, 'dist', 'platforms', 'ubuntu');
|
||||
return `echo "game ci cloud runner init"
|
||||
mkdir -p ${`${CloudRunnerFolders.projectBuildFolderAbsolute}/build`.replace(/\\/g, `/`)}
|
||||
cd ${CloudRunnerFolders.projectPathAbsolute}
|
||||
cp -r "${path.join(distFolder, 'default-build-script').replace(/\\/g, `/`)}" "/UnityBuilderAction"
|
||||
cp -r "${path.join(ubuntuPlatformsFolder, 'entrypoint.sh').replace(/\\/g, `/`)}" "/entrypoint.sh"
|
||||
cp -r "${path.join(ubuntuPlatformsFolder, 'steps').replace(/\\/g, `/`)}" "/steps"
|
||||
chmod -R +x "/entrypoint.sh"
|
||||
chmod -R +x "/steps"
|
||||
echo "game ci cloud runner start"
|
||||
/entrypoint.sh
|
||||
echo "game ci cloud runner push library to cache"
|
||||
chmod +x ${builderPath}
|
||||
node ${builderPath} -m cache-push --cachePushFrom ${
|
||||
CloudRunnerFolders.libraryFolderAbsolute
|
||||
} --artifactName lib-${guid} --cachePushTo ${linuxCacheFolder}/Library
|
||||
echo "game ci cloud runner push build to cache"
|
||||
node ${builderPath} -m cache-push --cachePushFrom ${
|
||||
CloudRunnerFolders.projectBuildFolderAbsolute
|
||||
} --artifactName build-${guid} --cachePushTo ${`${linuxCacheFolder}/build`.replace(/\\/g, `/`)}`;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user