Cloud Runner v0 - Reliable and trimmed down cloud runner (#353)

* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt
2022-04-11 00:00:37 +01:00
parent 2b399b2641
commit a61c02481f
65 changed files with 10262 additions and 2259 deletions
--- a/src/model/cloud-runner/remote-client/caching.test.ts
+++ b/src/model/cloud-runner/remote-client/caching.test.ts
@@ -0,0 +1,63 @@
+import fs from 'fs';
+import path from 'path';
+import BuildParameters from '../../build-parameters';
+import { Cli } from '../../cli/cli';
+import Input from '../../input';
+import UnityVersioning from '../../unity-versioning';
+import CloudRunner from '../cloud-runner';
+import { CloudRunnerSystem } from '../services/cloud-runner-system';
+import { Caching } from './caching';
+import { v4 as uuidv4 } from 'uuid';
+
+describe('Cloud Runner Caching', () => {
+  it('responds', () => {});
+});
+describe('Cloud Runner Caching', () => {
+  if (process.platform === 'linux') {
+    it('Simple caching works', async () => {
+      Cli.options = {
+        versioning: 'None',
+        projectPath: 'test-project',
+        unityVersion: UnityVersioning.read('test-project'),
+        targetPlatform: 'StandaloneLinux64',
+        cacheKey: `test-case-${uuidv4()}`,
+      };
+      Input.githubInputEnabled = false;
+      const buildParameter = await BuildParameters.create();
+      CloudRunner.buildParameters = buildParameter;
+
+      // create test folder
+      const testFolder = path.resolve(__dirname, Cli.options.cacheKey);
+      fs.mkdirSync(testFolder);
+
+      // crate cache folder
+      const cacheFolder = path.resolve(__dirname, `cache-${Cli.options.cacheKey}`);
+      fs.mkdirSync(cacheFolder);
+
+      // add test has file to test folders
+      fs.writeFileSync(path.resolve(testFolder, 'test.txt'), Cli.options.cacheKey);
+      await Caching.PushToCache(cacheFolder, testFolder, `${Cli.options.cacheKey}`);
+
+      // delete test folder
+      fs.rmdirSync(testFolder, { recursive: true });
+      await Caching.PullFromCache(
+        cacheFolder.replace(/\\/g, `/`),
+        testFolder.replace(/\\/g, `/`),
+        `${Cli.options.cacheKey}`,
+      );
+      await CloudRunnerSystem.Run(`du -h ${__dirname}`);
+      await CloudRunnerSystem.Run(`tree ${testFolder}`);
+      await CloudRunnerSystem.Run(`tree ${cacheFolder}`);
+
+      // compare validity to original hash
+      expect(fs.readFileSync(path.resolve(testFolder, 'test.txt'), { encoding: 'utf8' }).toString()).toContain(
+        Cli.options.cacheKey,
+      );
+      fs.rmdirSync(testFolder, { recursive: true });
+      fs.rmdirSync(cacheFolder, { recursive: true });
+
+      Input.githubInputEnabled = true;
+      delete Cli.options;
+    }, 1000000);
+  }
+});
--- a/src/model/cloud-runner/remote-client/caching.ts
+++ b/src/model/cloud-runner/remote-client/caching.ts
@@ -0,0 +1,171 @@
+import { assert } from 'console';
+import fs from 'fs';
+import path from 'path';
+import CloudRunner from '../cloud-runner';
+import CloudRunnerLogger from '../services/cloud-runner-logger';
+import { CloudRunnerFolders } from '../services/cloud-runner-folders';
+import { CloudRunnerSystem } from '../services/cloud-runner-system';
+import { LfsHashing } from '../services/lfs-hashing';
+import { RemoteClientLogger } from './remote-client-logger';
+import { Cli } from '../../cli/cli';
+import { CliFunction } from '../../cli/cli-functions-repository';
+// eslint-disable-next-line github/no-then
+const fileExists = async (fpath) => !!(await fs.promises.stat(fpath).catch(() => false));
+
+export class Caching {
+  @CliFunction(`cache-push`, `push to cache`)
+  static async cachePush() {
+    try {
+      const buildParameter = JSON.parse(process.env.BUILD_PARAMETERS || '{}');
+      CloudRunner.buildParameters = buildParameter;
+      await Caching.PushToCache(
+        Cli.options['cachePushTo'],
+        Cli.options['cachePushFrom'],
+        Cli.options['artifactName'] || '',
+      );
+    } catch (error: any) {
+      CloudRunnerLogger.log(`${error}`);
+    }
+  }
+
+  @CliFunction(`cache-pull`, `pull from cache`)
+  static async cachePull() {
+    try {
+      const buildParameter = JSON.parse(process.env.BUILD_PARAMETERS || '{}');
+      CloudRunner.buildParameters = buildParameter;
+      await Caching.PullFromCache(
+        Cli.options['cachePushFrom'],
+        Cli.options['cachePushTo'],
+        Cli.options['artifactName'] || '',
+      );
+    } catch (error: any) {
+      CloudRunnerLogger.log(`${error}`);
+    }
+  }
+
+  public static async PushToCache(cacheFolder: string, sourceFolder: string, cacheArtifactName: string) {
+    cacheArtifactName = cacheArtifactName.replace(' ', '');
+    const startPath = process.cwd();
+    try {
+      if (!(await fileExists(cacheFolder))) {
+        await CloudRunnerSystem.Run(`mkdir -p ${cacheFolder}`);
+      }
+      process.chdir(path.resolve(sourceFolder, '..'));
+
+      if (CloudRunner.buildParameters.cloudRunnerIntegrationTests) {
+        CloudRunnerLogger.log(
+          `Hashed cache folder ${await LfsHashing.hashAllFiles(sourceFolder)} ${sourceFolder} ${path.basename(
+            sourceFolder,
+          )}`,
+        );
+      }
+      // eslint-disable-next-line func-style
+      const formatFunction = function (format: string) {
+        const arguments_ = Array.prototype.slice.call(
+          [path.resolve(sourceFolder, '..'), cacheFolder, cacheArtifactName],
+          1,
+        );
+        return format.replace(/{(\d+)}/g, function (match, number) {
+          return typeof arguments_[number] != 'undefined' ? arguments_[number] : match;
+        });
+      };
+      await CloudRunnerSystem.Run(`zip -q -r ${cacheArtifactName}.zip ${path.basename(sourceFolder)}`);
+      assert(await fileExists(`${cacheArtifactName}.zip`), 'cache zip exists');
+      assert(await fileExists(path.basename(sourceFolder)), 'source folder exists');
+      if (CloudRunner.buildParameters.cachePushOverrideCommand) {
+        await CloudRunnerSystem.Run(formatFunction(CloudRunner.buildParameters.cachePushOverrideCommand));
+      }
+      await CloudRunnerSystem.Run(`mv ${cacheArtifactName}.zip ${cacheFolder}`);
+      RemoteClientLogger.log(`moved ${cacheArtifactName}.zip to ${cacheFolder}`);
+      assert(
+        await fileExists(`${path.join(cacheFolder, cacheArtifactName)}.zip`),
+        'cache zip exists inside cache folder',
+      );
+    } catch (error) {
+      process.chdir(`${startPath}`);
+      throw error;
+    }
+    process.chdir(`${startPath}`);
+  }
+  public static async PullFromCache(cacheFolder: string, destinationFolder: string, cacheArtifactName: string = ``) {
+    cacheArtifactName = cacheArtifactName.replace(' ', '');
+    const startPath = process.cwd();
+    RemoteClientLogger.log(`Caching for ${path.basename(destinationFolder)}`);
+    try {
+      if (!(await fileExists(cacheFolder))) {
+        await fs.promises.mkdir(cacheFolder);
+      }
+
+      if (!(await fileExists(destinationFolder))) {
+        await fs.promises.mkdir(destinationFolder);
+      }
+
+      const latestInBranch = await (await CloudRunnerSystem.Run(`ls -t "${cacheFolder}" | grep .zip$ | head -1`))
+        .replace(/\n/g, ``)
+        .replace('.zip', '');
+
+      process.chdir(cacheFolder);
+
+      const cacheSelection =
+        cacheArtifactName !== `` && (await fileExists(`${cacheArtifactName}.zip`)) ? cacheArtifactName : latestInBranch;
+      await CloudRunnerLogger.log(`cache key ${cacheArtifactName} selection ${cacheSelection}`);
+
+      // eslint-disable-next-line func-style
+      const formatFunction = function (format: string) {
+        const arguments_ = Array.prototype.slice.call(
+          [path.resolve(destinationFolder, '..'), cacheFolder, cacheArtifactName],
+          1,
+        );
+        return format.replace(/{(\d+)}/g, function (match, number) {
+          return typeof arguments_[number] != 'undefined' ? arguments_[number] : match;
+        });
+      };
+
+      if (CloudRunner.buildParameters.cachePullOverrideCommand) {
+        await CloudRunnerSystem.Run(formatFunction(CloudRunner.buildParameters.cachePullOverrideCommand));
+      }
+
+      if (await fileExists(`${cacheSelection}.zip`)) {
+        const resultsFolder = `results${CloudRunner.buildParameters.buildGuid}`;
+        await CloudRunnerSystem.Run(`mkdir -p ${resultsFolder}`);
+        RemoteClientLogger.log(`cache item exists ${cacheFolder}/${cacheSelection}.zip`);
+        const fullResultsFolder = path.join(cacheFolder, resultsFolder);
+        await CloudRunnerSystem.Run(`unzip -q ${cacheSelection}.zip -d ${path.basename(resultsFolder)}`);
+        RemoteClientLogger.log(`cache item extracted to ${fullResultsFolder}`);
+        assert(await fileExists(fullResultsFolder), `cache extraction results folder exists`);
+        const destinationParentFolder = path.resolve(destinationFolder, '..');
+
+        if (await fileExists(destinationFolder)) {
+          await fs.promises.rmdir(destinationFolder, { recursive: true });
+        }
+        await CloudRunnerSystem.Run(
+          `mv "${path.join(fullResultsFolder, path.basename(destinationFolder))}" "${destinationParentFolder}"`,
+        );
+        await CloudRunnerSystem.Run(`du -sh ${path.join(destinationParentFolder, path.basename(destinationFolder))}`);
+        const contents = await fs.promises.readdir(
+          path.join(destinationParentFolder, path.basename(destinationFolder)),
+        );
+        CloudRunnerLogger.log(
+          `There is ${contents.length} files/dir in the cache pulled contents for ${path.basename(destinationFolder)}`,
+        );
+      } else {
+        RemoteClientLogger.logWarning(`cache item ${cacheArtifactName} doesn't exist ${destinationFolder}`);
+        if (cacheSelection !== ``) {
+          RemoteClientLogger.logWarning(`cache item ${cacheArtifactName}.zip doesn't exist ${destinationFolder}`);
+          throw new Error(`Failed to get cache item, but cache hit was found: ${cacheSelection}`);
+        }
+      }
+    } catch (error) {
+      process.chdir(`${startPath}`);
+      throw error;
+    }
+    process.chdir(`${startPath}`);
+  }
+
+  public static async handleCachePurging() {
+    if (process.env.PURGE_REMOTE_BUILDER_CACHE !== undefined) {
+      RemoteClientLogger.log(`purging ${CloudRunnerFolders.purgeRemoteCaching}`);
+      fs.promises.rmdir(CloudRunnerFolders.cacheFolder, { recursive: true });
+    }
+  }
+}
--- a/src/model/cloud-runner/remote-client/index.ts
+++ b/src/model/cloud-runner/remote-client/index.ts
@@ -0,0 +1,95 @@
+import fs from 'fs';
+import CloudRunner from '../cloud-runner';
+import { CloudRunnerFolders } from '../services/cloud-runner-folders';
+import { Caching } from './caching';
+import { LfsHashing } from '../services/lfs-hashing';
+import { RemoteClientLogger } from './remote-client-logger';
+import path from 'path';
+import { assert } from 'console';
+import CloudRunnerLogger from '../services/cloud-runner-logger';
+import { CliFunction } from '../../cli/cli-functions-repository';
+import { CloudRunnerSystem } from '../services/cloud-runner-system';
+
+export class RemoteClient {
+  public static async bootstrapRepository() {
+    try {
+      await CloudRunnerSystem.Run(`mkdir -p ${CloudRunnerFolders.uniqueCloudRunnerJobFolderAbsolute}`);
+      await CloudRunnerSystem.Run(`mkdir -p ${CloudRunnerFolders.repoPathAbsolute}`);
+      await CloudRunnerSystem.Run(`mkdir -p ${CloudRunnerFolders.cacheFolderFull}`);
+      process.chdir(CloudRunnerFolders.repoPathAbsolute);
+      await RemoteClient.cloneRepoWithoutLFSFiles();
+      await RemoteClient.sizeOfFolder('repo before lfs cache pull', CloudRunnerFolders.repoPathAbsolute);
+      const lfsHashes = await LfsHashing.createLFSHashFiles();
+      if (fs.existsSync(CloudRunnerFolders.libraryFolderAbsolute)) {
+        RemoteClientLogger.logWarning(`!Warning!: The Unity library was included in the git repository`);
+      }
+      await Caching.PullFromCache(
+        CloudRunnerFolders.lfsCacheFolderFull,
+        CloudRunnerFolders.lfsFolderAbsolute,
+        `${lfsHashes.lfsGuidSum}`,
+      );
+      await RemoteClient.sizeOfFolder('repo after lfs cache pull', CloudRunnerFolders.repoPathAbsolute);
+      await RemoteClient.pullLatestLFS();
+      await RemoteClient.sizeOfFolder('repo before lfs git pull', CloudRunnerFolders.repoPathAbsolute);
+      await Caching.PushToCache(
+        CloudRunnerFolders.lfsCacheFolderFull,
+        CloudRunnerFolders.lfsFolderAbsolute,
+        `${lfsHashes.lfsGuidSum}`,
+      );
+      await Caching.PullFromCache(CloudRunnerFolders.libraryCacheFolderFull, CloudRunnerFolders.libraryFolderAbsolute);
+      await RemoteClient.sizeOfFolder('repo after library cache pull', CloudRunnerFolders.repoPathAbsolute);
+      await Caching.handleCachePurging();
+    } catch (error) {
+      throw error;
+    }
+  }
+
+  private static async sizeOfFolder(message: string, folder: string) {
+    CloudRunnerLogger.log(`Size of ${message}`);
+    await CloudRunnerSystem.Run(`du -sh ${folder}`);
+  }
+
+  private static async cloneRepoWithoutLFSFiles() {
+    try {
+      process.chdir(`${CloudRunnerFolders.repoPathAbsolute}`);
+      RemoteClientLogger.log(`Initializing source repository for cloning with caching of LFS files`);
+      await CloudRunnerSystem.Run(`git config --global advice.detachedHead false`);
+      RemoteClientLogger.log(`Cloning the repository being built:`);
+      await CloudRunnerSystem.Run(`git config --global filter.lfs.smudge "git-lfs smudge --skip -- %f"`);
+      await CloudRunnerSystem.Run(`git config --global filter.lfs.process "git-lfs filter-process --skip"`);
+      await CloudRunnerSystem.Run(
+        `git clone -q ${CloudRunnerFolders.targetBuildRepoUrl} ${path.resolve(
+          `..`,
+          path.basename(CloudRunnerFolders.repoPathAbsolute),
+        )}`,
+      );
+      await CloudRunnerSystem.Run(`git lfs install`);
+      assert(fs.existsSync(`.git`), 'git folder exists');
+      RemoteClientLogger.log(`${CloudRunner.buildParameters.branch}`);
+      await CloudRunnerSystem.Run(`git checkout ${CloudRunner.buildParameters.branch}`);
+      assert(fs.existsSync(path.join(`.git`, `lfs`)), 'LFS folder should not exist before caching');
+      RemoteClientLogger.log(`Checked out ${process.env.GITHUB_SHA}`);
+    } catch (error) {
+      throw error;
+    }
+  }
+
+  private static async pullLatestLFS() {
+    process.chdir(CloudRunnerFolders.repoPathAbsolute);
+    await CloudRunnerSystem.Run(`git config --global filter.lfs.smudge "git-lfs smudge -- %f"`);
+    await CloudRunnerSystem.Run(`git config --global filter.lfs.process "git-lfs filter-process"`);
+    await CloudRunnerSystem.Run(`git lfs pull`);
+    RemoteClientLogger.log(`pulled latest LFS files`);
+    assert(fs.existsSync(CloudRunnerFolders.lfsFolderAbsolute));
+  }
+
+  @CliFunction(`remote-cli`, `sets up a repository, usually before a game-ci build`)
+  static async runRemoteClientJob() {
+    const buildParameter = JSON.parse(process.env.BUILD_PARAMETERS || '{}');
+    RemoteClientLogger.log(`Build Params:
+      ${JSON.stringify(buildParameter, undefined, 4)}
+    `);
+    CloudRunner.buildParameters = buildParameter;
+    await RemoteClient.bootstrapRepository();
+  }
+}
--- a/src/model/cloud-runner/remote-client/remote-client-logger.ts
+++ b/src/model/cloud-runner/remote-client/remote-client-logger.ts
@@ -0,0 +1,19 @@
+import CloudRunnerLogger from '../services/cloud-runner-logger';
+
+export class RemoteClientLogger {
+  public static log(message: string) {
+    CloudRunnerLogger.log(`[Client] ${message}`);
+  }
+
+  public static logCliError(message: string) {
+    CloudRunnerLogger.log(`[Client][Error] ${message}`);
+  }
+
+  public static logCliDiagnostic(message: string) {
+    CloudRunnerLogger.log(`[Client][Diagnostic] ${message}`);
+  }
+
+  public static logWarning(message) {
+    CloudRunnerLogger.logWarning(message);
+  }
+}