Cloud runner v0.2 - continued quality of life improvements (#387)
* Update cloud-runner-aws-pipeline.yml * Update cloud-runner-k8s-pipeline.yml * yarn build * yarn build * correct branch ref * correct branch ref passed to target repo * Create k8s-tests.yml * Delete k8s-tests.yml * correct branch ref passed to target repo * correct branch ref passed to target repo * Always describe AWS tasks for now, because unstable error handling * Remove unused tree commands * Use lfs guid sum * Simple override cache push * Simple override cache push and pull override to allow pure cloud storage driven caching * Removal of early branch (breaks lfs caching) * Remove unused tree commands * Update action.yml * Update action.yml * Support cache and input override commands as input + full support custom hooks * Increase k8s timeout * replace filename being appended for unknclear reason * cache key should not contain whitespaces * Always try and deploy rook for k8s * Apply k8s files for rook * Update action.yml * Apply k8s files for rook * Apply k8s files for rook * cache test and action description for kuber storage class * Correct test and implement dependency health check and start * GCP-secret run, cache key * lfs smudge set explicit and undo explicit * Run using external secret provider to speed up input * Update cloud-runner-aws-pipeline.yml * Add nodejs as build step dependency * Add nodejs as build step dependency * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Cloud Runner Tests must be specified to capture logs from cloud runner for tests * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * Refactor and cleanup - no async input, combined setup/build, removed github logs for cli runs * better defaults for new inputs * better defaults * merge latest * force build update * use npm n to update node in unity builder * use npm n to update node in unity builder * use npm n to update node in unity builder * correct new line * quiet zipping * quiet zipping * default secrets for unity username and password * default secrets for unity username and password * ls active directory before lfs install * Get cloud runner secrets from * Get cloud runner secrets from * Cleanup setup of default secrets * Various fixes * Cleanup setup of default secrets * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * Various fixes * AWS secrets manager support * less caching logs * default k8s storage class to pd-standard * more readable build commands * Capture aws exit code 1 reliably * Always replace /head from branch * k8s default storage class to standard-rwo * cleanup * further cleanup input * further cleanup input * further cleanup input * further cleanup input * further cleanup input * folder sizes to inspect caching * dir command for local cloud runner test * k8s wait for pending because pvc will not create earlier * prefer k8s standard storage * handle empty string as cloud runner cluster input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * local-system is now used for cloud runner test implementation AND correctly unset test CLI input * fix unterminated quote * fix unterminated quote * do not share build parameters in tests - in cloud runner this will cause conflicts with resouces of the same name * remove head and heads from branch prefix * fix reversed caching direction of cache-push * fixes * fixes * fixes * cachePull cli * fixes * fixes * fixes * fixes * fixes * order cache test to be first * order cache test to be first * fixes * populate cache key instead of using branch * cleanup cli * garbage-collect-aws cli can iterate over aws resources and cli scans all ts files * import cli methods * import cli files explicitly * import cli files explicitly * import cli files explicitly * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * import cli methods * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * log parameters in cloud runner parameter test * Cloud runner param test before caching because we have a fast local cache test now * Using custom build path relative to repo root rather than project root * aws-garbage-collect at end of pipeline * aws-garbage-collect do not actually delete anything for now - just list * remove some legacy du commands * Update cloud-runner-aws-pipeline.yml * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * log contents after cache pull and fix some scenarios with duplicate secrets * PR comments * Replace guid with uuid package * use fileExists lambda instead of stat to check file exists in caching * build failed results in core error message * Delete sample.txt * cloud-runner-system prefix changed to cloud-runner * Update cloud-runner-aws-pipeline.yml * remove du from caching, should run manually if interested in size, adds too much runtime to job to include by default * github ephemeral pipeline support * github ephemeral pipeline support * Merge remote-tracking branch 'origin/main' into cloud-runner-develop # Conflicts: # dist/index.js.map # src/model/cloud-runner/providers/aws/aws-task-runner.ts # src/model/cloud-runner/providers/aws/index.ts * garbage collection * garbage collection * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * self hosted runner pipeline * ephemeral runner pipeline * ephemeral runner pipeline * ephemeral runner pipeline * download runner each time * download runner each time * download runner each time * garbage collect all older than 1d as part of cleanup * download runner each time * number container cpu and memory for aws * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * Skip printing size unless cloudRunnerIntegrationTests is true * transition zip usage in cache to uncompressed tar for speed * transition zip usage in cache to uncompressed tar for speed * transition zip usage in cache to uncompressed tar for speed * transition zip usage in cache to uncompressed tar for speed * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * per provider container defaults * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * baked in cloud formation template * better aws commands * better aws commands * parse number for cloud formation template * remove container resource defaults from actions yaml * remove container resource defaults from actions yaml * skip all input readers when cloud runner is local * prefer fs/promises * actually set aws cloud runner step as failure if unity build fails * default to 3gb of ram - webgl fails on 2
This commit is contained in:
Frostebite
GitHub
@@ -1,8 +1,7 @@
|
||||
import CloudRunnerLogger from '../../services/cloud-runner-logger';
|
||||
import * as core from '@actions/core';
|
||||
import * as SDK from 'aws-sdk';
|
||||
import * as fs from 'fs';
|
||||
import path from 'path';
|
||||
import { BaseStackFormation } from './cloud-formations/base-stack-formation';
|
||||
const crypto = require('crypto');
|
||||
|
||||
export class AWSBaseStack {
|
||||
@@ -14,7 +13,7 @@ export class AWSBaseStack {
|
||||
async setupBaseStack(CF: SDK.CloudFormation) {
|
||||
const baseStackName = this.baseStackName;
|
||||
|
||||
const baseStack = fs.readFileSync(path.join(__dirname, 'cloud-formations', 'base-setup.yml'), 'utf8');
|
||||
const baseStack = BaseStackFormation.formation;
|
||||
|
||||
// Cloud Formation Input
|
||||
const describeStackInput: SDK.CloudFormation.DescribeStacksInput = {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import * as fs from 'fs';
|
||||
import { TaskDefinitionFormation } from './cloud-formations/task-definition-formation';
|
||||
|
||||
export class AWSCloudFormationTemplates {
|
||||
public static getParameterTemplate(p1) {
|
||||
@@ -34,6 +34,6 @@ export class AWSCloudFormationTemplates {
|
||||
}
|
||||
|
||||
public static readTaskCloudFormationTemplate(): string {
|
||||
return fs.readFileSync(`${__dirname}/cloud-formations/task-def-formation.yml`, 'utf8');
|
||||
return TaskDefinitionFormation.formation;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import CloudRunnerSecret from '../../services/cloud-runner-secret';
|
||||
import { AWSCloudFormationTemplates } from './aws-cloud-formation-templates';
|
||||
import CloudRunnerLogger from '../../services/cloud-runner-logger';
|
||||
import { AWSError } from './aws-error';
|
||||
import CloudRunner from '../../cloud-runner';
|
||||
|
||||
export class AWSJobStack {
|
||||
private baseStackName: string;
|
||||
@@ -23,6 +24,20 @@ export class AWSJobStack {
|
||||
): Promise<CloudRunnerAWSTaskDef> {
|
||||
const taskDefStackName = `${this.baseStackName}-${buildGuid}`;
|
||||
let taskDefCloudFormation = AWSCloudFormationTemplates.readTaskCloudFormationTemplate();
|
||||
const cpu = CloudRunner.buildParameters.cloudRunnerCpu || '1024';
|
||||
const memory = CloudRunner.buildParameters.cloudRunnerMemory || '3072';
|
||||
taskDefCloudFormation = taskDefCloudFormation.replace(
|
||||
`ContainerCpu:
|
||||
Default: 1024`,
|
||||
`ContainerCpu:
|
||||
Default: ${Number.parseInt(cpu)}`,
|
||||
);
|
||||
taskDefCloudFormation = taskDefCloudFormation.replace(
|
||||
`ContainerMemory:
|
||||
Default: 2048`,
|
||||
`ContainerMemory:
|
||||
Default: ${Number.parseInt(memory)}`,
|
||||
);
|
||||
for (const secret of secrets) {
|
||||
secret.ParameterKey = `${buildGuid.replace(/[^\dA-Za-z]/g, '')}${secret.ParameterKey.replace(
|
||||
/[^\dA-Za-z]/g,
|
||||
@@ -85,7 +100,9 @@ export class AWSJobStack {
|
||||
},
|
||||
...secretsMappedToCloudFormationParameters,
|
||||
];
|
||||
|
||||
CloudRunnerLogger.log(
|
||||
`Starting AWS job with memory: ${CloudRunner.buildParameters.cloudRunnerMemory} cpu: ${CloudRunner.buildParameters.cloudRunnerCpu}`,
|
||||
);
|
||||
let previousStackExists = true;
|
||||
while (previousStackExists) {
|
||||
previousStackExists = false;
|
||||
@@ -101,14 +118,15 @@ export class AWSJobStack {
|
||||
}
|
||||
}
|
||||
}
|
||||
const createStackInput: SDK.CloudFormation.CreateStackInput = {
|
||||
StackName: taskDefStackName,
|
||||
TemplateBody: taskDefCloudFormation,
|
||||
Capabilities: ['CAPABILITY_IAM'],
|
||||
Parameters: parameters,
|
||||
};
|
||||
|
||||
try {
|
||||
await CF.createStack({
|
||||
StackName: taskDefStackName,
|
||||
TemplateBody: taskDefCloudFormation,
|
||||
Capabilities: ['CAPABILITY_IAM'],
|
||||
Parameters: parameters,
|
||||
}).promise();
|
||||
await CF.createStack(createStackInput).promise();
|
||||
CloudRunnerLogger.log('Creating cloud runner job');
|
||||
await CF.waitFor('stackCreateComplete', { StackName: taskDefStackName }).promise();
|
||||
} catch (error) {
|
||||
|
||||
@@ -58,14 +58,21 @@ class AWSTaskRunner {
|
||||
CloudRunnerLogger.log(
|
||||
`Cloud runner job status is running ${(await AWSTaskRunner.describeTasks(ECS, cluster, taskArn))?.lastStatus}`,
|
||||
);
|
||||
const output = await this.streamLogsUntilTaskStops(ECS, CF, taskDef, cluster, taskArn, streamName);
|
||||
const { output, shouldCleanup } = await this.streamLogsUntilTaskStops(
|
||||
ECS,
|
||||
CF,
|
||||
taskDef,
|
||||
cluster,
|
||||
taskArn,
|
||||
streamName,
|
||||
);
|
||||
const taskData = await AWSTaskRunner.describeTasks(ECS, cluster, taskArn);
|
||||
const exitCode = taskData.containers?.[0].exitCode;
|
||||
const wasSuccessful = exitCode === 0 || (exitCode === undefined && taskData.lastStatus === 'RUNNING');
|
||||
if (wasSuccessful) {
|
||||
CloudRunnerLogger.log(`Cloud runner job has finished successfully`);
|
||||
|
||||
return output;
|
||||
return { output, shouldCleanup };
|
||||
} else {
|
||||
if (taskData.stoppedReason === 'Essential container in task exited' && exitCode === 1) {
|
||||
throw new Error('Container exited with code 1');
|
||||
@@ -122,22 +129,24 @@ class AWSTaskRunner {
|
||||
const logBaseUrl = `https://${Input.region}.console.aws.amazon.com/cloudwatch/home?region=${CF.config.region}#logsV2:log-groups/log-group/${taskDef.taskDefStackName}`;
|
||||
CloudRunnerLogger.log(`You can also see the logs at AWS Cloud Watch: ${logBaseUrl}`);
|
||||
let shouldReadLogs = true;
|
||||
let shouldCleanup = true;
|
||||
let timestamp: number = 0;
|
||||
let output = '';
|
||||
while (shouldReadLogs) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 1500));
|
||||
const taskData = await AWSTaskRunner.describeTasks(ECS, clusterName, taskArn);
|
||||
({ timestamp, shouldReadLogs } = AWSTaskRunner.checkStreamingShouldContinue(taskData, timestamp, shouldReadLogs));
|
||||
({ iterator, shouldReadLogs, output } = await AWSTaskRunner.handleLogStreamIteration(
|
||||
({ iterator, shouldReadLogs, output, shouldCleanup } = await AWSTaskRunner.handleLogStreamIteration(
|
||||
kinesis,
|
||||
iterator,
|
||||
shouldReadLogs,
|
||||
taskDef,
|
||||
output,
|
||||
shouldCleanup,
|
||||
));
|
||||
}
|
||||
|
||||
return output;
|
||||
return { output, shouldCleanup };
|
||||
}
|
||||
|
||||
private static async handleLogStreamIteration(
|
||||
@@ -146,6 +155,7 @@ class AWSTaskRunner {
|
||||
shouldReadLogs: boolean,
|
||||
taskDef: CloudRunnerAWSTaskDef,
|
||||
output: string,
|
||||
shouldCleanup: boolean,
|
||||
) {
|
||||
const records = await kinesis
|
||||
.getRecords({
|
||||
@@ -153,9 +163,16 @@ class AWSTaskRunner {
|
||||
})
|
||||
.promise();
|
||||
iterator = records.NextShardIterator || '';
|
||||
({ shouldReadLogs, output } = AWSTaskRunner.logRecords(records, iterator, taskDef, shouldReadLogs, output));
|
||||
({ shouldReadLogs, output, shouldCleanup } = AWSTaskRunner.logRecords(
|
||||
records,
|
||||
iterator,
|
||||
taskDef,
|
||||
shouldReadLogs,
|
||||
output,
|
||||
shouldCleanup,
|
||||
));
|
||||
|
||||
return { iterator, shouldReadLogs, output };
|
||||
return { iterator, shouldReadLogs, output, shouldCleanup };
|
||||
}
|
||||
|
||||
private static checkStreamingShouldContinue(taskData: AWS.ECS.Task, timestamp: number, shouldReadLogs: boolean) {
|
||||
@@ -183,6 +200,7 @@ class AWSTaskRunner {
|
||||
taskDef: CloudRunnerAWSTaskDef,
|
||||
shouldReadLogs: boolean,
|
||||
output: string,
|
||||
shouldCleanup: boolean,
|
||||
) {
|
||||
if (records.Records.length > 0 && iterator) {
|
||||
for (let index = 0; index < records.Records.length; index++) {
|
||||
@@ -197,11 +215,18 @@ class AWSTaskRunner {
|
||||
shouldReadLogs = false;
|
||||
} else if (message.includes('Rebuilding Library because the asset database could not be found!')) {
|
||||
core.warning('LIBRARY NOT FOUND!');
|
||||
core.setOutput('library-found', 'false');
|
||||
} else if (message.includes('Build succeeded')) {
|
||||
core.setOutput('build-result', 'success');
|
||||
} else if (message.includes('Build fail')) {
|
||||
core.setOutput('build-result', 'failed');
|
||||
core.setFailed('unity build failed');
|
||||
core.error('BUILD FAILED!');
|
||||
} else if (message.includes(': Listening for Jobs')) {
|
||||
core.setOutput('cloud runner stop watching', 'true');
|
||||
shouldReadLogs = false;
|
||||
shouldCleanup = false;
|
||||
core.warning('cloud runner stop watching');
|
||||
}
|
||||
message = `[${CloudRunnerStatics.logPrefix}] ${message}`;
|
||||
if (CloudRunner.buildParameters.cloudRunnerIntegrationTests) {
|
||||
@@ -213,7 +238,7 @@ class AWSTaskRunner {
|
||||
}
|
||||
}
|
||||
|
||||
return { shouldReadLogs, output };
|
||||
return { shouldReadLogs, output, shouldCleanup };
|
||||
}
|
||||
|
||||
private static async getLogStream(kinesis: AWS.Kinesis, kinesisStreamName: string) {
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
AWSTemplateFormatVersion: '2010-09-09'
|
||||
Description: AWS Fargate cluster that can span public and private subnets. Supports
|
||||
public facing load balancers, private internal load balancers, and
|
||||
both internal and external service discovery namespaces.
|
||||
export class BaseStackFormation {
|
||||
public static readonly formation: string = `AWSTemplateFormatVersion: '2010-09-09'
|
||||
Description: Game-CI base stack
|
||||
Parameters:
|
||||
EnvironmentName:
|
||||
Type: String
|
||||
@@ -335,57 +334,58 @@ Outputs:
|
||||
Description: 'The connection endpoint for the database.'
|
||||
Value: !Ref EfsFileStorage
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:EfsFileStorageId
|
||||
Name: !Sub ${'${EnvironmentName}'}:EfsFileStorageId
|
||||
ClusterName:
|
||||
Description: The name of the ECS cluster
|
||||
Value: !Ref 'ECSCluster'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:ClusterName
|
||||
Name: !Sub${' ${EnvironmentName}'}:ClusterName
|
||||
AutoscalingRole:
|
||||
Description: The ARN of the role used for autoscaling
|
||||
Value: !GetAtt 'AutoscalingRole.Arn'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:AutoscalingRole
|
||||
Name: !Sub ${'${EnvironmentName}'}:AutoscalingRole
|
||||
ECSRole:
|
||||
Description: The ARN of the ECS role
|
||||
Value: !GetAtt 'ECSRole.Arn'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:ECSRole
|
||||
Name: !Sub ${'${EnvironmentName}'}:ECSRole
|
||||
ECSTaskExecutionRole:
|
||||
Description: The ARN of the ECS role tsk execution role
|
||||
Value: !GetAtt 'ECSTaskExecutionRole.Arn'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:ECSTaskExecutionRole
|
||||
Name: !Sub ${'${EnvironmentName}'}:ECSTaskExecutionRole
|
||||
|
||||
DeleteCFNLambdaExecutionRole:
|
||||
Description: Lambda execution role for cleaning up cloud formations
|
||||
Value: !GetAtt 'DeleteCFNLambdaExecutionRole.Arn'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:DeleteCFNLambdaExecutionRole
|
||||
Name: !Sub ${'${EnvironmentName}'}:DeleteCFNLambdaExecutionRole
|
||||
|
||||
CloudWatchIAMRole:
|
||||
Description: The ARN of the CloudWatch role for subscription filter
|
||||
Value: !GetAtt 'CloudWatchIAMRole.Arn'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:CloudWatchIAMRole
|
||||
Name: !Sub ${'${EnvironmentName}'}:CloudWatchIAMRole
|
||||
VpcId:
|
||||
Description: The ID of the VPC that this stack is deployed in
|
||||
Value: !Ref 'VPC'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:VpcId
|
||||
Name: !Sub ${'${EnvironmentName}'}:VpcId
|
||||
PublicSubnetOne:
|
||||
Description: Public subnet one
|
||||
Value: !Ref 'PublicSubnetOne'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:PublicSubnetOne
|
||||
Name: !Sub ${'${EnvironmentName}'}:PublicSubnetOne
|
||||
PublicSubnetTwo:
|
||||
Description: Public subnet two
|
||||
Value: !Ref 'PublicSubnetTwo'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:PublicSubnetTwo
|
||||
|
||||
Name: !Sub ${'${EnvironmentName}'}:PublicSubnetTwo
|
||||
ContainerSecurityGroup:
|
||||
Description: A security group used to allow Fargate containers to receive traffic
|
||||
Value: !Ref 'ContainerSecurityGroup'
|
||||
Export:
|
||||
Name: !Sub ${EnvironmentName}:ContainerSecurityGroup
|
||||
Name: !Sub ${'${EnvironmentName}'}:ContainerSecurityGroup
|
||||
`;
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
AWSTemplateFormatVersion: 2010-09-09
|
||||
export class TaskDefinitionFormation {
|
||||
public static readonly formation: string = `AWSTemplateFormatVersion: 2010-09-09
|
||||
Description: >-
|
||||
AWS Fargate cluster that can span public and private subnets. Supports public
|
||||
facing load balancers, private internal load balancers, and both internal and
|
||||
@@ -23,12 +24,12 @@ Parameters:
|
||||
Default: 80
|
||||
Description: What port number the application inside the docker container is binding to
|
||||
ContainerCpu:
|
||||
Type: Number
|
||||
Default: 1024
|
||||
Type: Number
|
||||
Description: How much CPU to give the container. 1024 is 1 CPU
|
||||
ContainerMemory:
|
||||
Type: Number
|
||||
Default: 2048
|
||||
Type: Number
|
||||
Description: How much memory in megabytes to give the container
|
||||
BUILDGUID:
|
||||
Type: String
|
||||
@@ -78,7 +79,7 @@ Resources:
|
||||
Properties:
|
||||
FilterPattern: ''
|
||||
RoleArn:
|
||||
'Fn::ImportValue': !Sub '${EnvironmentName}:CloudWatchIAMRole'
|
||||
'Fn::ImportValue': !Sub '${'${EnvironmentName}'}:CloudWatchIAMRole'
|
||||
LogGroupName: !Ref ServiceName
|
||||
DestinationArn:
|
||||
'Fn::GetAtt':
|
||||
@@ -98,9 +99,7 @@ Resources:
|
||||
Metadata:
|
||||
'AWS::CloudFormation::Designer':
|
||||
id: c6f18447-b879-4696-8873-f981b2cedd2b
|
||||
|
||||
# template secrets p2 - secret
|
||||
|
||||
TaskDefinition:
|
||||
Type: 'AWS::ECS::TaskDefinition'
|
||||
Properties:
|
||||
@@ -112,12 +111,12 @@ Resources:
|
||||
- Name: efs-data
|
||||
EFSVolumeConfiguration:
|
||||
FilesystemId:
|
||||
'Fn::ImportValue': !Sub '${EnvironmentName}:EfsFileStorageId'
|
||||
'Fn::ImportValue': !Sub '${'${EnvironmentName}'}:EfsFileStorageId'
|
||||
TransitEncryption: ENABLED
|
||||
RequiresCompatibilities:
|
||||
- FARGATE
|
||||
ExecutionRoleArn:
|
||||
'Fn::ImportValue': !Sub '${EnvironmentName}:ECSTaskExecutionRole'
|
||||
'Fn::ImportValue': !Sub '${'${EnvironmentName}'}:ECSTaskExecutionRole'
|
||||
TaskRoleArn:
|
||||
'Fn::If':
|
||||
- HasCustomRole
|
||||
@@ -153,69 +152,7 @@ Resources:
|
||||
awslogs-group: !Ref ServiceName
|
||||
awslogs-region: !Ref 'AWS::Region'
|
||||
awslogs-stream-prefix: !Ref ServiceName
|
||||
Metadata:
|
||||
'AWS::CloudFormation::Designer':
|
||||
id: dabb0116-abe0-48a6-a8af-cf9111c879a5
|
||||
DependsOn:
|
||||
- LogGroup
|
||||
Metadata:
|
||||
'AWS::CloudFormation::Designer':
|
||||
dabb0116-abe0-48a6-a8af-cf9111c879a5:
|
||||
size:
|
||||
width: 60
|
||||
height: 60
|
||||
position:
|
||||
x: 270
|
||||
'y': 90
|
||||
z: 1
|
||||
embeds: []
|
||||
dependson:
|
||||
- aece53ae-b82d-4267-bc16-ed964b05db27
|
||||
c6f18447-b879-4696-8873-f981b2cedd2b:
|
||||
size:
|
||||
width: 60
|
||||
height: 60
|
||||
position:
|
||||
x: 270
|
||||
'y': 210
|
||||
z: 1
|
||||
embeds: []
|
||||
7f809e91-9e5d-4678-98c1-c5085956c480:
|
||||
size:
|
||||
width: 60
|
||||
height: 60
|
||||
position:
|
||||
x: 60
|
||||
'y': 300
|
||||
z: 1
|
||||
embeds: []
|
||||
dependson:
|
||||
- aece53ae-b82d-4267-bc16-ed964b05db27
|
||||
- c6f18447-b879-4696-8873-f981b2cedd2b
|
||||
aece53ae-b82d-4267-bc16-ed964b05db27:
|
||||
size:
|
||||
width: 150
|
||||
height: 150
|
||||
position:
|
||||
x: 60
|
||||
'y': 90
|
||||
z: 1
|
||||
embeds: []
|
||||
4d2da56c-3643-46b8-aaee-e46e19f95fcc:
|
||||
source:
|
||||
id: 7f809e91-9e5d-4678-98c1-c5085956c480
|
||||
target:
|
||||
id: aece53ae-b82d-4267-bc16-ed964b05db27
|
||||
z: 11
|
||||
14eb957b-f094-4653-93c4-77b2f851953c:
|
||||
source:
|
||||
id: 7f809e91-9e5d-4678-98c1-c5085956c480
|
||||
target:
|
||||
id: c6f18447-b879-4696-8873-f981b2cedd2b
|
||||
z: 12
|
||||
85c57444-e5bb-4230-bc85-e545cd4558f6:
|
||||
source:
|
||||
id: dabb0116-abe0-48a6-a8af-cf9111c879a5
|
||||
target:
|
||||
id: aece53ae-b82d-4267-bc16-ed964b05db27
|
||||
z: 13
|
||||
`;
|
||||
}
|
||||
@@ -4,27 +4,89 @@ import Input from '../../../../input';
|
||||
import CloudRunnerLogger from '../../../services/cloud-runner-logger';
|
||||
|
||||
export class AwsCliCommands {
|
||||
@CliFunction(`aws-garbage-collect`, `garbage collect aws`)
|
||||
static async garbageCollectAws() {
|
||||
@CliFunction(`aws-list-stacks`, `List stacks`)
|
||||
static async awsListStacks(perResultCallback: any) {
|
||||
process.env.AWS_REGION = Input.region;
|
||||
CloudRunnerLogger.log(`Cloud Formation stacks`);
|
||||
const CF = new AWS.CloudFormation();
|
||||
const stacks =
|
||||
(await CF.listStacks().promise()).StackSummaries?.filter((_x) => _x.StackStatus !== 'DELETE_COMPLETE') || [];
|
||||
CloudRunnerLogger.log(`DescribeStacksRequest ${stacks.length}`);
|
||||
for (const element of stacks) {
|
||||
CloudRunnerLogger.log(JSON.stringify(element, undefined, 4));
|
||||
CloudRunnerLogger.log(`${element.StackName}`);
|
||||
if (perResultCallback) await perResultCallback(element);
|
||||
}
|
||||
CloudRunnerLogger.log(`ECS Clusters`);
|
||||
const ecs = new AWS.ECS();
|
||||
const clusters = (await ecs.listClusters().promise()).clusterArns || [];
|
||||
if (stacks === undefined) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
@CliFunction(`aws-list-tasks`, `List tasks`)
|
||||
static async awsListTasks(perResultCallback: any) {
|
||||
process.env.AWS_REGION = Input.region;
|
||||
CloudRunnerLogger.log(`ECS Clusters`);
|
||||
const ecs = new AWS.ECS();
|
||||
const clusters = (await ecs.listClusters().promise()).clusterArns || [];
|
||||
for (const element of clusters) {
|
||||
const input: AWS.ECS.ListTasksRequest = {
|
||||
cluster: element,
|
||||
};
|
||||
CloudRunnerLogger.log(JSON.stringify(await ecs.listTasks(input).promise(), undefined, 4));
|
||||
const list = (await ecs.listTasks(input).promise()).taskArns || [];
|
||||
if (list.length > 0) {
|
||||
const describeInput: AWS.ECS.DescribeTasksRequest = { tasks: list, cluster: element };
|
||||
const describeList = (await ecs.describeTasks(describeInput).promise()).tasks || [];
|
||||
if (describeList === []) {
|
||||
continue;
|
||||
}
|
||||
CloudRunnerLogger.log(`DescribeTasksRequest ${describeList.length}`);
|
||||
for (const taskElement of describeList) {
|
||||
if (taskElement === undefined) {
|
||||
continue;
|
||||
}
|
||||
taskElement.overrides = {};
|
||||
taskElement.attachments = [];
|
||||
CloudRunnerLogger.log(JSON.stringify(taskElement, undefined, 4));
|
||||
if (taskElement.createdAt === undefined) {
|
||||
CloudRunnerLogger.log(`Skipping ${taskElement.taskDefinitionArn} no createdAt date`);
|
||||
continue;
|
||||
}
|
||||
if (perResultCallback) await perResultCallback(taskElement, element);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@CliFunction(`aws-garbage-collect`, `garbage collect aws`)
|
||||
static async garbageCollectAws() {
|
||||
await AwsCliCommands.cleanup(false);
|
||||
}
|
||||
@CliFunction(`aws-garbage-collect-all`, `garbage collect aws`)
|
||||
static async garbageCollectAwsAll() {
|
||||
await AwsCliCommands.cleanup(true);
|
||||
}
|
||||
@CliFunction(`aws-garbage-collect-all-1d-older`, `garbage collect aws`)
|
||||
static async garbageCollectAwsAllOlderThanOneDay() {
|
||||
await AwsCliCommands.cleanup(true);
|
||||
}
|
||||
|
||||
private static async cleanup(deleteResources = false) {
|
||||
process.env.AWS_REGION = Input.region;
|
||||
const CF = new AWS.CloudFormation();
|
||||
const ecs = new AWS.ECS();
|
||||
await AwsCliCommands.awsListStacks(async (element) => {
|
||||
if (deleteResources) {
|
||||
if (element.StackName === 'game-ci' || element.TemplateDescription === 'Game-CI base stack') {
|
||||
CloudRunnerLogger.log(`Skipping ${element.StackName} ignore list`);
|
||||
|
||||
return;
|
||||
}
|
||||
const deleteStackInput: AWS.CloudFormation.DeleteStackInput = { StackName: element.StackName };
|
||||
await CF.deleteStack(deleteStackInput).promise();
|
||||
}
|
||||
});
|
||||
await AwsCliCommands.awsListTasks(async (taskElement, element) => {
|
||||
if (deleteResources) {
|
||||
await ecs.stopTask({ task: taskElement.taskArn || '', cluster: element }).promise();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -66,21 +66,24 @@ class AWSBuildEnvironment implements ProviderInterface {
|
||||
);
|
||||
|
||||
let postRunTaskTimeMs;
|
||||
let output = '';
|
||||
try {
|
||||
const postSetupStacksTimeMs = Date.now();
|
||||
CloudRunnerLogger.log(`Setup job time: ${Math.floor((postSetupStacksTimeMs - startTimeMs) / 1000)}s`);
|
||||
output = await AWSTaskRunner.runTask(taskDef, ECS, CF, environment, buildGuid, commands);
|
||||
const { output, shouldCleanup } = await AWSTaskRunner.runTask(taskDef, ECS, CF, environment, buildGuid, commands);
|
||||
postRunTaskTimeMs = Date.now();
|
||||
CloudRunnerLogger.log(`Run job time: ${Math.floor((postRunTaskTimeMs - postSetupStacksTimeMs) / 1000)}s`);
|
||||
} finally {
|
||||
await this.cleanupResources(CF, taskDef);
|
||||
if (shouldCleanup) {
|
||||
await this.cleanupResources(CF, taskDef);
|
||||
}
|
||||
const postCleanupTimeMs = Date.now();
|
||||
if (postRunTaskTimeMs !== undefined)
|
||||
CloudRunnerLogger.log(`Cleanup job time: ${Math.floor((postCleanupTimeMs - postRunTaskTimeMs) / 1000)}s`);
|
||||
}
|
||||
|
||||
return output;
|
||||
return output;
|
||||
} catch (error) {
|
||||
await this.cleanupResources(CF, taskDef);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async cleanupResources(CF: SDK.CloudFormation, taskDef: CloudRunnerAWSTaskDef) {
|
||||
|
||||
@@ -108,8 +108,8 @@ class KubernetesJobSpecFactory {
|
||||
workingDir: `${workingDirectory}`,
|
||||
resources: {
|
||||
requests: {
|
||||
memory: buildParameters.cloudRunnerMemory,
|
||||
cpu: buildParameters.cloudRunnerCpu,
|
||||
memory: buildParameters.cloudRunnerMemory || '750M',
|
||||
cpu: buildParameters.cloudRunnerCpu || '1',
|
||||
},
|
||||
},
|
||||
env: [
|
||||
|
||||
Reference in New Issue
Block a user