Skip to content

Commit 2f0c972

Browse files
committed
dashboard/app: use machines with more memory for batch processing
8 cores, 12 gig machines can't process quarter long aggregation for some reason. The process requires only 1 gig and do the stream processing. It is something else I don't fully understand. Local experiments show high memory consumptions by the git processes.
1 parent 1c65791 commit 2f0c972

File tree

1 file changed

+9
-3
lines changed

1 file changed

+9
-3
lines changed

dashboard/app/batch_main.go

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ func createScriptJob(ctx context.Context, projectID, jobNamePrefix, script strin
4646
ComputeResource: &batchpb.ComputeResource{
4747
// CpuMilli is milliseconds per cpu-second. This means the task requires 8 whole CPUs.
4848
CpuMilli: 8000,
49-
MemoryMib: 12 * 1024,
49+
MemoryMib: 30 * 1024,
5050
},
5151
MaxRunDuration: &durationpb.Duration{
5252
Seconds: timeout,
@@ -62,8 +62,14 @@ func createScriptJob(ctx context.Context, projectID, jobNamePrefix, script strin
6262
Instances: []*batchpb.AllocationPolicy_InstancePolicyOrTemplate{{
6363
PolicyTemplate: &batchpb.AllocationPolicy_InstancePolicyOrTemplate_Policy{
6464
Policy: &batchpb.AllocationPolicy_InstancePolicy{
65-
ProvisioningModel: batchpb.AllocationPolicy_SPOT,
66-
MachineType: "c3-highcpu-8",
65+
// Spot machines may also be used here.
66+
// The problem with spot machines is the high probability of preemption on long runs.
67+
// Quarter long coverage merge costs 4 hours on 8 cores and we trigger it only once/week.
68+
// TODO(tarasmadan): switch back to the spot machines in combination with higher retry count
69+
ProvisioningModel: batchpb.AllocationPolicy_STANDARD,
70+
// Quarter long aggregation OOMs on machine with 16 gigs and 8 cores.
71+
// It is important to use machine with min 32 gigs for 8 cores.
72+
MachineType: "e2-standard-8",
6773
},
6874
},
6975
InstallOpsAgent: true,

0 commit comments

Comments
 (0)