Skip to content
This repository was archived by the owner on Nov 1, 2023. It is now read-only.

Commit 909b130

Browse files
nharper285AdamL-Microsoftchkeitakananbmgreisen
authored
JobResult Table Re-design - Improved task- and machine- granularity (#3539)
* Release 8.7.1 (hotfix) (#3459) * Remove the retention policy setting (#3452) --------- Co-authored-by: Cheick Keita <[email protected]> * Revert "Release 8.7.1 (hotfix) (#3459)" (#3468) This reverts commit c69deed. * Redo 8.7.1 (#3469) * Redo-8.7.1-hotfix --------- Co-authored-by: Cheick Keita <[email protected]> * Support custom ado fields that mark work items as duplicate (#3467) * Add field to ado config for checking duplicate work items * Make duplicate fields nullable and add it to python models * Update broken tests * Update docs to include new ado_duplicate_fields property * Update readme with archive message (#3408) Co-authored-by: Adam <[email protected]> * Bump tokio from 1.30.0 to 1.32.0 in /src/proxy-manager (#3425) Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.30.0 to 1.32.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.30.0...tokio-1.32.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump tokio from 1.30.0 to 1.32.0 in /src/agent (#3424) Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.30.0 to 1.32.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](tokio-rs/tokio@tokio-1.30.0...tokio-1.32.0) --- updated-dependencies: - dependency-name: tokio dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove unnecessary method argument (#3473) * Bump elsa from 1.8.1 to 1.9.0 in /src/agent (#3411) Bumps [elsa](https://github.com/manishearth/elsa) from 1.8.1 to 1.9.0. - [Commits](Manishearth/elsa@v1.8.1...v1.9.0) --- updated-dependencies: - dependency-name: elsa dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump tempfile from 3.7.1 to 3.8.0 in /src/agent (#3437) Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0. - [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md) - [Commits](Stebalien/tempfile@v3.7.1...v3.8.0) --- updated-dependencies: - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump tempfile from 3.7.1 to 3.8.0 in /src/proxy-manager (#3436) Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.7.1 to 3.8.0. - [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md) - [Commits](Stebalien/tempfile@v3.7.1...v3.8.0) --- updated-dependencies: - dependency-name: tempfile dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Updating requirements.txt to accept >= onefuzztypes. (#3477) * Updating requirements.txt to accept >= onefuzztypes. * Trying to loosen restriction. * Bump notify from 6.0.1 to 6.1.1 in /src/agent (#3435) Bumps [notify](https://github.com/notify-rs/notify) from 6.0.1 to 6.1.1. - [Release notes](https://github.com/notify-rs/notify/releases) - [Changelog](https://github.com/notify-rs/notify/blob/main/CHANGELOG.md) - [Commits](notify-rs/notify@notify-6.0.1...notify-6.1.1) --- updated-dependencies: - dependency-name: notify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump azure_* crates (#3478) * Release 8.8.0 (#3466) * Release 8.8.0 * Bump clap from 4.3.21 to 4.4.2 in /src/agent (#3484) Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.3.21...v4.4.2) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gimli from 0.27.3 to 0.28.0 in /src/agent (#3414) Bumps [gimli](https://github.com/gimli-rs/gimli) from 0.27.3 to 0.28.0. - [Changelog](https://github.com/gimli-rs/gimli/blob/master/CHANGELOG.md) - [Commits](gimli-rs/gimli@0.27.3...0.28.0) --- updated-dependencies: - dependency-name: gimli dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump clap from 4.3.21 to 4.4.2 in /src/proxy-manager (#3474) Bumps [clap](https://github.com/clap-rs/clap) from 4.3.21 to 4.4.2. - [Release notes](https://github.com/clap-rs/clap/releases) - [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md) - [Commits](clap-rs/clap@v4.3.21...v4.4.2) --- updated-dependencies: - dependency-name: clap dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump winreg from 0.50.0 to 0.51.0 in /src/agent (#3434) Bumps [winreg](https://github.com/gentoo90/winreg-rs) from 0.50.0 to 0.51.0. - [Release notes](https://github.com/gentoo90/winreg-rs/releases) - [Changelog](https://github.com/gentoo90/winreg-rs/blob/master/CHANGELOG.md) - [Commits](gentoo90/winreg-rs@v0.50.0...v0.51.0) --- updated-dependencies: - dependency-name: winreg dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Adam <[email protected]> * Starting integration tests (#3438) * Starting integration tests * Ready to test the test * Parametrize test * checkpoint * Test works * Run integration tests in pipeline * fmt * . * -p * Install clang * quotes not required in yaml? * Hopefully fixed windows? * Try without killondrop * lint * small test * another test * Reuse core name * Wrong step * bump tokio? * Try with rust * make build happy * Bump pete and small clean up * Clean up and make the test pass regularly * fix broken ci * Lower the poll timeout * Set the timeout in a nicer way * fix windows * fmt * Include and copy pdbs * Ignore if pdb is missing on linux * It takes too long for coverage to be generated * lint * Only warn on missing coverage since it's flaky * Fix windows build * Small clean up * Try lowering the poll delay * fix coverage * PR comments * . * Apparently make is missing? * Remove aggressive step skipping in CI * Fix sed checks for CLI versioning (#3486) * Fix sed checks for CLI versioning * Fix. * Fix. * Changing build_cli * Trying greater than * Tring once more. * Trying major minor * trying to replace major minor * Using major minor * Bump bytes from 1.4.0 to 1.5.0 in /src/agent (#3488) Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](tokio-rs/bytes@v1.4.0...v1.5.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve area/iteration path validation (#3489) * Add more comprehensive checks and better error messages to area/iteration path validation * Join invalid chars with space instead of comma * Make tree path validation more testable * Add error code for invalid ADO project in config * Write unit tests for tree path validation * Format tree path unit tests * Merge escape character and control character checks and clarify error message * Improve handling of unexpected breakpoints (#3493) * Improve handling of unexpected breakpoints * fmt * Update azure_* crates (#3503) * Fuzz coverage recording (#3322) * Fuzz coverage recording * Update cargo.toml * Update src/agent/coverage/fuzz/fuzz_targets/fuzz_target_record_coverage.rs Co-authored-by: George Pollard <[email protected]> * Fix fuzz --------- Co-authored-by: George Pollard <[email protected]> * Reporting coverage on task start up (#3502) * Reporting coverage on task start up * Moving metric up. * Remove feature flag from heartbeat metrics. (#3505) * Update archive notice. (#3507) * Add onefuzz service version to job created events (#3504) * Tevoinea/add version checking in local tasks (#3517) * Compare task version to service version * Swallow output when looking for appropriate name * Create directories if they don't exist in the template (#3522) * Create directories if they don't exist in the template * fmt * Support for retention policies on containers (#3501) - [x] ability to specify a retention period on a container, which applies to newly-created blobs - [x] specify default retention periods in templates from CLI side There's a small breaking change to the Python JobHelper class. * Bump rayon from 1.7.0 to 1.8.0 in /src/agent (#3520) Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.7.0 to 1.8.0. - [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md) - [Commits](rayon-rs/rayon@rayon-core-v1.7.0...rayon-core-v1.8.0) --- updated-dependencies: - dependency-name: rayon dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump insta from 1.31.0 to 1.32.0 in /src/agent (#3521) Bumps [insta](https://github.com/mitsuhiko/insta) from 1.31.0 to 1.32.0. - [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md) - [Commits](mitsuhiko/insta@1.31.0...1.32.0) --- updated-dependencies: - dependency-name: insta dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Disable `repro` and `debug` VM CLI commands. (#3494) * Disable and VM CLI commands. * Formatting. * More formatting. * More formatting. * Removing Repro check. * Make modules case insenstive on windows (#3527) * Make modules and coverage allowlist case insensitive on Windows * Tests and fmt * PR comments * fmt * Debugging missing file coverage * fmt * Broken linux test * Add a case insensitive transformer for better perf * cargo fix * Update windows interceptor list (#3528) * Template creation command (#3531) * Tasks are selectable * Almost there * It works * fmt * remove dead code * Remove unnecessary comments * Improve instructions * fix bug * Add some dummy values for paths * Terminate process on timeout in windows for the coverage task (#3529) * Terminate process on timeout in windows for the coverage task * set the timeout before we start the debugger * split the target launch from the debugger initialization wait for the process to finish on a separate thread * fix build * move comments * Ignore regression update when the work item is in some states (#3532) * Ignore regression update when the work item is in some states * format * formatting * don't hide messages in the poison queue * fix typo * update regression logic update test_template to support regression * build fix * mypy fix * build fix * move regression ignore state under ADODuplicateTemplate * replace extend with append * update set_tcp_keepalive * mke mypy happy * copy ADODuplicateTemplate.OnDuplicate.RegressionIgnoreStates * Updating IterationCount to be Task-based. * Changing to machine_id based * Fixing repro event name. * Updating iteration pr. * Single entry results. * Retry. * trying with unique guid. * Generic string type. * putting it back. * removing old update code. * removing comment. * Attempting to partition on task_id, machine_id, and event type. * Using replace and update. * Add logging statement. * UPdating such that we query. * attempting to try update. * Trying different update mechanism. * Checking previous value. * cleanup. * Removing old model. * Case guard for better readability. * Fix import ordering. * Removing duplicate code. * accidentally didn't include update. * Constructing the wrong way. * Moving back to standard switch. * Removing old code. * Removing more old code. * Using constants. * Addressing comments. * Corrected. * Updating. * Comment. * Replacing with Update to deal with edge cases. * Using timestamp. * Adding CreatedAt field. * Comparing timestamps. * Setting propery. * Adding check for CreatedAt. * Making created_at optional. * Trying again. * Remove. * Add log statement. * Remove function handle.: * Updating * Concat names * Set version. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Adam <[email protected]> Co-authored-by: Cheick Keita <[email protected]> Co-authored-by: Kanan B <[email protected]> Co-authored-by: Marc Greisen <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: George Pollard <[email protected]> Co-authored-by: Teo Voinea <[email protected]> Co-authored-by: George Pollard <[email protected]>
1 parent 0390822 commit 909b130

File tree

6 files changed

+87
-106
lines changed

6 files changed

+87
-106
lines changed

‎src/ApiService/ApiService/Functions/QueueJobResult.cs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,12 @@ public async Async.Task Run([QueueTrigger("job-result", Connection = "AzureWebJo
3131

3232
var job = await _jobs.Get(task.JobId);
3333
if (job == null) {
34-
_log.LogWarning("invalid {JobId}", task.JobId);
34+
_log.LogWarning("invalid message {JobId}", task.JobId);
35+
return;
36+
}
37+
38+
if (jr.CreatedAt == null) {
39+
_log.LogWarning("invalid message, no created_at field {JobId}", task.JobId);
3540
return;
3641
}
3742

@@ -52,7 +57,7 @@ public async Async.Task Run([QueueTrigger("job-result", Connection = "AzureWebJo
5257
return;
5358
}
5459

55-
var jobResult = await _context.JobResultOperations.CreateOrUpdate(job.JobId, jobResultType, value);
60+
var jobResult = await _context.JobResultOperations.CreateOrUpdate(job.JobId, jr.TaskId, jr.MachineId, jr.CreatedAt.Value, jr.Version, jobResultType, value);
5661
if (!jobResult.IsOk) {
5762
_log.LogError("failed to create or update with job result {JobId}", job.JobId);
5863
}

‎src/ApiService/ApiService/OneFuzzTypes/Model.cs‎

Lines changed: 18 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -34,19 +34,6 @@ public enum HeartbeatType {
3434
TaskAlive,
3535
}
3636

37-
[SkipRename]
38-
public enum JobResultType {
39-
NewCrashingInput,
40-
NoReproCrashingInput,
41-
NewReport,
42-
NewUniqueReport,
43-
NewRegressionReport,
44-
NewCoverage,
45-
NewCrashDump,
46-
CoverageData,
47-
RuntimeStats,
48-
}
49-
5037
public record HeartbeatData(HeartbeatType Type);
5138

5239
public record TaskHeartbeatEntry(
@@ -55,12 +42,14 @@ public record TaskHeartbeatEntry(
5542
[property: Required] Guid MachineId,
5643
HeartbeatData[] Data);
5744

58-
public record JobResultData(JobResultType Type);
45+
public record JobResultData(string Type);
5946

6047
public record TaskJobResultEntry(
6148
Guid TaskId,
6249
Guid? JobId,
6350
Guid MachineId,
51+
DateTime? CreatedAt,
52+
double Version,
6453
JobResultData Data,
6554
Dictionary<string, double> Value
6655
);
@@ -921,26 +910,24 @@ public record SecretAddress<T>(Uri Url) : ISecret<T> {
921910
public record SecretData<T>(ISecret<T> Secret) {
922911
}
923912

913+
[SkipRename]
914+
public enum JobResultType {
915+
CoverageData,
916+
RuntimeStats,
917+
}
918+
924919
public record JobResult(
925-
[PartitionKey][RowKey] Guid JobId,
920+
[PartitionKey] Guid JobId,
921+
[RowKey] string TaskIdMachineIdMetric,
922+
Guid TaskId,
923+
Guid MachineId,
924+
DateTime CreatedAt,
926925
string Project,
927926
string Name,
928-
double NewCrashingInput = 0,
929-
double NoReproCrashingInput = 0,
930-
double NewReport = 0,
931-
double NewUniqueReport = 0,
932-
double NewRegressionReport = 0,
933-
double NewCrashDump = 0,
934-
double InstructionsCovered = 0,
935-
double TotalInstructions = 0,
936-
double CoverageRate = 0,
937-
double IterationCount = 0
938-
) : EntityBase() {
939-
public JobResult(Guid JobId, string Project, string Name) : this(
940-
JobId: JobId,
941-
Project: Project,
942-
Name: Name, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) { }
943-
}
927+
string Type,
928+
double Version,
929+
Dictionary<string, double> MetricValue
930+
) : EntityBase();
944931

945932
public record JobConfig(
946933
string Project,

‎src/ApiService/ApiService/onefuzzlib/JobResultOperations.cs‎

Lines changed: 47 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -2,99 +2,75 @@
22
using Microsoft.Extensions.Logging;
33
using Polly;
44
namespace Microsoft.OneFuzz.Service;
5+
using System.Net;
56

67
public interface IJobResultOperations : IOrm<JobResult> {
78

8-
Async.Task<JobResult?> GetJobResult(Guid jobId);
9-
Async.Task<OneFuzzResultVoid> CreateOrUpdate(Guid jobId, JobResultType resultType, Dictionary<string, double> resultValue);
9+
Async.Task<JobResult?> GetJobResult(Guid jobId, Guid taskId, Guid machineId, string metricType);
10+
Async.Task<OneFuzzResultVoid> CreateOrUpdate(Guid jobId, Guid taskId, Guid machineId, DateTime createdAt, double version, string resultType, Dictionary<string, double> resultValue);
1011

1112
}
1213
public class JobResultOperations : Orm<JobResult>, IJobResultOperations {
1314

15+
const string COVERAGE_DATA = "CoverageData";
16+
const string RUNTIME_STATS = "RuntimeStats";
17+
1418
public JobResultOperations(ILogger<JobResultOperations> log, IOnefuzzContext context)
1519
: base(log, context) {
1620
}
1721

18-
public async Async.Task<JobResult?> GetJobResult(Guid jobId) {
19-
return await SearchByPartitionKeys(new[] { jobId.ToString() }).SingleOrDefaultAsync();
22+
public async Async.Task<JobResult?> GetJobResult(Guid jobId, Guid taskId, Guid machineId, string metricType) {
23+
var data = QueryAsync(Query.SingleEntity(jobId.ToString(), string.Concat(taskId, "-", machineId, "-", metricType)));
24+
return await data.FirstOrDefaultAsync();
2025
}
2126

22-
private JobResult UpdateResult(JobResult result, JobResultType type, Dictionary<string, double> resultValue) {
23-
24-
var newResult = result;
25-
double newValue;
26-
switch (type) {
27-
case JobResultType.NewCrashingInput:
28-
newValue = result.NewCrashingInput + resultValue["count"];
29-
newResult = result with { NewCrashingInput = newValue };
30-
break;
31-
case JobResultType.NewReport:
32-
newValue = result.NewReport + resultValue["count"];
33-
newResult = result with { NewReport = newValue };
34-
break;
35-
case JobResultType.NewUniqueReport:
36-
newValue = result.NewUniqueReport + resultValue["count"];
37-
newResult = result with { NewUniqueReport = newValue };
38-
break;
39-
case JobResultType.NewRegressionReport:
40-
newValue = result.NewRegressionReport + resultValue["count"];
41-
newResult = result with { NewRegressionReport = newValue };
42-
break;
43-
case JobResultType.NewCrashDump:
44-
newValue = result.NewCrashDump + resultValue["count"];
45-
newResult = result with { NewCrashDump = newValue };
46-
break;
47-
case JobResultType.CoverageData:
48-
double newCovered = resultValue["covered"];
49-
double newTotalCovered = resultValue["features"];
50-
double newCoverageRate = resultValue["rate"];
51-
newResult = result with { InstructionsCovered = newCovered, TotalInstructions = newTotalCovered, CoverageRate = newCoverageRate };
52-
break;
53-
case JobResultType.RuntimeStats:
54-
double newTotalIterations = resultValue["total_count"];
55-
newResult = result with { IterationCount = newTotalIterations };
56-
break;
57-
default:
58-
_logTracer.LogWarning($"Invalid Field {type}.");
59-
break;
60-
}
61-
_logTracer.LogInformation($"Attempting to log new result: {newResult}");
62-
return newResult;
63-
}
64-
65-
private async Async.Task<bool> TryUpdate(Job job, JobResultType resultType, Dictionary<string, double> resultValue) {
27+
private async Async.Task<bool> TryUpdate(Job job, Guid taskId, Guid machineId, DateTime createdAt, double version, string resultType, Dictionary<string, double> resultValue) {
6628
var jobId = job.JobId;
29+
var taskIdMachineIdMetric = string.Concat(taskId, "-", machineId, "-", resultType);
6730

68-
var jobResult = await GetJobResult(jobId);
69-
70-
if (jobResult == null) {
71-
_logTracer.LogInformation("Creating new JobResult for Job {JobId}", jobId);
72-
73-
var entry = new JobResult(JobId: jobId, Project: job.Config.Project, Name: job.Config.Name);
31+
var oldEntry = await GetJobResult(jobId, taskId, machineId, resultType);
7432

75-
jobResult = UpdateResult(entry, resultType, resultValue);
76-
77-
var r = await Insert(jobResult);
78-
if (!r.IsOk) {
79-
throw new InvalidOperationException($"failed to insert job result {jobResult.JobId}");
33+
if (oldEntry == null) {
34+
_logTracer.LogInformation($"attempt to insert new job result {taskId} and taskId+machineId+metricType {taskIdMachineIdMetric}");
35+
var newEntry = new JobResult(JobId: jobId, TaskIdMachineIdMetric: taskIdMachineIdMetric, TaskId: taskId, MachineId: machineId, CreatedAt: createdAt, Project: job.Config.Project, Name: job.Config.Name, resultType, Version: version, resultValue);
36+
var result = await Insert(newEntry);
37+
if (!result.IsOk) {
38+
throw new InvalidOperationException($"failed to insert job result with taskId {taskId} and taskId+machineId+metricType {taskIdMachineIdMetric}");
8039
}
81-
_logTracer.LogInformation("created job result {JobId}", jobResult.JobId);
82-
} else {
83-
_logTracer.LogInformation("Updating existing JobResult entry for Job {JobId}", jobId);
84-
85-
jobResult = UpdateResult(jobResult, resultType, resultValue);
40+
return true;
41+
}
8642

87-
var r = await Update(jobResult);
88-
if (!r.IsOk) {
89-
throw new InvalidOperationException($"failed to insert job result {jobResult.JobId}");
90-
}
91-
_logTracer.LogInformation("updated job result {JobId}", jobResult.JobId);
43+
ResultVoid<(HttpStatusCode Status, string Reason)> r;
44+
switch (resultType) {
45+
case COVERAGE_DATA:
46+
case RUNTIME_STATS:
47+
if (oldEntry.CreatedAt < createdAt) {
48+
oldEntry = oldEntry with { CreatedAt = createdAt, MetricValue = resultValue };
49+
r = await Update(oldEntry);
50+
if (!r.IsOk) {
51+
throw new InvalidOperationException($"failed to replace job result with taskId {taskId} and machineId+metricType {taskIdMachineIdMetric}");
52+
}
53+
} else {
54+
_logTracer.LogInformation($"received an out-of-date metric. skipping.");
55+
}
56+
break;
57+
default:
58+
_logTracer.LogInformation($"attempt to update job result {taskId} and taskId+machineId+metricType {taskIdMachineIdMetric}");
59+
oldEntry.MetricValue["count"]++;
60+
oldEntry = oldEntry with { MetricValue = oldEntry.MetricValue };
61+
r = await Update(oldEntry);
62+
if (!r.IsOk) {
63+
throw new InvalidOperationException($"failed to update job result with taskId {taskId} and machineId+metricType {taskIdMachineIdMetric}");
64+
}
65+
break;
9266
}
9367

68+
9469
return true;
70+
9571
}
9672

97-
public async Async.Task<OneFuzzResultVoid> CreateOrUpdate(Guid jobId, JobResultType resultType, Dictionary<string, double> resultValue) {
73+
public async Async.Task<OneFuzzResultVoid> CreateOrUpdate(Guid jobId, Guid taskId, Guid machineId, DateTime createdAt, double version, string resultType, Dictionary<string, double> resultValue) {
9874

9975
var job = await _context.JobOperations.Get(jobId);
10076
if (job == null) {
@@ -106,7 +82,7 @@ public async Async.Task<OneFuzzResultVoid> CreateOrUpdate(Guid jobId, JobResultT
10682
_logTracer.LogInformation("attempt to update job result {JobId}", job.JobId);
10783
var policy = Policy.Handle<InvalidOperationException>().WaitAndRetryAsync(50, _ => new TimeSpan(0, 0, 5));
10884
await policy.ExecuteAsync(async () => {
109-
success = await TryUpdate(job, resultType, resultValue);
85+
success = await TryUpdate(job, taskId, machineId, createdAt, version, resultType, resultValue);
11086
_logTracer.LogInformation("attempt {success}", success);
11187
});
11288
return OneFuzzResultVoid.Ok;

‎src/agent/Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎src/agent/onefuzz-result/Cargo.toml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,14 @@ license = "MIT"
99
[dependencies]
1010
anyhow = { version = "1.0", features = ["backtrace"] }
1111
async-trait = "0.1"
12+
chrono = { version = "0.4", default-features = false, features = [
13+
"clock",
14+
"std",
15+
"serde"
16+
] }
1217
reqwest = "0.11"
1318
serde = "1.0"
1419
storage-queue = { path = "../storage-queue" }
1520
uuid = { version = "1.4", features = ["serde", "v4"] }
1621
onefuzz-telemetry = { path = "../onefuzz-telemetry" }
1722
log = "0.4"
18-

‎src/agent/onefuzz-result/src/job_result.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33

44
use anyhow::Result;
55
use async_trait::async_trait;
6+
use chrono::DateTime;
7+
pub use chrono::Utc;
68
use onefuzz_telemetry::warn;
79
use reqwest::Url;
810
use serde::{self, Deserialize, Serialize};
@@ -32,6 +34,8 @@ struct JobResult {
3234
job_id: Uuid,
3335
machine_id: Uuid,
3436
machine_name: String,
37+
created_at: DateTime<Utc>,
38+
version: f64,
3539
data: JobResultData,
3640
value: HashMap<String, f64>,
3741
}
@@ -103,7 +107,8 @@ impl JobResultSender for TaskJobResultClient {
103107
let job_id = self.context.state.job_id;
104108
let machine_id = self.context.state.machine_id;
105109
let machine_name = self.context.state.machine_name.clone();
106-
110+
let created_at = chrono::Utc::now();
111+
let version = 1.0;
107112
let _ = self
108113
.context
109114
.queue_client
@@ -112,6 +117,8 @@ impl JobResultSender for TaskJobResultClient {
112117
job_id,
113118
machine_id,
114119
machine_name,
120+
created_at,
121+
version,
115122
data,
116123
value,
117124
})

0 commit comments

Comments
 (0)