Skip to content

Implemented sparse fieldsets support on the backend #1152 #1715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Eric-Wasson merged 6 commits into from
Dec 3, 2025
Merged

Implemented sparse fieldsets support on the backend #1152 #1715

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d897121
Implemented fetching sparse fieldsets of JSON API standard in new API
Eric-Wasson Oct 23, 2025
cd61bff
Added sparse fieldset support for the aggregated data
Eric-Wasson Oct 27, 2025
70ff641
Implemented sparse fieldsets PR feedback and improved handling of agg…
Eric-Wasson Oct 29, 2025
1818850
Fixed sparse fieldsets array handling and one small aggregate data error
Eric-Wasson Nov 28, 2025
11bd146
Merge branch 'dev' into 1152-implement-fetching-sparse-fieldsets
Eric-Wasson Dec 2, 2025
cf0e5c9
Fixed merge errors
Eric-Wasson Dec 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 17 additions & 8 deletions src/inc/apiv2/common/AbstractBaseAPI.class.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ protected function getUpdateHandlers($id, $current_user): array {
* Overridable function to aggregate data in the object. Currently only used for Tasks
* returns the aggregated data in key value pairs
*/
public static function aggregateData(object $object): array {
public static function aggregateData(object $object, array $sparseFieldsets = null): array {
return [];
}

Expand Down Expand Up @@ -551,7 +551,7 @@ protected function obj2Array(object $obj): array {
* @throws NotFoundExceptionInterface
* @throws ContainerExceptionInterface
*/
protected function obj2Resource(object $obj, array $expandResult = []): array {
protected function obj2Resource(object $obj, array $expandResult = [], array $sparseFieldsets = null): array {
// Convert values to JSON supported types
$features = $obj->getFeatures();
$kv = $obj->getKeyValueDict();
Expand All @@ -569,6 +569,16 @@ protected function obj2Resource(object $obj, array $expandResult = []): array {
if ($feature['private'] === true) {
continue;
}

// If sparse fieldsets (https://jsonapi.org/format/#fetching-sparse-fieldsets) is used, return only the requested data
if (is_array($sparseFieldsets)) {
Copy link
Contributor

@jessevz jessevz Oct 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try to put the three if statements into a single if statement

if (array_key_exists($this->getObjectTypeName($obj), $sparseFieldsets)) {
if (!in_array($feature['alias'], $sparseFieldsets[$this->getObjectTypeName($obj)])) {
continue;
}
}
}

// Hide the primaryKey from the attributes since this is used as indentifier (id) in response
if ($feature['pk'] === true) {
continue;
Expand All @@ -581,8 +591,7 @@ protected function obj2Resource(object $obj, array $expandResult = []): array {
$attributes[$feature['alias']] = $apiClass::db2json($feature, $kv[$name]);
}

//TODO: only aggregate data when it has been included
$aggregatedData = $apiClass::aggregateData($obj);
$aggregatedData = $apiClass::aggregateData($obj, $sparseFieldsets);
$attributes = array_merge($attributes, $aggregatedData);

/* Build JSON::API relationship resource */
Expand Down Expand Up @@ -1420,23 +1429,23 @@ protected static function getOneResource(object $apiClass, object $object, Reque
// Convert objects to data resources
foreach ($objects as $object) {
// Create object
$newObject = $apiClass->obj2Resource($object, $expandResult);
$newObject = $apiClass->obj2Resource($object, $expandResult, $request->getQueryParams()['fields'] ?? null);

// For compound document, included resources
foreach ($expands as $expand) {
if (array_key_exists($object->getId(), $expandResult[$expand])) {
$expandResultObject = $expandResult[$expand][$object->getId()];
if (is_array($expandResultObject)) {
foreach ($expandResultObject as $expandObject) {
$includedResources[] = $apiClass->obj2Resource($expandObject);
$includedResources[] = $apiClass->obj2Resource($expandObject, [], $request->getQueryParams()['fields'] ?? null);
}
}
else {
if ($expandResultObject === null) {
// to-only relation which is nullable
continue;
}
$includedResources[] = $apiClass->obj2Resource($expandResultObject);
$includedResources[] = $apiClass->obj2Resource($expandResultObject, [], $request->getQueryParams()['fields'] ?? null);
}
}
}
Expand Down Expand Up @@ -1483,4 +1492,4 @@ protected static function getMetaResponse(array $meta, Request $request, Respons
static public function getAvailableMethods(): array {
return ["GET", "POST", "PATCH", "DELETE"];
}
}
}
7 changes: 4 additions & 3 deletions src/inc/apiv2/common/AbstractModelAPI.class.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -747,29 +747,30 @@ public static function getManyResources(object $apiClass, Request $request, Resp
// Convert objects to data resources
foreach ($objects as $object) {
// Create object
$newObject = $apiClass->obj2Resource($object, $expandResult);
$newObject = $apiClass->obj2Resource($object, $expandResult, $request->getQueryParams()['fields'] ?? null);

// For compound document, included resources
foreach ($expands as $expand) {
if (array_key_exists($object->getId(), $expandResult[$expand])) {
$expandResultObject = $expandResult[$expand][$object->getId()];
if (is_array($expandResultObject)) {
foreach ($expandResultObject as $expandObject) {
$includedResources = self::addToRelatedResources($includedResources, $apiClass->obj2Resource($expandObject));
$includedResources = self::addToRelatedResources($includedResources, $apiClass->obj2Resource($expandObject, [], $request->getQueryParams()['fields'] ?? null));
}
} else {
if ($expandResultObject === null) {
// to-only relation which is nullable
continue;
}
$includedResources = self::addToRelatedResources($includedResources, $apiClass->obj2Resource($expandResultObject));
$includedResources = self::addToRelatedResources($includedResources, $apiClass->obj2Resource($expandResultObject, [], $request->getQueryParams()['fields'] ?? null));
}
}
}

// Add to result output
$dataResources[] = $newObject;
}

$baseUrl = Util::buildServerUrl();
//build last link
$lastParams = $request->getQueryParams();
Expand Down
57 changes: 34 additions & 23 deletions src/inc/apiv2/model/tasks.routes.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,36 +160,47 @@ protected function createObject(array $data): int {
return $task->getId();
}

//TODO make aggregate data queryable and not included by default
static function aggregateData(object $object): array {
//TODO make aggregate data queryable
static function aggregateData(object $object, array $sparseFieldsets = null): array {
$aggregatedData = [];
$keyspace = $object->getKeyspace();
$keyspaceProgress = $object->getKeyspaceProgress();

$aggregatedData["dispatched"] = Util::showperc($keyspaceProgress, $keyspace);
$aggregatedData["searched"] = Util::showperc(TaskUtils::getTaskProgress($object), $keyspace);

$qF = new QueryFilter(Chunk::TASK_ID, $object->getId(), "=");
$chunks = Factory::getChunkFactory()->filter([Factory::FILTER => $qF]);

$activeAgents = [];
foreach ($chunks as $chunk) {
if (time() - max($chunk->getSolveTime(), $chunk->getDispatchTime()) < SConfig::getInstance()->getVal(DConfig::CHUNK_TIMEOUT) && $chunk->getProgress() < 10000) {
$activeAgents[$chunk->getAgentId()] = true;
}
if(is_array($sparseFieldsets) && array_key_exists('task', $sparseFieldsets) && in_array("dispatched", $sparseFieldsets['task']) ) {
$aggregatedData["dispatched"] = Util::showperc($keyspaceProgress, $keyspace);
}

//status 1 is running, 2 is idle and 3 is completed
$status = 2;
if ($keyspaceProgress >= $keyspace && $keyspaceProgress > 0) {
$status = 3;

if(is_array($sparseFieldsets) && array_key_exists('task', $sparseFieldsets) && in_array("searched", $sparseFieldsets['task']) ) {
Copy link
Contributor

@jessevz jessevz Oct 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the first part of the check: " if(is_array($sparseFieldsets) && array_key_exists('task', $sparseFieldsets)". is now done twice, I would say do that part as on outer if statement, and then afterwards just check individually for "searched" and "status"

$aggregatedData["searched"] = Util::showperc(TaskUtils::getTaskProgress($object), $keyspace);
}

if(is_array($sparseFieldsets) && array_key_exists('task', $sparseFieldsets) && in_array("activeAgents", $sparseFieldsets['task']) ) {
$qF = new QueryFilter(Chunk::TASK_ID, $object->getId(), "=");
$chunks = Factory::getChunkFactory()->filter([Factory::FILTER => $qF]);

$activeAgents = [];
foreach ($chunks as $chunk) {
if (time() - max($chunk->getSolveTime(), $chunk->getDispatchTime()) < SConfig::getInstance()->getVal(DConfig::CHUNK_TIMEOUT) && $chunk->getProgress() < 10000) {
$activeAgents[$chunk->getAgentId()] = true;
}
}

$aggregatedData["activeAgents"] = array_keys($activeAgents);
}
elseif (count($activeAgents) > 0) {
$status = 1;

if(is_array($sparseFieldsets) && array_key_exists('task', $sparseFieldsets) && in_array("searched", $sparseFieldsets['task']) ) {
Copy link
Contributor

@jessevz jessevz Oct 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "searched" here, should be "status".

//status 1 is running, 2 is idle and 3 is completed
$status = 2;
if ($keyspaceProgress >= $keyspace && $keyspaceProgress > 0) {
$status = 3;
}
elseif (count($activeAgents) > 0) {
$status = 1;
}

$aggregatedData["status"] = $status;
}

$aggregatedData["activeAgents"] = array_keys($activeAgents);
$aggregatedData["status"] = $status;

return $aggregatedData;
}

Expand Down