Skip to content

Update sync to use pagination #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fulldecent wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update sync to use pagination #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 35 additions & 23 deletions sync.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,32 +89,44 @@
$localColumns = []; // column names on local side to prepare SQL statements

// fetch the API result
$results = $doGet($client, $settings['endpoint']);
if ($results === false) {
$io->error(sprintf('Failed to sync data for endpoint: %s', $settings['endpoint']));
}
$page = 1;
while (true) {
$separator = (strpos($settings['endpoint'], '?') === false) ? '?' : '&';
$url = sprintf('%s%spage=%s&size=500', $settings['endpoint'], $separator, $page);
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The page size is hardcoded to 500. Consider making this configurable through a constant or configuration parameter to allow flexibility for different environments or API rate limits.

Copilot uses AI. Check for mistakes.
$results = $doGet($client, $url);
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The $doGet function doesn't handle Guzzle HTTP exceptions. If an HTTP error (like 500, 401, 403) or network error occurs during pagination, it will cause an uncaught exception that crashes the entire sync process. This is particularly problematic in a pagination loop where partial data may have already been collected. Consider wrapping the API call in a try-catch block to handle errors gracefully, similar to how it's done at line 312 for the teams sync.

Copilot uses AI. Check for mistakes.

if ($results === false) {
$io->error(sprintf('Failed to sync data for endpoint: %s', $settings['endpoint']));
break;
Copy link

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When an API error occurs mid-pagination, the function breaks out of the loop and continues processing with partial data from previous pages. This could lead to incomplete or inconsistent data in the database. Consider returning early or throwing an exception instead of breaking, to prevent processing partial results.

Suggested change
break;
return;

Copilot uses AI. Check for mistakes.
}

// prepare the array of all entities for the local database by mapping columns
foreach ($results as $entity) {
$newEntity = [];
foreach ($settings['mapping'] as $kimaiField => $localField) {
$key = $localField;
$value = $entity[$kimaiField];
// some values need to be converted to local format (eg. datetime)
if (is_callable($localField)) {
$tmp = call_user_func($localField, $entity, $kimaiField);
$key = $tmp[0];
$value = $tmp[1];
}
$newEntity[$key] = $value;
if (empty($results)) {
break;
}
if (count($localColumns) === 0) {
$localColumns = array_keys($newEntity);

// prepare the array of all entities for the local database by mapping columns
foreach ($results as $entity) {
$newEntity = [];
foreach ($settings['mapping'] as $kimaiField => $localField) {
$key = $localField;
$value = $entity[$kimaiField];
// some values need to be converted to local format (eg. datetime)
if (is_callable($localField)) {
$tmp = call_user_func($localField, $entity, $kimaiField);
$key = $tmp[0];
$value = $tmp[1];
}
$newEntity[$key] = $value;
}
if (count($localColumns) === 0) {
$localColumns = array_keys($newEntity);
}
$apiEntities[$entity['id']] = $newEntity;
}
$apiEntities[$entity['id']] = $newEntity;
}

unset($results);
$page++;
unset($results);
}

if (count($apiEntities) === 0) {
$io->success('No data found to sync: ' . $title);
Expand Down Expand Up @@ -238,7 +250,7 @@

$syncConfig['Timesheets'] = [
'table' => 'timesheet',
'endpoint' => 'timesheets?user=all&modified_after=' . $since->format('Y-m-d\TH:i:s') . '&size=' . PHP_INT_MAX,
'endpoint' => 'timesheets?user=all&modified_after=' . $since->format('Y-m-d\TH:i:s'),
'mapping' => [
'id' => 'kimai_id',
'activity' => 'activity',
Expand Down