Leantime data provider changed to use APIData plugin instead of built in API. #242

tuj · 2025-10-08T18:45:56Z

Link to ticket

https://leantime.itkdev.dk/TimeTable/TimeTable?showTicketModal=5538#/tickets/showTicket/5538

Description

Changes leantime data provider to use the APIData Plugin: https://github.com/ITK-Leantime/data-api instead of the built in API.
Every fetch and update action is a job that can be executed as either synchronous or asynchronous.
Removed DAMA testing bundle.

TODOs

Manual start of project sync from UI.
Tests.
Use the deleted endpoint of APIData Plugin for handling deleted entities.
Call monitoringUrl when updating jobs are started.
Use ThrottlingHttpClient?

Checklist

My code is covered by test cases.
My code passes our test (all our tests).
My code passes our static analysis suite.
My code passes our continuous integration process.

turegjorup · 2025-10-09T08:23:46Z

src/Command/SyncCommand.php

    protected function configure(): void
    {
+        $this->addOption("job", 'j', InputOption::VALUE_NONE, "Use async job handling");
+        $this->addOption("modified", "m", InputOption::VALUE_NONE, "Only items modified since last update");


modified implies Datetime as value, should maybe be only-modifed tom imply Bool

turegjorup · 2025-10-09T08:24:02Z

src/Command/SyncCommand.php


-        $io->info('Processing projects');
+        $jobHandling = $input->getOption("job");
+        $modified = $input->getOption("modified");


As above, modified implies Datetime as value, should maybe be only-modifed tom imply Bool

turegjorup · 2025-10-09T08:30:30Z

src/Entity/Trait/FetchDateTrait.php

+trait FetchDateTrait
+{
+    #[ORM\Column(type: Types::DATETIME_MUTABLE, nullable: true)]
+    private ?\DateTimeInterface $fetchTime = null;


When can this be null? For data that is sync'ed from a data provider we should always have a date.

For legacy data we can either set 1970-01-01 00:00 or NOW() as default value in the db migration. For new entities we can set new DatetomeImmutable() as default in the constructor.

Not allowing null values will also simplify some of the queries in the repository methods.

turegjorup · 2025-10-09T08:48:29Z

src/MessageHandler/LeantimeUpdateHandler.php

+                $message->modified,
+            );
+        } catch (\Exception $e) {
+            throw new UnrecoverableMessageHandlingException($e->getMessage());


Question - should we also do $this->logger->error(..)?

I think throwing UnrecoverableMessageHandlingException will put the message in the failed queue, so the error is visible there. But even so, if somebody just logs at the logs when debugging ad don't see errors it might be confusing.

turegjorup · 2025-10-09T09:12:16Z

src/Entity/Trait/FetchDateTrait.php

+
+trait FetchDateTrait
+{
+    #[ORM\Column(type: Types::DATETIME_MUTABLE, nullable: true)]


Given the queries we do for oldest fetch time we should consider an index on this.

I have changed approach to not use fetchDate to find modifiedAfter.

turegjorup · 2025-10-09T09:18:04Z

src/Service/JiraApiService.php

 use Symfony\Contracts\HttpClient\HttpClientInterface;

-class JiraApiService implements DataProviderServiceInterface
+class JiraApiService implements DataProviderInterface


Suggested change

class JiraApiService implements DataProviderInterface

#[\Deprecated(message: "The Jira data provider is deprcated and will be removed", since: "x.y")]

class JiraApiService implements DataProviderInterface

I remove the Service as it does not need to exist with the changes in this PR and it does not work.

turegjorup · 2025-10-09T09:24:30Z

src/Service/DataProviderService.php

+
+    private function linkToTicket(string $ticketId, DataProvider $dataProvider): string
+    {
+        return $dataProvider->getUrl() . "/errorpage/#/tickets/showTicket/" . $ticketId;


This is technically only valid for Leantime, so does not belong in a generic data provider class. But I don't know how important we consider the support for multiple providers to be going forward?

turegjorup · 2025-10-09T09:32:36Z

src/Model/Upsert/UpsertIssueData.php

+
+use App\Enum\IssueStatusEnum;
+
+class UpsertIssueData


I Don't see the point of having Upsert in the class name for any of these models, but that is semantics and subjective

I will leave it as it ensures that the name does not collide with other IssueData class.

What about ProviderIssueDataas we talked about?

Forgot about that talk :D

turegjorup · 2025-10-09T09:33:54Z

src/Repository/IssueRepository.php

+        return $qb->getQuery()->getSingleColumnResult();
+    }
+
+    public function getOldestFetchTime(DataProvider $dataProvider, ?array $projectTrackerProjectIds = null): ?\DateTimeInterface


I would expect this to be getNewestFetchTime? If we always fetch from the oldest and forward will we not always sync everything. Or do I have it the wrong way round?

codecov-commenter · 2025-11-04T10:57:21Z

Codecov Report

❌ Patch coverage is 58.65052% with 239 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@fdecc1a). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/Service/DataProviderService.php	71.65%	53 Missing ⚠️
src/Service/LeantimeApiService.php	77.36%	43 Missing ⚠️
src/Command/SyncCommand.php	0.00%	34 Missing ⚠️
src/MessageHandler/LeantimeUpdateHandler.php	0.00%	15 Missing ⚠️
src/Controller/ProjectController.php	0.00%	12 Missing ⚠️
src/Controller/PlanningController.php	0.00%	11 Missing ⚠️
src/MessageHandler/LeantimeDeleteHandler.php	0.00%	11 Missing ⚠️
src/Command/SyncDeletedCommand.php	0.00%	10 Missing ⚠️
src/Command/SyncModifiedCommand.php	0.00%	10 Missing ⚠️
src/Repository/ProjectRepository.php	0.00%	9 Missing ⚠️
... and 12 more

Additional details and impacted files

@@            Coverage Diff             @@
##             develop     #242   +/-   ##
==========================================
  Coverage           ?   17.38%           
  Complexity         ?     1662           
==========================================
  Files              ?      202           
  Lines              ?     6880           
  Branches           ?        0           
==========================================
  Hits               ?     1196           
  Misses             ?     5684           
  Partials           ?        0

Flag	Coverage Δ
	`17.38% <58.65%> (?)`
unittests	`17.38% <58.65%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

turegjorup

Consider if we can lower this in docker-compose.server.override.yml

    phpfpm:
        environment:
            - PHP_MEMORY_LIMIT=512M

turegjorup · 2025-11-04T11:41:40Z

src/Command/SyncDeletedCommand.php

+    name: 'app:data-providers:sync-deleted',
+    description: 'Sync Data Provider deleted data, that has been deleted within the last hour, as jobs. Scheduled to run every 15 minutes.',
+)]
+#[AsPeriodicTask(frequency: '15 minutes')]


I havn't tried it, but it should be possible to do something like #[SomeAttribute('%env(MY_ENV_VAR)%')], so

Suggested change

#[AsPeriodicTask(frequency: '15 minutes')]

#[AsPeriodicTask(frequency: '%env(SYNC_FREQUENCY)%')]

And then set SYNC_FREQUENCY='15 minutes' in .env so it's configurable per installation

turegjorup · 2025-11-04T11:45:25Z

src/Command/SyncDeletedCommand.php

+    {
+        // Look at entries modified within the last hour.
+        $deletedAfter = new \DateTime();
+        $deletedAfter->sub(new \DateInterval('P1D'));


P1Dis one DAY, but comment and description says one HOUR. I'm fine with either, as long as code and comment/description match.

Nice to have: P1D as default, but possible to give another value as command argument.

turegjorup · 2025-11-04T11:51:07Z

src/Command/SyncModifiedCommand.php

+    name: 'app:data-providers:sync-modified',
+    description: 'Sync Data Provider data, that has been modified within the last hour, as jobs. Scheduled to run every 15 minutes.',
+)]
+#[AsPeriodicTask(frequency: '15 minutes')]


As above. If we can move 15 minutes to .env we should

turegjorup · 2025-11-04T12:05:04Z

src/Entity/Trait/SynchronizedEntityTrait.php

+    #[ORM\Column(type: Types::DATETIME_MUTABLE, nullable: true)]
+    private ?\DateTimeInterface $fetchDate = null;
+
+    #[ORM\Column(type: Types::DATETIME_MUTABLE, nullable: true)]
+    private ?\DateTimeInterface $sourceModifiedDate = null;
+
+    public function getFetchDate(): ?\DateTimeInterface
+    {
+        return $this->fetchDate;
+    }
+
+    public function setFetchDate(?\DateTimeInterface $fetchDate): void
+    {
+        $this->fetchDate = $fetchDate;
+    }
+
+    public function getSourceModifiedDate(): ?\DateTimeInterface
+    {
+        return $this->sourceModifiedDate;
+    }
+
+    public function setSourceModifiedDate(?\DateTimeInterface $sourceModifiedDate): void
+    {
+        $this->sourceModifiedDate = $sourceModifiedDate;
+    }


I'm torn between wanting to use an interface and wanting to use DateTimeImmutable (which has no interface). I would do this, but it's only a suggestion. Pick the one you prefer.

Suggested change

#[ORM\Column(type: Types::DATETIME_MUTABLE, nullable: true)]

private ?\DateTimeInterface $fetchDate = null;

#[ORM\Column(type: Types::DATETIME_MUTABLE, nullable: true)]

private ?\DateTimeInterface $sourceModifiedDate = null;

public function getFetchDate(): ?\DateTimeInterface

{

return $this->fetchDate;

}

public function setFetchDate(?\DateTimeInterface $fetchDate): void

{

$this->fetchDate = $fetchDate;

}

public function getSourceModifiedDate(): ?\DateTimeInterface

{

return $this->sourceModifiedDate;

}

public function setSourceModifiedDate(?\DateTimeInterface $sourceModifiedDate): void

{

$this->sourceModifiedDate = $sourceModifiedDate;

}

#[ORM\Column(type: Types::DATETIME_IMMUTABLE, nullable: true)]

private ?\DateTimeImmutable $fetchDate = null;

#[ORM\Column(type: Types::DATETIME_IMMUTABLE, nullable: true)]

private ?\DateTimeImmutable $sourceModifiedDate = null;

public function getFetchDate(): ?\DateTimeImmutable

{

return $this->fetchDate;

}

public function setFetchDate(?\DateTimeInterface $fetchDate): void

{

if (!$fetchDate instanceof \DateTimeImmutable) {

$fetchDate = \DateTimeImmutable::createFromInterface($fetchDate);

}

$this->fetchDate = $fetchDate;

}

public function getSourceModifiedDate(): ?\DateTimeImmutable

{

return $this->sourceModifiedDate;

}

public function setSourceModifiedDate(?\DateTimeImmutable $sourceModifiedDate): void

{

$this->sourceModifiedDate = $sourceModifiedDate;

}

I think I will stick with DateTimeInterface, since this is the approach we use in the other parts of the codebase.

src/MessageHandler/EntityRemovedFromDataProviderHandler.php

jeppekroghitk · 2025-11-05T07:57:54Z

src/Controller/PlanningController.php

+        }

-        return new JsonResponse(['issuesSynced' => $issuesSynced], 200);
+        return new JsonResponse(['issuesSynced' => 0], 200);


Does it make sense to return this?

Not really... will remove it :)

tuj · 2025-11-05T11:10:13Z

Consider if we can lower this in docker-compose.server.override.yml
    phpfpm:
        environment:
            - PHP_MEMORY_LIMIT=512M

I have removed the override value. We will have to see it we need more than 128M

…ger exists in the source

…nstead

tuj changed the title ~~Feature/new leantime api~~ Leantime data provider changed to use APIData plugin instead of built in API. Oct 8, 2025

turegjorup reviewed Oct 9, 2025

View reviewed changes

tuj self-assigned this Oct 10, 2025

tuj added the enhancement New feature or request label Oct 10, 2025

tuj requested a review from jeppekroghitk November 4, 2025 11:28

tuj marked this pull request as ready for review November 4, 2025 11:49

turegjorup approved these changes Nov 4, 2025

View reviewed changes

jeppekroghitk reviewed Nov 5, 2025

View reviewed changes

jeppekroghitk approved these changes Nov 5, 2025

View reviewed changes

jeppekroghitk force-pushed the feature/new-leantime-api branch from 641257b to 22bbd84 Compare December 4, 2025 13:22

tuj added 18 commits December 11, 2025 09:46

5538: Started work on new synchronization integration

b27ba16

5538: Added version sync

d09b89f

5538: Fixed worklog sync

c994bd2

5538: Fixed sync fields

441b12f

5388: Cleaned up

0de9709

5388: Started job handling

12add71

5388: Fixed sync/async handling

837a8ed

5388: Changed default value

abaa657

5388: Added fetch date to entities and modified filter to api calls

c06680b

5388: Cleaned up duplicated code pieces

6ffff10

5388: Added entityManager clear when running in sync mode

845ba63

5538: Started work on fixing issues raised in review

e6e1d56

5538: Cleaned up

ed66d84

5538: Added schedule to commands

aa2bc14

5538: Avoid updating if source modified has not changed

3da2f26

5538: Fixed issues raised by psalm

59e3f74

5538: Renamed models

23ec8be

5538: Added schedule to command comment

8950d60

tuj and others added 29 commits December 11, 2025 09:47

5538: Fixed response

9a7f8e9

5538: Applied coding standards

abecc06

5538: Added source deleted date to show that a project/worklog no lon…

d7a255e

…ger exists in the source

5538: Fixed css for warning triangle

70df593

Edited API_PATH_DATA, leantime path is case sensitive

0692e28

Fallback to empty worker in issue if null

26dc911

Included sync of users

1f4ed46

Added command config

0967a52

Added worker flag to sync command

1645fdb

Added worker to api service

f2f3f64

Added upsert method for worker

3f84aad

Added message, handler and model for worker data

ddf4d7b

edit oops

3e42463

coding standards

8cebb79

Updated changelog

895223d

Made worker optional on issue

b4a522a

Re-added mis-deleted line break

07f26c4

Removed unnecessary setdefault values

c264a78

Updated changelog

0717bec

Fixed http-foundation CVE-2025-64500

7e52ecf

Fixed merge conflicts

1071f8d

Docker nginx config

9a5d21d

Call deleteAll method in syncDeletedCommand

c9fe10d

Added more descriptive message in NotFoundException

468ea82

Removed worker from deleteAsJob

3f8a06a

Coding standards

6d0bb1f

Fixed CVE-2025-66578

46a4b23

Removed robrichards/xmlseclibs from composer.json, and updated deps i…

d1d82c4

…nstead

Updated through container

013fa32

jeppekroghitk force-pushed the feature/new-leantime-api branch from 4145988 to 013fa32 Compare December 11, 2025 08:48

	class JiraApiService implements DataProviderInterface
	#[\Deprecated(message: "The Jira data provider is deprcated and will be removed", since: "x.y")]
	class JiraApiService implements DataProviderInterface

	#[AsPeriodicTask(frequency: '15 minutes')]
	#[AsPeriodicTask(frequency: '%env(SYNC_FREQUENCY)%')]

Leantime data provider changed to use APIData plugin instead of built in API. #242

Are you sure you want to change the base?

Leantime data provider changed to use APIData plugin instead of built in API. #242

Uh oh!

Conversation

tuj commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Link to ticket

Description

TODOs

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turegjorup Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Nov 4, 2025

Codecov Report

Uh oh!

turegjorup left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuj commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tuj commented Oct 8, 2025 •

edited

Loading

turegjorup Oct 9, 2025 •

edited

Loading