Brainstorming sur les issues

[mheit-dlier]

Je demande respectueusement d’analyser les issues rencontrés la dernière fois et proposer des solutions bien argumentées

[MOUSAAB-ABOUJAAFAR]

• What is the most effective approach to integrate Apache Kafka and Apache NiFi to batch and streaming data for real-time processing ?

[khadyja-R]

@MOUSAAB-ABOUJAAFAR ,As we discussed last session The most effective approach is to use Apache NiFi as the data ingestion and flow orchestration layer, and Apache Kafka handles real-time processing and scalability,

[assmae-elhamydy]

oui @khadyja-R , Apache NiFi capte les données depuis diverses sources,les transforme, les nettoie et les enrichit,puis les achemine avec souplesse vers Apache Kafka,qui permet un traitement en streaming à grande échelle.
Le résultat c'est Un pipeline hybride, intelligent et adaptable, où chaque donnée trouve son chemin,

[AbdoMis]

Hi @mheit-dlier and everyone,

Following up on our last meeting regarding the project's ingestion, we discussed potential architectures (Kappa vs Lambda) and relevant tools. We agreed to implement a Lambda architecture with two pipelines: one for batch processing and another for real-time streaming. We also have the flexibility to use either kafka or both kafka and nifi.
This summarizes our discussion.

Please share any alternative ideas, architectures, or additions. Thanks!

[khadyja-R]

Thanks for the summary @AbdoMis!
Since we agreed on the Lambda architecture with batch and real-time pipelines, I think it would be great to review the current PR that follows this approach and discuss whether it’s ready to merge what do you @ALL think?
That could help us move closer to our first release.
Also, let me know what tasks we can focus on today ?

[AbdoMis]

yes 👍 , I agree with you @khadyja-R. If we all agree on this approach, we should start.

[aminaelbayyadi]

Thanks for the summary @AbdoMis and for the suggestion @khadyja-R!

I agree with reviewing the PR based on the Lambda architecture. It will allow us to validate the technical choices and move forward toward the first release.

As for today's tasks, could we:

Identify any blockers in the PR?

Split the code review or testing tasks among us?

Discuss a plan for integrating Kafka and NiFi into each pipeline?

Looking forward to your feedback!

[AbdoMis]

Thank you @aminaelbayyadi, we will proceed with these tasks to move toward the first release, and if there are any issues we will discuss it here.