Skip to content

Karabo Dev Meetings

Jump to bottom
Andreas Wassmer edited this page Jan 15, 2026 · 61 revisions

Meeting Minutes 2026

14/01/26

present: Rohini (RJ), Chris Finlay (CF), Massimo (MS), Emmanuel (ES), Andreas (AW)

written input from Delberin (DA) and Dev Null (DN)

minutes: AW

Roundtable discussion

  • AW:

  • DA:

    • Currently working on pinning libboost-python 1.88.0 to expose remaining blockers, validating PyBDSF builds (build-time vs runtime Boost requirements confirmed), and testing clean local environments before CI integration.
    • Highest supported Python version is currently Python 3.10. Higher versions are blocked by unresolved dependency compatibility in the Conda stack.
    • Current challenges: Remaining dependency resolution issues around NumPy 2.2, especially identifying which package still prevents full solver convergence after confirming Boost itself is not the blocker.

  • DN: Their Karabo docker image works on Daint's GH200 GPUs, but OSKAR is not optimized for arm64. Thus performance is disappointing.

  • EM: Could solve issue with CSCS. Data from SDC 3 is now publicly available. Will report on SKACH Winter Meeting.

  • MS:

    • Currently working on transition for image cleaning.
    • Adding user feature sky model from FITS.

  • RJ:

    • Had a chat with DN how to include their work in SRCNet.
    • Have a look at conference announced in Slack channel. Appears to be a computing conference in the first place.

  • CF: We keep integration of tabascal on hold until sdp transition.

Actions

  • LG on sick leave. AW will force push PR to main since there are only minor changes.

Meeting Minutes 2025

17/12/25

present: Lukas (LG), Dev Null (DN), Rohini (RJ), Massimo (MS), Emmanuel (ES), Andreas (AW)

minutes: AW

Roundtable discussion

  • ES: There will be a presentation of Karabo at the Winter SKACH Meeting. He thanks the commitee for accepting the late submission.

  • MS:

    • SKA SDP runs next to OSKAR and RASCIL
    • Started with testing HI pipeline, will update notebook
    • Current status of work: see github branch rascil-to-sdp

  • DN:

    • Worked on getting GPU into spack container
    • Compiled showcase based on source finding notebooks

  • DA:

    • Built dev packages against NumPy 2.2; PyBDSF now builds successfully.
    • Boost isn’t blocking; next step: pin libboost-python=1.88.0 to find remaining dependency issues.
    • Using conda-forge libboost- sub-packages* (may drop generic boost).
    • PyBDSF needs libboost-python-devel (build) + libboost-python (runtime).

  • AW: stil working on CI/CD fails. Github action runner cannot find mpirun, although it's installed from the dependencies.

Actions:

  • MS and AW will have chat regarding presentation at SKACH meeting.

03/12/25

present: Andreas (AW), Chris (CF)

minutes: AW

There was no meeting, just a short chat between AW and CF.

Roundtable discussion

  • AW: Broken CI pipeline seems to work. Fails occasionally due to a time out.
  • AW: No idea for topic for SKACH winter meeting.

19/11/25

present: Lukas (LG), Dev Null (DN), Rohini (RJ), Pascal (PH), Hatem (HG), Massimo (MS), Emmanuel (ES)

minutes: ES

Roundtable discussion

  • AW: Automated tests don't work. AW is taking care of this.

  • DA: Update on RASCIL + NumPy ≥ 2. The current blocker is chain bdsf → boost → python-casacore, because boost hard-pins NumPy < 2.0. Suggests two options:

    1. Make bdsf optional. This would allow for NumPy >= 2.0 for many users.
    2. Rebuild Boost/Python/casacore stack (private): This means we would need to maintain our own builds.
    • CF: Maybe all that is required is the pybdsf feedstock maintained by Karabo needs to be updated? I see that it is pinned to numpy==1.23.

  • MS: Work is progressing smoothly.

    • Imaging: integrate sdp imager, RASCIL works in parallel until full transition.
    • Some functions of imaging are not done yet but it is progressing

  • DN: Issues with tests failing on certain JupyterHub instances (memory leaks) in Swiss SLC; Australian setup works fine.

    • Reviewed latest OSKAR version: update expected to be straightforward (except NumPy).
    • Attempted OpenCL for OSKAR: performance decreased instead of improving.
    • Discussion:
      • RJ: Should more tests be added to catch such issues?
      • DN: Cause unknown; more experimentation needed before adding tests

  • LG: Minor update: some compatibility issues with NumPy updates; progress ongoing.

  • HG:

    • Asks, what do we do for winter meeting?
      • RJ: Would be good to present something, but difficult to decide what. We should discuss this with AW for the topics.
    • Regarding funding & task allocation: Need to coordinate next steps with new reallocated funding (after refused projects). Discuss HES-SO contributions. Can there be any SRC tasks?
      • RJ: Possible allocation to Karabo; SRC work organized under SRCnet global, FHNW tasks already defined.
    • HES-SO will have additional engineer in January. Tasks to be discussed
      • RJ: Carolyn should be included in planning. Start coordination via email.
    • Is Kubernetes expertise still needed?
      • RJ: Yes.

Actions:

  • Evaluate feasibility of making bdsf optional and/or rebuilding Boost stack
  • AW, GH and RJ: Initiate email discussion on winter meeting topic.

15/10/25

present: Lukas (LG), Dev Null (DN), Chris (CF), Rohini (RJ), Pascal (PH), André (AC), Delberin Ali (DA), Andreas (AW), Massimo (MS), Emmanuel (ES)

minutes: AW

Roundtable discussion

  • AW: Karabo is used on the Euler-Cluster at ETHZ. It's a student project with scope HI simulations.
  • AW: CI pipeline fixed. It's unknown what caused the pipeline to fail. The script to update the citation file is deactivated. It messed up the hash for the tag, because it finishes before everython is built..
  • DN: Has a container running ARM version. Has no GPU support yet. DN is also working with the beam code and they do some test runs.
  • MS: Started on the transition from RASCIL to sdp package. Seems doable.
  • CF: Runs tests on CSCS with Dev's container.

01/10/25

present: Lukas (LG), Dev Null (DN), Chris (CF), Rohini (RJ), Pascal (PH), André (AC), Delberin Ali (DA) apologies: Andreas, Massimo, Emmanuel

minutes: RJ

Roundtable discussion

  • CF: conversations with AW ongoing, documentation about new functionality ongoing
  • LG: CSCS object storage being used, but more object storage issues emerged with failing CIs. These were fixed but few other small issues related to versioneer. Release pending.
  • DA: Numpy version upgrade to 2+, updating CUDA version 12.0+. Looking into Github Actions
  • DN: Spack support can help alleviate this. Note RASCIL support doesn't exist for Numpy 2+ and is no longer supported. https://artefact.skao.int/#browse/browse:pypi-all:rascil%2F2.1.0%2Frascil-2.1.0-py3-none-any.whl

https://github.com/d3v-null/Karabo-Pipeline/tree/karabo-jupyter-docker Note spack-overlay directory. Demo in 2 weeks with notes on Conda vs spack, comparing runtimes, etc. LG: We should drop py3.9 support as soon as we have py3.10.

03/09/25

present: Lukas (LG), Dev Null (DN), Chris (CF), Rohini (RJ), Pascal (PH), Andreas (AW)

minutes: AW

Roundtable discussion

  • CF: Chris gives short update/summary of the poll done at Swiss SKA Days. There was feedback from 11 participants. Most of them state that an easy access to visibilities would be beneficial.
  • AW: Working on RFI integration together with Chris. But was busy with ESA project.
  • DN: Wanted to demonstrate Karabo at their EoR group. Did not really work. Visibilities look ok, but dirty images from WSClean don't.
  • LG:
    • The service on CSCS was decommissioned. The data needs to be migrated to S3. A ticket was opened. Need then to rewrite the code in Karabo.
    • Quota issue on Conda Forge is solved. LG removed some of the unused stuff.
    • Will have meeting with Delberin. He will continue to work on Karabo.

Actions:

  • DN connect to CF regarding the dirty images
  • Need to rewrite the download class to work with S3 -> LG
  • Need to update to Python 3.11, Numpy 2 and Cuda 2

20/08/25

present: Lukas (LG), Emmanuel(ES), Chris (CF), Rohini (RJ), Andreas (AW)

minutes: AW

Roundtable discussion

  • ES: Work on data storage started and is ongoing.
  • CF: Poll is ready for SKACH days. Any input is welcome. Will be sent out to SKACH mailing list.
  • RJ: No input related to Karabo
  • LG: Did another release. Needs to fix CI pipeline. Quota on Conda Forge exceeded. We either need to switch to a paid version or need to delete older versions.
  • AW: Setting up demo for RFI integration. Struggling with compatibility issues.

06/08/25

present: Lukas (LG), Massimo (MS), Chris (CF), Pascal (PH), Dev (DN), Rohini (RJ), Andreas (AW)

minutes: AW

Roundtable discussion

  • MS: Started with the simulation part. Replacing the backend for the visibilities.
  • DN: Still working on the transition to Python 3.10. Identified main issue: there is an outdated docker imager with old conda-build version. We don't need this image anymore.
  • CF: Nothing new, submitted talk and workshop for Swiss SKA days. Putting together a poll regarding the use of Karabo.
  • RJ:
    • Also presenting a SRC workshop at Swiss SKA days.
    • Reagrding the hosting SDC data: it is possible to host it on CSCS object storage
  • LG: Reading and managing PR
  • AW: Working on the class for RFI integration, it will need more testing.

23/07/25

present: Lukas (LG), Emmanuel (ES), Chris (CF), Pascal (PH), Dev (DN), Rohini (RJ) (MS on holiday)

minutes: RJ

Roundtable discussion

  • LG + PH: Focused on field test activities, not much to report
  • DN: python3.10 PR is ready for review! RJ to poke DA, else LH will take a look https://github.com/i4Ds/Karabo-Pipeline/pull/653 DN needs CSCS access (for some MWA data processing on CSCS not directly related to Karabo) but also for testing ARM compatibility eventually, this might be via access to Michele's project on CSCS
  • CF: automating the satellite frequency selection to avoid user interaction.
  • ES: Working with the SDC3a/b team. There was lots of simulated data generated and this has garnered a lot of international interest. Looking into how/where this can be stored and hosted to give collaborators access to this ~10 TB of data. public access but anon access not required acknowledging FAIR data principles. In conversation with Victor. RJ: possibly could be hosted on object storage at CSCS.
  • RJ: Note Swiss SKA days submission deadline this week, please consider submitting Karabo/other abstracts.

09/07/25

present: Lukas (LG), Emmanuel (ES), Massimo (MS), Chris (CF), Pascal (PH), Dev (DN)

minutes: LG

Roundtable discussion

  • DN: Python 3.10 WiP. Docker has old conda version, will be updated.
  • MS: Upgrade ska-sdp-*: WiP Held meeting with SDC3: Ideas for Karabo, documented but bring it up at a later point
  • PH: CronJob now doesn't cause issues anymore
  • CF: Tabascal: Together with Andreas, created a class for RFI-signal, which collects infos for telescope, etc which does create a config-file to run Tabascal. Made some adaptions to run in Python 3.9, but 3.10 coming soon! Organizing a session at swiss SKA-days. Maybe get form to SKACH how they use Karabo, who does use it. Survey to get user-base of Karabo. Go trough results on ska-days.

25/06/25

present: Rohini (RJ), Lukas (LG), Emmanuel (ES), Massimo (MS), Chris (CF), Andreas (AW)

minutes: AW

Roundtable discussion

  • Status of actions:
    • Review backlog in order to identify tasks for MS --> will have a look at RASCIL to ska-sdp transition
    • PH: Review PR from Lukas --> was merged
  • CF: separated simulation code from tabascal into new repo'tabsim'. This is the code which we need for Karabo.
  • MS: Not ready yet to include SDC3 data. Will work on RASCL to ska-sdp transition.
  • AW:
    • Solved problem with timestamps in RASCIL and OSKAR visibilities. Cannot fully get rid of differences because that's how RASCIL works. We'll add an information to the user.
    • Had meeting with CF. Is about to connect Karabo to tabascal.
    • Updated the Karabo installation on Renkulab to latest 0.33.2. Added example on how to use beam shapes.

Holiday absences

  • AW from July 8 to August 3

Actions

  • AW: Do we have a Karabo version running Python 3.10 that can be installed via conda?

11/06/25

present: Rohini (RJ), Lukas (LG), Pascal (PH), Emmanuel (ES), Massimo (MS), Chris (CF), Rohit (RS), André (AC), Andreas (AW) and Dev Null (DN)

minutes: AW

We welcome Dev Null. They're visiting us before attending the SKAO Science Meeting in Goerlitz.

Roundtable discussion

  • MS: Needs some task to start. First test on their infrastructre went well.
  • RJ: Had PI meeting last week. Got new backlog for 3 months
  • RS: Will present some work done based on Karabo at SKAO conference.
  • DA: Transition to Python 3.10 almost done. Got issues from some tests. But theyseem to be resolved. The version is on i4DS channel in Anaconda and labeld as "dev".
  • LG: Waits for open PR to be reviewed.
  • CF: Having a meeting with AW today regarding RFI integration into Karabo
  • DN: Investigate switching from conda to SPACK. First tests are very promising. Will give lighter images and ARM support.
  • AW: working on solving the problem of diverging timestamps in RASCIL and OSKAR visibilities. They are due to the fact that RASCIL sets the observation time to the next (not nearest) date when the hour angle of the phase center is off 0. This can lead to difference in timestamp up to almost 24 hours.

Holiday absences

  • AW from July 8 to August 3

Actions

  • Review backlog in order to identify tasks for MS
  • PH: Review PR from Lukas

28/05/25

present:

minutes: AW

Roundtable discussion

  • RS: Reviewing the referees comments and submitting paper update.
    • May propose a project which involves more development in Karabo
  • DA: Could build a container with Karabo and Python 3.10. The container can be downloaded. Needs testing.
    • There is also a Conda package. Currently, there are 3 version on conda-forge
  • CF: There are Docker containers running tabascal. There also container on ARM on CSCS.
  • LG: Is working on a Karabo version which runs on Gornergrat. Pascal will review it.
    • Did also do some bug fixing and updating the docu
  • RJ: PI planning is up to come next week.
  • AW: Jennifer (ETHZ) was able to fix issue #606 (Dirty images are scaled differently). This is an important part towards fixing issue 626 (RASCIL simulation leads to black dirty images #626).
  • AW: Karabo benchmark finished on Euler. The results show that using GPU speeds up calculations by a factor of 3. This holds for calculations with different frequency channels. The result are to be published in the review of the paper.

Actions

  • Talk to Delberin regarding the 3 Karabo package version.
  • Brainstroming with Chris regarding tabascal. Scheduled for June 11

14/05/25

present: Rohini (RJ), Lukas (LG), Pascal (PH), Emmanuel (ES), Massimo (MS), Chris (CF), Andreas (AW)

minutes: AW

Workshop tabascal

We're happy to welcome Chris Finlay to the Karabo Dev team. He gives an introduction to his tabascal software. A brainstorming session will be planned to see how to integrate its functionality into Karabo. AW and CF will participate. Ideas shall be shared and discussed with the dev team.

Roundtable discussion

  • RJ: There was a serious outage of Gornergrat. The reason is yet unknown. CSCS helps recover the computing infrastructure.
  • AW:
    • We have access to new CSCS infrastructre with ARM architecture. We will use the account to test the Karabo ARM version.
    • Is running benchmarks on Euler. Goal is to compare CPU vs. GPU performance as requested for paper.
    • contacted casacore team if they supply a ARM version. There's an answer from Peter Williams:

I'm not aware of anyone else with specific plans to add ARM64 support. But, conda-forge is all about contributions from the user community, and there is a lot of automation to support these kinds of changes! In particular, I think that to make this happen we "just" need to add casacore to the "arch rebuild" list of packages via a PR analogous to this example: conda-forge/conda-forge-pinning-feedstock#7358 See also this brief knowledgebase item. Once casacore is added to the rebuild list, I believe the migrator will start filing PRs to add ARM64 builds as needed. Some of these might then need some work to actually get them building correctly, but the migrator can at least take care of the mechanical aspects of adding new build arches.

Actions

  • AW: Figure out what Peter's answer means for Karabo
  • AW and CF: Agree on date for brainstorming session

30/04/25

present: Rohini (RJ), Lukas (LG), Pascal (PH), Emmanuel (ES), Massimo (MS), Delberin (DA), Andreas (AW)

minutes: AW

Roundtable discussion

  • MS could install Karabo and run some of the examples successfully.
    • He will use our workshop provided on Renkulab in order to present Karabo
  • HES-SO plans to add data set of SDC 3 to the Karabo data sets
  • DA finished upgrading to Python 3.10. Runs locally, it's not yet production ready. He is about to build a Docker image.
    • Python 3.11 is more work to do, needs some major dependency updates
  • RJ/DN were working on SRCNet related topics
  • There is now a password for the Miro board. Ask RJ for it if you need writing access.
  • AW in contact with Chris, wants to learn more about tabascal
    • Karabo runs on EPFL's Kuma cluster. But there is no support for GPU if OSKAR dirty imager is userd.

Actions

  • In mid-term we should address updating to 3.11
  • Also mid-term: update SDP tools (we're still on 0.1.3)
  • AW: will have a look at OSKAR issue
  • Contact casacore team if they supply a ARM version (AW)

16/04/25

present: Rohini (RJ), Lukas (LG), Pascal (PH), Rohit (RS), Emmanuel (ES), Massimo, (MS), Andreas (AW)

minutes: AW

Roundtable discussion

  • Kick off meeting with new participants. Welcome to Emmanuel and Massimo from HES-SO!
  • RJ shows Karabo mid term roadmap. Link to miro board is here: https://miro.com/app/board/uXjVICV_wqU=/
    • Development will focus on 3 categories: Functionality improvements, operational improvements, bugs & maintenance
  • LG reports that shift to Python 3.10 is almost done. Shift to 3.11 is postponed. There are some breaking changes to deal with.
  • Efforts towards running Karabo on Gornergrat. There will be no direct access to cluster like on ordinary CSCS infrastructre.
  • Dev Null and RS are working on data ingestion and data processing
  • Dev Null are testing SPACK which should give ARM support
  • Karabo version 1.0 release still an option. Need thoroughful API testing first.
  • AW is joining forces with Chris Finlay (now EPFL) on RFI integration.
    • Presentation planned @ Swiss SKA Day in Winterthur
  • HES-SO ready to start working on Karabo.
    • Getting familiar with software.
    • Massimo will be developer.

Actions No immediate actions defined

09/01/25

Asynch: Andreas (AW), Delberin (DA)

Roundtable discussion Andreas:

  • Added new ALMA cycles for upcoming slots in 2025.
  • WiP: Adding RFI simulations into Karabo. Reached out to Chris Finlay. He's the author of tabascal package. Will have a chat about how to proceed.

Delberin:

  • Finished first draft of Karabo dev build in Feedstock for Python3.10. Needs some wrap-up before merging into main and do prod-builds. Python3.11 ATM not possible because of RASCIL 1.0.0. Need to upgrade RASCIL version first before going to 3.11+.
  • WiP: Create Karabo wheel with Python3.10 deps.

Meeting Minutes 2024

28/11

Present: Lukas (LG), Michel (MP), Rohini (RJ)

Roundtable discussion

  • Lukas: some comments on Andreas' PR about TelescopeClass, but changes should make it into this month's release
  • Lukas (on behalf of Delberin): Discussion about some issues with wsclean and everybeam, resolution can be patch old versions or build new versions of both? Have gone for the second option which is hopefully backwards compatible. RASCIL has some issues, probably a few weeks to get through the last bits and to get to python 3.10
  • Michel: Updated PI24 run scripts have been merged, relevant docs done and demo WIP, should be wrapped in PI24
  • Lukas: Note Daint decommissioning EoY, esp if we need to do more dataset generation, where will it happen? Eiger or Alps?
    • Rohini: Maybe ingesting visibility data is an option, or using Karabo dataset generation as a way to validate Alps delivery
  • Exploring MSv4 data ingestion wrt SRCNet (it may not be as hierarchical as MSv2)

21/11

Present: Lukas (LG), Andreas (AW), Michel (MP), Rohini (RJ)

Roundtable discussion

  • Andreas: working on Telescope Class, discussions ongoing in PR.
  • Michel: Working on wrap-up from Karabo runs for SRCNet, small PR incoming. Workflow streamlining to generate data, ingest metadata, have it be discoverable. Discovery possible via SCS (simple cone search) or a TAP client + ADQL.
  • Lukas: PR review, supporting others. Delberin working on upgraded python support, assuming happy path should be in soon, perhaps not for next release.
  • Monthly release should include new AA layouts, Telescope class PR, maybe data ingestion scripts
  • Rohini: Not a lot of obvious Karabo work in SRCNet for PI25. Some data ingestion could be continued. We could work on adding noise to Karabo data, but need science user input.
    • Lukas: Adding MSv4 could be an option. But might be too soon, unsure how metadata will be handled.
    • Michel: Could do noise addition, if it can get prioritised. Or Karabo run + data ingestion to test the SRCNodes.
    • Rohini: Could be added to test campaign preparation ticket (Not too much to update on the paper atm)

07/11

Present: Lukas (LG), Simon (SF), Andreas (AW), Michel (MP), Delberin (DA), Rohini (RJ)

Minutes

Roundtable updates

  • Andreas: changes to the telescope class (due to discrepancy between OSKAR/RASCIL usage). For ex, baselines returned should now be the same irrespective of backend used. waiting on CI pipelines to clear. and review of course
  • Simon: First pass of paper complete, to be discussed with Rohit. Publication location not decided yet. Contributors welcome!
  • Lukas: CI is failing since yesterday due to some breaking change elsewhere from some dependency. Availability is low in the coming 2 weeks so need someone to look into this. (starting point can be going back to last successful run in github actions and the diff of dependency resolution since then)
  • Michel - data ingestion SRCNet side ongoing (related PR open, waiting for review)
  • Delberin - discussing with Simon, currently working on wsclean compatibility
  • Rohini - work in progress to define the SRCNet work plan, including for Karabo

31/10

Present: Lukas (LG), Simon (SF), Andreas (AW), Michel (MP), Rohini (RJ)

Agenda

  • Roundtable update
  • Followup discussion after OSKAR/RASCIL meeting

Minutes

Roundtable update

  1. MP finishing up Karabo run, run went well, just took a while. Visibility data (1.5 TB for this run) doubles in size after cleaning the image, due to wsclean implementation, needs further investigation. Okay for now, but a problem as datasets get larger. Not able to create full image cubes, issue created (https://github.com/i4Ds/Karabo-Pipeline/issues/624). SF: Dask could be an option to consider.
  2. Simon and Rohit continuing work on the paper.
  3. Delberin and Simon in touch about compilation issues.
  4. AW worked on telescope discrepancy issue wrt OSKAR and RASCIL. Working on this, still a shift of 0.1 degree present, sometimes with a telescope center shift. MP: source telescope configurations might be different so this could be a source of discrepancy.

Followup discussion after OSKAR/RASCIL meeting

Recap of the few important points

  • originally OSKAR thought for SKA-LOW sims, development progress is limited, no concrete long term roadmap but things are implemented/fixed based on need. Recommendation to not use vis file format, just use MS. (vis to dump to disk but that's about it)
  • RASCIL is python based (thus slow), started as a PoC, but then evolved accidentally to be used for sims. RASCIL-main retirement happening soon and migration of core functionality to SDP packages. no new simulation functionality planned.

Discussion

24/10

Present: Lukas (LG), Simon (SF), Andreas (AW)

Agenda

  • Roundtable update
  • Date for followup discussion after OSKAR/RASCIL meeting?

Minutes

  1. Delberin is still stuck with the compiling error related to the everybeam package.
  2. How should we handle user requests? Would be cool if we could respond to them. But we still suffer from a lack of human resources.
  3. How to go on after OSKAR/RASCIL meeting was not discussed. Does the outcome of the meeting affect future Karabo development? Maybe we should bring it up in next Dev meeting.
  4. Working on the Karabo paper ongoing. Simon finished another chapter.

Actions

  1. Simon - will have a look at the error and discuss ideas with Delberin.
  2. No immediate actions
  3. Schedule update for next Karabo Dev meeting. AW will ask MP for presentation.
  4. No immediate actions.

17/10

Present: Michel (MP), Lukas (LG), Pascal (PH), Andreas (AW), Delberin (DA), Rohini (RJ)

Minutes

  • Lukas - meeting with OSKAR/RASCIL devs
  • Andreas - Jennifer chat this week, HI science would require HPC infra so might be a good idea to deploy to Alps
  • Michel - Followup discussion after OSKAR/RASCIL meeting needed. AA layout integration complete (all SKA LOW and MID layouts), to be merged.
  • Delberin: python version upgrade nearly complete, 2 packages outstanding Async update
  • Simon and Rohit making progress on the paper 🎉

Actions

  • RJ: Followup meeting from OSKAR/RASCIL discussion for Karabo roadmap impact

10/10/2024

Present: Michel, Lukas, Pascal, Simon

Minutes

  • Michel PR #620 merged
  • Michel suspects RASCIL interferometry doesn’t work anymore (#626). Weird issue, needs to be looked at.
  • Simon worked on paper
  • Should we automatically convert .vis to MS when required? Might take a lot of time & space.
  • We should show user feedback for long running tasks.

Actions

SF: Create issue for automatic conversions of visibility formats (#627) SF: User Feedback for long running tasks. (#)

03/10

Meeting was cancelled due to a lot of absences and no need for a meeting among the people who were available.

26/09 ‘All-hands’ dev meeting

Present: Michel, Lukas, Simon, Vincenzo, Andreas, Delberin, Pascal, Rohini

Agenda/minutes

Time Item Discussion Actions
14.00 Roundtable updates    
14.15 MilestonesClose out milestones? (other than icebox)Make milestone for v1.0 - What should go into this? https://github.com/i4Ds/Karabo-Pipeline/milestones We should close out 0.23 and 0.24 milestones. Check issues dont get deleted when they are removed for a milestone.V1.0MP: how strict would we be with breaking changes with v1.0 and beyondSF: Want to send a message that this software is usableLG: should define the public api, even if breaking changes come in v1.0+ with an API review for public APIsOther things for v1.0 AW: should running on CSCS infra (alps) be linked to this?RJ: link to python version?Probably no to bothSF: focusing more on PR angle when thinking about which issues should be resolved for v1.0SF: would be good to get a test science user to run Karabo, go over documentation and give feedbackJan 2025 might be okay (SF: probably not later than this) RJ: accept pending invite!API review (public/private) (issue 609)Find test science user to run Karabo, go over documentation and give feedbackRJ: Create v1.0 milestoneAll: add issues to this milestone to be reviewed in future dev meeting
14.30 LabelsWhat do we mean by prio-low/med/highDoes it imply things getting into the next release?Can we use them to identify good-first-issues? SF: Originally prio-high were meant to be the to-do list, prio-med is next steps, and prio low is similar to iceboxLG: need some ownership/responsiblity for labels and relabeling as neededLG: add a work-in-progress label to indicate work that has been picked upMP: rename prios to must/should/could haveLG: some way to group larger topics spanning multiple issues/releases(milestones could work but not ideal) Remove proj-pinochio and proj-sdc3 labelsrename prios to must/should/could have
14.45 Dev meeting formatSuggestions here welcome!Roundtable updates, identify items for longer discussion as a meet after Going around based on issues wasn’t the best approach in the pastwork to make Karabo more accessible, visible, etc that doesn’t entail PRs to the repoMinutes on the wiki, note actions? Separate user feedback/use case meetings as we need them? MP: Look at new issues made in the last week and look at labels  

User/Stakeholder meetings

Sync with OSKAR, RASCIL teams

Oct 15th

People

Celeste Lu, Danielle Fenech, Mark Ashdown, Fred Dulwich, Michel Plüss, Pascal Herzog, Lukas Gehrig, Rohini Joshi, Jeremy Coles

Agenda, Minutes

  • Introductions
  • Intro to Karabo and what we have done to expand on the backends, how/why we wrap it to provide a package

Using slides: https://confluence.skatelescope.org/pages/viewpage.action?pageId=280401927&preview=/280401927/284182692/Karabo.pdf

  • OSKAR, RASCIL roadmaps, plans
    • Dependency management plans ?
    • Tell us a bit more about RASCIL retirement but core functionality retained in different repository structure, how/where and the plans for this?

Fred: Current users? Rohini/Lukas: Some SDC use cases, and users at ETH, and EPFL mainly SRCNet at the moment

Fred: note there are existing datasets out there from existing instruments. Also, OSKAR originally built for SKA-Low, simulating dish arrays with gaussian beams wasn't the original intention :)

Mark: Development work process is distributed? Fred: Collaborative effort, yes

Celeste: RASCIL python models mostly moved out, RASCIL misuse also prevalent. RASCIL-main is what's left after moving the imaging tooling, data models, python models etc. This will be archived ~March, Gitlab for understanding the latest status wrt RASCIL

Don't touch RASCIL, also don't touch OSKAR 😁

Semantic versioning? Discussion ongoing, push to move to version 1 and be more intentional about versioning

Celeste: Other pipelines being looked at in DP - processing CoP Mark: Analysis pipelines not simulation pipelines https://confluence.skatelescope.org/display/SE/Processing+Community+of+Practice

  • What is the current usage of OSKAR/RASCIL/Casa tools in the Construction ARTs, or DP ART?

  • Appetite/need for simulated data generation pipeline for SDP? Other feature work for the upcoming PI? Communication and discussion welcome here, SDP side more useful to have smaller (than AA*) but precise simulations

  • Q&A

    • MPI support for multi-node clusters? Dask hasn't worked quite well for RASCIL. OSKAR originally had MPI but this was messy and no planned support for this going forward.
    • Difference between the visibility data formats? Don't touch .vis files 😁 they have a very specific function and intended use, ie dump to disk and load into OSKAR or python when you don't want to use MS interaction tools like casacore. For interoperability, use MS files.

Note some useful links:

Clone this wiki locally