Added search command to search for rosdep keys. by StefanFabian · Pull Request #997 · ros-infrastructure/rosdep

StefanFabian · 2025-03-26T17:47:47Z

I've had to open the rosdep repo one time too many to look for a package key because I couldn't guess the correct one.
Hence, I've added a search command for rosdep which fuzzy searches the sources for a key or system package matching the search terms.
It allows one error per search term provided after rosdep search.
This could probably be made a parameter but typically this should be enough.

Examples:

$ rosdep search libpcl-dev
Closest keys:
  libfcl-dev

Closest packages:
  libpcl-all-dev: libpcl-dev [debian]
  libpcl-all-dev: libpcl-dev [ubuntu]
  acl: libacl-devel [fedora]
  acl: libacl-devel [rhel]

$ rosdep search pinoccio
Closest keys:
  kinematics_interface_pinocchio
  pinocchio

… found.

ros-discourse · 2025-04-10T08:21:24Z

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/rosdep-search/43148/1

peci1

I've tested this locally and it works very nice! Just one more bug I found, you have a fix for it as a suggestion.

I've tested on U20.04/Noetic and tried searching both Noetic and Jazzy packages (via --rosdistro). Both worked as expected.

peci1 · 2025-04-10T08:42:08Z

Once this is merged, it should be followed by a flood of tutorial/docs updates :)

…ose packages. Co-authored-by: Martin Pecka <peci1@seznam.cz>

StefanFabian · 2025-04-16T14:12:52Z

@cottsay any chance you could review this or assign someone who can do a review?

kscottz · 2025-04-24T21:47:05Z

Thanks for the contribution @StefanFabian

I ran the tests again to make sure we're good.

Scott is busy with Kilted so I pinged @christophebedard to do a quick second review. Let's get this out the door.

kscottz · 2025-04-24T21:52:15Z

@StefanFabian looks like an issue with importing regex in the tests.

StefanFabian · 2025-04-25T13:51:48Z

Ah, yes. As I see it, there are three options.
Option 1: Use the shipped re module, but that does not allow errors, so it wouldn't have fuzzy search, which I think is quite helpful.
Option 2: Add python3-regex as a dependency (I have already made a PR in rosdistro to add the key, but I couldn't find a package for MacOS).
Option 3: Make regex optional by using a try-catch import and falling back to a search using only re.

Which would you prefer?

kscottz · 2025-04-28T22:32:46Z

@StefanFabian This is just my preference but option number 3, " Make regex optional by using a try-catch import and falling back to a search using only re" seems like it has the lowest maintainer overhead with the most cross platform support.

christophebedard · 2025-04-29T00:42:04Z

Sorry for the delay. I'll try to take a look tomorrow

christophebedard · 2025-04-30T00:06:02Z

If it works with the built-in re module, I think we should just stick to that and not (conditionally or not) use/depend on regex.

christophebedard · 2025-04-30T01:00:51Z

If it works with the built-in re module, I think we should just stick to that and not (conditionally or not) use/depend on regex.

Although I see that fuzzy matching is just not (easily) possible with the built-in re module, so yeah let's keep regex as an option.

christophebedard

Some minor comments.

I think I understand the difference between "keys" and "packages" (i.e., searching actual rosdep keys vs searching what a key might point to), but I think it might be a bit confusing to users if it's not explained. That could be left for some eventual user documentation.

christophebedard · 2025-04-30T00:27:18Z

+rosdep search <searchstrings>...
+  search for a key in the rosdep database. E.g. using the system package name.
+


From what I understand, you can provide multiple "search strings" and the potential results have to match (fuzzy or not) all of them: it's not an OR, it's an AND. I think rosdep is pretty unclear about a lot of its options and commands, so I'd prefer to make that crystal clear here 😁

Also, I think making it clear that this searches system packages and ROS packages (even though it's implied by "key in the rosdep database") would also help.

Lastly, we should mention somewhere (possibly in user documentation and not here) that providing --rosdistro= is required in order to search ROS packages. I know this is always the case for rosdep, but again this is often unclear to users, especially new users.

Are you sure --rosdistro= is necessary?
I can find ROS packages just fine without it.

I will make the help string more clear 👍

It gets filled from ROS_DISTRO env var.

Good point, but it's not available if you haven't sourced ROS (ros_environment) in your terminal.

Anyway, like I was saying, this is the case for all commands that need a ROS distro. I just feel like this isn't very clear, but we can always make it clearer later.

christophebedard · 2025-04-30T01:02:31Z

+        view = sources_loader.get_source(view_name=view_name)
+        if not isinstance(view, CachedDataSource):
+            if options.verbose:
+                print('Skipping non-cached source %s' % view_name)


Can we tell users what to do in this case?

Honestly, I don't know what to do there.
I'm skipping them because I don't know the structure if they're not this type.
This print is purely for debugging purposes.
I think a typical installation wouldn't encounter it, at least I never did.

christophebedard · 2025-04-30T01:29:19Z

Oh, and some tests would be great! At least just for the main use-cases.

Co-authored-by: Christophe Bedard <bedard.christophe@gmail.com>

StefanFabian · 2025-04-30T09:09:26Z

@christophebedard Thank you for the review!
I've addressed your concerns, added tests, and handling of more dependency key structures.

Let me know if there's anything else that needs to be improved.

christophebedard · 2025-05-06T00:44:08Z

There are new flake8 errors caused by unnecessary global (because the variable doesn't get re-assigned), probably because of a newer flake8 version. They're causing CI to not fully run: https://github.com/ros-infrastructure/rosdep/actions/runs/14750900000/job/41502835987?pr=997#step:5:286. I think we can just remove global in this case.

It would be simpler to just fix them in this PR. Could you do that?

christophebedard · 2025-05-06T00:59:03Z

+                    results = [regex.search(pkg) for regex in regexes]
+                    if all(results):
+                        error = count_errors(results)
+                        close_pkgs.append({'key': key, 'pkg': pkg, 'os': os_entry, 'error': error})


I'm a bit confused by the intended or unintended use of the os info for packages:

$ rosdep search tracetools --rosdistro=rolling WARNING: ROS_PYTHON_VERSION is unset. Defaulting to 3 Closest keys: tracetools_image_pipeline tracetools tracetools_launch tracetools_read tracetools_test tracetools_trace tracetools_acceleration tracetools_analysis Closest packages: ros2trace_analysis: ros/rolling/tracetools_analysis [osx]

Why is it only listing osx? As a user, I would think this means that this package is only available on macOS. Could we instead list all OSes, e.g., [osx, debian, rhel, ubuntu]? The code seems to be stopping after finding results for the first available OS.

I've decided to stop when a match is found because usually it's not relevant if there's another os with a matching name, since you usually know what you are looking for.
The main use case here is also looking for the package name in your distro's package manager to find the associated key, and in that case, only that would match anyway.
Debian and Ubuntu, for example, would almost always both have the same results, and this would unnecessarily bloat the output.
If I were to also list the other OS, it would be confusing if I didn't also list the matching package name for each OS, and this could get quite a long list if the search string matches many packages.

What about searching just for the currently used OS (or the one passed in --os) ?

Yeah, I think that would make a lot of sense. Not sure what the proper way to do it is, but looks like you could use OsDetect + the OS override info.

I've decided to stop when a match is found because usually it's not relevant if there's another os with a matching name, since you usually know what you are looking for.

If the OS information doesn't really matter, it shouldn't be shown IMO.

Okay, I have changed it to use the os information and only look through packages for the current OS.
This is also updated in the help string, and I've removed the information for which OS the package was found as this is now no longer relevant.

Co-authored-by: Christophe Bedard <bedard.christophe@gmail.com>

…rrent os.

StefanFabian · 2025-05-12T10:05:31Z

I think I've addressed all feedback. If there's something that still needs to be changed, please let me know.

peci1

Great, thanks a lot, it works for me with the updates!

christophebedard

Thanks for iterating! Looks good to me now.

@nuclearsandwich @cottsay I don't have permissions on this repo: could one of you approve the CI run and possibly review and/or merge this? Note that this PR includes the same changes as in #1004, since they were necessary to prevent CI from failing fast due to unrelated flake8 errors.

codecov · 2025-05-16T18:48:53Z

Codecov Report

Attention: Patch coverage is 72.56637% with 31 lines in your changes missing coverage. Please review.

Project coverage is 71.76%. Comparing base (4cb03bb) to head (a0527e0).
Report is 37 commits behind head on master.

Files with missing lines	Patch %	Lines
src/rosdep2/main.py	72.56%	16 Missing and 15 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #997      +/-   ##
==========================================
- Coverage   74.22%   71.76%   -2.47%     
==========================================
  Files          44       44              
  Lines        3376     3488     +112     
  Branches        0      686     +686     
==========================================
- Hits         2506     2503       -3     
+ Misses        870      808      -62     
- Partials        0      177     +177

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

christophebedard · 2025-05-20T20:23:44Z

@StefanFabian can you take a look at the CI failure for the Run tests / pytest / pytest (macos-latest, 3.12) job? https://github.com/ros-infrastructure/rosdep/actions/runs/15075300653/job/42381233133?pr=997

StefanFabian · 2025-05-21T15:50:16Z

@christophebedard I think it was caused by the search command using the os of the test runner.
I've made the os explicit in the tests, so they should run the same on any platform.

christophebedard · 2025-05-21T18:55:04Z

Thanks, that makes sense.

@nuclearsandwich could you approve a new CI run? 😅

christophebedard · 2025-06-06T15:20:31Z

CI looks good. Just waiting on an additional review or merge, but people are a bit busy.

cottsay

I didn't do a close review here, but I don't see anything dangerous.
Thanks for your contribution!

Co-authored-by: Martin Pecka <peci1@seznam.cz> Co-authored-by: Christophe Bedard <bedard.christophe@gmail.com> Co-authored-by: Steven! Ragnarök <steven@nuclearsandwich.com>

StefanFabian · 2025-06-20T14:02:16Z

Any timeline on when the next release can be expected? @christophebedard @cottsay @kscottz

christophebedard · 2025-06-20T17:28:51Z

Not sure about the timeline, but @nuclearsandwich is working on it.

StefanFabian added 2 commits March 26, 2025 18:18

Added search command to search for rosdep keys.

cf849bb

Use fuzzy count to reduce the number of results if exact matches were…

40f592d

… found.

peci1 approved these changes Apr 10, 2025

View reviewed changes

Comment thread src/rosdep2/main.py Outdated

Fix: Show total count of close pkgs not keys for truncated list of cl…

d019487

…ose packages. Co-authored-by: Martin Pecka <peci1@seznam.cz>

kscottz requested a review from christophebedard April 24, 2025 21:45

kscottz mentioned this pull request Apr 24, 2025

Add documentation for search feature #998

Open

Added fallback with note to re module if regex module is not available.

1cef306

christophebedard reviewed Apr 30, 2025

View reviewed changes

StefanFabian and others added 5 commits April 30, 2025 09:37

FIx typo.

2968296

Co-authored-by: Christophe Bedard <bedard.christophe@gmail.com>

Remove whitespace.

4046d5e

Co-authored-by: Christophe Bedard <bedard.christophe@gmail.com>

Updated docs.

85ef174

Reduced method complexity and added handling of more possible cases.

0c1b581

Added test.

9067b32

StefanFabian requested a review from christophebedard April 30, 2025 09:09

christophebedard reviewed May 6, 2025

View reviewed changes

StefanFabian and others added 2 commits May 6, 2025 15:43

Apply suggestions from code review

0b5390d

Co-authored-by: Christophe Bedard <bedard.christophe@gmail.com>

Fixed unrelated global flake8 error.

5e91d2f

Use os information to filter packages and only search packages for cu…

556789d

…rrent os.

StefanFabian requested a review from christophebedard May 12, 2025 10:07

peci1 approved these changes May 14, 2025

View reviewed changes

christophebedard approved these changes May 15, 2025

View reviewed changes

nuclearsandwich added 2 commits May 16, 2025 11:44

Merge remote-tracking branch 'origin/master' into StefanFabian/master

ef521ab

Merge remote-tracking branch 'origin/master' into StefanFabian/master

e1308b4

Specify os in test to obtain reproducible results.

a0527e0

christophebedard approved these changes Jun 7, 2025

View reviewed changes

cottsay approved these changes Jun 9, 2025

View reviewed changes

cottsay merged commit 0a38d49 into ros-infrastructure:master Jun 9, 2025
12 of 14 checks passed

cottsay mentioned this pull request Dec 11, 2025

Drop more over-aggressive asserts from test_rosdep_main #1020

Merged

		rosdep search <searchstrings>...
		search for a key in the rosdep database. E.g. using the system package name.

Uh oh!

Conversation

StefanFabian commented Mar 26, 2025

Uh oh!

ros-discourse commented Apr 10, 2025

Uh oh!

peci1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

peci1 commented Apr 10, 2025

Uh oh!

StefanFabian commented Apr 16, 2025

Uh oh!

kscottz commented Apr 24, 2025

Uh oh!

kscottz commented Apr 24, 2025

Uh oh!

StefanFabian commented Apr 25, 2025

Uh oh!

kscottz commented Apr 28, 2025

Uh oh!

christophebedard commented Apr 29, 2025

Uh oh!

christophebedard commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christophebedard commented Apr 30, 2025

Uh oh!

christophebedard left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christophebedard commented Apr 30, 2025

Uh oh!

StefanFabian commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christophebedard commented May 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StefanFabian commented May 12, 2025

Uh oh!

peci1 left a comment

Choose a reason for hiding this comment

Uh oh!

christophebedard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

christophebedard commented Apr 30, 2025 •

edited

Loading

StefanFabian commented Apr 30, 2025 •

edited

Loading

christophebedard left a comment •

edited

Loading

codecov Bot commented May 16, 2025 •

edited

Loading