Duplicate code detection & refactoring across repositories

[philwinder]

Interesting discussion on linkedin about how there's an opportunity to perform duplication detection across repositories.

Russ' problem was that of duplicated code across microservices when working with similar data structures:

it would be very cool if I could for example extract code from multiple repositories into shared libraries and then reimport those shared definitions in coordinated operations. Doing this kind of thing manually is just a pain. Error prone, repetitive, drudge work.

The first step is to be able to see the problem. So my initial thought is to include some kind of simple "duplication detector" that runs within and across indexes.

Once we can see it, then maybe we can add a tool to the MCP server to pull out selected offending code from specific codebases and have a demo where we're asking the LLM to write unifying libraries.

[topiaruss]

This is well expressed.
Narrowing it a bit, priority is to find and handle areas of duplication that represent touch-points between services. REST APIs, message queues, kafka, db-schemas, etc. Ensuring that signature changes are coherent, and that small changes like the addition of a field does not force coordinated (meaning simultaneous or orchestrated-sequential) update of large parts of a system.
That's a general and broad description, but it would be great to find some real world cases using dataclasses or similar between services, as probably the simplest starting point.

[philwinder]

Quick update @topiaruss: I wrote a very simple microservice-oriented demo: https://docs.helixml.tech/kodit/demos/go-simple-microservice/

First step towards the deduplication goal.