percona-toolkit install doesn't include go tools

[vinokrishna]

brew config AND brew doctor output OR brew gist-logs <formula> link

% brew config
HOMEBREW_VERSION: 5.0.9-16-g4dbe056
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4dbe056c37871a93716e80d59da1e892ea4c5132
Last commit: 88 minutes ago
Branch: main
Core tap JSON: 06 Jan 11:25 UTC
Core cask tap JSON: 06 Jan 11:25 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.vmPhAERrU7/org.xquartz:0
HOMEBREW_DOWNLOAD_CONCURRENCY: 24
HOMEBREW_FORBID_PACKAGES_FROM_PATHS: set
HOMEBREW_MAKE_JOBS: 12
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.4.8 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.4.8/bin/ruby
CPU: dodeca-core 64-bit arm_brava
Clang: 17.0.0 build 1700
Git: 2.50.1 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 8.7.1 => /usr/bin/curl
macOS: 15.7.2-arm64
CLT: 26.0.0.0.1.1757719676
Xcode: N/A
Rosetta 2: false

% brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed casks are deprecated or disabled.
You should find replacements for the following casks:
  copyq

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed manually.
You should find replacements for the following formulae:
  mongodb-community@6.0

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  icu4c@76
  icu4c@77

Verification

• My brew doctor output says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.
• I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.
• My issue is not about a failure to build a formula from source.

What were you trying to do (and why)?

I am trying to install percona-toolkit package from brew as follows and it is not installing all the tools from the package. Pls Refer - https://docs.percona.com/percona-toolkit.

For example, pt-galera-log-explainer tool under go was not installed. The list of tools that were missing under golang are here - https://github.com/percona/percona-toolkit/tree/v3.7.1/src/go:
pt-galera-log-explainer
pt-k8s-debug-collector
pt-mongodb-index-check
pt-mongodb-query-digest
pt-mongodb-summary
pt-pg-summary
pt-secure-collect

The warnings received in brew doctor on my machine are irrelevant here:
% brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed casks are deprecated or disabled.
You should find replacements for the following casks:
copyq

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed manually.
You should find replacements for the following formulae:
mongodb-community@6.0

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
icu4c@76
icu4c@77

What happened (include all command output)?

% brew install percona-toolkit
==> Fetching downloads for: percona-toolkit
✔︎ Bottle Manifest percona-toolkit (3.7.1)                                                                                                                                              Downloaded   36.3KB/ 36.3KB
✔︎ Bottle Manifest libcbor (0.13.0)                                                                                                                                                     Downloaded    9.3KB/  9.3KB
✔︎ Bottle libcbor (0.13.0)                                                                                                                                                              Downloaded   43.7KB/ 43.7KB
✔︎ Bottle Manifest libfido2 (1.16.0_1)                                                                                                                                                  Downloaded   19.9KB/ 19.9KB
✔︎ Bottle Manifest zlib (1.3.1)                                                                                                                                                         Downloaded   10.7KB/ 10.7KB
✔︎ Bottle zlib (1.3.1)                                                                                                                                                                  Downloaded  141.3KB/141.3KB
✔︎ Bottle Manifest mysql-client (9.5.0)                                                                                                                                                 Downloaded   22.0KB/ 22.0KB
✔︎ Bottle Manifest perl-dbd-mysql (5.013_1)                                                                                                                                             Downloaded   23.0KB/ 23.0KB
✔︎ Bottle libfido2 (1.16.0_1)                                                                                                                                                           Downloaded  363.8KB/363.8KB
✔︎ Bottle perl-dbd-mysql (5.013_1)                                                                                                                                                      Downloaded  140.6KB/140.6KB
✔︎ Bottle percona-toolkit (3.7.1)                                                                                                                                                       Downloaded    2.2MB/  2.2MB
✔︎ Bottle mysql-client (9.5.0)                                                                                                                                                          Downloaded   36.2MB/ 36.2MB
==> Installing dependencies for percona-toolkit: libcbor, libfido2, zlib, mysql-client and perl-dbd-mysql
==> Installing percona-toolkit dependency: libcbor
==> Pouring libcbor--0.13.0.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/libcbor/0.13.0: 36 files, 217.3KB
==> Installing percona-toolkit dependency: libfido2
==> Pouring libfido2--1.16.0_1.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/libfido2/1.16.0_1: 566 files, 1.4MB
==> Installing percona-toolkit dependency: zlib
==> Pouring zlib--1.3.1.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/zlib/1.3.1: 14 files, 413.6KB
==> Installing percona-toolkit dependency: mysql-client
==> Pouring mysql-client--9.5.0.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/mysql-client/9.5.0: 125 files, 134.2MB
==> Installing percona-toolkit dependency: perl-dbd-mysql
==> Pouring perl-dbd-mysql--5.013_1.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/perl-dbd-mysql/5.013_1: 14 files, 464.1KB
==> Installing percona-toolkit
==> Pouring percona-toolkit--3.7.1.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/percona-toolkit/3.7.1: 229 files, 9.0MB
==> Running `brew cleanup percona-toolkit`...
Disable this behaviour by setting `HOMEBREW_NO_INSTALL_CLEANUP=1`.
Hide these hints with `HOMEBREW_NO_ENV_HINTS=1` (see `man brew`).

 % ls /opt/homebrew/Cellar/percona-toolkit/3.7.1/bin/
pt-align			pt-eustack-resolver		pt-index-usage			pt-pmp				pt-slave-delay			pt-table-sync
pt-archiver			pt-fifo-split			pt-ioprofile			pt-query-digest			pt-slave-find			pt-table-usage
pt-config-diff			pt-find				pt-kill				pt-replica-find			pt-slave-restart		pt-upgrade
pt-deadlock-logger		pt-fingerprint			pt-mext				pt-replica-restart		pt-stalk			pt-variable-advisor
pt-diskstats			pt-fk-error-logger		pt-mysql-summary		pt-show-grants			pt-summary			pt-visual-explain
pt-duplicate-key-checker	pt-heartbeat			pt-online-schema-change		pt-sift				pt-table-checksum

What did you expect to happen?

I expect brew to install the missing Go tools mentioned above as well. i.e under here - https://github.com/percona/percona-toolkit/tree/v3.7.1/src/go. As a workaround, I had to compile them manually.

Step-by-step reproduction instructions (by running brew commands)

% brew install percona-toolkit
...

% which pt-galera-log-explainer
pt-galera-log-explainer not found

[vinokrishna]

Also, while compiling manually the toolkit, it wasn't able to find the Perl-DBD-MySQL though it was installed already. I had to proceed with make anyway

percona-toolkit % perl Makefile.PL
Checking if your kit is complete...
Warning: the following files are missing in your kit:
	CONTRIBUTE.md
	Gopkg.lock
	Gopkg.toml
Please inform the author.
Warning: prerequisite DBD::mysql 3 not found.
Generating a Unix-style Makefile
Writing Makefile for Percona::Toolkit
Writing MYMETA.yml and MYMETA.json

percona-toolkit % brew install perl-dbd-mysql
✔︎ JSON API formula.jws.json                                                                                                                                                            Downloaded   32.0MB/ 32.0MB
✔︎ JSON API cask.jws.json                                                                                                                                                               Downloaded   15.3MB/ 15.3MB
Warning: perl-dbd-mysql 5.013_1 is already installed and up-to-date.
To reinstall 5.013_1, run:
  brew reinstall perl-dbd-mysql

[svetasmirnova]

Hi! Percona Toolkit maintainer is here. Looks like reason for the build failure in the top dir is because Makefile does not recognize operating system.

GOOS=darwin GOARCH=arm64 make

should fix the issue.

[svetasmirnova]

I am not an expert in Homebrew formulas and do not have a Mac to test, but I think the fix could be something as I proposed in #261451

[cho-m]

This is an upstream issue as make install doesn't install Go binaries if following install steps at https://github.com/percona/percona-toolkit/tree/3.x?tab=readme-ov-file#installing

Specifically, the following (https://github.com/percona/percona-toolkit/blob/3.x/Makefile.PL#L15)

    EXE_FILES => [ <bin/*> ],

is expanded by MakeMaker when running perl Makefile.PL so the Go binaries are not included in the generated Makefile.

---

A workaround is make BIN_DIR=#{bin} which installs the Go binaries directly into the install path rather than the staging path.

Another workaround is to build the Go binaries first, make -C src/go build && perl Makefile.PL && ...

However, even if we use a workaround, the upstream build script or documented install steps should be fixed to handle Go binaries.

[svetasmirnova]

@cho-m I agree that installation instructions should be fixed, but upstream only supports Linux. Installation instructions perfectly work on Linux, because they do not need to set the environment GOOS=darwin GOARCH=arm64 before building. See also my comments above.

I don't mind adding pre-checks for the operating system and setting these variables in the Makefile.PL or src/go/Makefile, but, since macOS is not supported, I cannot make it part of the pre-release quality control tests and guarantee it would not be broken by some later changes.

[cho-m]

The problem isn't OS-specific.

It can be reproduced on Linux using same steps I mentioned at #261451 (comment)

For example, spinning up a clean Docker Ubuntu image:

$ apt update && apt install -y curl perl-modules
$ curl -LO https://downloads.percona.com/downloads/percona-toolkit/3.7.1/source/tarball/percona-toolkit-3.7.1.tar.gz
$ tar zxvf percona-toolkit-3.7.1.tar.gz
$ cd percona-toolkit-3.7.1
$ perl Makefile.PL
$ grep '^EXE_FILES' Makefile
EXE_FILES = bin/pt-align bin/pt-archiver bin/pt-config-diff bin/pt-deadlock-logger bin/pt-diskstats bin/pt-duplicate-key-checker bin/pt-eustack-resolver bin/pt-fifo-split bin/pt-find bin/pt-fingerprint bin/pt-fk-error-logger bin/pt-heartbeat bin/pt-index-usage bin/pt-ioprofile bin/pt-kill bin/pt-mext bin/pt-mysql-summary bin/pt-online-schema-change bin/pt-pmp bin/pt-query-digest bin/pt-replica-find bin/pt-replica-restart bin/pt-show-grants bin/pt-sift bin/pt-slave-delay bin/pt-slave-find bin/pt-slave-restart bin/pt-stalk bin/pt-summary bin/pt-table-checksum bin/pt-table-sync bin/pt-table-usage bin/pt-upgrade bin/pt-variable-advisor bin/pt-visual-explain

Same issue as EXE_FILES lacks Go binaries so make install does not install those binaries.

[svetasmirnova]

Same issue as EXE_FILES lacks Go binaries so make install does not install those binaries.

EXE_FILES are for Perl staff only. In the end of the Makefile you should find:

1508 # --- MakeMaker postamble section:
1509 gotools:
1510     cd src/go && $(MAKE) build

And this part builds Go tools successfully on Linux:

(sveta) sveta@s76:~/src/percona/percona-toolkit$ ls bin/pt-galera-log-explainer bin/pt-k8s-debug-collector bin/pt-mongodb-* bin/pt-secure-collect bin/pt-pg-summary 
ls: cannot access 'bin/pt-galera-log-explainer': No such file or directory
ls: cannot access 'bin/pt-k8s-debug-collector': No such file or directory
ls: cannot access 'bin/pt-mongodb-*': No such file or directory
ls: cannot access 'bin/pt-secure-collect': No such file or directory
ls: cannot access 'bin/pt-pg-summary': No such file or directory
(sveta) sveta@s76:~/src/percona/percona-toolkit$ perl Makefile.PL 
Checking if your kit is complete...
Warning: the following files are missing in your kit:
	CONTRIBUTE.md
	Gopkg.lock
	Gopkg.toml
Please inform the author.
Generating a Unix-style Makefile
Writing Makefile for Percona::Toolkit
Writing MYMETA.yml and MYMETA.json
(sveta) sveta@s76:~/src/percona/percona-toolkit$ make
cp lib/Lmo/Role.pm blib/lib/Lmo/Role.pm
cp lib/EventTimeline.pm blib/lib/EventTimeline.pm
...
cd src/go && make build
make[1]: Entering directory '/home/sveta/src/percona/percona-toolkit/src/go'
Building binaries in /home/sveta/src/percona/percona-toolkit/src/go/../../bin as version ->
make[1]: Leaving directory '/home/sveta/src/percona/percona-toolkit/src/go'
Manifying 36 pod documents
(sveta) sveta@s76:~/src/percona/percona-toolkit$ ls bin/pt-galera-log-explainer bin/pt-k8s-debug-collector bin/pt-mongodb-* bin/pt-secure-collect bin/pt-pg-summary 
bin/pt-galera-log-explainer  bin/pt-mongodb-query-digest  bin/pt-secure-collect
bin/pt-k8s-debug-collector   bin/pt-mongodb-summary
bin/pt-mongodb-index-check   bin/pt-pg-summary

For other platforms, we have either GOOS or GOARCH environments or special build instructions:

(sveta) sveta@s76:~/src/percona/percona-toolkit$ make -C src/go
make: Entering directory '/home/sveta/src/percona/percona-toolkit/src/go'
Please use `make <target>` where <target> is one of:
  help                      Display this help message.
  env-up                    Start MongoDB docker containers cluster
  env-down                  Clean-up MongoDB docker containers cluster
  linux-amd64               Build Go tools for linux-amd64.
  linux-arm64               Build Go tools for linux-arm64.
  linux-386                 Build Go tools for linux-386
  darwin-amd64              Build Go tools for darwin-amd64 (MacOS)
  darwin-arm64              Build Go tools for darwin-arm64 (MacOS)
  build                     Build Go tools for current platform
  style                     Check code style
  test                      Run tests
  format                    Format source code.
  vet                       Run vet on Go code

To build binaries use: VERSION=m.n.p make <target>
Example: VERSION=3.4.0 make linux-amd64
The binaries generation uses pwd command to set the base directory. Please run make from the same directory where the Makefile is located.
make: Leaving directory '/home/sveta/src/percona/percona-toolkit/src/go'

[cho-m]

For other platforms, we have either GOOS or GOARCH environments or special build instructions:

There is no need to use GOOS or GOARCH unless cross-compiling. Go knows how to build binaries natively.

You can run make or make gotools or make -C src/go build and create the binaries on all Go-supported platforms natively.

Building binaries has never been a problem. It is the installation.

---

For example, following shows make gotools will build the binaries:

❯ export PERL5LIB=/opt/homebrew/opt/perl-dbd-mysql/libexec/lib/perl5

❯ cd /tmp

❯ curl https://downloads.percona.com/downloads/percona-toolkit/3.7.1/source/tarball/percona-toolkit-3.7.1.tar.gz -sLO

❯ tar zxf ./percona-toolkit-3.7.1.tar.gz

❯ cd percona-toolkit-3.7.1

❯ perl Makefile.PL
Checking if your kit is complete...
Warning: the following files are missing in your kit:
	CONTRIBUTE.md
	Gopkg.lock
	Gopkg.toml
Please inform the author.
Generating a Unix-style Makefile
Writing Makefile for Percona::Toolkit
Writing MYMETA.yml and MYMETA.json

❯ make gotools
cd src/go && make build
Building binaries in /tmp/percona-toolkit-3.7.1/src/go/../../bin as version v3.7.0-2

❯ file bin/pt-galera-log-explainer
bin/pt-galera-log-explainer: Mach-O 64-bit executable arm64

---

Another example that make does the same:

❯ cd /tmp

❯ mkdir percona-toolkit

❯ tar zxf ./percona-toolkit-3.7.1.tar.gz -C percona-toolkit

❯ cd percona-toolkit/percona-toolkit-3.7.1

❯ perl Makefile.PL
Checking if your kit is complete...
Warning: the following files are missing in your kit:
	CONTRIBUTE.md
	Gopkg.lock
	Gopkg.toml
Please inform the author.
Generating a Unix-style Makefile
Writing Makefile for Percona::Toolkit
Writing MYMETA.yml and MYMETA.json

❯ make
cp lib/OptionParser.pm blib/lib/OptionParser.pm
cp lib/DiskstatsGroupByDisk.pm blib/lib/DiskstatsGroupByDisk.pm
cp lib/TableSyncStream.pm blib/lib/TableSyncStream.pm
...
cp bin/pt-visual-explain blib/script/pt-visual-explain
"/usr/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/pt-visual-explain
cd src/go && make build
Building binaries in /private/tmp/percona-toolkit/percona-toolkit-3.7.1/src/go/../../bin as version v3.7.0-2
Manifying 35 pod documents

❯ ls -tr bin | tail -5
pt-k8s-debug-collector
pt-mongodb-query-digest
pt-mongodb-summary
pt-galera-log-explainer
pt-mongodb-index-check

❯ file bin/pt-galera-log-explainer
bin/pt-galera-log-explainer: Mach-O 64-bit executable arm64

[svetasmirnova]

@cho-m Thank you for the explanation. I think you just do:

perl Makefile.PL
make install

And this is why Go tools are ignored. I will test it and be back.

[cho-m]

Splitting make and make install doesn't change result (it also builds the Go binaries twice)

perl Makefile.PL INSTALL_BASE=$PWD/tmp
make
make install

❯ ls tmp/bin/*galera*
zsh: no matches found: tmp/bin/*galera*

❯ ls tmp/bin/*upgrade*
tmp/bin/pt-upgrade

❯ ls bin/*galera*
bin/pt-galera-log-explainer

---

If you want to keep same documented steps, then BIN_DIR is the closest thing that works (though haven't checked if anything breaks from overriding this):

perl Makefile.PL INSTALL_BASE=$PWD/tmp
make install BIN_DIR=$PWD/tmp/bin

❯ ls tmp/bin/*galera*
tmp/bin/pt-galera-log-explainer