openssl-sys crate doesn't build with vendored feature enabled

[MichaelPaddon]

The crate builds fine using vanilla cargo.

The build script fails when it is trying to copy the openssl source:

Build script process failed with exit code 101
--stdout:
cargo:rustc-cfg=const_fn
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_NO_VENDOR
X86_64_UNKNOWN_LINUX_GNU_OPENSSL_NO_VENDOR unset
cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
OPENSSL_NO_VENDOR unset

--stderr:
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', external/crate_index__openssl-src-111.22.0-1.1.1q/src/lib.rs:507:32

When I disable sandboxing, the copy succeeds, but the make fails.

[fmorency]

Got bitten by this as well. Is there a known workaround?

[bazaglia]

I've tried compiling openssl with Bazel and providing it in the crate.annotation, similarly to what is done here:

rules_rust/examples/crate_universe/WORKSPACE.bazel

Lines 180 to 192 in 21eed19

	"openssl-sys": [crate.annotation(
	build_script_data = [
	"@openssl//:gen_dir",
	"@openssl//:openssl",
	],
	build_script_data_glob = ["build/**/*.c"],
	build_script_env = {
	"OPENSSL_DIR": "$(execpath @openssl//:gen_dir)",
	"OPENSSL_STATIC": "1",
	},
	data = ["@openssl"],
	deps = ["@openssl"],
	)],

But I still got the same error as @MichaelPaddon shared. I also tried using the openssl from my machine. I build on RBE but the openssl paths are the same on the host and execution machines:

"openssl-sys": [crate.annotation(
    build_script_env = {
        "OPENSSL_LIB_DIR": "/usr/lib/x86_64-linux-gnu",
        "OPENSSL_INCLUDE_DIR": "/usr/include/openssl",
        "OPENSSL_STATIC": "1",
    },
)],

It still didn't work. For some reason, it cannot open the directories specified. One interesting thing I noticed: when I disable the sandbox, it does go a bit further, but fails when trying to configure/compile openssl.

The only workaround I could find so far is to force dynamic linking instead of static link the openssl library. The annotation looks like that:

"openssl-sys": [crate.annotation(
    build_script_env = {
        "OPENSSL_NO_VENDOR": "1",
    },
)],

But that is far from ideal. Hopefully, someone will find a fix to this issue.

I also tried to disable the build script completely by setting gen_build_script and providing the static library in the annotation deps. With this, the openssl-sys crate succeeds, but the openssl fails because of missing types: the Rust compilation fails with 2.000+ errors complaining of type not found in ffi.

[CinchBlue]

I managed to get this working with static linking, at least on aarch64-apple-darwin with Rust 1.70. Here is what my structure looks like:

# //external/bazel/openssl/BUILD.build.bazel

"""An openssl build file based on a snippet found in the github issue:
https://github.com/bazelbuild/rules_foreign_cc/issues/337
"""

load("@rules_foreign_cc//foreign_cc:defs.bzl", "configure_make", "configure_make_variant")

# Read https://wiki.openssl.org/index.php/Compilation_and_Installation.

filegroup(
    name = "all_srcs",
    srcs = glob(["**"]),
)

CONFIGURE_OPTIONS = [
    "no-comp",
    "no-idea",
    "no-weak-ssl-ciphers",
    "no-shared",
]

LIB_NAME = "openssl"

MAKE_TARGETS = [
    "build_libs",
    "install_dev",
]

config_setting(
    name = "clang_cl_compiler",
    flag_values = {
        "@bazel_tools//tools/cpp:compiler": "clang-cl",
    },
    visibility = ["//visibility:public"],
)

alias(
    name = "openssl",
    actual = select({
        ":clang_cl_compiler": "openssl_clang_cl",
        "//conditions:default": "openssl_default",
    }),
    visibility = ["//visibility:public"],
)

configure_make_variant(
    name = "openssl_clang_cl",
    build_data = [
        "@nasm//:nasm",
        "@perl//:perl",
    ],
    configure_command = "Configure",
    configure_in_place = True,
    configure_options = CONFIGURE_OPTIONS + [
        # Target
        "VC-WIN64A",
        # Override environment variables set by higher-level Bazel targets.
        # Most importantly, set the C++ compiler to be `cl` (as opposed to 
        #`clang-cl`). We're unsure whether OpenSSL can be built using
        # clang-cl. 
        # TODO(#3): Investigate whether OpenSSL can
        # build with clang-cl.
        "CC=\"cl\"",
        "ASFLAGS=\"-g\"",
        "CFLAGS=\"/W3 /wd4090 /nologo /O2 /Zi /MP\"",
        "LDFLAGS=\"/nologo /debug /Zi\"",
        "ARFLAGS=\"/nologo\"",
        "LD=\"link\"",
    ],
    configure_prefix = "$PERL",
    env = {
        "PATH": "$(dirname $(execpath @nasm//:nasm)):$PATH",
        "PERL": "$(execpath @perl//:perl)",
    },
    lib_name = LIB_NAME,
    lib_source = ":all_srcs",
    out_static_libs = [
        "libssl.lib",
        "libcrypto.lib",
    ],
    targets = MAKE_TARGETS,
    toolchain = "@rules_foreign_cc//toolchains:preinstalled_nmake_toolchain",
)

configure_make(
    name = "openssl_default",
    configure_command = "config",
    configure_in_place = True,
    configure_options = CONFIGURE_OPTIONS,
    env = select({
        "@platforms//os:macos": {"AR": ""},
        "//conditions:default": {},
    }),
    lib_name = LIB_NAME,
    lib_source = ":all_srcs",
    # Note that for Linux builds libssl must be listed before
    # libcrypto on the linker command-line.
    out_static_libs = [
        "libssl.a",
        "libcrypto.a",
    ],
    targets = MAKE_TARGETS,
)

filegroup(
    name = "gen_dir",
    srcs = [":openssl"],
    output_group = "gen_dir",
    visibility = ["//visibility:public"],
)

# //WORKSPACE.bazel

############################################################
# General dependencies
############################################################

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# bazel-skylib is on an older version unless imported
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "bazel_skylib",
    sha256 = "cd55a062e763b9349921f0f5db8c3933288dc8ba4f76dd9416aac68acee3cb94",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.5.0/bazel-skylib-1.5.0.tar.gz",
        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.5.0/bazel-skylib-1.5.0.tar.gz",
    ],
)
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()

# rules_perl for OpenSSL
git_repository(
    name = "rules_perl",
    remote = "https://github.com/bazelbuild/rules_perl.git",
    commit = "366b6aa76b12056a9e0cc23364686f25dcc41702",
)
load("@rules_perl//perl:deps.bzl", "perl_register_toolchains", "perl_rules_dependencies")
perl_rules_dependencies()
perl_register_toolchains()


# OpenSSL + foreign_cc rules
# Group the sources of the library so that rules in rules_foreign_cc have access to it
http_archive(
    name = "rules_foreign_cc",
    sha256 = "476303bd0f1b04cc311fc258f1708a5f6ef82d3091e53fd1977fa20383425a6a",
    strip_prefix = "rules_foreign_cc-0.10.1",
    url = "https://github.com/bazelbuild/rules_foreign_cc/releases/download/0.10.1/rules_foreign_cc-0.10.1.tar.gz",
)
load("@rules_foreign_cc//foreign_cc:repositories.bzl", "rules_foreign_cc_dependencies")
rules_foreign_cc_dependencies(register_built_pkgconfig_toolchain = True)
http_archive(
    name = "openssl",
    urls = ["https://github.com/openssl/openssl/archive/OpenSSL_1_1_1d.tar.gz"],
    sha256 = "23011a5cc78e53d0dc98dfa608c51e72bcd350aa57df74c5d5574ba4ffb62e74",
    strip_prefix = "openssl-OpenSSL_1_1_1d",
    build_file = "bazel/openssl/BUILD.bazel"
)

# rules_rust
http_archive(
    name = "rules_rust",
    sha256 = "6357de5982dd32526e02278221bb8d6aa45717ba9bbacf43686b130aa2c72e1e",
    urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.30.0/rules_rust-v0.30.0.tar.gz"],
)
load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")
rules_rust_dependencies()
rust_register_toolchains(
    edition = "2021",
    versions = [
        "1.70.0",
    ]
)

# crate_universe
load("@rules_rust//crate_universe:repositories.bzl", "crate_universe_dependencies")
crate_universe_dependencies()
load("@rules_rust//crate_universe:defs.bzl", "crate", "crates_repository", "splicing_config")
crates_repository(
    name = "crate_index",
    cargo_lockfile = "//:Cargo.lock",
    lockfile = "//:cargo-bazel-lock.json",
    manifests = [
        "//:Cargo.toml",
    ],
    annotations = {
        "openssl-sys": [crate.annotation(
            build_script_data = [
                "@openssl//:gen_dir",
                "@openssl//:openssl",
            ],
            build_script_data_glob = ["build/**/*.c"],
            build_script_env = {
                # "OPENSSL_DIR": "$(location @openssl//:gen_dir)",
                # "OPENSSL_LIB_DIR": "$(execpath @openssl//:gen_dir)/lib",
                "OPENSSL_STATIC": "1",
                # "OPENSSL_NO_VENDOR": "1",
            },
            data = ["@openssl"],
            deps = ["@openssl"],
        )],
    },
    splicing_config = splicing_config(
        resolver_version = "2",
    )
)
load("@crate_index//:defs.bzl", "crate_repositories")
crate_repositories()

NOTE:

• DO NOT set OPENSSL_DIR or OPENSSL_LIB_DIR as they are indeed set correctly. (If you do set them with something like $(location @openssl//:gen_dir), it will not resolve to the correct directory but instead resolve to some copy_openssl directory.
• DO NOT set OPENSSL_NO_VENDOR as this given BUILD.bazel file will install the libraries into the system using the make command within the OpenSSL repo
• Unset the vendored feature for openssl and openssl-sys.

[erenon]

For the record, here's a working version with bzlmod. The key idea: build openssl itself with rules_foreign_cc statically, then make openssl-sys pick it up, as it was not vendored.

# MODULE.bazel:
bazel_dep(name = "rules_foreign_cc", version = "0.12.0")

http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "openssl",
    build_file = "BUILD.openssl.bazel",
    strip_prefix = "openssl-3.4.0",
    urls = ["https://github.com/openssl/openssl/releases/download/openssl-3.4.0/openssl-3.4.0.tar.gz"],
    integrity = "sha256-4V3agv4v6BOdwqwho21MoB1TE8dfmfRsTooncJtylL8="
)

# ... load `crate`, add `openssl`, etc.

crate.annotation(
  crate = "openssl-sys",
  build_script_data = [
    "@@_main~_repo_rules~openssl//:gen_dir",
  ],
  build_script_env = {
    "OPENSSL_NO_VENDOR": "1",
    "OPENSSL_LIB_DIR": "$(location @@_main~_repo_rules~openssl//:gen_dir)/lib64",
    "OPENSSL_INCLUDE_DIR": "$(location @@_main~_repo_rules~openssl//:gen_dir)/include",
    "OPENSSL_STATIC": "1",
  },
)

# BUILD.openssl.bazel
# Inspired by https://github.com/bazel-contrib/rules_foreign_cc/blob/main/examples/third_party/openssl/BUILD.openssl.bazel

load("@rules_foreign_cc//foreign_cc:defs.bzl", "configure_make")

filegroup(
    name = "all_srcs",
    srcs = glob(
        include = ["**"],
        exclude = ["*.bazel"],
    ),
)

# https://wiki.openssl.org/index.php/Compilation_and_Installation
CONFIGURE_OPTIONS = [
    "no-comp",
    "no-idea",
    "no-weak-ssl-ciphers",
    "no-shared",
]

MAKE_TARGETS = [
    "build_libs",
    "install_dev",
]

configure_make(
    name = "openssl",
    args = ["-j12"],
    configure_command = "config",
    configure_in_place = True,
    configure_options = CONFIGURE_OPTIONS,
    lib_name = "openssl",
    lib_source = ":all_srcs",
    out_lib_dir = "lib64",
    #out_binaries = ["openssl"],
    out_shared_libs = [],
    out_static_libs = ["libssl.a"],
    targets = MAKE_TARGETS,
    visibility = ["//visibility:public"],
)

filegroup(
    name = "gen_dir",
    srcs = [":openssl"],
    output_group = "gen_dir",
    visibility = ["//visibility:public"],
)

[laduke]

Anyone have a recent example of this? I can't get anything to work. It's probably me.
Can't use the openssl from registry ? https://registry.bazel.build/modules/openssl

---

yeah. I can't figure out how to refer to openssl (from bcr) in the crate annotation. #2736 seemed related.