Bazel 使用maven管理依赖库

JDK11 获取到 JDK17编译的依赖

有下面依赖 repository.bzl

load("@rules_jvm_external//:defs.bzl", "maven_install")

def java_maven_repository():
    maven_install(
        artifacts = [
            "org.slf4j:slf4j-api:1.7.25",
            "org.slf4j:slf4j-simple:1.7.25",
            "com.google.guava:guava:30.1-jre",
            "info.picocli:picocli:4.7.0",
            "com.alibaba:fastjson:1.2.83",
	    "com.github.java-json-tools:json-schema-validator:2.2.14",
            "org.eclipse.xtext:org.eclipse.xtext:2.28.0",
            "org.eclipse.xtext:org.eclipse.xtext.util:2.28.0",
            "org.eclipse.xtext:org.eclipse.xtext.common.types:2.28.0",
            "org.eclipse.xtend:org.eclipse.xtend.lib:2.28.0",
            "org.eclipse.emf:org.eclipse.emf.edit:2.17.0",
            "org.eclipse.platform:org.eclipse.core.runtime:3.17.100",
            "org.eclipse.platform:org.eclipse.core.commands:3.9.700",
            "org.eclipse.platform:org.eclipse.core.resources:3.13.700",
            "org.eclipse.birt.runtime:javax.xml:1.3.4.v201005080400",
            "org.eclipse.birt.runtime.3_7_1:org.apache.xerces:2.9.0",
            "javax.activation:activation:1.1.1",
            "org.eclipse.sphinx.plugins:org.eclipse.sphinx.platform:0.11.2-20230505.071004-2",
            "org.eclipse.sphinx.plugins:org.eclipse.sphinx.emf:0.11.2-20230505.073217-2",
   
        ],
        fetch_sources = False,
        repositories = [
            "https://repo1.maven.org/maven2/",
        ],
    )

使用依赖库: BUILD

java_binary(
    name = "apconfig",
    srcs = glob([
     ...
    ]),
    resources = glob([
      ...
    ]),
    resource_strip_prefix = "...",
    main_class = "TestMain",
    deps = [
      ":xtext_resources",
      ":arxml_resources",

      "@maven//:org_slf4j_slf4j_api",
      "@maven//:org_slf4j_slf4j_simple",
      "@maven//:org_eclipse_birt_runtime_javax_xml",
      "@maven//:javax_activation_activation",
      "@maven//:org_eclipse_birt_runtime_3_7_1_org_apache_xerces",
      "@maven//:com_google_guava_guava",
      "@maven//:info_picocli_picocli",
      "@maven//:com_alibaba_fastjson",
      "@maven//:com_github_java_json_tools_json_schema_validator",
      "@maven//:com_github_java_json_tools_jackson_coreutils",
      "@maven//:com_github_java_json_tools_json_schema_core",
      "@maven//:com_fasterxml_jackson_core_jackson_databind",

      "@maven//:org_eclipse_xtext_org_eclipse_xtext",
      "@maven//:com_google_inject_guice",
      "@maven//:org_antlr_antlr_runtime",
      "@maven//:org_eclipse_emf_org_eclipse_emf_ecore",
      "@maven//:org_eclipse_emf_org_eclipse_emf_common",
      "@maven//:org_eclipse_xtext_org_eclipse_xtext_util",

      "@maven//:org_eclipse_emf_org_eclipse_emf_edit",

      "@maven//:org_eclipse_xtext_org_eclipse_xtext_common_types",

      "@maven//:org_eclipse_xtend_org_eclipse_xtend_lib",
      "@maven//:org_eclipse_xtext_org_eclipse_xtext_xbase_lib",

      "@maven//:org_eclipse_platform_org_eclipse_core_runtime",
      "@maven//:org_eclipse_platform_org_eclipse_core_commands",
      "@maven//:org_eclipse_platform_org_eclipse_core_resources",
    ],
)

编译正常，运行时报错:

Caused by: java.lang.UnsupportedClassVersionError: 
org/eclipse/core/runtime/jobs/ISchedulingRule has been compiled by a more recent version of 
the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes 
class file versions up to 55.0

使用javap查看class文件:

javap -v ./org/eclipse/core/runtime/jobs/ISchedulingRule.class 
Classfile /root/.cache/bazel/_bazel_root/74701efd2f2b9f7ec0e1a36d2d8b0c9b/execroot/__main__/bazel-out/k8-fastbuild/bin/t/org/eclipse/core/runtime/jobs/ISchedulingRule.class
  Last modified Dec 31, 2009; size 237 bytes
  MD5 checksum 36395a293069bd6508d359e7ecadb119
  Compiled from "ISchedulingRule.java"
public interface org.eclipse.core.runtime.jobs.ISchedulingRule
  minor version: 0
  major version: 61
  flags: (0x0601) ACC_PUBLIC, ACC_INTERFACE, ACC_ABSTRACT

查看maven上jar文件: https://repo1.maven.org/maven2/org/eclipse/platform/org.eclipse.core.jobs/3.14.0/org.eclipse.core.jobs-3.14.0.jar

确实更新了，使用JDK17编译:/META-INF/MANIFEST.MF

Manifest-Version: 1.0
Created-By: Maven Archiver 3.6.0
Build-Jdk-Spec: 17
Bundle-ManifestVersion: 2
Bundle-Name: %pluginName
Bundle-SymbolicName: org.eclipse.core.jobs; singleton:=true
Bundle-Version: 3.14.0.v20230317-0901

更新时间: https://repo1.maven.org/maven2/org/eclipse/platform/org.eclipse.core.jobs/

3.13.300/                                         2023-03-13 10:43         -  
3.14.0/                                           2023-06-09 14:43         -

但在repository.bzl中指定使用org/eclipse/platform/org.eclipse.core.jobs的版本。

在repository.bzl添加: "org.eclipse.platform:org.eclipse.core.jobs:3.13.300",

BUILD添加: "@maven//:org_eclipse_platform_org_eclipse_core_jobs", (可不添加)

还其他依赖存在相同问题。

"org.eclipse.platform:org.eclipse.core.expressions:3.8.200",
"org.eclipse.platform:org.eclipse.core.filesystem:1.9.400",
"org.eclipse.platform:org.eclipse.core.jobs:3.13.300",
"org.eclipse.xtext:org.eclipse.xtext.xbase.lib:2.28.0",
"org.eclipse.emf:org.eclipse.emf.ecore:2.28.0",

在repository.bzl添加所有依赖库并指定版本。

maven_install_json 锁定下载版本

参考文档: https://github.com/bazelbuild/rules_jvm_external#multiple-maven_installjson-files

在maven_install配置name，生成指定name的maven_install.json

load("@rules_jvm_external//:defs.bzl", "maven_install")
  
def java_maven():
    maven_install(
        name="xx_maven",
        artifacts = [
            "org.slf4j:slf4j-api:1.7.25",
            "org.slf4j:slf4j-simple:1.7.25",
            ...
        ],
        fetch_sources = False,
        repositories = [
            "https://repo1.maven.org/maven2/",
        ],
        fail_on_missing_checksum = True,
        maven_install_json = "//third_party:maven_java/xx_maven_install.json",
    )

生成 maven_install.json文件:

命令: bazel run @xx_maven//:pin

bazel run @xx_maven//:pin
输出:
INFO: Analyzed target @xx_maven//:pin (1 packages loaded, 1 target configured).
INFO: Found 1 target...
INFO: Elapsed time: 24.800s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Successfully pinned resolved artifacts for @xx_maven in <当前目录>/xx_maven_install.json. This file should be checked in your version control system.

Next, please update your WORKSPACE file by adding the maven_install_json attribute and loading pinned_maven_install from @xx_maven//:defs.bzl.

For example:

=============================================================

maven_install(
    artifacts = # ...,
    repositories = # ...,
    maven_install_json = "//:xx_maven_install.json",
)

load("@xx_maven//:defs.bzl", "pinned_maven_install")
pinned_maven_install()

=============================================================

To update xx_maven_install.json, run this command to re-pin the unpinned repository:

    bazel run @unpinned_xx_maven//:pin

在maven_install使用maven_install.json:

maven_install_json = "//third_party:maven_java/xx_maven_install.json",

在WORKSPACE中java_maven()后添加: pinned_maven_install()

load("//third_party:maven_java/repository.bzl", "java_maven")
java_maven()

load("@xx_maven//:defs.bzl", "pinned_maven_install")
pinned_maven_install()

如果该操作，编译jar包时报错:

bazel build <jar_target>
输出:
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
ERROR: /root/.cache/bazel/_bazel_root/74701efd2f2b9f7ec0e1a36d2d8b0c9b/external/maven/BUILD:546:8: no such package '@joda_time_joda_time_2_10_5//file': The repository '@joda_time_joda_time_2_10_5' could not be resolved: Repository '@joda_time_joda_time_2_10_5' is not defined and referenced by '@maven//:joda_time_joda_time_2_10_5_extension'
ERROR: Analysis of target '//<jar_target>:' failed; build aborted: 
INFO: Elapsed time: 7.717s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (41 packages loaded, 416 targets configured)

因为maven_install指定了name，对应的BUILD中也要使用对应name的maven。

java_binary(
    name = "apconfig",
    srcs = glob([
      ...
    ]),
    resources = glob([
      "resources/plugin.properties",
    ]),
    resource_strip_prefix = "resources/",
    main_class = "XXConfig",
    deps = [
      "@xx_maven//:org_slf4j_slf4j_api",
      "@xx_maven//:org_slf4j_slf4j_simple",
      ...
    ],
)

更新maven_install: bazel run @unpinned_xx_maven//:pin

bazel run @unpinned_xx_maven//:pin
输出:
INFO: Analyzed target @unpinned_xx_maven//:pin (1 packages loaded, 1 target configured).
INFO: Found 1 target...
INFO: Elapsed time: 30.135s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Successfully pinned resolved artifacts for @xx_maven in <当前目录>/xx_maven_install.json. This file should be checked in your version control system.

Next, please update your WORKSPACE file by adding the maven_install_json attribute and loading pinned_maven_install from @xx_maven//:defs.bzl.

For example:

=============================================================

maven_install(
    artifacts = # ...,
    repositories = # ...,
    maven_install_json = "//:xx_maven_install.json",
)

load("@xx_maven//:defs.bzl", "pinned_maven_install")
pinned_maven_install()

=============================================================

To update xx_maven_install.json, run this command to re-pin the unpinned repository:

    bazel run @unpinned_xx_maven//:pin

如果执行: bazel run @xx_maven//:pin 会报错

INFO: Analyzed target @xx_maven//:pin (0 packages loaded, 1 target configured).
INFO: Found 1 target...
ERROR: /root/.cache/bazel/_bazel_root/74701efd2f2b9f7ec0e1a36d2d8b0c9b/external/xx_maven/BUILD:4:14: @xx_maven//:pin: missing input file 'external/xx_maven/pin', owner: '@xx_maven//:pin'
ERROR: /root/.cache/bazel/_bazel_root/74701efd2f2b9f7ec0e1a36d2d8b0c9b/external/xx_maven/BUILD:4:14: 1 input file(s) do not exist
INFO: Elapsed time: 0.121s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Pinning artifacts and integration with Bazel’s downloader

参考文档: https://github.com/bazelbuild/rules_jvm_external#multiple-maven_installjson-files

rules_jvm_external supports pinning artifacts and their SHA-256 checksums into a maven_install.json file that can be checked into your repository.

Without artifact pinning, in a clean checkout of your project, rules_jvm_external executes the full artifact resolution and fetching steps (which can take a bit of time) and does not verify the integrity of the artifacts against their checksums. The downloaded artifacts also cannot be shared across Bazel workspaces.

By pinning artifact versions, you can get improved artifact resolution and build times, since using maven_install.json enables rules_jvm_external to integrate with Bazel’s downloader that caches files on their sha256 checksums. It also improves resiliency and integrity by tracking the sha256 checksums and original artifact urls in the JSON file.

Since all artifacts are persisted locally in Bazel’s cache, it means that fully offline builds are possible after the initial bazel fetch @maven//.... The artifacts are downloaded with http_file which supports netrc for authentication. Your ~/.netrc will be included automatically. To pass machine login credentials in the ~/.netrc file to coursier, specify use_credentials_from_home_netrc_file = True in your maven_install rule. For additional credentials, add them in the repository URLs passed to maven_install (so they will be included in the generated JSON). Alternatively, pass an array of additional_netrc_lines to maven_install for authentication with credentials from outside the workspace.

To get started with pinning artifacts, run the following command to generate the initial maven_install.json at the root of your Bazel workspace:

$ bazel run @maven//:pin

Then, specify maven_install_json in maven_install and load pinned_maven_install from @maven//:defs.bzl:

maven_install(
    # artifacts, repositories, ...
    maven_install_json = "//:maven_install.json",
)

load("@maven//:defs.bzl", "pinned_maven_install")
pinned_maven_install()

Note: The //:maven_install.json label assumes you have a BUILD file in your project’s root directory. If you do not have one, create an empty BUILD file to fix issues you may see. See #242

Note: If you’re using an older version of rules_jvm_external and haven’t repinned your dependencies, you may see a warning that you lock file “does not contain a signature of the required artifacts” then don’t worry: either ignore the warning or repin the dependencies.

Updating maven_install.json

Whenever you make a change to the list of artifacts or repositories and want to update maven_install.json, run this command to re-pin the unpinned @maven repository:

$ bazel run @unpinned_maven//:pin

Without re-pinning, maven_install will not pick up the changes made to the WORKSPACE, as maven_install.json is now the source of truth.

Note that the repository is @unpinned_maven instead of @maven. When using artifact pinning, each maven_install repository (e.g. @maven) will be accompanied by an unpinned repository. This repository name has the @unpinned_ prefix (e.g.@unpinned_maven or @unpinned_<your_maven_install_name>). For example, if your maven_install is named @foo, @unpinned_foo will be created.

Requiring lock file repinning when the list of artifacts changes

It can be easy to forget to update the maven_install.json lock file when updating artifacts in a maven_install. Normally, rules_jvm_external will print a warning to the console and continue the build when this happens, but by setting the fail_if_repin_required attribute to True, this will be treated as a build error, causing the build to fail. When this attribute is set, it is possible to update the maven_install.json file using:

# To repin everything:
REPIN=1 bazel run @unpinned_maven//:pin

# To only repin rules_jvm_external:
RULES_JVM_EXTERNAL_REPIN=1 bazel run @unpinned_maven//:pin

Alternatively, it is also possible to modify the fail_if_repin_required attribute in your WORKSPACE file, run bazel run @unpinned_maven//:pin and then reset the fail_if_repin_required attribute.

Custom location for maven_install.json

You can specify a custom location for maven_install.json by changing the maven_install_json attribute value to point to the new file label. For example:

maven_install(
    name = "maven_install_in_custom_location",
    artifacts = ["com.google.guava:guava:27.0-jre"],
    repositories = ["https://repo1.maven.org/maven2"],
    maven_install_json = "@rules_jvm_external//tests/custom_maven_install:maven_install.json",
)

load("@maven_install_in_custom_location//:defs.bzl", "pinned_maven_install")
pinned_maven_install()

Future artifact pinning updates to maven_install.json will overwrite the file at the specified path instead of creating a new one at the default root directory location.

Multiple maven_install.json files

If you have multiple maven_install declarations, you have to alias pinned_maven_install to another name to prevent redefinitions:

maven_install(
    name = "foo",
    maven_install_json = "//:foo_maven_install.json",
    # ...
)

load("@foo//:defs.bzl", foo_pinned_maven_install = "pinned_maven_install")
foo_pinned_maven_install()

maven_install(
    name = "bar",
    maven_install_json = "//:bar_maven_install.json",
    # ...
)

load("@bar//:defs.bzl", bar_pinned_maven_install = "pinned_maven_install")
bar_pinned_maven_install()