将 native library 打包进 jar 里


有一个 Java 写的程序,需要在 Spark 上跑,所以运行之前没法设置环境变量,也没办法设置 Java 运行参数(不知道是否真的不能设置,但甲方这么说,无奈啊),需要在运行的时候把 native library 加载起来。之前没有怎么写过 Java ,琢磨了一下反正也写出来了。

Tensorflow Java 1.14.0 的 bug

Tensorflow 在运行的时候, Linux 报:

Exception in thread 'main' java.lang.UnsatisfiedLinkError: /tmp/tensorflow_native_libraries-1680337452521-0/libtensorflow_jni.so: libtensorflow_framework.so.1: 无法打开共享文件:没有那个文件或目录
  at ...
  at java.lang.System.load(System.java:1100)
  at org.tensorflow.NativeLibrary.load(NativeLibrary.java:101)
  at ...

看这个 PR tensorflow/tensorflow#32829 , tensorflow 解压了 libtensorflow_framework.so ,但 libtensorflow_jni.so 链接到的是 libtensorflow_framework.so.1 这个文件。

在这个版本的话,要想修复这个问题有很多种办法,比如用 maven-plugin-shade 在打包的时候把这个类替换掉。但是甲方还是不同意~另一个简单的方法就是抢先在 tensorflow 初始化前先把 libtensorflow_framework.so 加载进来,同 tf 加载的套路一样,从资源文件 jar 里把 so 解压出来,然后调用 System.load 加载就 OK 了。

JSCIP 的编译和打包

之所以之前那么加载是 OK 的,是因为如果一个动态链接库已经被 Java 加载了,那么依赖它的动态链接库就可以直接用内存中的这个库,而不会再重新从文件系统中再加载。这就意味着不能直接用 conda-forge 打包的 SCIP ,或者是直接用 conda-forge 的工具链去编译。(倒不是说 conda-forge 的工具链不能编译,而是 conda-forge 里包的依赖关系做得太烂,很多 native 的包依赖高版本 glibc ,而没有把 sysroot 写进依赖里。)

目标机器是 CentOS 7 ,为了简单本地搭了一下环境:

cmake .. -DTBB_DIR="$(pwd)/../../tbb2019_20190320oss" -DBOOST_INCLUDEDIR=/home/azuk/sciptest/scip2/boost_1_81_0   -DBOOST_LIBRARYDIR=/home/azuk/sciptest/scip2/boost_1_81_0/libs

编译 papilo 报错了,但是不想研究为啥,因为 libscip 编译出来了。但是 SCIP 不应该依赖 SoPlex 吗,我不理解啊。。

之后就是 ldd 把依赖找出来,打进一个 jar 包里,等需要用的时候解压出来就行了。

参考了 tensorflow 的 NativeLibrary.java 瞎编的:

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;

public class NativeLibrary {
    private static boolean loaded = false;
    private static File tempPath = null;

    public static void load() {
        if (loaded) return;
        log(String.format("classpath: %s", System.getProperty ("java.class.path")));
        switch (os()) {
            case "windows":
                loadLibrary("tbb.dll");
                loadLibrary("libscip.dll");
                loadLibrary("jscip.dll");
                break;
            case "linux":
                loadLibrary("libtinfo.so.5");
                loadLibrary("libz.so.1");
                loadLibrary("libpord-5.3.so");
                loadLibrary("libquadmath.so.0");
                loadLibrary("libgfortran.so.3");
                loadLibrary("libgmp.so.10");
                loadLibrary("libgmpxx.so.4");
                loadLibrary("libblas.so.3");
                loadLibrary("libbz2.so.1");
                loadLibrary("libdl.so.2");
                loadLibrary("libreadline.so.6");
                loadLibrary("libtbb.so.2");
                loadLibrary("libgomp.so.1");
                loadLibrary("libscotcherr.so.0");
                loadLibrary("libscotch.so.0");
                loadLibrary("libmetis.so.0");
                loadLibrary("libmpiseq-5.3.so");
                loadLibrary("libesmumps.so.0");
                loadLibrary("libscotchmetis.so.0");
                loadLibrary("libopenblas.so.0");
                loadLibrary("libmumps_common-5.3.so");
                loadLibrary("libdmumps-5.3.so");
                loadLibrary("libipopt.so.3");
                loadLibrary("libscip.so.8.0");
                loadLibrary("libjscip.so");
                break;
        }
        loaded = true;
    }

    private static void log(String msg) {
        System.err.println("NativeLibrary: " + msg);
    }
    private static void loadLibrary(String libName) {
        final String jniLibName = libName;
        log(String.format("Load library %s", jniLibName));
        final String resourceName = makeResourceName(jniLibName);


        InputStream jniResource = NativeLibrary.class.getResourceAsStream(resourceName);
        if (jniResource == null) {
            jniResource = NativeLibrary.class.getClass().getResourceAsStream(resourceName);
        }
        if (jniResource == null) {
            throw new UnsatisfiedLinkError("Native library not found");
        }

        String libPath;
        try {
            if (tempPath == null) {
                tempPath = createTemporaryDirectory();
                tempPath.deleteOnExit();
            }
            final String tempDirectory = tempPath.getCanonicalPath();
            libPath = extractResource(jniResource, jniLibName, tempDirectory);
        } catch (IOException e) {
            throw new UnsatisfiedLinkError("Unable to extract native library.");
        }

        System.load(libPath);
    }

    private static String os() {
        final String p = System.getProperty("os.name").toLowerCase();
        if (p.contains("linux")) {
            return "linux";
        } else if (p.contains("os x") || p.contains("darwin")) {
            return "darwin";
        } else if (p.contains("windows")) {
            return "windows";
        } else {
            return p.replaceAll("\\s", "");
        }
    }

    private static String architecture() {
        final String arch = System.getProperty("os.arch").toLowerCase();
        return (arch.equals("amd64")) ? "x86_64" : arch;
    }

    private static String makeResourceName(String baseName) {
        return String.format("/%s-%s/", os(), architecture()) + baseName;
    }

    private static String extractResource(
            InputStream resource, String resourceName, String extractToDirectory) throws IOException {
        final File dst = new File(extractToDirectory, resourceName);
        dst.deleteOnExit();
        final String dstPath = dst.toString();
        final long nbytes = copy(resource, dst);
        return dstPath;
    }

    private static long copy(InputStream src, File dstFile) throws IOException {
        FileOutputStream dst = new FileOutputStream(dstFile);
        try {
            byte[] buffer = new byte[1 << 20]; // 1MB
            long ret = 0;
            int n = 0;
            while ((n = src.read(buffer)) >= 0) {
                dst.write(buffer, 0, n);
                ret += n;
            }
            return ret;
        } finally {
            dst.close();
            src.close();
        }
    }

    private static File createTemporaryDirectory() {
        File baseDirectory = new File(System.getProperty("java.io.tmpdir"));
        String directoryName = "jscip-" + System.currentTimeMillis() + "-";
        for (int attempt = 0; attempt < 1000; attempt++) {
            File temporaryDirectory = new File(baseDirectory, directoryName + attempt);
            if (temporaryDirectory.mkdir()) {
                return temporaryDirectory;
            }
        }
        throw new IllegalStateException(
                "Could not create a temporary directory (tried to make "
                        + directoryName
                        + "*) to extract TensorFlow native libraries.");
    }

}

报错日志

最后是编译时的报错日志,反正能用不研究了:

[ 15%] Linking CXX executable ../../../bin/soplex_c_testing
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::Matchers::Floating::WithinUlpsMatcher::describe() const':
TestMain.cpp:(.text+0xc155): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::ReusableStringStream::~ReusableStringStream()':
TestMain.cpp:(.text+0x192c2): undefined reference to `std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::ReusableStringStream::ReusableStringStream()':
TestMain.cpp:(.text+0x194b1): undefined reference to `std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: TestMain.cpp:(.text+0x19553): undefined reference to `std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: TestMain.cpp:(.text+0x195c2): undefined reference to `std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::clara::detail::BasicResult<Catch::clara::detail::ParseResultType> Catch::clara::detail::convertInto<unsigned int>(std::string const&, unsigned int&)':
TestMain.cpp:(.text._ZN5Catch5clara6detail11convertIntoIjEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_[_ZN5Catch5clara6detail11convertIntoIjEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_]+0x20): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::clara::detail::BasicResult<Catch::clara::detail::ParseResultType> Catch::clara::detail::convertInto<long>(std::string const&, long&)':
TestMain.cpp:(.text._ZN5Catch5clara6detail11convertIntoIlEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_[_ZN5Catch5clara6detail11convertIntoIlEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_]+0x20): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::clara::detail::BasicResult<Catch::clara::detail::ParseResultType> Catch::clara::detail::convertInto<double>(std::string const&, double&)':
TestMain.cpp:(.text._ZN5Catch5clara6detail11convertIntoIdEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_[_ZN5Catch5clara6detail11convertIntoIdEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_]+0x20): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/TestMain.cpp.o: in function `Catch::clara::detail::BasicResult<Catch::clara::detail::ParseResultType> Catch::clara::detail::convertInto<int>(std::string const&, int&)':
TestMain.cpp:(.text._ZN5Catch5clara6detail11convertIntoIiEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_[_ZN5Catch5clara6detail11convertIntoIiEENS1_11BasicResultINS1_15ParseResultTypeEEERKSsRT_]+0x20): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/papilo/core/PresolveTest.cpp.o: in function `papilo::ParallelColDetection<double>::execute(papilo::Problem<double> const&, papilo::ProblemUpdate<double> const&, papilo::Num<double> const&, papilo::Reductions<double>&, papilo::Timer const&)':
PresolveTest.cpp:(.text._ZN6papilo20ParallelColDetectionIdE7executeERKNS_7ProblemIdEERKNS_13ProblemUpdateIdEERKNS_3NumIdEERNS_10ReductionsIdEERKNS_5TimerE[_ZN6papilo20ParallelColDetectionIdE7executeERKNS_7ProblemIdEERKNS_13ProblemUpdateIdEERKNS_3NumIdEERNS_10ReductionsIdEERKNS_5TimerE]+0x374): undefined reference to `__cxa_throw_bad_array_new_length'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/unit_test.dir/papilo/core/PresolveTest.cpp.o: in function `papilo::ParallelRowDetection<double>::execute(papilo::Problem<double> const&, papilo::ProblemUpdate<double> const&, papilo::Num<double> const&, papilo::Reductions<double>&, papilo::Timer const&)':
PresolveTest.cpp:(.text._ZN6papilo20ParallelRowDetectionIdE7executeERKNS_7ProblemIdEERKNS_13ProblemUpdateIdEERKNS_3NumIdEERNS_10ReductionsIdEERKNS_5TimerE[_ZN6papilo20ParallelRowDetectionIdE7executeERKNS_7ProblemIdEERKNS_13ProblemUpdateIdEERKNS_3NumIdEERNS_10ReductionsIdEERKNS_5TimerE]+0x5ff): undefined reference to `__cxa_throw_bad_array_new_length'
collect2: error: ld returned 1 exit status
make[2]: *** [papilo/test/CMakeFiles/unit_test.dir/build.make:443: papilo/test/unit_test] Error 1
make[1]: *** [CMakeFiles/Makefile2:1329: papilo/test/CMakeFiles/unit_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `operator delete(void*, unsigned long)'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `std::overflow_error::overflow_error(char const*)'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `__cxa_throw_bad_array_new_length'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `std::runtime_error::runtime_error(char const*)'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `std::invalid_argument::invalid_argument(char const*)'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `operator delete[](void*, unsigned long)'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: ../../../lib/libsoplexshared.so.6.0.2.0: undefined reference to `std::domain_error::domain_error(char const*)'
collect2: error: ld returned 1 exit status
make[2]: *** [soplex/tests/c_interface/CMakeFiles/soplex_c_testing.dir/build.make:112: bin/soplex_c_testing] Error 1
make[1]: *** [CMakeFiles/Makefile2:1553: soplex/tests/c_interface/CMakeFiles/soplex_c_testing.dir/all] Error 2
[ 15%] Building C object scip/src/CMakeFiles/libscip.dir/scip/scipgithash.c.o
[ 15%] Building C object scip/src/CMakeFiles/scip.dir/scip/scipgithash.c.o
[ 15%] Linking CXX shared library ../../lib/libscip.so
[ 15%] Linking CXX executable ../../bin/scip
[ 44%] Built target libscip
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/scip.dir/lpi/lpi_spx2.cpp.o: in function `soplex::SPxBasisBase<double>::readBasis(std::istream&, soplex::NameSet const*, soplex::NameSet const*)':
lpi_spx2.cpp:(.text._ZN6soplex12SPxBasisBaseIdE9readBasisERSiPKNS_7NameSetES5_[_ZN6soplex12SPxBasisBaseIdE9readBasisERSiPKNS_7NameSetES5_]+0x606): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: lpi_spx2.cpp:(.text._ZN6soplex12SPxBasisBaseIdE9readBasisERSiPKNS_7NameSetES5_[_ZN6soplex12SPxBasisBaseIdE9readBasisERSiPKNS_7NameSetES5_]+0x743): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: CMakeFiles/scip.dir/lpi/lpi_spx2.cpp.o: in function `soplex::SPxSolverBase<double>::factorize()':
lpi_spx2.cpp:(.text._ZN6soplex13SPxSolverBaseIdE9factorizeEv[_ZN6soplex13SPxSolverBaseIdE9factorizeEv]+0x352): undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream()'
collect2: error: ld returned 1 exit status
make[2]: *** [scip/src/CMakeFiles/scip.dir/build.make:5864: bin/scip] Error 1
make[1]: *** [CMakeFiles/Makefile2:2505: scip/src/CMakeFiles/scip.dir/all] Error 2
^Cmake[2]: *** [soplex/src/CMakeFiles/soplex.dir/build.make:83: soplex/src/CMakeFiles/soplex.dir/soplexmain.cpp.o] Interrupt
make[1]: *** [CMakeFiles/Makefile2:1496: soplex/src/CMakeFiles/soplex.dir/all] Interrupt
make: *** [Makefile:183: all] Interrupt