Fixing Mamba for Multiple User Environment

When it comes to python environment management, I would not hesitate to suggest conda as a sysadmin. It just works most of the time for python packages, python itself, and other native stuff. The problem is, conda works, but it’s too slow. Conda finally went beyond my endurance solving a maybe complex environment for a full hour, so I chose mamba, a reimplementation of conda package manager written in C++.

Mamba is great, it’s fast and solving the packages the right way. But as mamba being a reimplementation, we cannot expect it works like an alias. When I tried to set up a multi-user conda environment on my server, a strange error happened:

(base) ~$ mamba install ipython
Looking for: ['ipython']

filesystem error: cannot set file time: Operation not permitted [/opt/mambaforge/pkgs/cache/09cdf8bf.json]

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/mambaforge/lib/python3.10/site-packages/conda/exceptions.py", line 1118, in __call__
        return func(*args, **kwargs)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 936, in exception_converter
        raise e
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 929, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 887, in _wrapped_main
        result = do_call(parsed_args, p)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 750, in do_call
        exit_code = install(args, parser, "install")
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 497, in install
        index = load_channels(pool, channels, repos)
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 129, in load_channels
        index = get_index(
      File "/opt/mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 110, in get_index
        is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)
    RuntimeError: filesystem error: cannot set file time: Operation not permitted [/opt/mambaforge/pkgs/cache/09cdf8bf.json]

(Only newer versions of mamba shows the “cannot set file time” prompt.)

Mamba failed to update the package index. After some googling, I found the two issues in mamba-org/mamba: #488 RuntimeError on cache in multi-users case, #1123 Resolving from cache fails when cache dir rights are applied with ACL’s. It seems the current work around is to delete package index manually before installing new packages, but why? After some tests, conda seems to work without this bug.

I’m not familiar with C++ STL, so I decided to build it locally and debug. dlist is an instance of api.DownloadTargetList, defined in C++ source libmambapy pointing to class MultiDownloadTarget in fetch.cpp. A quick scan of the source was of no help, I decided to debug it with gdb.

Build and install libmamba, libmambapy and mamba for debug:

(base) $ # install packages from environment-dev.yml
(mdev) $ cmake -B build/ \
      -DCMAKE_BUILD_TYPE=Debug \
      -DBUILD_LIBMAMBA=ON \
      -DBUILD_SHARED=ON \
      -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
      -DCMAKE_PREFIX_PATH=$CONDA_PREFIX \
      -DBUILD_LIBMAMBAPY=ON
(mdev) $ make install -C build/ -j $(($(nproc) + 1))
(mdev) $ pip install -e libmambapy/ --no-deps
(mdev) $ pip install -e mamba/ --no-deps

The compiled mamba in different environment use a different path for cache, mamba will print the path in verbose mode. (base for $CONDA_PREFIX/pkgs/cache, others for $CONDA_PREFIX/env/ENV_NAME/pkgs/cache) According to Anaconda documentation for Installing for multiple users, set up the environment:

$ # create a new group and user
$ sudo groupadd testg
$ sudo useradd -m testu
$ sudo usermod -aG testg testu
$ sudo usermod -aG testg $USER
$ # trigger the creation of index cache
$ # e.g. mamba install ipython
$ # change the ownership
$ sudo chgrp -R /opt/mambaforge
$ sudo chmod 770 -R /opt/mambaforge
$ # oh we are not the files' owner
$ sudo chown testu /opt/mambaforge/pkgs/cache/*
$ # make the cache expired
$ sudo find /opt/mambaforge/pkgs/cache -exec touch -d "7 days ago" {} +

Conda use a magic hash for each channel index cache json. So after creating the cache file, setting the group write bit on the cache, everyone in the group can update the index themselves. Mamba does so, but likely it use a different hash function, the files are different from which conda uses.

Attach gdb to the process:

(mdev) ~$ gdb -q --args python -m mamba.mamba install ipython
gef➤  catch throw
Catchpoint 1 (throw)
gef➤  r

When mamba got 304 Not Modified from HTTP server, it just updates the timestamps. But std::filesystem::last_write_time led to the problem. On Linux, th underlying utimensat() updates the timestamps of a file, but if you are not the file owner, or you have no appropriate privilege, the timestamps cannot be modified to a time except now, even if you have write access to the file. A quick strace verified my thought.

std::filesystem::last_write_time is not a perfect function. It works in most scenes, but it’s also limited. To fix the problem, simply substitute the function with OS specific functions with appropriate parameters, like utimensat(dirfd, pathname, NULL, flags) on POSIX systems. We got a library function to avoid OS specific syscalls, but in the end we fix the bug by re-introducing the syscalls, oops. Taking boost::filesystem::last_write_time as a reference, a hack maybe applied to the std::time_t new_time parameter to touch the file, but since it’s an undocumented behavior we cannot guarantee that in different platforms or libc it works.

BTW, conda works because it directly calls os.utime [source]. Yes, os.time is available on Windows! [source] Python beats C++ again, LOL. Another reference is GNULib lib/utimens.c. It handles utimensat on each unix platform correctly, and a Windows shim was provided for the POSIX API.

A quick dirty fix for this: hooking it

#include <fcntl.h>
#include <sys/stat.h>
#include <stddef.h>
#define __USE_GNU
#include <dlfcn.h>

int utimensat(int dirfd, const char *pathname, const struct timespec times[2], int flags) {
    int (*outimensat)(int, const char *, const struct timespec[2], int) = dlsym(RTLD_NEXT, "utimensat");
    return outimensat(dirfd, pathname, (struct timespec*)NULL, flags);
}

Compile and load it when using mamba:

$ cc fixmamba.c -fPIC -shared -o fixmamba.so -ldl
$ LD_PRELOAD="$(pwd)/fixmamba.so" mamba install ipython