Arch Linux User Repository

For future commenters:

TLDR:

https://aur.archlinux.org/pkgbase/ceph | From source build (slow)

https://aur.archlinux.org/pkgbase/ceph-bin | Pre-built binaries (fast)

Unlike the original community version, this repo builds ceph from source. Ceph is a large, complicated project so this takes several hours on a good build server.

To get a similar experience to how community/ceph worked (pre-built binaries) use ceph-bin instead.

I have a fix for this now, I'll push a -6 sometime today, if I don't forgot later after the build finishes

I think this is basically down to the fact that ceph v19 bundles an absolutely ancient version of liburing (https://github.com/axboe/liburing/tree/liburing-0.7), which gcc has decided it doesn't like for whatever reason.

You can work around this with -DWITH_SYSTEM_LIBURING=ON and adding liburing to makedepends, but I don't know if you'll hit a bug later down. For what it's worth, this is fixed in v20, but I'm still waiting for 20.2.1 to release. I'll take a look at this on the weekend

Oh found an error way up in the build log:

[73/2624] Performing build step for 'liburing_ext'
FAILED: [code=1] liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build /build/ceph/src/ceph-19.2.3/src/liburing/src/liburing.a /build/ceph/src/ceph-19.2.3/build/liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build 
cd /build/ceph/src/ceph-19.2.3/src/liburing && /usr/bin/cmake -P /build/ceph/src/ceph-19.2.3/build/liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build-RelWithDebInfo.cmake && /usr/bin/cmake -E touch /build/ceph/src/ceph-19.2.3/build/liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build
CMake Error at /build/ceph/src/ceph-19.2.3/build/liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build-RelWithDebInfo.cmake:37 (message):
  Command failed: 2

   '/usr/bin/make' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe  -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/ceph/src=/usr/src/debug/ceph -fPIC' '-C' 'src' '-s'

  See also

    /build/ceph/src/ceph-19.2.3/build/liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build.log


-- Log output is:
     CC setup.ol
In file included from setup.c:12:
include/liburing.h:409:66: warning: ‘struct open_how’ declared inside parameter list will not be visible outside of this definition or declaration
  409 |                                         const char *path, struct open_how *how)
      |                                                                  ^~~~~~~~
include/liburing.h: In function ‘io_uring_prep_openat2’:
include/liburing.h:411:67: error: invalid application of ‘sizeof’ to incomplete type ‘struct open_how’
  411 |         io_uring_prep_rw(IORING_OP_OPENAT2, sqe, dfd, path, sizeof(*how),
      |                                                                   ^
make: *** [Makefile:44: setup.ol] Error 1

CMake Error at /build/ceph/src/ceph-19.2.3/build/liburing_ext-prefix/src/liburing_ext-stamp/liburing_ext-build-RelWithDebInfo.cmake:47 (message):
  Stopping after outputting logs.

I'm seeing a build failure but not any errors?

objcopy --localize-hidden `sed -n 's/^  *\([a-zA-Z0-9_]*\);$/-G \1/p' libpmem2.link` ../debug/libpmem2/libpmem2_unscoped.o ../debug/libpmem2/libpmem2_all.o
ar rv ../debug/libpmem2/../libpmem2.a ../debug/libpmem2/libpmem2_all.o
ar: creating ../debug/libpmem2/../libpmem2.a
a - ../debug/libpmem2/libpmem2_all.o
/usr/bin/cc -Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs -Wl,-z,relro -Wl,--fatal-warnings -Wl,--warn-common  -Wl,-rpath=/usr/local/lib/pmdk_debug:: -L../debug/libpmem2/.. -shared -Wl,--version-script=libpmem2.link,-soname,libpmem2.so.1 -o ../debug/libpmem2/../libpmem2.so.1.0.0 ../debug/libpmem2/ravl.o ../debug/libpmem2/libpmem2.o ../debug/libpmem2/badblocks.o ../debug/libpmem2/badblocks_none.o ../debug/libpmem2/config.o ../debug/libpmem2/deep_flush.o ../debug/libpmem2/errormsg.o ../debug/libpmem2/memops_generic.o ../debug/libpmem2/map.o ../debug/libpmem2/map_posix.o ../debug/libpmem2/persist.o ../debug/libpmem2/persist_posix.o ../debug/libpmem2/pmem2_utils.o ../debug/libpmem2/usc_none.o ../debug/libpmem2/source.o ../debug/libpmem2/source_posix.o ../debug/libpmem2/vm_reservation.o ../debug/libpmem2/vm_reservation_posix.o ../debug/libpmem2/ravl_interval.o ../debug/libpmem2/auto_flush_linux.o ../debug/libpmem2/deep_flush_linux.o ../debug/libpmem2/extent_linux.o ../debug/libpmem2/pmem2_utils_linux.o ../debug/libpmem2/pmem2_utils_none.o ../debug/libpmem2/region_namespace_none.o ../debug/libpmem2/alloc.o ../debug/libpmem2/fs_posix.o ../debug/libpmem2/os_posix.o ../debug/libpmem2/os_thread_posix.o ../debug/libpmem2/out.o ../debug/libpmem2/util.o ../debug/libpmem2/util_posix.o ../debug/libpmem2/init.o ../debug/libpmem2/cpu.o ../debug/libpmem2/memcpy_nt_avx.o ../debug/libpmem2/memcpy_nt_sse2.o ../debug/libpmem2/memset_nt_avx.o ../debug/libpmem2/memset_nt_sse2.o ../debug/libpmem2/memcpy_t_avx.o ../debug/libpmem2/memcpy_t_sse2.o ../debug/libpmem2/memset_t_avx.o ../debug/libpmem2/memset_t_sse2.o ../debug/libpmem2/memcpy_nt_avx512f.o ../debug/libpmem2/memset_nt_avx512f.o ../debug/libpmem2/memcpy_t_avx512f.o ../debug/libpmem2/memset_t_avx512f.o    -pthread 
ln -sf libpmem2.so.1.0.0 ../debug/libpmem2/../libpmem2.so.1
ln -sf libpmem2.so.1 ../debug/libpmem2/../libpmem2.so
make[2]: Leaving directory '/build/ceph/src/ceph-19.2.3/src/pmdk/src/libpmem2'
/usr/bin/make -C librpmem
make[2]: Entering directory '/build/ceph/src/ceph-19.2.3/src/pmdk/src/librpmem'
NOTE: Skipping librpmem because libfabric (version >= 1.4.2) is missing -- see src/librpmem/README for details
mkdir -p ../nondebug/librpmem
make[2]: Leaving directory '/build/ceph/src/ceph-19.2.3/src/pmdk/src/librpmem'
/usr/bin/make -C librpmem DEBUG=1
make[2]: Entering directory '/build/ceph/src/ceph-19.2.3/src/pmdk/src/librpmem'
NOTE: Skipping librpmem because libfabric (version >= 1.4.2) is missing -- see src/librpmem/README for details
mkdir -p ../debug/librpmem
make[2]: Leaving directory '/build/ceph/src/ceph-19.2.3/src/pmdk/src/librpmem'
make[1]: Leaving directory '/build/ceph/src/ceph-19.2.3/src/pmdk/src'
ninja: build stopped: subcommand failed.
==> ERROR: A failure occurred in build().
    Aborting...

I've pushed a -5 for the v19 series, rebuilding for python 3.14.

As an update on v20:

• I'm still waiting for https://github.com/ceph/ceph/pull/65947 (rgw data loss fix) to be merged to main, and ideally backported to v20
• There are a couple other serious-ish issues with v20 I'm waiting to see the outcome of, see the bug tracker, so I'm probably going to wait until 20.2.1 if the v19 build remains viable

I've pushed a -4 rebuild with the fix for the allow_ec_overwrites issue.

Just another FYI

https://tracker.ceph.com/issues/74156

Non-LTS kerenels will leak folios when using the in kernel cephfs driver.

So, try to keep your cephfs clients on a pre 6.15 kernel

Just a heads up to Ceph users, there is currently a data corruption bug in EC pools if you have set allow_ec_overwrites to true on the pool.

Thanks to https://github.com/insanemal for bringing this to my attention.

You can follow along with the upstream bug tracker here: https://tracker.ceph.com/issues/70390

or this package's issue here: https://github.com/bazaah/aur-ceph/issues/34

To address any new corruption you can...

1. Ensure that ceph config get osd bluestore_elastic_shared_blobs is false (default is true)

or

1. Verify that no EC pool has allow_ec_overwrites set to true (default is false)

Changing either of these will not affect current data corruption, and so far the only known fix is to redeploy the OSD after making of the above changes

Thanks for the kind words :)

I was expecting v20 to already be released, but something seems to be holding it up, and we're getting pretty close to the holiday season, so it may only happen next year, now.

@pbazaah Thanks again for all your work. I've got the ISA erasure coding plugin working.

It's fantastic. Thanks for your ongoing work. I look forward to 20.x releases which apparently bring huge speed ups to erasure coded pools.