Arch Linux User Repository

Thank you very much for your work on the python-celery package, @hashworks!

Paperless-ngx 1.17.0 now works just fine without any package modification or downgrades!

I've released python-celery 5.3.1 with some test skips.

@hashworks thanks for celery custom build. works great on my paperless-ngx just installed. before i had paperless-task-queue.service fails issue cause the billiard version 4.1.0

many many thanks :)

FYI: I tested 1.16.2 with python-billiard 4.1.0 and python-celery 5.3.1 (custom build, some tests are failing ATM), seems to work fine.

Update to paperless-1.16.1 is now available. The new version of celery containing the fix for python-billiard has finally been published! We still have to wait for the archlinux maintainers of python-celery to update the package... You might therefore want to wait before upgrading python-billiard.

@amo

@dp20eic: Getting the python-environment right needs triggering a rebuilt of python packages, as advised in the pinned comment. Also, it seems that we need to apply the patch that @atomicfs kindly provided. This is due to celery taking ages to stay up-to-date as mentioned in the other pinned comment. As for the migration: It should be enough to put your old configuration file, your old sqlite file and the old persistent directories under /var/lib/paperless/ in place before starting the systemd units.

I was yesterday frustrated, so I stop my tests, today I saw, that the installation at my new LXC Container was finished correct, but I'm not able to add/upload new pdf to paperless-ngx, after I read all comments here, I think it is python-billiard-4.x which make problems. I remove this Version and try to install the old one, but anytime I get this error

================================================================================================================= FAILURES ==================================================================================================================
___________________________________________________________________________________________________________ test_spawn.test_start ___________________________________________________________________________________________________________

self = <t.unit.test_spawn.test_spawn object at 0x7f8fad938050>

    def test_start(self):
        ctx = get_context('spawn')

        p = ctx.Process(target=task_from_process, args=('opa',))
>       p.start()

t/unit/test_spawn.py:17: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
billiard/process.py:124: in start
    self._popen = self._Popen(self)
billiard/context.py:341: in _Popen
    return Popen(process_obj)
billiard/popen_spawn_posix.py:39: in __init__
    super(Popen, self).__init__(process_obj)
billiard/popen_fork.py:24: in __init__
    self._launch(process_obj)
billiard/popen_spawn_posix.py:49: in _launch
    tracker_fd = semaphore_tracker.getfd()
billiard/semaphore_tracker.py:38: in getfd
    self.ensure_running()
billiard/semaphore_tracker.py:62: in ensure_running
    spawnv_passfds(exe, args, fds_to_pass)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = '/usr/bin/python', args = [['/usr/bin/python', '-c', 'from billiard.semaphore_tracker import main;main(20)'], [b'/usr/bin/python'], True, (8, 20), None, None, ...], passfds = [8, 20]

    def spawnv_passfds(path, args, passfds):
        passfds = sorted(passfds)
        errpipe_read, errpipe_write = os.pipe()
        try:
            args = [
                args, [fsencode(path)], True, tuple(passfds), None, None,
                -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
                False, False]
            if sys.version_info >= (3, 9):
                args.extend((None, None, None, -1))  # group, extra_groups, user, umask
            args.append(None)  # preexec_fn
>           return _posixsubprocess.fork_exec(*args)
E           TypeError: fork_exec() takes exactly 23 arguments (21 given)

billiard/compat.py:230: TypeError
============================================================================================================= warnings summary ==============================================================================================================
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
t/unit/test_spawn.py::test_spawn::test_start
  /home/dp20eic/.cache/paru/clone/python-billiard3/src/billiard-3.6.4.0/billiard/pool.py:1753: DeprecationWarning: isSet() is deprecated, use is_set() instead
    return self._event.isSet()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================================== short test summary info ==========================================================================================================
FAILED t/unit/test_spawn.py::test_spawn::test_start - TypeError: fork_exec() takes exactly 23 arguments (21 given)
=========================================================================================== 1 failed, 18 passed, 51 skipped, 8 warnings in 1.30s ============================================================================================
==> FEHLER: Ein Fehler geschah in check().
    Breche ab...
Fehler: kann ‚python-billiard3-3.6.4.0-1‘ nicht bauen: 
Fehler: kann Pakete nicht bauen: python-billiard3-3.6.4.0-1

So I comment the check at PKGBUILD and I was able to install python-billiard3-3.6.4.0-1 after restart paperless-ngx I was able to add/upload new PDF Files

So now I try again to copy, as you describe, from the old to the new installation.

With kind regards Bernd

Edit: The restore from the old installation into the new was successful :)

1.14.5 replaces python-django-ipware with python-ipware in its dependencies. Since these two are mutually exclusive and therefore conflicting with one another, you will be prompted to replace python-django-ipware with python-ipware, which I just submitted to the AUR. If you run into a deadlock where you can't remove python-django-ipware because it is still required by the (yet to be upgraded) paperless-ngx package, just force-remove python-django-ipware with pacman -Rdd python-django-ipware before relaunching the paperless package upgrade.

@dp20eic: Getting the python-environment right needs triggering a rebuilt of python packages, as advised in the pinned comment. Also, it seems that we need to apply the patch that @atomicfs kindly provided. This is due to celery taking ages to stay up-to-date as mentioned in the other pinned comment. As for the migration: It should be enough to put your old configuration file, your old sqlite file and the old persistent directories under /var/lib/paperless/ in place before starting the systemd units.

@amo kommentierte 2023-05-14 21:42 (CEST)

Hi, I try it out today on my production LXC Container, but I could not migrate the old installation, I'm an end user and have no idea about python I only install it, using package manager. After install the new concurrent-log-handler I get the next missing dependents corsheaders so I install django-cors-header and then magic_ahupp but package magic was installed in pythen-3.10 and python-3.11. So I stop this test and restore the last running backup.

I create a new LXC Container with arch-linux, did an update and install paperless-ngx from the scratch, and it works without a problem, in this LXC Container is no old python-3.10 so I think in my other installation is the python Environment not correct, as I say I'm an end user :) Now I must check how to migrate the Data to the new Container.

with kind regards Bernd

I had to do a bit of back-porting on python-billiard-3.6.4.0-3 to get it working with python 3.11, so in case somebody finds it usefull, here are the steps.

Download python-billiard-3.6.4.0-3 PKGBUILD file and patch it:

diff --git a/PKGBUILD b/PKGBUILD
index 47eac64..5a31009 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -12,12 +12,19 @@ license=('BSD')
 depends=('python')
 makedepends=('python-setuptools')
 checkdepends=('python-pytest-runner' 'python-case' 'python-psutil')
-source=("https://pypi.io/packages/source/b/billiard/billiard-$pkgver.tar.gz")
-sha512sums=('9f2a44024e45c4e7667472a0a3ade7caae755ec7c163929433f818e4b87165218b9cc9545bdee2254e7159c206e525306e383d38906b00fd5428bd1616c39b8f')
+source=(
+  "https://pypi.io/packages/source/b/billiard/billiard-$pkgver.tar.gz"
+  "python_311.diff"
+)
+sha512sums=(
+  '9f2a44024e45c4e7667472a0a3ade7caae755ec7c163929433f818e4b87165218b9cc9545bdee2254e7159c206e525306e383d38906b00fd5428bd1616c39b8f'
+  "SKIP"
+)

 prepare() {
   cd billiard-$pkgver
   sed -i 's/< *5/<6/' requirements/test.txt
+  patch --forward -p1 < "$srcdir/python_311.diff"
 }

 build() {

Create a patch file called python_311.diff with content:

diff --git a/billiard/compat.py b/billiard/compat.py
index b5ce7c7..20c93d5 100644
--- a/billiard/compat.py
+++ b/billiard/compat.py
@@ -3,6 +3,7 @@ from __future__ import absolute_import
 import errno
 import numbers
 import os
+import subprocess
 import sys

 from .five import range, zip_longest
@@ -224,9 +225,13 @@ else:
                 args, [fsencode(path)], True, tuple(passfds), None, None,
                 -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
                 False, False]
+            if sys.version_info >= (3, 11):
+                args.append(-1)  # process_group
             if sys.version_info >= (3, 9):
                 args.extend((None, None, None, -1))  # group, extra_groups, user, umask
             args.append(None)  # preexec_fn
+            if sys.version_info >= (3, 11):
+                args.append(subprocess._USE_VFORK)
             return _posixsubprocess.fork_exec(*args)
         finally:
             os.close(errpipe_read)

Build it makepkg -s