Arch Linux User Repository

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

Updates:

-Added a warning to upx.sh.in about skipping binaries with mimetypes that may not be supported. Research needs to be done to find out if other mimetypes are compatible with UPX compression.

If I interpret this correctly, compressing application/x-sharedlib does not work, while compressing application/x-pie-executable should.

-New instructions regarding clean chroot building with PGO on the wiki.

@brikler Aha, another filetype that is excluded. Yes, you could get upx to compress this by adding it to the list. Again, I don't know if it is safe to do so. You should look up if anyone has tried upx on shared libraries.

After the holidays, I think what I will do with the script in this package is add a catch-all case that will ouput something like:

Skipping upx compression for unsupported binary: $binary $filetype

Of course any filetypes that can be proven safe to upx compress I will add to the list!

[tom@donar upx]$ file --brief --mime-type upx
application/x-sharedlib

is it helpful?

edit your idea solves the problem :)

        case "$(file --brief --mime-type "$binary")" in
                'application/x-executable' | 'application/x-dosexec' | 'application/x-sharedlib') 

thank you for your help :)

@brikler could you run:

file --brief --mime-type "$binary"

On that unpacked upx binary? (put it in place of $binary). I'm pretty sure you're going to get "application/x-pie-executable".

I've tried to run a a few PIE programs compressed by UPX, but they crash. This type of executable was excluded by the pacman developers when the upx script was part of it. I don't know if that is because it is known to be unsafe, or just an oversight. I found very little about PIE and UPX by google search, but what I could find suggests that it may be unsafe.

@quequotion upx.sh doesn't work. i have build upx withe "upx" in the option array but the binary wasn't compressed. a really simple test unpack the package and compress the binary:

[tom@donar upx]$ upx --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1958664 ->    493528   25.20%   linux/amd64   upx                           

Packed 1 file.
[tom@donar upx]$ upx -d --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1958664 <-    493528   25.20%   linux/amd64   upx

Unpacked 1 file.

if it where compressed there would be throw a "cant pack exception"

[tom@donar upx]$ upx --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx: upx: CantPackException: bad e_shstrndx

Packed 0 files.

@brikler please post the error messages you received, if any. posting the code is not necessary, but if you do post code, put it between two lines of ``` to make a code block (AUR supports markdown).

here is an example

I'm looking into this, but I believe this code works. It is exactly the same code as was removed from pacman.

I added a catchall statement to the case switch:

*)
    echo "$(file --brief --mime-type "$binary")"
    ;;

This showed executables having the filetype "application/x-pie-executable". I'm not sure if it is safe to compress a PIE executable with UPX.

the upx script doesn't work as expected i noticed because i had build some packages with set upx in the option array but the binaries was never compressed.

tidy_upx() {
    if check_option "upx" "y"; then
        msg2 "$(gettext "Compressing binaries with %s...")" "UPX"
        local binary
        find . -type f -perm -u+w 2>/dev/null | while read -r binary ; do
            case "$(file --brief --mime-type "$binary")" in
                'application/x-executable' | 'application/x-dosexec')
                    upx "${UPXFLAGS[@]}" "$binary" &>/dev/null ||
                        warning "$(gettext "Could not compress binary : %s")" "${binary/$pkgdir\//}"
                    ;;
            esac
        done
    fi
}

it seems binary never reached the upx comman https://aur.archlinux.org/cgit/aur.git/tree/upx.sh.in?h=makepkg-optimize

Good news, buildenv is now extendable: https://git.archlinux.org/pacman.git/commit/?id=508b4e3ec0cb3e365942f4dc0626edda4789932b

makepkg-optimize 5-1 is now a supplement to pacman-git!

As such, the package now conflicts() and replaces() pacman-buildenv_ext-git, which is deprecated, and depends() on pacman-git.

To avoid the possibility of pacman removing itself, remove old versions of makepkg-optimize and install pacman-git before upgrading.

If you use an AUR helper, you can probably install them simultaneously:

pikaur -S makepkg-optimize pacman-git

See the wiki article for details.

wolf

Long story short, the concept may get in there, someday...

I have grown tired of waiting. I've uploaded a patched "pacman-buildenv_ext-git" and "makepkg-optimize" to demonstrate how this {c,sh}ould work in the future: pacman patched to allow extensions to BUILDENV and separate packages of supplemental BUILDENV and Tidy extensions.