Arch Linux User Repository

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

it seems the settings in /usr/share/makepkg/buildenv/lto.sh are not honered by gcc, as example a build with -flto set in makepkg.conf an on only lto.sh

<h1>-flto and -fuse-linker-plugin set in makepkg.conf. from config.log</h1>

gcc -o conftest -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5

<h1>only in lto.sh from config.log</h1>

configure:3574: gcc -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5

and the build process breaks when -fuse-linker-plugin is in LDFLAGS but is working in CFLAGS

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

Updates:

-Added a warning to upx.sh.in about skipping binaries with mimetypes that may not be supported. Research needs to be done to find out if other mimetypes are compatible with UPX compression.

If I interpret this correctly, compressing application/x-sharedlib does not work, while compressing application/x-pie-executable should.

-New instructions regarding clean chroot building with PGO on the wiki.

@brikler Aha, another filetype that is excluded. Yes, you could get upx to compress this by adding it to the list. Again, I don't know if it is safe to do so. You should look up if anyone has tried upx on shared libraries.

After the holidays, I think what I will do with the script in this package is add a catch-all case that will ouput something like:

Skipping upx compression for unsupported binary: $binary $filetype

Of course any filetypes that can be proven safe to upx compress I will add to the list!

[tom@donar upx]$ file --brief --mime-type upx
application/x-sharedlib

is it helpful?

edit your idea solves the problem :)

        case "$(file --brief --mime-type "$binary")" in
                'application/x-executable' | 'application/x-dosexec' | 'application/x-sharedlib') 

thank you for your help :)

@brikler could you run:

file --brief --mime-type "$binary"

On that unpacked upx binary? (put it in place of $binary). I'm pretty sure you're going to get "application/x-pie-executable".

I've tried to run a a few PIE programs compressed by UPX, but they crash. This type of executable was excluded by the pacman developers when the upx script was part of it. I don't know if that is because it is known to be unsafe, or just an oversight. I found very little about PIE and UPX by google search, but what I could find suggests that it may be unsafe.

@quequotion upx.sh doesn't work. i have build upx withe "upx" in the option array but the binary wasn't compressed. a really simple test unpack the package and compress the binary:

[tom@donar upx]$ upx --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1958664 ->    493528   25.20%   linux/amd64   upx                           

Packed 1 file.
[tom@donar upx]$ upx -d --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1958664 <-    493528   25.20%   linux/amd64   upx

Unpacked 1 file.

if it where compressed there would be throw a "cant pack exception"

[tom@donar upx]$ upx --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx: upx: CantPackException: bad e_shstrndx

Packed 0 files.

@brikler please post the error messages you received, if any. posting the code is not necessary, but if you do post code, put it between two lines of ``` to make a code block (AUR supports markdown).

here is an example

I'm looking into this, but I believe this code works. It is exactly the same code as was removed from pacman.

I added a catchall statement to the case switch:

*)
    echo "$(file --brief --mime-type "$binary")"
    ;;

This showed executables having the filetype "application/x-pie-executable". I'm not sure if it is safe to compress a PIE executable with UPX.

the upx script doesn't work as expected i noticed because i had build some packages with set upx in the option array but the binaries was never compressed.

tidy_upx() {
    if check_option "upx" "y"; then
        msg2 "$(gettext "Compressing binaries with %s...")" "UPX"
        local binary
        find . -type f -perm -u+w 2>/dev/null | while read -r binary ; do
            case "$(file --brief --mime-type "$binary")" in
                'application/x-executable' | 'application/x-dosexec')
                    upx "${UPXFLAGS[@]}" "$binary" &>/dev/null ||
                        warning "$(gettext "Could not compress binary : %s")" "${binary/$pkgdir\//}"
                    ;;
            esac
        done
    fi
}

it seems binary never reached the upx comman https://aur.archlinux.org/cgit/aur.git/tree/upx.sh.in?h=makepkg-optimize