Arch Linux User Repository

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

@quequotion

if saw this in makepkg.conf and it's not necessary

-T "$(getconf _NPROCESSORS_ONLN)"
and
zstd -c -z -q --ultra -T0 -22 -)

because xz can do the same with:

#xz --long-help
  -T, --threads=NUM   use at most NUM threads; the default is 1; set to 0
                      to use as many threads as there are processor cores

#man zstd
zstdmt is equivalent to zstd -T0

@Anonymous Descriptions added

@brikler I've filed an issue upstream to see if we can get some clarification about what mimetypes are safe to compress. Also, UPX just had its first new release in two years, so there may be new capabilities. Looking at the code, it seems like application/x-sharedlib should be covered. Still unsure about application/x-pie-executable.

Could you add descriptions for the optdepends and what they are used for?

@quequotion always outputs "warning, not compressing... this message should appear when 'application/x-executable', 'application/x-dosexec', 'application/x-sharedlib' should be compressed and the system are using more x-sharedlib then other types… so it's probably a x-sharedlib to compress

[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-sharedlib | wc
   1109    1109   26616
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-executable | wc
    427     427   10675
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-sharedlib | wc
   1109    1109   26616
[tom@donar ~]$ file --brief --mime-type /usr/lib/* | grep application/x-executable | wc
      1       1      25

https://aur.archlinux.org/packages/makepkg-optimize/?O=20&PP=10#comment-675946 https://aur.archlinux.org/packages/makepkg-optimize/?O=20&PP=10#comment-675582

@brikler

I wonder if stripping even works on a compressed binary..

The scripts should be executed in alphanumerical order (this is why lto.sh.in became ZZ-lto.sh.in); strip.sh should be happening before upx.sh anyway..

I'll look into it.

EDIT: I have been unable to compress any binaries in any packages with upx, as makepkg-optimize always outputs "warning, not compressing..." even when I deleted that code from upx.sh. I experienced an alternate reality in which no changes to upx.sh had any effect on the execution of upx.sh and I am at a loss to explain that.

EDIT again: fixed the "alternate reality" problem; apparently packaging had duplicated the tidy scripts as the executable scripts; should also fix strip and upx running out of order

  -> Compressing binaries with UPX...
  -> Stripping unneeded symbols from binaries and libraries...

imo should the binaries be stripped before they will be compressed because less to compress when stripped, right?

@bartus

I've made two updates since, changing how old used profiles are handled. I suspect this was the weak point: the old conditions allowed for a scenario in which both a directory for generated profiles and an old directory of used profiles may exist simultaneously at build time--in which case pgo would silently fail to utilize the generated profiles and build an unoptimized package. Now, if a folder for newly generated files exists, and a folder of old used profiles also exists, the old used profiles will be deleted and the previously generated profiles will be used to optimize.

Perhaps it has been the case that I always deleted the old profiles by hand, so it had not been a problem for me before.

This patch seems redundant (works the same before and after, for me). You should just stick to a single convention for conditionals ([[ is ~25% faster than [ on my system). Also you should consider using shellcheck to sanitize you scripts ( use with care - sometimes it missreport intended behaviors as errors), especially expanding multiple variables in a procedure call without quotes and ${} syntax.

@brikler

You're right. I'll try to make that change soon. 'auto' hasn't been around long; I hadn't gotten the memo.

@blackout

Are you referring to this? It is indeed possible to specify makepkg-optimize's options in /etc/makepkg.conf, but I recommended using the provided /etc/makepkg-optimize.conf and specifying the config file to when building (--config /etc/makepkg-optimize.conf). I do this so that running makepkg or makechrootpkg (without any flags) on a PKGBUILD makes "standard" (unoptimized) packages. It's less hassle than editing makepkg.conf for every build (not all packages work with all optimizations and sometimes I need a 'clean' build for comparison).

By the way, it makes my day every time I see a new name in these comments! Good to know people are actually using makepkg-optimize!

@blackout i know how to optimize but somebody don't know about and this was a hint because the linker can also run in parallel processes with -flto=auto

-flto=auto can replace ltoflags+="=$(getconf _NPROCESSORS_ONLN) in ZZ-lto.sh.in