Arch Linux User Repository

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

@brikler

You still need to follow the instructions on the wiki to enable the clang compiler (set CC and CXX, etc), but ZZ-lto.sh.in will now set the appropriate flags depending on which compiler you have enabled.

I've also added a new build environment option lto-thin. This does -flto=thin for clang or uses the default behavior for gcc (formerly -fwhopr).

The build environment option lto will do monolithic link time optimization for both; this can result in greater optimization, but requires greater resources and takes more time.

I looked into the compatibility of the pgo build environment option's flags, and it seems like they should be fine as is; clang claims to support the gcc flags in addition to its own.

@quequotion i don't know how makepkg decide the compiler to use but it should be possible to catch this choice before the compiler works because some packages where compiled with clang by default, as example mesa

@brikler, I read it!

clang's -flto=thin attempts to make lto more scalable by breaking up compilation (using less memory at the cost of somewhat less thorough optimization), similar to gcc's -fwhopr. This is sometimes necessary when compiling very large programs with lto. This may also decrease compile time, as less effort is made to find optimization opportunities in distant parts of the code. EDIT: I have since learned that clang's lto uses a number of threads equal to the number of cores by default, and the flag to control this is --thinlto-jobs=N.

gcc's -flto=n parallelizes the optimization threads, which may also decrease compile time but requires a significantly greater amount of free memory as it searches through larger sections of code (and in multiple threads). EDIT: I have since learned that -fwhopr was scrapped and incorporated as the default behavior of -flto=n; the original behavior can be restored by additionally specifying -flto-partition=none

They are both lto methods, but their goals and results differ.

-flto=thin is not supported by gcc, and -flto=n is not supported by clang. In order to support both, I'd have to come up with some means (in ''lto.sh.in'') of distinguishing which compiler is intended to be used. Perhaps there is something in makepkg's code I could use.

There are actually a lot of other lto-related flags to consider as well. I might make a few other changes to the lto macro, but it's troublesome that the compilers are using mutually incompatible options (makes my job harder).

On a hunch, I went looking for pgo in clang, and found that they are using different flags on the command line and different terms in their documentation. Rather than "profile guided optimization", clang's devs are calling it "profiling with instrumentation".

At the moment makepkg-optimize's macros can only be said to support gcc. I'll look into adding support for clang, and making sure gcc-specific flags aren't getting sent to non-compliant compilers.

Edit: Looking at the archwiki page for clang, in the header it mentions having to edit makepkg.conf to set up the compiler. Doesn't seem like makepkg has any built-in mechanism to set the compiler. This could be an opportunity for libmakepkg--perhaps something like a supplemental buildenv script could be used to switch compilers and filter flags. Not yet sure if I'm talking about a few scripts (maybe another aur package, ie 'makepkg-compilers') or a massive change to makepkg itself (to be proposed, revised, argued, and fought over on the mailing list; then perhaps, someday, implemented).

it seems llvm/clang doesn't support -flto=nCore but the build doesn't stop if set.

-rw-r--r-- 1 tom tom 373988 10.04.2019 09:11 upx-3.95-1-x86_64.pkg.tar.xz.clangLTO
-rw-r--r-- 1 tom tom 376536 10.04.2019 09:14 upx-3.95-1-x86_64.pkg.tar.xz.clangTHINLTO
-rw-r--r-- 1 tom tom 371856 10.04.2019 10:24 upx-3.95-1-x86_64.pkg.tar.xz.clangWithoutLTO
-rw-r--r-- 1 tom tom 368732 06.04.2019 16:30 upx-3.95-1-x86_64.pkg.tar.xz.gccLTO
-rw-r--r-- 1 tom tom 373300 10.04.2019 09:53 upx-3.95-1-x86_64.pkg.tar.xz.gccWithoutLTO

btw -flto=thindoes the same as -flto=nCore but it differs how it is be done …please read the linked blog article in my previous post for a more reliable explanation

brikler, the LTO macro is set in lto.sh.in, which installs to /usr/share/makepkg/buildenv/lto.sh.

Not sure how or if I could add code here to check which compiler was to be used and change the flag accordingly; you can edit it by hand in the meantime.

-flto=thin doesn't do the same thing as -flto=nCPUs; did llvm never support =nCPUs to begin with?

is it possible to make it work with llvm/clang without touch makepkg.conf?

=> loop optimization is enabled by default

==> i doesn't found llvm/clang support pgo

===>llvm/clang doesn't use the "-flto=nCores" flag, instead they use "-flto=thin", if is compiled with this option (llvm/clang 8 isn't compiled with) http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html

@brikler What profile-guided optimization does is collect statistics on the execution of a program, then use the data to generate more optimal code when recompiling the program. The result should be a faster, smaller binary that achieves whatever the original code intends. Of course, as the second version of the code is machine-generated, that cannot always be guaranteed; the benefits also depend on how well the profiling run exemplifies typical use of the program.

In order to use PGO, a package must be built, installed, used, and then rebuilt and reinstalled. During the first run, profiles are generated and stored in $PROFDEST/$pkgbase.gen. Once you've used every feature of the software that you wish to optimize, rebuild the package to move the profiles to $PROFDEST/$pkgbase.used and apply them, creating optimized binaries. It is a time-consuming process, but the benefits are generally proportional to the effort.

It is possible to use the collected profiles to make improvements to the code by hand, such as is done by developers of the kernel, kernel modules, real-time control software, etc. Unless you intend to do so, it is not necessary to keep the $pkgbase.used folder.

so i must keep the optimized profile up to the next with pgo compiled version come? i thought it's code optimization not for run time?

@brikler Systemd is trying to access its profiles, but your home directory isn't available until you log in. There is a new "Package Output" variable for this in makepkg-optimize.conf, PROFDEST. If unset, profiles are stored in the build directory.

To profile systemd, you'll need to set PROFDEST to somewhere in the root file system; I use /mnt/pgo/.

next "problem": i have build systemd with pgo and after reboot my sreen was spammed with this kind of message:

…
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-send.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-vacuum.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-verify.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/lookup3.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/mmap-cache.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/sd-journal.c.gcda:Cannot open

how to avoid this?