Package Details: makepkg-optimize 15-1

Git Clone URL: https://aur.archlinux.org/makepkg-optimize.git (read-only)
Package Base: makepkg-optimize
Description: Supplemental build and packaging optimizations for makepkg
Upstream URL: https://wiki.archlinux.org/index.php/Makepkg-optimize
Licenses: GPL
Submitter: quequotion
Maintainer: quequotion
Last Packager: quequotion
Votes: 9
Popularity: 0.95
First Submitted: 2016-03-20 15:08
Last Updated: 2019-11-01 03:50

Pinned Comments

quequotion commented on 2019-02-27 07:49

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman-git. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, link time optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

Latest Comments

« First ‹ Previous 1 2 3 4 Next › Last »

quequotion commented on 2019-06-28 02:09

brikler, the LTO macro is set in lto.sh.in, which installs to /usr/share/makepkg/buildenv/lto.sh.

Not sure how or if I could add code here to check which compiler was to be used and change the flag accordingly; you can edit it by hand in the meantime.

-flto=thin doesn't do the same thing as -flto=nCPUs; did llvm never support =nCPUs to begin with?

brikler commented on 2019-06-25 14:57

is it possible to make it work with llvm/clang without touch makepkg.conf?

=> loop optimization is enabled by default

==> i doesn't found llvm/clang support pgo

===>llvm/clang doesn't use the "-flto=nCores" flag, instead they use "-flto=thin", if is compiled with this option (llvm/clang 8 isn't compiled with) http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html

quequotion commented on 2019-03-10 10:15

@brikler What profile-guided optimization does is collect statistics on the execution of a program, then use the data to generate more optimal code when recompiling the program. The result should be a faster, smaller binary that achieves whatever the original code intends. Of course, as the second version of the code is machine-generated, that cannot always be guaranteed; the benefits also depend on how well the profiling run exemplifies typical use of the program.

In order to use PGO, a package must be built, installed, used, and then rebuilt and reinstalled. During the first run, profiles are generated and stored in $PROFDEST/$pkgbase.gen. Once you've used every feature of the software that you wish to optimize, rebuild the package to move the profiles to $PROFDEST/$pkgbase.used and apply them, creating optimized binaries. It is a time-consuming process, but the benefits are generally proportional to the effort.

It is possible to use the collected profiles to make improvements to the code by hand, such as is done by developers of the kernel, kernel modules, real-time control software, etc. Unless you intend to do so, it is not necessary to keep the $pkgbase.used folder.

brikler commented on 2019-03-10 07:45

so i must keep the optimized profile up to the next with pgo compiled version come? i thought it's code optimization not for run time?

quequotion commented on 2019-03-10 01:01

@brikler Systemd is trying to access its profiles, but your home directory isn't available until you log in. There is a new "Package Output" variable for this in makepkg-optimize.conf, PROFDEST. If unset, profiles are stored in the build directory.

To profile systemd, you'll need to set PROFDEST to somewhere in the root file system; I use /mnt/pgo/.

brikler commented on 2019-03-09 12:58

next "problem": i have build systemd with pgo and after reboot my sreen was spammed with this kind of message:

…
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-send.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-vacuum.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-verify.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/lookup3.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/mmap-cache.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/sd-journal.c.gcda:Cannot open

how to avoid this?

quequotion commented on 2019-03-08 14:55

@brikler The flags were not getting set because of a typo!

lto.sh.in line 14:

-buildenv_functionss+=('buildenv_lto')
+buildenv_functions+=('buildenv_lto')

No idea how long that'd been there. Thanks for the help!

The flags are where they need to be. I test built a few things to make sure; these flags work. They have a long history.

Note that some packages may fail to compile with LTO, for example pulseaudio-git.

brikler commented on 2019-03-06 13:34

it seems the settings in /usr/share/makepkg/buildenv/lto.sh are not honered by gcc, as example a build with -flto set in makepkg.conf an on only lto.sh

<h1>-flto and -fuse-linker-plugin set in makepkg.conf. from config.log</h1>

gcc -o conftest -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5

<h1>only in lto.sh from config.log</h1>

configure:3574: gcc -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5

and the build process breaks when -fuse-linker-plugin is in LDFLAGS but is working in CFLAGS

quequotion commented on 2019-02-27 07:49

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman-git. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, link time optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

quequotion commented on 2019-02-14 09:09

Updates:

-Added a warning to upx.sh.in about skipping binaries with mimetypes that may not be supported. Research needs to be done to find out if other mimetypes are compatible with UPX compression.

If I interpret this correctly, compressing application/x-sharedlib does not work, while compressing application/x-pie-executable should.

-New instructions regarding clean chroot building with PGO on the wiki.