Any chance for including BOLT binary optimiser, it's has some substantial improvement potential over pgo+lto. https://www.phoronix.com/scan.php?page=news_item&px=LLVM-Lands-BOLT https://github.com/facebookincubator/BOLT/blob/main/bolt/docs/OptimizingClang.md
Search Criteria
Package Details: makepkg-optimize 21-1
Package Actions
Git Clone URL: | https://aur.archlinux.org/makepkg-optimize.git (read-only, click to copy) |
---|---|
Package Base: | makepkg-optimize |
Description: | Supplemental build and packaging optimizations for makepkg |
Upstream URL: | https://wiki.archlinux.org/index.php/Makepkg-optimize |
Licenses: | GPL |
Submitter: | quequotion |
Maintainer: | quequotion |
Last Packager: | quequotion |
Votes: | 15 |
Popularity: | 0.134235 |
First Submitted: | 2016-03-20 15:08 (UTC) |
Last Updated: | 2022-01-09 15:05 (UTC) |
Dependencies (6)
- pacman (pacman-git, pacman-arch, pacman-hearts)
- openmp (openmp-svn, openmp-nvptx) (optional) – Parallelize loops
- optipng (optional) – Optimize PNG files
- polly (llvm-git) (optional) – Polyhedral model optimization for clang
- svgo (optional) – Optimize SVG files
- upx (upx-git) (optional) – Compress executables
Required by (0)
Sources (20)
- buildenv_ext.conf
- cflags.conf
- cmake-flags.conf
- compress-param_max.conf
- cxxflags.conf
- debug-makeflags.conf
- destdirs_ext.conf
- graphite.sh.in
- ldflags.conf
- makeflags.conf
- optipng-exec.sh.in
- optipng.sh.in
- pgo.sh.in
- pkgopts-param_ext.conf
- pkgopts_ext.conf
- rice.sh.in
- svgo-exec.sh.in
- svgo.sh.in
- upx-exec.sh.in
- upx.sh.in
Latest Comments
bartus commented on 2022-02-21 14:29 (UTC) (edited on 2022-02-21 14:34 (UTC) by bartus)
quequotion commented on 2022-01-09 13:09 (UTC) (edited on 2022-01-09 13:50 (UTC) by quequotion)
@j1simon
I will err on the side of letting upstream deal with it, and have updated makepkg-optimize removing my own lto routine.
This means you may use one provided by pacman, in 'OPTIONS', and report problems with it to pacman's developers.
Edit: I am still shipping mine in git, btw, if you want to make some comparisons.
j1simon commented on 2022-01-09 11:56 (UTC)
@quequotion in short, if I want to compile using makepkg-optimize with lto
, should I activate it in BUILDENV
, OPTIONS
or both?
quequotion commented on 2022-01-09 09:57 (UTC) (edited on 2022-01-09 12:57 (UTC) by quequotion)
@j1simon
I finally understand what you are asking about.
Ten months ago, Allan McRae added lto, as a build option, upstream.
This was not available in pacman when I published makepkg-optimize and has probably only recently come out in a stable release.
Looking over his script, it will probably work, but I wonder about the choices that went into this implementation.
First of all, it just sets "-flto". For GCC, that actually means flto "thin" (a strategy is used to reduce memory usage during compilation that can reduce optimization; GCC decided to change this to their default while clang has not, last I checked). My version distinguishes actual flto from flto-thin.
Another issue is that it makes no distinction between GCC or clang, which passes up an opportunity to save time with GCC by disabling fat objects..
Finally, it does nothing with the LTO plugin. To be honest, my implementation here is ancient. I can no longer find the thing it is based on (an older article about LTO in the Gentoo wiki, which no longer exists, couldn't drege up whatever scripts their build system uses now). I do not know if the plugin is relevant today; it was always hard to get anyone to notice its existence.
Edit: plus one, it shouldn't be an issue, but it was an issue: I had renamed this file "ZZ-lto.sh" because of a timing issue with setting the flags in addition to flags set by other libmakepkg routines (such as those provided here).
j1simon commented on 2021-12-29 08:31 (UTC)
What's the point of lto
in BUILDENV
and OPTIONS
?
nihilistzsche commented on 2021-06-08 07:55 (UTC) (edited on 2021-06-08 08:12 (UTC) by nihilistzsche)
https://bugs.archlinux.org/task/66226
According to this bug those options needed to be added for polly to work with Arch's clang.
EDIT:
Upon further reflection the better change is to add the necessary flags to the polly line and add check_buildoption "polly" "y" && to the beginning of the line.
In addition, the -freg-struct-return needs to be moved into the line check_buildoption "rice" "y" && because clang does not support it on x86_64 so should not be used I think with rice-clang.
brikler commented on 2021-03-06 17:47 (UTC) (edited on 2021-03-06 18:06 (UTC) by brikler)
@tjuh upx is more or less useless because most binaries are application/x-pie-executable
and application/x-sharedlib
, and this will fail with a NotCompressibleException
or a CantPackException
Tjuh commented on 2021-03-05 01:00 (UTC)
Alright, thanks for clarifying. So does that mean that using upx to compress pkgs with makepkg is useless, since it will pretty much always fail?
quequotion commented on 2021-03-04 03:38 (UTC)
@Tjuh
@brikler is correct, upx itself does not discriminate, but some kinds of executables will fail to run if compressed.
To be honest, documentation on this topic is lacking. There are a few links in these comments, as well as instructions for how to add new filetypes.
If you find any new filetype that works (compressed executables run), let me know.
Same if you find a problem with my script. I have been over this particular part of it countless times. Upstream discarded this, maybe they had reasons (never found any discussion of it).
brikler commented on 2021-03-02 13:14 (UTC)
@Tjuh makepkg-optimze tests the file before compress but upx doesn't test and you can compress every compressible file. you can test the file type with: file --brief --mime-type <fileToTest>
Tjuh commented on 2021-03-02 12:34 (UTC)
Are you sure about that? I tried a couple of times to manually compress files which makepkg gave a warning about not being able to compress and they compressed just fine with upx --ultra-brute filename.
brikler commented on 2021-03-02 09:24 (UTC) (edited on 2021-03-02 09:26 (UTC) by brikler)
@Tjuh what do you mean? not compressed binaries? makepkg-optimize only compress binaries from the type application/x-executable
but not application/x-sharedlib
and the most binaries are from the type x-sharedlib
Tjuh commented on 2021-03-01 17:38 (UTC)
Have not been able to properly use upx in months and think because of this pkg, can anyone confirm please?
homocomputeris commented on 2020-07-03 15:36 (UTC) (edited on 2020-07-03 15:37 (UTC) by homocomputeris)
oxipng
claims to be significantly faster than optipng
. Could you consider adding it as an optional dependency?
quequotion commented on 2020-06-11 11:40 (UTC) (edited on 2020-06-11 11:42 (UTC) by quequotion)
@brikler
See commentary on the additional buildenv options; because clang and gcc are inexplicably not using equivalent flags for equivalent functions, and there is no mechanism for makepkg to detect which toolchain is to be used, I have been forced to make a completely different set of options for clang.
You want lto-thin-clang
and poly
.
brikler commented on 2020-06-05 14:28 (UTC)
to day i wanted to try lto-thin but it seems clang doesn't work with graphite
clang-10: error: unknown argument: '-fgraphite-identity'
clang-10: error: unknown argument: '-floop-interchange'
clang-10: error: unknown argument: '-floop-nest-optimize'
clang-10: error: unknown argument: '-floop-parallelize-all'
clang-10: error: unknown argument: '-ftree-loop-distribution'
clang-10: error: unknown argument: '-ftree-parallelize-loops=2'
clang-10: error: unsupported argument 'auto' to option 'flto='
BUILDENV=(!distcc color !ccache check !sign lto-thin graphite !pgo)
[tom@donar compiz]$ pacman -Q clang
clang 10.0.0-3
brikler commented on 2020-03-09 18:02 (UTC) (edited on 2020-03-11 08:18 (UTC) by brikler)
@quequotion
if saw this in makepkg.conf and it's not necessary
-T "$(getconf _NPROCESSORS_ONLN)"
and
zstd -c -z -q --ultra -T0 -22 -)
because xz can do the same with:
#xz --long-help
-T, --threads=NUM use at most NUM threads; the default is 1; set to 0
to use as many threads as there are processor cores
#man zstd
zstdmt is equivalent to zstd -T0
quequotion commented on 2020-03-04 08:03 (UTC)
@Anonymous Descriptions added
@brikler
I've filed an issue upstream to see if we can get some clarification about what mimetypes are safe to compress. Also, UPX just had its first new release in two years, so there may be new capabilities. Looking at the code, it seems like application/x-sharedlib
should be covered. Still unsure about application/x-pie-executable
.
commented on 2020-02-22 13:08 (UTC)
Could you add descriptions for the optdepends and what they are used for?
brikler commented on 2020-01-07 17:33 (UTC)
@quequotion
always outputs "warning, not compressing...
this message should appear when 'application/x-executable', 'application/x-dosexec', 'application/x-sharedlib' should be compressed and the system are using more x-sharedlib
then other types… so it's probably a x-sharedlib
to compress
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-sharedlib | wc
1109 1109 26616
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-executable | wc
427 427 10675
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-sharedlib | wc
1109 1109 26616
[tom@donar ~]$ file --brief --mime-type /usr/lib/* | grep application/x-executable | wc
1 1 25
https://aur.archlinux.org/packages/makepkg-optimize/?O=20&PP=10#comment-675946 https://aur.archlinux.org/packages/makepkg-optimize/?O=20&PP=10#comment-675582
quequotion commented on 2019-12-15 06:10 (UTC) (edited on 2020-03-04 14:26 (UTC) by quequotion)
@brikler
I wonder if stripping even works on a compressed binary..
The scripts should be executed in alphanumerical order (this is why lto.sh.in became ZZ-lto.sh.in); strip.sh should be happening before upx.sh anyway..
I'll look into it.
EDIT: I have been unable to compress any binaries in any packages with upx, as makepkg-optimize always outputs "warning, not compressing..." even when I deleted that code from upx.sh. I experienced an alternate reality in which no changes to upx.sh had any effect on the execution of upx.sh and I am at a loss to explain that.
EDIT again: fixed the "alternate reality" problem; apparently packaging had duplicated the tidy scripts as the executable scripts; should also fix strip and upx running out of order
brikler commented on 2019-12-13 11:58 (UTC)
-> Compressing binaries with UPX...
-> Stripping unneeded symbols from binaries and libraries...
imo should the binaries be stripped before they will be compressed because less to compress when stripped, right?
quequotion commented on 2019-11-24 15:59 (UTC)
@bartus
I've made two updates since, changing how old used profiles are handled. I suspect this was the weak point: the old conditions allowed for a scenario in which both a directory for generated profiles and an old directory of used profiles may exist simultaneously at build time--in which case pgo would silently fail to utilize the generated profiles and build an unoptimized package. Now, if a folder for newly generated files exists, and a folder of old used profiles also exists, the old used profiles will be deleted and the previously generated profiles will be used to optimize.
Perhaps it has been the case that I always deleted the old profiles by hand, so it had not been a problem for me before.
bartus commented on 2019-11-24 13:06 (UTC) (edited on 2019-11-24 13:11 (UTC) by bartus)
This patch seems redundant (works the same before and after, for me). You should just stick to a single convention for conditionals ([[
is ~25% faster than [
on my system). Also you should consider using shellcheck
to sanitize you scripts ( use with care - sometimes it missreport intended behaviors as errors), especially expanding multiple variables in a procedure call without quotes and ${}
syntax.
quequotion commented on 2019-11-15 08:47 (UTC)
@brikler
You're right. I'll try to make that change soon. 'auto' hasn't been around long; I hadn't gotten the memo.
@blackout
Are you referring to this? It is indeed possible to specify makepkg-optimize's options in /etc/makepkg.conf
, but I recommended using the provided /etc/makepkg-optimize.conf
and specifying the config file to when building (--config /etc/makepkg-optimize.conf
). I do this so that running makepkg
or makechrootpkg
(without any flags) on a PKGBUILD
makes "standard" (unoptimized) packages. It's less hassle than editing makepkg.conf
for every build (not all packages work with all optimizations and sometimes I need a 'clean' build for comparison).
By the way, it makes my day every time I see a new name in these comments! Good to know people are actually using makepkg-optimize!
brikler commented on 2019-11-14 16:22 (UTC)
@blackout i know how to optimize but somebody don't know about and this was a hint because the linker can also run in parallel processes with -flto=auto
-flto=auto
can replace ltoflags+="=$(getconf _NPROCESSORS_ONLN)
in ZZ-lto.sh.in
blackout commented on 2019-11-14 15:08 (UTC)
@brikler you have to understand the package. check ZZ-lto.sh.in makeflags.conf
after installation you have to check the makepkg.conf and you have the optimizations commented out. so you can replace your original file with the optimizations.
RTM
brikler commented on 2019-11-09 10:23 (UTC)
if found this options perhaps it is interesting for makepkg-optimize?
If you specify the optional n, the optimization and code generation done at link time is executed in parallel using n parallel jobs by utilizing an installed make program. The environment variable MAKE may be used to override the program used.
You can also specify -flto=jobserver to use GNU make’s job server mode to determine the number of parallel jobs. This is useful when the Makefile calling GCC is already executing in parallel. You must prepend a ‘+’ to the command recipe in the parent Makefile for this to work. This option likely only works if MAKE is GNU make. Even without the option value, GCC tries to automatically detect a running GNU make’s job server.
Use -flto=auto to use GNU make’s job server, if available, or otherwise fall back to autodetection of the number of CPU threads present in your system.
bartus commented on 2019-11-01 09:05 (UTC) (edited on 2019-11-01 12:39 (UTC) by bartus)
@quequotion: I saw this first in Alads commit that strips parallel
off of aur-srcver
particular this line showing wait -n
usage.
quequotion commented on 2019-11-01 03:53 (UTC)
bartus commented on 2019-10-30 20:58 (UTC) (edited on 2019-10-30 20:59 (UTC) by bartus)
Fix the case statement in upx
and add multiprocessing
to upx
optipng
and svgo
git am < <(curl -s 'https://github.com/bartoszek/AUR-makepkg-optimize/commit/f1964eb728aee09524357d58585037b18ea50725.patch')
Inspired by aurutils
bartus commented on 2019-10-24 11:14 (UTC)
pacman=5.2
just landed in [core]
, you can update depends[]
quequotion commented on 2019-09-24 16:16 (UTC) (edited on 2019-09-26 00:45 (UTC) by quequotion)
@brikler
Finally had a chance to test this in the wild; checking $CC
and $CXX
only really works if they are set in makepkg.conf. Packages (see pcsx2-git) may set them in build()
, after the libmakepkg scripts have run; the only solution seems to be having distinct options for gcc and clang; ie lto-clang
and lto-thin-clang
...
EDIT: There are now separate options for clang: lto-clang, lto-thin-clang, polly, rice-clang. I'm having a heck of a time proving they work, but they should. polly
will not, because it requires the polly package for llvm, which is only available in llvm-git (aur).
quequotion commented on 2019-07-05 12:01 (UTC) (edited on 2019-09-24 16:12 (UTC) by quequotion)
@brikler
You still need to follow the instructions on the wiki to enable the clang compiler (set CC
and CXX
, etc), but ZZ-lto.sh.in will now set the appropriate flags depending on which compiler you have enabled.
I've also added a new build environment option lto-thin. This does -flto=thin
for clang or uses the default behavior for gcc (formerly -fwhopr
).
The build environment option lto will do monolithic link time optimization for both; this can result in greater optimization, but requires greater resources and takes more time.
I looked into the compatibility of the pgo build environment option's flags, and it seems like they should be fine as is; clang claims to support the gcc flags in addition to its own.
brikler commented on 2019-07-04 07:26 (UTC)
@quequotion i don't know how makepkg decide the compiler to use but it should be possible to catch this choice before the compiler works because some packages where compiled with clang by default, as example mesa
quequotion commented on 2019-07-03 21:07 (UTC) (edited on 2019-07-10 08:24 (UTC) by quequotion)
@brikler, I read it!
clang's -flto=thin
attempts to make lto more scalable by breaking up compilation (using less memory at the cost of somewhat less thorough optimization), similar to gcc's -fwhopr
. This is sometimes necessary when compiling very large programs with lto. This may also decrease compile time, as less effort is made to find optimization opportunities in distant parts of the code. EDIT: I have since learned that clang's lto uses a number of threads equal to the number of cores by default, and the flag to control this is --thinlto-jobs=N
.
gcc's -flto=n
parallelizes the optimization threads, which may also decrease compile time but requires a significantly greater amount of free memory as it searches through larger sections of code (and in multiple threads). EDIT: I have since learned that -fwhopr
was scrapped and incorporated as the default behavior of -flto=n
; the original behavior can be restored by additionally specifying -flto-partition=none
They are both lto methods, but their goals and results differ.
-flto=thin
is not supported by gcc, and -flto=n
is not supported by clang. In order to support both, I'd have to come up with some means (in ''lto.sh.in'') of distinguishing which compiler is intended to be used. Perhaps there is something in makepkg's code I could use.
There are actually a lot of other lto-related flags to consider as well. I might make a few other changes to the lto macro, but it's troublesome that the compilers are using mutually incompatible options (makes my job harder).
On a hunch, I went looking for pgo in clang, and found that they are using different flags on the command line and different terms in their documentation. Rather than "profile guided optimization", clang's devs are calling it "profiling with instrumentation".
At the moment makepkg-optimize's macros can only be said to support gcc. I'll look into adding support for clang, and making sure gcc-specific flags aren't getting sent to non-compliant compilers.
Edit: Looking at the archwiki page for clang, in the header it mentions having to edit makepkg.conf to set up the compiler. Doesn't seem like makepkg has any built-in mechanism to set the compiler. This could be an opportunity for libmakepkg--perhaps something like a supplemental buildenv script could be used to switch compilers and filter flags. Not yet sure if I'm talking about a few scripts (maybe another aur package, ie 'makepkg-compilers') or a massive change to makepkg itself (to be proposed, revised, argued, and fought over on the mailing list; then perhaps, someday, implemented).
brikler commented on 2019-07-01 06:28 (UTC) (edited on 2019-07-03 12:30 (UTC) by brikler)
it seems llvm/clang doesn't support -flto=nCore
but the build doesn't stop if set.
-rw-r--r-- 1 tom tom 373988 10.04.2019 09:11 upx-3.95-1-x86_64.pkg.tar.xz.clangLTO
-rw-r--r-- 1 tom tom 376536 10.04.2019 09:14 upx-3.95-1-x86_64.pkg.tar.xz.clangTHINLTO
-rw-r--r-- 1 tom tom 371856 10.04.2019 10:24 upx-3.95-1-x86_64.pkg.tar.xz.clangWithoutLTO
-rw-r--r-- 1 tom tom 368732 06.04.2019 16:30 upx-3.95-1-x86_64.pkg.tar.xz.gccLTO
-rw-r--r-- 1 tom tom 373300 10.04.2019 09:53 upx-3.95-1-x86_64.pkg.tar.xz.gccWithoutLTO
btw -flto=thin
does the same as -flto=nCore
but it differs how it is be done …please read the linked blog article in my previous post for a more reliable explanation
quequotion commented on 2019-06-28 02:09 (UTC) (edited on 2019-06-28 02:35 (UTC) by quequotion)
brikler, the LTO macro is set in lto.sh.in
, which installs to /usr/share/makepkg/buildenv/lto.sh
.
Not sure how or if I could add code here to check which compiler was to be used and change the flag accordingly; you can edit it by hand in the meantime.
-flto=thin doesn't do the same thing as -flto=nCPUs; did llvm never support =nCPUs to begin with?
brikler commented on 2019-06-25 14:57 (UTC) (edited on 2019-06-26 18:21 (UTC) by brikler)
is it possible to make it work with llvm/clang without touch makepkg.conf?
=> loop optimization is enabled by default
==> i doesn't found llvm/clang support pgo
===>llvm/clang doesn't use the "-flto=nCores" flag, instead they use "-flto=thin", if is compiled with this option (llvm/clang 8 isn't compiled with)
http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html
quequotion commented on 2019-03-10 10:15 (UTC) (edited on 2019-03-10 11:52 (UTC) by quequotion)
@brikler What profile-guided optimization does is collect statistics on the execution of a program, then use the data to generate more optimal code when recompiling the program. The result should be a faster, smaller binary that achieves whatever the original code intends. Of course, as the second version of the code is machine-generated, that cannot always be guaranteed; the benefits also depend on how well the profiling run exemplifies typical use of the program.
In order to use PGO, a package must be built, installed, used, and then rebuilt and reinstalled. During the first run, profiles are generated and stored in $PROFDEST/$pkgbase.gen
. Once you've used every feature of the software that you wish to optimize, rebuild the package to move the profiles to $PROFDEST/$pkgbase.used
and apply them, creating optimized binaries. It is a time-consuming process, but the benefits are generally proportional to the effort.
It is possible to use the collected profiles to make improvements to the code by hand, such as is done by developers of the kernel, kernel modules, real-time control software, etc. Unless you intend to do so, it is not necessary to keep the $pkgbase.used
folder.
brikler commented on 2019-03-10 07:45 (UTC) (edited on 2019-03-10 07:49 (UTC) by brikler)
so i must keep the optimized profile up to the next with pgo compiled version come? i thought it's code optimization not for run time?
quequotion commented on 2019-03-10 01:01 (UTC) (edited on 2019-03-10 01:02 (UTC) by quequotion)
@brikler
Systemd is trying to access its profiles, but your home directory isn't available until you log in. There is a new "Package Output" variable for this in makepkg-optimize.conf, PROFDEST
. If unset, profiles are stored in the build directory.
To profile systemd, you'll need to set PROFDEST
to somewhere in the root file system; I use /mnt/pgo/.
brikler commented on 2019-03-09 12:58 (UTC)
next "problem": i have build systemd with pgo and after reboot my sreen was spammed with this kind of message:
…
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-send.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-vacuum.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-verify.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/lookup3.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/mmap-cache.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/sd-journal.c.gcda:Cannot open
how to avoid this?
quequotion commented on 2019-03-08 14:55 (UTC) (edited on 2019-03-08 17:23 (UTC) by quequotion)
@brikler The flags were not getting set because of a typo!
lto.sh.in line 14:
-buildenv_functionss+=('buildenv_lto')
+buildenv_functions+=('buildenv_lto')
No idea how long that'd been there. Thanks for the help!
The flags are where they need to be. I test built a few things to make sure; these flags work. They have a long history.
Note that some packages may fail to compile with LTO, for example pulseaudio-git.
brikler commented on 2019-03-06 13:34 (UTC) (edited on 2019-03-06 13:45 (UTC) by brikler)
it seems the settings in /usr/share/makepkg/buildenv/lto.sh are not honered by gcc, as example a build with -flto set in makepkg.conf an on only lto.sh
<h1>-flto and -fuse-linker-plugin set in makepkg.conf. from config.log</h1>gcc -o conftest -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5
<h1>only in lto.sh from config.log</h1>configure:3574: gcc -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5
and the build process breaks when -fuse-linker-plugin is in LDFLAGS but is working in CFLAGS
quequotion commented on 2019-02-27 07:49 (UTC) (edited on 2022-01-09 13:20 (UTC) by quequotion)
makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..
Note: Over-optimization is a thing, and it is not good.
See the wiki article for details.
Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!
quequotion commented on 2019-02-14 09:09 (UTC) (edited on 2019-02-16 12:20 (UTC) by quequotion)
Updates:
-Added a warning to upx.sh.in
about skipping binaries with mimetypes that may not be supported. Research needs to be done to find out if other mimetypes are compatible with UPX compression.
If I interpret this correctly, compressing application/x-sharedlib
does not work, while compressing application/x-pie-executable
should.
-New instructions regarding clean chroot building with PGO on the wiki.
quequotion commented on 2018-12-30 12:11 (UTC)
@brikler Aha, another filetype that is excluded. Yes, you could get upx to compress this by adding it to the list. Again, I don't know if it is safe to do so. You should look up if anyone has tried upx on shared libraries.
After the holidays, I think what I will do with the script in this package is add a catch-all case that will ouput something like:
Skipping upx compression for unsupported binary: $binary $filetype
Of course any filetypes that can be proven safe to upx compress I will add to the list!
brikler commented on 2018-12-26 17:18 (UTC) (edited on 2018-12-27 08:34 (UTC) by brikler)
[tom@donar upx]$ file --brief --mime-type upx
application/x-sharedlib
is it helpful?
edit your idea solves the problem :)
case "$(file --brief --mime-type "$binary")" in
'application/x-executable' | 'application/x-dosexec' | 'application/x-sharedlib')
thank you for your help :)
quequotion commented on 2018-12-26 08:35 (UTC)
@brikler could you run:
file --brief --mime-type "$binary"
On that unpacked upx binary? (put it in place of $binary). I'm pretty sure you're going to get "application/x-pie-executable".
I've tried to run a a few PIE programs compressed by UPX, but they crash. This type of executable was excluded by the pacman developers when the upx script was part of it. I don't know if that is because it is known to be unsafe, or just an oversight. I found very little about PIE and UPX by google search, but what I could find suggests that it may be unsafe.
brikler commented on 2018-12-21 09:45 (UTC) (edited on 2018-12-21 09:59 (UTC) by brikler)
@quequotion upx.sh doesn't work. i have build upx withe "upx" in the option array but the binary wasn't compressed. a really simple test unpack the package and compress the binary:
[tom@donar upx]$ upx --exact upx
File size Ratio Format Name
-------------------- ------ ----------- -----------
1958664 -> 493528 25.20% linux/amd64 upx
Packed 1 file.
[tom@donar upx]$ upx -d --exact upx
File size Ratio Format Name
-------------------- ------ ----------- -----------
1958664 <- 493528 25.20% linux/amd64 upx
Unpacked 1 file.
if it where compressed there would be throw a "cant pack exception"
[tom@donar upx]$ upx --exact upx
File size Ratio Format Name
-------------------- ------ ----------- -----------
upx: upx: CantPackException: bad e_shstrndx
Packed 0 files.
quequotion commented on 2018-12-20 06:22 (UTC) (edited on 2018-12-20 09:11 (UTC) by quequotion)
@brikler please post the error messages you received, if any. posting the code is not necessary, but if you do post code, put it between two lines of ``` to make a code block (AUR supports markdown).
here is an example
I'm looking into this, but I believe this code works. It is exactly the same code as was removed from pacman.
I added a catchall statement to the case switch:
*)
echo "$(file --brief --mime-type "$binary")"
;;
This showed executables having the filetype "application/x-pie-executable". I'm not sure if it is safe to compress a PIE executable with UPX.
brikler commented on 2018-12-19 14:38 (UTC) (edited on 2018-12-20 17:11 (UTC) by brikler)
the upx script doesn't work as expected i noticed because i had build some packages with set upx in the option array but the binaries was never compressed.
tidy_upx() {
if check_option "upx" "y"; then
msg2 "$(gettext "Compressing binaries with %s...")" "UPX"
local binary
find . -type f -perm -u+w 2>/dev/null | while read -r binary ; do
case "$(file --brief --mime-type "$binary")" in
'application/x-executable' | 'application/x-dosexec')
upx "${UPXFLAGS[@]}" "$binary" &>/dev/null ||
warning "$(gettext "Could not compress binary : %s")" "${binary/$pkgdir\//}"
;;
esac
done
fi
}
it seems binary never reached the upx comman https://aur.archlinux.org/cgit/aur.git/tree/upx.sh.in?h=makepkg-optimize
quequotion commented on 2018-11-28 02:34 (UTC) (edited on 2019-02-27 07:36 (UTC) by quequotion)
Good news, buildenv is now extendable: https://git.archlinux.org/pacman.git/commit/?id=508b4e3ec0cb3e365942f4dc0626edda4789932b
makepkg-optimize 5-1 is now a supplement to pacman-git!
As such, the package now conflicts() and replaces() pacman-buildenv_ext-git, which is deprecated, and depends() on pacman-git.
To avoid the possibility of pacman removing itself, remove old versions of makepkg-optimize and install pacman-git before upgrading.
If you use an AUR helper, you can probably install them simultaneously:
pikaur -S makepkg-optimize pacman-git
See the wiki article for details.
quequotion commented on 2016-11-14 17:18 (UTC) (edited on 2018-08-30 19:08 (UTC) by quequotion)
wolf
Long story short, the concept may get in there, someday...
I have grown tired of waiting. I've uploaded a patched "pacman-buildenv_ext-git" and "makepkg-optimize" to demonstrate how this {c,sh}ould work in the future: pacman patched to allow extensions to BUILDENV and separate packages of supplemental BUILDENV and Tidy extensions.
Pinned Comments
quequotion commented on 2019-02-27 07:49 (UTC) (edited on 2022-01-09 13:20 (UTC) by quequotion)
makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..
Note: Over-optimization is a thing, and it is not good.
See the wiki article for details.
Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!