Package Details: makepkg-optimize 21-1

Git Clone URL: https://aur.archlinux.org/makepkg-optimize.git (read-only, click to copy)
Package Base: makepkg-optimize
Description: Supplemental build and packaging optimizations for makepkg
Upstream URL: https://wiki.archlinux.org/index.php/Makepkg-optimize
Licenses: GPL
Submitter: quequotion
Maintainer: quequotion
Last Packager: quequotion
Votes: 15
Popularity: 0.134235
First Submitted: 2016-03-20 15:08 (UTC)
Last Updated: 2022-01-09 15:05 (UTC)

Pinned Comments

quequotion commented on 2019-02-27 07:49 (UTC) (edited on 2022-01-09 13:20 (UTC) by quequotion)

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

Latest Comments

bartus commented on 2022-02-21 14:29 (UTC) (edited on 2022-02-21 14:34 (UTC) by bartus)

Any chance for including BOLT binary optimiser, it's has some substantial improvement potential over pgo+lto. https://www.phoronix.com/scan.php?page=news_item&px=LLVM-Lands-BOLT https://github.com/facebookincubator/BOLT/blob/main/bolt/docs/OptimizingClang.md

quequotion commented on 2022-01-09 13:09 (UTC) (edited on 2022-01-09 13:50 (UTC) by quequotion)

@j1simon

I will err on the side of letting upstream deal with it, and have updated makepkg-optimize removing my own lto routine.

This means you may use one provided by pacman, in 'OPTIONS', and report problems with it to pacman's developers.

Edit: I am still shipping mine in git, btw, if you want to make some comparisons.

j1simon commented on 2022-01-09 11:56 (UTC)

@quequotion in short, if I want to compile using makepkg-optimize with lto, should I activate it in BUILDENV, OPTIONS or both?

quequotion commented on 2022-01-09 09:57 (UTC) (edited on 2022-01-09 12:57 (UTC) by quequotion)

@j1simon

I finally understand what you are asking about.

Ten months ago, Allan McRae added lto, as a build option, upstream.

This was not available in pacman when I published makepkg-optimize and has probably only recently come out in a stable release.

Looking over his script, it will probably work, but I wonder about the choices that went into this implementation.

First of all, it just sets "-flto". For GCC, that actually means flto "thin" (a strategy is used to reduce memory usage during compilation that can reduce optimization; GCC decided to change this to their default while clang has not, last I checked). My version distinguishes actual flto from flto-thin.

Another issue is that it makes no distinction between GCC or clang, which passes up an opportunity to save time with GCC by disabling fat objects..

Finally, it does nothing with the LTO plugin. To be honest, my implementation here is ancient. I can no longer find the thing it is based on (an older article about LTO in the Gentoo wiki, which no longer exists, couldn't drege up whatever scripts their build system uses now). I do not know if the plugin is relevant today; it was always hard to get anyone to notice its existence.

Edit: plus one, it shouldn't be an issue, but it was an issue: I had renamed this file "ZZ-lto.sh" because of a timing issue with setting the flags in addition to flags set by other libmakepkg routines (such as those provided here).

j1simon commented on 2021-12-29 08:31 (UTC)

What's the point of lto in BUILDENV and OPTIONS?

nihilistzsche commented on 2021-06-08 07:55 (UTC) (edited on 2021-06-08 08:12 (UTC) by nihilistzsche)

https://bugs.archlinux.org/task/66226

According to this bug those options needed to be added for polly to work with Arch's clang.

EDIT:

Upon further reflection the better change is to add the necessary flags to the polly line and add check_buildoption "polly" "y" && to the beginning of the line.

In addition, the -freg-struct-return needs to be moved into the line check_buildoption "rice" "y" && because clang does not support it on x86_64 so should not be used I think with rice-clang.

brikler commented on 2021-03-06 17:47 (UTC) (edited on 2021-03-06 18:06 (UTC) by brikler)

@tjuh upx is more or less useless because most binaries are application/x-pie-executable and application/x-sharedlib, and this will fail with a NotCompressibleException or a CantPackException

Tjuh commented on 2021-03-05 01:00 (UTC)

Alright, thanks for clarifying. So does that mean that using upx to compress pkgs with makepkg is useless, since it will pretty much always fail?

quequotion commented on 2021-03-04 03:38 (UTC)

@Tjuh

@brikler is correct, upx itself does not discriminate, but some kinds of executables will fail to run if compressed.

To be honest, documentation on this topic is lacking. There are a few links in these comments, as well as instructions for how to add new filetypes.

If you find any new filetype that works (compressed executables run), let me know.

Same if you find a problem with my script. I have been over this particular part of it countless times. Upstream discarded this, maybe they had reasons (never found any discussion of it).

brikler commented on 2021-03-02 13:14 (UTC)

@Tjuh makepkg-optimze tests the file before compress but upx doesn't test and you can compress every compressible file. you can test the file type with: file --brief --mime-type <fileToTest>

Tjuh commented on 2021-03-02 12:34 (UTC)

Are you sure about that? I tried a couple of times to manually compress files which makepkg gave a warning about not being able to compress and they compressed just fine with upx --ultra-brute filename.

brikler commented on 2021-03-02 09:24 (UTC) (edited on 2021-03-02 09:26 (UTC) by brikler)

@Tjuh what do you mean? not compressed binaries? makepkg-optimize only compress binaries from the type application/x-executable but not application/x-sharedlib and the most binaries are from the type x-sharedlib

Tjuh commented on 2021-03-01 17:38 (UTC)

Have not been able to properly use upx in months and think because of this pkg, can anyone confirm please?

homocomputeris commented on 2020-07-03 15:36 (UTC) (edited on 2020-07-03 15:37 (UTC) by homocomputeris)

oxipng claims to be significantly faster than optipng. Could you consider adding it as an optional dependency?

quequotion commented on 2020-06-11 11:40 (UTC) (edited on 2020-06-11 11:42 (UTC) by quequotion)

@brikler

See commentary on the additional buildenv options; because clang and gcc are inexplicably not using equivalent flags for equivalent functions, and there is no mechanism for makepkg to detect which toolchain is to be used, I have been forced to make a completely different set of options for clang.

You want lto-thin-clang and poly.

brikler commented on 2020-06-05 14:28 (UTC)

to day i wanted to try lto-thin but it seems clang doesn't work with graphite

  clang-10: error: unknown argument: '-fgraphite-identity'
    clang-10: error: unknown argument: '-floop-interchange'
    clang-10: error: unknown argument: '-floop-nest-optimize'
    clang-10: error: unknown argument: '-floop-parallelize-all'
    clang-10: error: unknown argument: '-ftree-loop-distribution'
    clang-10: error: unknown argument: '-ftree-parallelize-loops=2'
    clang-10: error: unsupported argument 'auto' to option 'flto='
BUILDENV=(!distcc color !ccache check !sign lto-thin graphite !pgo)
[tom@donar compiz]$ pacman -Q clang
clang 10.0.0-3

brikler commented on 2020-03-09 18:02 (UTC) (edited on 2020-03-11 08:18 (UTC) by brikler)

@quequotion

if saw this in makepkg.conf and it's not necessary

-T "$(getconf _NPROCESSORS_ONLN)"
and
zstd -c -z -q --ultra -T0 -22 -)

because xz can do the same with:

#xz --long-help
  -T, --threads=NUM   use at most NUM threads; the default is 1; set to 0
                      to use as many threads as there are processor cores
#man zstd
zstdmt is equivalent to zstd -T0

quequotion commented on 2020-03-04 08:03 (UTC)

@Anonymous Descriptions added

@brikler I've filed an issue upstream to see if we can get some clarification about what mimetypes are safe to compress. Also, UPX just had its first new release in two years, so there may be new capabilities. Looking at the code, it seems like application/x-sharedlib should be covered. Still unsure about application/x-pie-executable.

commented on 2020-02-22 13:08 (UTC)

Could you add descriptions for the optdepends and what they are used for?

brikler commented on 2020-01-07 17:33 (UTC)

@quequotion always outputs "warning, not compressing... this message should appear when 'application/x-executable', 'application/x-dosexec', 'application/x-sharedlib' should be compressed and the system are using more x-sharedlib then other types… so it's probably a x-sharedlib to compress

[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-sharedlib | wc
   1109    1109   26616
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-executable | wc
    427     427   10675
[tom@donar ~]$ file --brief --mime-type /usr/bin/* | grep application/x-sharedlib | wc
   1109    1109   26616
[tom@donar ~]$ file --brief --mime-type /usr/lib/* | grep application/x-executable | wc
      1       1      25

https://aur.archlinux.org/packages/makepkg-optimize/?O=20&PP=10#comment-675946 https://aur.archlinux.org/packages/makepkg-optimize/?O=20&PP=10#comment-675582

quequotion commented on 2019-12-15 06:10 (UTC) (edited on 2020-03-04 14:26 (UTC) by quequotion)

@brikler

I wonder if stripping even works on a compressed binary..

The scripts should be executed in alphanumerical order (this is why lto.sh.in became ZZ-lto.sh.in); strip.sh should be happening before upx.sh anyway..

I'll look into it.

EDIT: I have been unable to compress any binaries in any packages with upx, as makepkg-optimize always outputs "warning, not compressing..." even when I deleted that code from upx.sh. I experienced an alternate reality in which no changes to upx.sh had any effect on the execution of upx.sh and I am at a loss to explain that.

EDIT again: fixed the "alternate reality" problem; apparently packaging had duplicated the tidy scripts as the executable scripts; should also fix strip and upx running out of order

brikler commented on 2019-12-13 11:58 (UTC)

  -> Compressing binaries with UPX...
  -> Stripping unneeded symbols from binaries and libraries...

imo should the binaries be stripped before they will be compressed because less to compress when stripped, right?

quequotion commented on 2019-11-24 15:59 (UTC)

@bartus

I've made two updates since, changing how old used profiles are handled. I suspect this was the weak point: the old conditions allowed for a scenario in which both a directory for generated profiles and an old directory of used profiles may exist simultaneously at build time--in which case pgo would silently fail to utilize the generated profiles and build an unoptimized package. Now, if a folder for newly generated files exists, and a folder of old used profiles also exists, the old used profiles will be deleted and the previously generated profiles will be used to optimize.

Perhaps it has been the case that I always deleted the old profiles by hand, so it had not been a problem for me before.

bartus commented on 2019-11-24 13:06 (UTC) (edited on 2019-11-24 13:11 (UTC) by bartus)

This patch seems redundant (works the same before and after, for me). You should just stick to a single convention for conditionals ([[ is ~25% faster than [ on my system). Also you should consider using shellcheck to sanitize you scripts ( use with care - sometimes it missreport intended behaviors as errors), especially expanding multiple variables in a procedure call without quotes and ${} syntax.

quequotion commented on 2019-11-15 08:47 (UTC)

@brikler

You're right. I'll try to make that change soon. 'auto' hasn't been around long; I hadn't gotten the memo.

@blackout

Are you referring to this? It is indeed possible to specify makepkg-optimize's options in /etc/makepkg.conf, but I recommended using the provided /etc/makepkg-optimize.conf and specifying the config file to when building (--config /etc/makepkg-optimize.conf). I do this so that running makepkg or makechrootpkg (without any flags) on a PKGBUILD makes "standard" (unoptimized) packages. It's less hassle than editing makepkg.conf for every build (not all packages work with all optimizations and sometimes I need a 'clean' build for comparison).

By the way, it makes my day every time I see a new name in these comments! Good to know people are actually using makepkg-optimize!

brikler commented on 2019-11-14 16:22 (UTC)

@blackout i know how to optimize but somebody don't know about and this was a hint because the linker can also run in parallel processes with -flto=auto

-flto=auto can replace ltoflags+="=$(getconf _NPROCESSORS_ONLN) in ZZ-lto.sh.in

blackout commented on 2019-11-14 15:08 (UTC)

@brikler you have to understand the package. check ZZ-lto.sh.in makeflags.conf

after installation you have to check the makepkg.conf and you have the optimizations commented out. so you can replace your original file with the optimizations.

RTM

brikler commented on 2019-11-09 10:23 (UTC)

if found this options perhaps it is interesting for makepkg-optimize?

If you specify the optional n, the optimization and code generation done at link time is executed in parallel using n parallel jobs by utilizing an installed make program. The environment variable MAKE may be used to override the program used.

You can also specify -flto=jobserver to use GNU make’s job server mode to determine the number of parallel jobs. This is useful when the Makefile calling GCC is already executing in parallel. You must prepend a ‘+’ to the command recipe in the parent Makefile for this to work. This option likely only works if MAKE is GNU make. Even without the option value, GCC tries to automatically detect a running GNU make’s job server.

Use -flto=auto to use GNU make’s job server, if available, or otherwise fall back to autodetection of the number of CPU threads present in your system. 

https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

bartus commented on 2019-11-01 09:05 (UTC) (edited on 2019-11-01 12:39 (UTC) by bartus)

@quequotion: I saw this first in Alads commit that strips parallel off of aur-srcver particular this line showing wait -n usage.

quequotion commented on 2019-11-01 03:53 (UTC)

@bartus

Thanks for the assist!

What part of aurutils did you get the idea from?

bartus commented on 2019-10-30 20:58 (UTC) (edited on 2019-10-30 20:59 (UTC) by bartus)

Fix the case statement in upx and add multiprocessing to upx optipng and svgo

git am < <(curl -s 'https://github.com/bartoszek/AUR-makepkg-optimize/commit/f1964eb728aee09524357d58585037b18ea50725.patch')

Inspired by aurutils

bartus commented on 2019-10-24 11:14 (UTC)

pacman=5.2 just landed in [core], you can update depends[]

quequotion commented on 2019-09-24 16:16 (UTC) (edited on 2019-09-26 00:45 (UTC) by quequotion)

@brikler

Finally had a chance to test this in the wild; checking $CC and $CXX only really works if they are set in makepkg.conf. Packages (see pcsx2-git) may set them in build(), after the libmakepkg scripts have run; the only solution seems to be having distinct options for gcc and clang; ie lto-clang and lto-thin-clang...

EDIT: There are now separate options for clang: lto-clang, lto-thin-clang, polly, rice-clang. I'm having a heck of a time proving they work, but they should. polly will not, because it requires the polly package for llvm, which is only available in llvm-git (aur).

quequotion commented on 2019-07-05 12:01 (UTC) (edited on 2019-09-24 16:12 (UTC) by quequotion)

@brikler

You still need to follow the instructions on the wiki to enable the clang compiler (set CC and CXX, etc), but ZZ-lto.sh.in will now set the appropriate flags depending on which compiler you have enabled.

I've also added a new build environment option lto-thin. This does -flto=thin for clang or uses the default behavior for gcc (formerly -fwhopr).

The build environment option lto will do monolithic link time optimization for both; this can result in greater optimization, but requires greater resources and takes more time.

I looked into the compatibility of the pgo build environment option's flags, and it seems like they should be fine as is; clang claims to support the gcc flags in addition to its own.

brikler commented on 2019-07-04 07:26 (UTC)

@quequotion i don't know how makepkg decide the compiler to use but it should be possible to catch this choice before the compiler works because some packages where compiled with clang by default, as example mesa

quequotion commented on 2019-07-03 21:07 (UTC) (edited on 2019-07-10 08:24 (UTC) by quequotion)

@brikler, I read it!

clang's -flto=thin attempts to make lto more scalable by breaking up compilation (using less memory at the cost of somewhat less thorough optimization), similar to gcc's -fwhopr. This is sometimes necessary when compiling very large programs with lto. This may also decrease compile time, as less effort is made to find optimization opportunities in distant parts of the code. EDIT: I have since learned that clang's lto uses a number of threads equal to the number of cores by default, and the flag to control this is --thinlto-jobs=N.

gcc's -flto=n parallelizes the optimization threads, which may also decrease compile time but requires a significantly greater amount of free memory as it searches through larger sections of code (and in multiple threads). EDIT: I have since learned that -fwhopr was scrapped and incorporated as the default behavior of -flto=n; the original behavior can be restored by additionally specifying -flto-partition=none

They are both lto methods, but their goals and results differ.

-flto=thin is not supported by gcc, and -flto=n is not supported by clang. In order to support both, I'd have to come up with some means (in ''lto.sh.in'') of distinguishing which compiler is intended to be used. Perhaps there is something in makepkg's code I could use.

There are actually a lot of other lto-related flags to consider as well. I might make a few other changes to the lto macro, but it's troublesome that the compilers are using mutually incompatible options (makes my job harder).

On a hunch, I went looking for pgo in clang, and found that they are using different flags on the command line and different terms in their documentation. Rather than "profile guided optimization", clang's devs are calling it "profiling with instrumentation".

At the moment makepkg-optimize's macros can only be said to support gcc. I'll look into adding support for clang, and making sure gcc-specific flags aren't getting sent to non-compliant compilers.

Edit: Looking at the archwiki page for clang, in the header it mentions having to edit makepkg.conf to set up the compiler. Doesn't seem like makepkg has any built-in mechanism to set the compiler. This could be an opportunity for libmakepkg--perhaps something like a supplemental buildenv script could be used to switch compilers and filter flags. Not yet sure if I'm talking about a few scripts (maybe another aur package, ie 'makepkg-compilers') or a massive change to makepkg itself (to be proposed, revised, argued, and fought over on the mailing list; then perhaps, someday, implemented).

brikler commented on 2019-07-01 06:28 (UTC) (edited on 2019-07-03 12:30 (UTC) by brikler)

it seems llvm/clang doesn't support -flto=nCore but the build doesn't stop if set.

-rw-r--r-- 1 tom tom 373988 10.04.2019 09:11 upx-3.95-1-x86_64.pkg.tar.xz.clangLTO
-rw-r--r-- 1 tom tom 376536 10.04.2019 09:14 upx-3.95-1-x86_64.pkg.tar.xz.clangTHINLTO
-rw-r--r-- 1 tom tom 371856 10.04.2019 10:24 upx-3.95-1-x86_64.pkg.tar.xz.clangWithoutLTO
-rw-r--r-- 1 tom tom 368732 06.04.2019 16:30 upx-3.95-1-x86_64.pkg.tar.xz.gccLTO
-rw-r--r-- 1 tom tom 373300 10.04.2019 09:53 upx-3.95-1-x86_64.pkg.tar.xz.gccWithoutLTO

btw -flto=thindoes the same as -flto=nCore but it differs how it is be done …please read the linked blog article in my previous post for a more reliable explanation

quequotion commented on 2019-06-28 02:09 (UTC) (edited on 2019-06-28 02:35 (UTC) by quequotion)

brikler, the LTO macro is set in lto.sh.in, which installs to /usr/share/makepkg/buildenv/lto.sh.

Not sure how or if I could add code here to check which compiler was to be used and change the flag accordingly; you can edit it by hand in the meantime.

-flto=thin doesn't do the same thing as -flto=nCPUs; did llvm never support =nCPUs to begin with?

brikler commented on 2019-06-25 14:57 (UTC) (edited on 2019-06-26 18:21 (UTC) by brikler)

is it possible to make it work with llvm/clang without touch makepkg.conf?

=> loop optimization is enabled by default

==> i doesn't found llvm/clang support pgo

===>llvm/clang doesn't use the "-flto=nCores" flag, instead they use "-flto=thin", if is compiled with this option (llvm/clang 8 isn't compiled with) http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html

quequotion commented on 2019-03-10 10:15 (UTC) (edited on 2019-03-10 11:52 (UTC) by quequotion)

@brikler What profile-guided optimization does is collect statistics on the execution of a program, then use the data to generate more optimal code when recompiling the program. The result should be a faster, smaller binary that achieves whatever the original code intends. Of course, as the second version of the code is machine-generated, that cannot always be guaranteed; the benefits also depend on how well the profiling run exemplifies typical use of the program.

In order to use PGO, a package must be built, installed, used, and then rebuilt and reinstalled. During the first run, profiles are generated and stored in $PROFDEST/$pkgbase.gen. Once you've used every feature of the software that you wish to optimize, rebuild the package to move the profiles to $PROFDEST/$pkgbase.used and apply them, creating optimized binaries. It is a time-consuming process, but the benefits are generally proportional to the effort.

It is possible to use the collected profiles to make improvements to the code by hand, such as is done by developers of the kernel, kernel modules, real-time control software, etc. Unless you intend to do so, it is not necessary to keep the $pkgbase.used folder.

brikler commented on 2019-03-10 07:45 (UTC) (edited on 2019-03-10 07:49 (UTC) by brikler)

so i must keep the optimized profile up to the next with pgo compiled version come? i thought it's code optimization not for run time?

quequotion commented on 2019-03-10 01:01 (UTC) (edited on 2019-03-10 01:02 (UTC) by quequotion)

@brikler Systemd is trying to access its profiles, but your home directory isn't available until you log in. There is a new "Package Output" variable for this in makepkg-optimize.conf, PROFDEST. If unset, profiles are stored in the build directory.

To profile systemd, you'll need to set PROFDEST to somewhere in the root file system; I use /mnt/pgo/.

brikler commented on 2019-03-09 12:58 (UTC)

next "problem": i have build systemd with pgo and after reboot my sreen was spammed with this kind of message:

…
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-send.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-vacuum.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/journal-verify.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/lookup3.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/mmap-cache.c.gcda:Cannot open
profiling:/home/tom/compile/systemd/systemd.gen/src/journal/da05bd7@@journal-client@sta/sd-journal.c.gcda:Cannot open

how to avoid this?

quequotion commented on 2019-03-08 14:55 (UTC) (edited on 2019-03-08 17:23 (UTC) by quequotion)

@brikler The flags were not getting set because of a typo!

lto.sh.in line 14:

-buildenv_functionss+=('buildenv_lto')
+buildenv_functions+=('buildenv_lto')

No idea how long that'd been there. Thanks for the help!

The flags are where they need to be. I test built a few things to make sure; these flags work. They have a long history.

Note that some packages may fail to compile with LTO, for example pulseaudio-git.

brikler commented on 2019-03-06 13:34 (UTC) (edited on 2019-03-06 13:45 (UTC) by brikler)

it seems the settings in /usr/share/makepkg/buildenv/lto.sh are not honered by gcc, as example a build with -flto set in makepkg.conf an on only lto.sh

<h1>-flto and -fuse-linker-plugin set in makepkg.conf. from config.log</h1>

gcc -o conftest -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -flto=4 -fuse-linker-plugin -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5

<h1>only in lto.sh from config.log</h1>

configure:3574: gcc -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -fgraphite-identity -floop-nest-optimize -ftree-loop-distribution -ftree-vectorize -fprofile-correction -fprofile-use -fprofile-dir=/home/tom/compile/iwd/iwd.used -mf16c -mavx -lpthread -pthread -march=native -Os -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fopenmp -Wno-error -w -fno-plt -Wl,-O3,--sort-common,--as-needed,-z,relro,--hash-style=gnu,-fuse-ld=gold conftest.c >&5

and the build process breaks when -fuse-linker-plugin is in LDFLAGS but is working in CFLAGS

quequotion commented on 2019-02-27 07:49 (UTC) (edited on 2022-01-09 13:20 (UTC) by quequotion)

makepkg-optimize is a collection of libmakepkg tidy, buildenv, and executable extensions, and a supplement to pacman. These enable various optimization routines for building and packaging such as upx compression, profile guided optimization, polyhedral model optimization, etc..

Note: Over-optimization is a thing, and it is not good.

See the wiki article for details.

Note to packagers: makepkg-optmize's macros may be enabled or disabled in options() as well!

quequotion commented on 2019-02-14 09:09 (UTC) (edited on 2019-02-16 12:20 (UTC) by quequotion)

Updates:

-Added a warning to upx.sh.in about skipping binaries with mimetypes that may not be supported. Research needs to be done to find out if other mimetypes are compatible with UPX compression.

If I interpret this correctly, compressing application/x-sharedlib does not work, while compressing application/x-pie-executable should.

-New instructions regarding clean chroot building with PGO on the wiki.

quequotion commented on 2018-12-30 12:11 (UTC)

@brikler Aha, another filetype that is excluded. Yes, you could get upx to compress this by adding it to the list. Again, I don't know if it is safe to do so. You should look up if anyone has tried upx on shared libraries.

After the holidays, I think what I will do with the script in this package is add a catch-all case that will ouput something like:

Skipping upx compression for unsupported binary: $binary $filetype

Of course any filetypes that can be proven safe to upx compress I will add to the list!

brikler commented on 2018-12-26 17:18 (UTC) (edited on 2018-12-27 08:34 (UTC) by brikler)

[tom@donar upx]$ file --brief --mime-type upx
application/x-sharedlib

is it helpful?

edit your idea solves the problem :)

        case "$(file --brief --mime-type "$binary")" in
                'application/x-executable' | 'application/x-dosexec' | 'application/x-sharedlib') 

thank you for your help :)

quequotion commented on 2018-12-26 08:35 (UTC)

@brikler could you run:

file --brief --mime-type "$binary"

On that unpacked upx binary? (put it in place of $binary). I'm pretty sure you're going to get "application/x-pie-executable".

I've tried to run a a few PIE programs compressed by UPX, but they crash. This type of executable was excluded by the pacman developers when the upx script was part of it. I don't know if that is because it is known to be unsafe, or just an oversight. I found very little about PIE and UPX by google search, but what I could find suggests that it may be unsafe.

brikler commented on 2018-12-21 09:45 (UTC) (edited on 2018-12-21 09:59 (UTC) by brikler)

@quequotion upx.sh doesn't work. i have build upx withe "upx" in the option array but the binary wasn't compressed. a really simple test unpack the package and compress the binary:

[tom@donar upx]$ upx --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1958664 ->    493528   25.20%   linux/amd64   upx                           

Packed 1 file.
[tom@donar upx]$ upx -d --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1958664 <-    493528   25.20%   linux/amd64   upx

Unpacked 1 file.

if it where compressed there would be throw a "cant pack exception"

[tom@donar upx]$ upx --exact upx
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx: upx: CantPackException: bad e_shstrndx

Packed 0 files.

quequotion commented on 2018-12-20 06:22 (UTC) (edited on 2018-12-20 09:11 (UTC) by quequotion)

@brikler please post the error messages you received, if any. posting the code is not necessary, but if you do post code, put it between two lines of ``` to make a code block (AUR supports markdown).

here is an example

I'm looking into this, but I believe this code works. It is exactly the same code as was removed from pacman.

I added a catchall statement to the case switch:

*)
    echo "$(file --brief --mime-type "$binary")"
    ;;

This showed executables having the filetype "application/x-pie-executable". I'm not sure if it is safe to compress a PIE executable with UPX.

brikler commented on 2018-12-19 14:38 (UTC) (edited on 2018-12-20 17:11 (UTC) by brikler)

the upx script doesn't work as expected i noticed because i had build some packages with set upx in the option array but the binaries was never compressed.

tidy_upx() {
    if check_option "upx" "y"; then
        msg2 "$(gettext "Compressing binaries with %s...")" "UPX"
        local binary
        find . -type f -perm -u+w 2>/dev/null | while read -r binary ; do
            case "$(file --brief --mime-type "$binary")" in
                'application/x-executable' | 'application/x-dosexec')
                    upx "${UPXFLAGS[@]}" "$binary" &>/dev/null ||
                        warning "$(gettext "Could not compress binary : %s")" "${binary/$pkgdir\//}"
                    ;;
            esac
        done
    fi
}

it seems binary never reached the upx comman https://aur.archlinux.org/cgit/aur.git/tree/upx.sh.in?h=makepkg-optimize

quequotion commented on 2018-11-28 02:34 (UTC) (edited on 2019-02-27 07:36 (UTC) by quequotion)

Good news, buildenv is now extendable: https://git.archlinux.org/pacman.git/commit/?id=508b4e3ec0cb3e365942f4dc0626edda4789932b

makepkg-optimize 5-1 is now a supplement to pacman-git!

As such, the package now conflicts() and replaces() pacman-buildenv_ext-git, which is deprecated, and depends() on pacman-git.

To avoid the possibility of pacman removing itself, remove old versions of makepkg-optimize and install pacman-git before upgrading.

If you use an AUR helper, you can probably install them simultaneously:

pikaur -S makepkg-optimize pacman-git

See the wiki article for details.

quequotion commented on 2016-11-14 17:18 (UTC) (edited on 2018-08-30 19:08 (UTC) by quequotion)

wolf

Long story short, the concept may get in there, someday...

I have grown tired of waiting. I've uploaded a patched "pacman-buildenv_ext-git" and "makepkg-optimize" to demonstrate how this {c,sh}ould work in the future: pacman patched to allow extensions to BUILDENV and separate packages of supplemental BUILDENV and Tidy extensions.

wolf commented on 2016-06-30 12:36 (UTC)

> Neither would be necessary, were my proposal to pacman-dev approved what as the reason it was not approved? I went throught the mailing list and saw no response to your patch. Did you get back any feedback at all? It sounds pretty useful.

quequotion commented on 2016-04-01 06:29 (UTC)

For now, makepkg-optimize installs an alternative script and configuration file. Neither would be necessary, were my proposal to pacman-dev approved: https://lists.archlinux.org/pipermail/pacman-dev/2016-March/020960.html

quequotion commented on 2016-03-22 15:23 (UTC)

>DeadMetaler Edit makepkg-optmize.conf, add lto to BUILDENV=(........)

DeadMetaler commented on 2016-03-21 23:20 (UTC) (edited on 2016-03-22 08:34 (UTC) by DeadMetaler)

How to use lto for any package?

quequotion commented on 2016-03-20 21:26 (UTC) (edited on 2016-03-20 21:26 (UTC) by quequotion)

>checksums I cheated, it's true. Must remember to always update pkgsums and then scrinfo before add/commit/push. >type-o How'd that happen?! >xz This is a little complicated. Is the threading implementation actually implemented now? or is the option still a place holder? Either way, all available cores is said to be the default. As for the compression, I might go with "--best -e" because I have the CPU for it, but maybe just "--best" for the package release--that will probably inflate tiny packages though. Also, the problems with makepkg-optimize.conf are a temporary concern. Once the next pacman release rolls out, I will change this to copy makepkg.conf from the user's own original. The current file represents some of my own settings.

FadeMind commented on 2016-03-20 20:09 (UTC) (edited on 2016-03-20 20:10 (UTC) by FadeMind)

And one more: COMPRESSXZ=(xz -T 0 -7 -c -z -) using all cores and it is high level compression for xz instead of default COMPRESSXZ=(xz -c -z -) In line 25 is typo #-- The the package required by makepkg to download VCS sources Simple double the at begin line in makepkg.conf file code.

FadeMind commented on 2016-03-20 19:54 (UTC)

checksums are not corrected. FIXED sha512sums=('67fb9359cd5dc52413ad2ca7e91b445826d7cf6b7ad3db0d2b3845310ab0bdfc41c2e70cec564e26269521ac313f7202c44996a24266380e48e88927bd1012e2' 'b4896082fd7fc4a69abc5fc98bf22c92a94cef9ccae5d3e5d1825cd3f04dcff1bce622a3f7fdc00bd43b5e1d091825d119b3a91d354f1ab58e6a8fe3af72b0e2' 'e8d9112c8ff66136451d2746a91c2c43083541322ca5430a0bc6bf03e65ace5ff00296ffea4582075a325d02ddf1d880483025d1fcfba2ce749e12d6418b413b' '3b88f57cd26efbab719e6a3c2a3f49f786c919cd85c8d9d89ef724d1d0a37287f499a729c340c032ed8d4f6f581d2ddb7d417c0ae9195ae6683f4906edfa6da5' '7c54d9c425ec6b6baf8ff4cd1580c6235fe01a612959bc5ae86eafaf9923e45039b8c303e2fcea113b65183ffd00fa06caf5e578f7fab146e02da3e926423e54' 'db55e9d4302035447854ec34d44d454a1a68882437129c00de388f56413cff71e133848df5a09962c40f2bd0203f5e5c692af4e100e67072da4b504ef4752ca4' '60ca209609ddbf617e63c7103d675da17671efc91db066a1e11dd1df78f9c8abf267f1cd417ff3b5db59411cf6e7df134e6bd8b06c73442410f14c13b1f5dd53' '16c2b0e666234f1a493e7456f4c387fbabd24b10514c93fa4338520e8f537e9a12de0d833a6148fe871452d87f6e1b3454b2bde332b9ba4c4e7680616b7eeede' 'dd609146c46b36bf3b9d7ed81935b64708cc5a37210f39e54162c603f1cae2bc9d8110dec92d61939c63245549b0a4548b17a15f9746abdd64faa65421288a2e' 'cef2f8f3e82fa668c490f305bb01a344391958ca3f0be95da09e6008b30ecf996803dfda863a73e340c71937c7691d115bfe7e109c71d71eb65b5052bb179504' '6df2ac2133f72ac2f3533fa9e3cc9337849dd5afbba2fd66b5d4d8b0b65a892c04771ceb6407eac357755fef87ac940d7848ccad915c387e465ad46b1ed496a3' '9e04d88c43cc25d21854ffc536cf9fd682d150fed08440a44a60b4e67285c28163cb069182355e718b2a6b6df8d536d8b32dcaac76f04d346cd658850b46769d')