Arch Linux User Repository

Peter @ptr1337, consider using oxipng instead of optipng

Why?

1) It does optimizes better.

You can see that optipng is one of the dependencies of Calibre. Indeed, it's used internally by it:

https://github.com/kovidgoyal/calibre/blob/eb161166b05f33eb0ff6bda97bac1fafadca77c5/src/calibre/utils/img.py#L621-L625

CLI utility of calibre ebook-polish is using it. My experience is that after running epub-polish on EPUB book with PNG images, the latter can be further optimized with oxipng:

oxipng --opt max extracted_epub/**.png

So it means that optipng doesn't optimally encodes PNGs. Worth noting that oxipng with --opt max doesn't use Zopfli, there is separate CLI switch for it, i.e. it's not why oxipng compresses files better.

2) oxipng is faster (multithreaded, uses all avaliable cores/threads by default).

As stated in the project's README:

The core goal of rewriting OptiPNG was to implement multithreading, which would be very difficult to do within the existing C codebase of OptiPNG. This also served as an opportunity to choose a more modern, safer language (Rust).

Multi-threading would be especially useful if you need to optimize only a few images (< $(nproc)), because you won't be able to utilize all your threads with single-threaded optipng, unlike with oxipng

makepkg-optimize.conf isn't properly created

CFLAGS="-march=native -O3 -pipe -fno-plt -fexceptions \
#CFLAGS="-march=native -O3 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fno-plt -fopenmp -pthread -Wno-error -w" #
#CFLAGS="-march=native -Og -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fno-plt" #
        -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security \
        -fstack-clash-protection -fcf-protection"

Mold --separate-debug-file and -fuse-ld=mold needs to be in one section with -Wl

#!/usr/bin/bash
#
#   mold.sh - Compile with the fast mold linker
#

[[ -n "$LIBMAKEPKG_BUILDENV_MOLD_SH" ]] && return
LIBMAKEPKG_BUILDENV_MOLD_SH=1

LIBRARY=${LIBRARY:-'/usr/share/makepkg'}

source "$LIBRARY/util/option.sh"

build_options+=('mold')
buildenv_functions+=('buildenv_mold')

buildenv_mold() {
    if
    check_buildoption "mold" "y"; then
            LDFLAGS="${LDFLAGS/-Wl,/-fuse-ld=mold -Wl,}"
        LDFLAGS+=",--separate-debug-file"
    fi
}

clang pgo: https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization isn't easy like pgo in gcc because more opportunity

@brikler

Actually I have never changed anything on the PGO thing from the upstream makepkg-optimize and I did tested it just with gcc - im sorry. Clang uses/needs other options for that. If I have time I will look into this!

it doesn't work as expected. the first compilation breaks with:

clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.used' [-Wunused-command-line-argument]
clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.used' [-Wunused-command-line-argument]
error: error: Could not read profile default.profdata: No such file or directoryCould not read profile default.profdata: No such file or directory

1 error generated.
make[1]: *** [Makefile:92: help.o] Error 1
make[1]: *** Waiting for unfinished jobs....
1 error generated.
make[1]: *** [Makefile:92: main.o] Error 1
make: *** [Makefile:31: all] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

PROFDEST was set in the PKGBUILD, and upx.genwas on the correct place.

there is something wrong in general, because its not on me to create gen and used folder.

edit1: it works with gcc, but not with 'clang'

edit2: a propos UNITY_BUILD it is more or less a developer tool, and it will not work on every project, but if, the time to compile a project can be reduced up to 50%. more about unity build: https://mesonbuild.com/Unity-builds.html

Cause of incremental builds I have something here: https://github.com/ptr1337/dotfiles/blob/master/scripts/toolchain/build_llvm.bash#L242

But it is not very well tested. It does work so for but has some issues and it also uses "check" at the compilation.

@brikler

You need to add in your makepkg.conf for example: PROFDEST=/home/ptr1337/Documents/pgo

In addition your need to create the folder for it. After you did compile the binary it should create a directory with packagename.gen and then you need to use your compiled binary, so that it does create profiles. So running for example at you upx based commands, then it will create profiles. As soon profiles are there it will be merged at the next compilation and change the folder to .used.

Here is it also explained: https://wiki.archlinux.org/index.php/Makepkg-optimize

Because CMAKE_UNITY Ive never watched into it and personally never saw any usage of it but i will take a watch.

Thank you

i noticed pgo doesn't work, there where no <paketname>.gen nor <paketname>.used folder created.

how to debug?

edit upx is my test binary :) this error messages appear when compile upx

clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
clang-14clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
clang-14clang-14: : : warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]
clang-14: warning: argument unused during compilation: '-fprofile-dir=/home/tom/compile/upx/upx.gen' [-Wunused-command-line-argument]

@ptr1337

thank you for this explanation :)

a'porpos clang, have you tried cmakes unity build function to build clang? does it work? it will safe a lot time to build clang, if this function work. https://cmake.org/cmake/help/latest/variable/CMAKE_UNITY_BUILD.html