Arch Linux User Repository

@Netboy: You are entirely correct. Same for nanosvg. I completely missed the addition of these. I'll update the sources so these can be efficiently managed/cached as well 👍

@Omar007: wayland-protocols should vendored as submodule. It's been set like that since February 2022.

@roosterteeth: I don't know the details of how yay deals with things and what it's supposed to do here but this suggests at least that it does not perform a clean build.

Since I'm not familiar with yay, I can't help you with that. Check the man-pages for options on how to have it perform clean builds or on how to clean the build environment it is using. If builds fail in general (which seems to be the case if it fails on some completely different file as well) this is even more important.
(side-note; 'stuck on a different file' is basically at the level of 'it doesn't work'. Provide exact errors/logs if you can)

I got a patch for my iGPU in /r/vfio because of a memory bug and I'm trying to apply the patch which I think works but I ran yay -aS looking-glass-git linux-vfio vfio-isolate --rebuild again gives mkdir: cannot create directory ‘host/build’: File exists. I don't understand what your all talking about below but I tried to delete the files and it is still stuck is there a fix? I already tried to delete some and https://aur.archlinux.org/packages/looking-glass as well but it's stuck on a different file

@ipaqmaster;

You're over-complicating this when all I'm requesting is that you apply this harmless 2-byte -p fix to the PKGBUILD file.

I'm not. Again, all of this works fine if you're caching correctly and doing the build itself clean. I build numerous packages including this one completely automatically and none have issues (barring any that don't build in a clean environment in the first place (usually dependency related) but I try to push fixes upstream for those as I encounter them).

In this case it may seem like a harmless 2-byte addition but it's not about the specific change, it's about the principle. I'm not very interested in nor do I have the time to cover and test potential use-cases outside of the standard build processes and managing compatibility for custom things that don't conform to the clean build process (incl. proper caching). If you (or anyone) can show me that anything I maintain/provide does not build in a clean environment as documented for Arch Linux (whether you cache or not, I do not care) or does not conform to upstream defaults where it should, I'll happily make corrections or changes as needed.

I don't need a lesson on build caching here, you don't need to worry about our local build environment ...

The request seems to suggest I do need to worry about it because you're here asking me to make changes to something that works fine when build in a clean environment (incl. proper caching).

(Including this one! after applying a mkdir -p patch)

Well, if that is the fix your environment needs, then keep that local modification. There's nothing stopping anyone from doing w/e they want locally. Just don't expect anything from me if you're deviating from a clean build environment.
It's not like I don't do the same occasionally but unless the problem persists with a bare and clean (re)build and provably doesn't work, it's my own problem to deal with.

The single issue I'm having here is that your PKGBUILD's mkdir line in the build() function stops the entire build process if it sees a directory it already created in an earlier run. This is an inexcusable reason to halt an entire build over.

Again, if you're building cleanly (even with caching), it doesn't halt. It does for your specific process apparently but I don't know what that is. Look at it from the other perspective; if this fails, I know the build environment wasn't clean. Regardless of the use of caches or not if/when this happens on my side for instance, there is something I need to fix as that means I f*ed the clean build environment somehow and I can no longer give a guarantee that what I pushed to the AUR can and has indeed been cleanly build at least once.

If you want a clean build you can wipe a workspace yourself or invoke make clean otherwise it's a waste of resources to obliterate other hard work (Particularly the dependencies of this project which take a moment and infrequently change).

There's no need to waste anything. I'm using caches just fine without it breaking builds. I even apply custom patches here and there completely without the need to change caches or pipelines.

I can't communicate this any clearer our build server is excperiencing NO issues nor re-build issues with this AUR package except when your PKGBUILD halts the build because of mkdir throwing a fatal error over a non-fatal directory's existence before it can even start compiling.

Again, that's a problem specifically in your environment. Performing the build cleanly using caches of all sources, dependencies, etc, will work every time.

You heard correctly - no issue with subsequent builds in a workspace it previously used. Not in the slightest. Yes, if something went wrong it would be on me locally to handle that build failure in its persistent workspace, not you as the AUR package maintainer.

Not visibly at least. You wouldn't be the first one to msg me because they (re)build, have a package but it doesn't run, crashes, has weird behaviour, etc, and in the end it just turned out their build environment wasn't clean. I have wasted enough time on bullshit like that. Which is why I expect someone to perform a clean build as documented first before I do anything more now.

Not only does reusing the workspace save pulling 250MiB of submodules from github every single time the package gets built (Which could be many times a day if the looking-glass project team are busy!)

Cache your sources.

but it also saves recompiling all these unchanged dependencies every single time this package gets an update.

See ccache and it's makepkg integration.

Not only does this also cut build times by more than half and spares the network traffic of initializing the submodules from scratch every build which would also be beneficial for metered and low-bandwidth connections not having to pull that 250MiB every single run. Let alone potentially getting rate limited by github with enough back to back builds. In subsequent runs even git knows to update the submodules rather than pull them entirely from scratch which would be a total waste.

If your caching is set up properly, none of this is a factor.

This is the only package failing subsequent builds in our pipelines of which the single lone root cause is the mkdir in PKGBUILD here throwing exit code(1) when it sees a directory it already created. You're suggesting we discard the build environment every single time downloading 250MB of dependency projects and wasting cpu cycles recompiling them with each new looking-glass commit? The PKGBUILD shouldn't be the single point of failure in a re-build by getting upset over the existence of a folder it created in a previous run.

Not a single time I've suggested to just throw away your workspace or caches, quite the opposite! I've suggested correcting your environment to cache the correct things, splitting out your caches to optimize and selectively be able to clean specific caches (if ever needed) thus further optimizing bandwidth and CPU use and finally to use a clean build environment.

To work around this PKGBUILD problem we've had to move the job to its own separate pipeline where sed -i 's/mkdir "/mkdir -p "/g' PKGBUILD is called before building. It would be really nice if it could join the other 200+ packages in our build pipeline.

So you really do want me to worry about your local build environment!

There's not much else to say here - our build environments for all packages are sound including this one - PKGBUILD in its current state halts everything over non-fatal directory presence without locally tweaking it to not do that. Please add -p to PKGBUILD's mkdir command so it doesn't crash and burn on subsequent runs anymore 🙇

Please fix your local build environment so it doesn't build in a dirty environment 🙇

TL/DR; You can cache everything without a problem, just keep the build environment clean. If this is what you have to go through, you're currently doing the caching incorrectly and technically building all your packages in an unclean manner/build environment. I do not see how that needs to be my problem to deal with. It works fine if you cache properly and (re)build cleanly.

TL/DR 2; It's not about the -p flag addition, it's about the principle of adding custom things for specific end-users. Here it's maybe 'just' a -p flag but then why should I say no the the next person wanting me to do include a custom/debug make flag or a rm -rf to clean up things or something else for their environment to work?

You're over-complicating this when all I'm requesting is that you apply this harmless 2-byte -p fix to the PKGBUILD file. I don't need a lesson on build caching here, you don't need to worry about our local build environment as it's making hundreds of flawless packages (Including this one! after applying a mkdir -p patch) without having to recompile hours and hours of work through each run (Strictly where applicable).

The single issue I'm having here is that your PKGBUILD's mkdir line in the build() function stops the entire build process if it sees a directory it already created in an earlier run. This is an inexcusable reason to halt an entire build over. If you want a clean build you can wipe a workspace yourself or invoke make clean otherwise it's a waste of resources to obliterate other hard work (Particularly the dependencies of this project which take a moment and infrequently change). I can't communicate this any clearer our build server is excperiencing NO issues nor re-build issues with this AUR package except when your PKGBUILD halts the build because of mkdir throwing a fatal error over a non-fatal directory's existence before it can even start compiling.

Subsequent builds of both the latest commit and older commits (Potentially with new local patches and hotfixes for example) work entirely fine with -p added to mkdir in this PKGBUILD. You heard correctly - no issue with subsequent builds in a workspace it previously used. Not in the slightest. Yes, if something went wrong it would be on me locally to handle that build failure in its persistent workspace, not you as the AUR package maintainer.

Not only does reusing the workspace save pulling 250MiB of submodules from github every single time the package gets built (Which could be many times a day if the looking-glass project team are busy!) but it also saves recompiling all these unchanged dependencies every single time this package gets an update. Not only does this also cut build times by more than half and spares the network traffic of initializing the submodules from scratch every build which would also be beneficial for metered and low-bandwidth connections not having to pull that 250MiB every single run. Let alone potentially getting rate limited by github with enough back to back builds. In subsequent runs even git knows to update the submodules rather than pull them entirely from scratch which would be a total waste.

This is the only package failing subsequent builds in our pipelines of which the single lone root cause is the mkdir in PKGBUILD here throwing exit code(1) when it sees a directory it already created. You're suggesting we discard the build environment every single time downloading 250MB of dependency projects and wasting cpu cycles recompiling them with each new looking-glass commit? The PKGBUILD shouldn't be the single point of failure in a re-build by getting upset over the existence of a folder it created in a previous run.

To work around this PKGBUILD problem we've had to move the job to its own separate pipeline where sed -i 's/mkdir "/mkdir -p "/g' PKGBUILD is called before building. It would be really nice if it could join the other 200+ packages in our build pipeline.

There's not much else to say here - our build environments for all packages are sound including this one - PKGBUILD in its current state halts everything over non-fatal directory presence without locally tweaking it to not do that. Please add -p to PKGBUILD's mkdir command so it doesn't crash and burn on subsequent runs anymore 🙇

@ipaqmaster: I'm also building automatically and verify with a clean build env. Obv. I can't tell how you've set up your automated build system but if you're doing caching for efficiency, there are several levels at which you could and should do it. Please note that if you're caching certain levels/layers, your builds are no longer clean and you may encounter inexplicable problems that are yours and yours only to deal with (which seems to be the case here as again, a clean environment works 100% of the time).
TL/DR; you can do caching but caching the whole tree is not a good way to go about it (caching should be separated into different levels) and issues are yours to deal with. This is sticking to both upstream instructions as exact as possible to prevent issues there.

In regards to setting up/playing with caching, at the first level you can do source caching. This is safe and with checksums, git, etc. this should just speed up the process without any issues or consequences to a (clean) build. You can always cache the source and only update/fetch changes as needed before going into the build process. This is basically agnostic to any package and should always be fine (barring any PKGBUILD that grabs/defines sources in a custom way). Look into SRCDEST (makepkg) for information on how to play with this and separating things out from your working/build dir.
Compile caching is the next level. Starting at this point things are not exactly clean anymore so it's a good practice to separate things into separate caches so you can clear this one individually without affecting your source cache if anything goes wrong. Look into using ccache. It works with both gcc and clang and integrates with makepkg (BUILDENV).
Lastly, you could do a build artifact cache, again separate from the other caches. Look into PKGDEST and SRCPKGDEST. Though if you store the resulting package, I'd say this is rather pointless so I would not bother with this.

Yes, I'm building this one automatically / periodically in Jenkins and it's the only one throwing a fuss in 200+ AUR packages.

This particular package doesn't take very long to build so maybe it can be made an exception by not utilizing a workspace but this seems very network and cpu cycle inefficient.

Large enough projects take hours to build from source and many AUR packages fetch project's from source and compile them on the local machine for a final zst for pacman to use. It would be stupid to rebuild every single common unchanged C component for these programs every single invocation no?

Are you suggesting this project is one which must be built from scratch every single time? I added -p to that mkdir line in the PKGBUILD's build function and it successfully made a new zst package every time with makepkg after deleting the most recently built zst but also worked with makepkg -f without deleting the existing zst. I'm not sure if there's something else I'm missing?

As the host didn't need to git-clone looking-glass, LGMP, PureSpice, cimgui, imgui nor rebuild any of their many pieces it's able to spit out a result more quickly and without downloading ~248.9MiB of git objects every single build attempt. That and it only took 27.6 seconds to build the package instead of 1m 9s (Fiber internet. Result would likely be even slower on a non-fiber connection)

// My bad, I always mistake -p for -v when talking about the -p mkdir flag. -p is what I tested with for the first comment and it solved everything in my case.

@ipaqmaster; it should not. You getting that error suggests that you're not building in a clean environment/chroot (e.g. https://wiki.archlinux.org/title/DeveloperWiki:Building_in_a_clean_chroot ). That is fine but in that case any breakage is for you to deal with.

Fwiw, adding -v is not going to change anything. You might be thinking of -p maybe? But in that case be aware that it will create parent directories and it does not fail (exit code always 0). That means if any expected dir in the path is missing, it'll fail somewhere later on with no indication this is actually where it failed (failure here would've told you the file structure is not as expected). But again, that's all for you to deal with if you choose to not build in a clean environment.

Subsequent builds fail due to line 55 in PKGBUILD.

mkdir: cannot create directory ‘client/build’: File exists
==> ERROR: A failure occurred in build().
    Aborting...

The mkdir "${b}" in PKGBUILD line 55 should have -p added.

(E: Corrected -v to -p)