Continuing improvements to continuous integration

[cboulay]

I recently updated the GitHub Actions yaml. The current behaviour is as follows:

• On pull request, build for all 3 platforms
• On push of a tag that starts with 'v' (e.g., v1.13.0), build, then create release and upload packages to release.

I'm fairly happy with this though I think there are a couple things that should be improved.

• Get rid of other CIs (appveyor and travis). 'latest' Windows, Mac, and Linux are enough for the applications.
  • This is trivial. I just want to hear any objections before I delete the configuration files.
• Get rid of hard-coding of application name and version in artifact files and package names.

The second problem has a couple potential solutions.

• The first solution is potentially complicated but could work today: capture the project name and project version from the CMakeCache.txt and use those to build filenames. I have an example of how to capture the names with Powershell here, but I have no idea how to use those to build the upload-artifact with: path:.
• The second solution is probably much simpler but we have to wait. Both upload-artifact and download-artifact actions are in preview for v2 which will support file globbing. I don't know what the syntax will be when that comes, but presumably we can just provide some wildcards (build/*.7z) and we don't have to worry about the package name.
  • I don't know how to use these filenames in softprops/action-gh-release with: files:.

[tstenner]

I really like appveyor, it has Qt preinstalled and the syntax is a lot easier than everything else.

Setting the names is possible through a somewhat hacky workaround: environment variables can be set by printing a special string (something like gh_actions_set_var::VARNAME=VALUE) to stdout, e.g. from the CMake / CPack string. Those can only be used in shell scripts, not as yaml variables (so, same problem as with the Powershell script).

The preview for the upload artifact is usable as of yesterday, so we could start using it.

[cboulay]

I really like appveyor, it has Qt preinstalled and the syntax is a lot easier than everything else.

I really don't want to maintain 2 different CI configurations when it isn't necessary. If we switch to AppVeyor exclusively, including using it for MacOS (available since ~Nov 2019)...

• Is the syntax that much easier when sh is no longer exclusive to the Linux builds? Actually I don't even know if it's sh because I can't find any docs on using AppVeyor for MacOS.
• The syntax for setting up a GitHub release and uploading artifacts seems easier than GitHub actions, but requires creating and configuring an appveyor account, generate OAuth token for their GitHub credentials, maybe more? This is maybe too much overhead for developers of other apps who we hope will use this template.

Can you try adding MacOS builds and creating GitHub releases and uploading artifacts to AppVeyor to see how easy it is? And I'll try cleaning up the GitHub actions script using v2-preview for uploading and downloading artifacts. Then we can re-evaluate which one is best.

I'll work on this in the AudioCapture app so we don't collide our GitHub releases.

[cboulay]

I made some progress in the AudioCapture app. It's a fair bit shorter and the yml file is now generic so it should work in any multi-platform app without editing. I'm still getting a ghost "Unspecified" target being built for MacOS and there are some bugs with packaging (though it still seems to upload the artifact ok 🤷‍♂ ).

https://github.com/labstreaminglayer/App-AudioCapture/blob/master/.github/workflows/cppcmake.yml

[tstenner]

I really don't want to maintain 2 different CI configurations when it isn't necessary.

For liblsl I like having several options, but for the apps I agree.

Is the syntax that much easier when sh is no longer exclusive to the Linux builds? Actually I don't even know if it's sh because I can't find any docs on using AppVeyor for MacOS.

Even with sh also running on OS X I think the Appveyor config is more straightforward than github actions.

The syntax for setting up a GitHub release and uploading artifacts seems easier than GitHub actions, but requires creating and configuring an appveyor account, generate OAuth token for their GitHub credentials, maybe more? This is maybe too much overhead for developers of other apps who we hope will use this template.

The question isn't whether Appveyor or Github Actions is better, but if we'd rather have developers struggling with the OAuth+Github integration or the Github Actions syntax.
The documentation and integration for Github Actions is getting better rapidly and for most apps it shouldn't be necessary to touch with the CI config, so I guess Github Actions might be easier to use.

I'm still getting a ghost "Unspecified" target being built for MacOS and there are some bugs with packaging (though it still seems to upload the artifact ok man_shrugging ).

The COMPONENT argument for install() etc. tells CMake which component to put the file into, and if it's missing it's put into the "Unspecified" component. Whenever I get an unspecified component I just unpack it to see which install call needs the parameter.

[cboulay]

The COMPONENT argument for install() etc. tells CMake which component to put the file into, and if it's missing it's put into the "Unspecified" component. Whenever I get an unspecified component I just unpack it to see which install call needs the parameter.

The bzip file is empty.

--target install works fine. It correctly tries to install only the AudioCapture target, and it finds the .app to run macdeployqt on:

-- Running Qt Deploy Tool for /Volumes/STORE/Tools/Neurophys/labstreaminglayer/Apps/AudioCapture/build/AudioCapture.app/Contents/MacOS/AudioCapture...

It's only --target package that's a problem.

CPack: Create package using TBZ2
CPack: Install projects
CPack: - Run preinstall target for: AudioCapture
CPack: - Install project: AudioCapture []
CPack: -   Install component: AudioCapture
CPack: -   Install component: Unspecified
'/usr/local/opt/qt/bin/macdeployqt' '/Volumes/STORE/Tools/Neurophys/labstreaminglayer/Apps/AudioCapture/build/_CPack_Packages/Darwin/TBZ2/lsl-0.1-Darwin/Unspecified/AudioCapture/AudioCapture.app' '-verbose=1' '-always-overwrite'
CMake Warning at /Volumes/STORE/Tools/Neurophys/labstreaminglayer/Apps/AudioCapture/build/cmake_install.cmake:69 (message):
  Error: Could not find app bundle
  "/Volumes/STORE/Tools/Neurophys/labstreaminglayer/Apps/AudioCapture/build/_CPack_Packages/Darwin/TBZ2/lsl-0.1-Darwin/Unspecified/AudioCapture/AudioCapture.app"

I'm not very good with CPack but I'll try and track this down. If you have any ideas then let me know.

[cboulay]

The culprit is this line: set(CPACK_ARCHIVE_COMPONENT_INSTALL ON). If I comment that out then I no longer get the Unspecified target and macdeployqt correctly bundles the .app bundle.

However, the archive it creates is called lsl-0.1-Darwin.tar.bz2.
If I also comment out the line immediately below (set(CPACK_PACKAGE_NAME lsl)) and for good measure the line above (set(CPACK_PACKAGE_INSTALL_DIRECTORY "lsl")) then it gives a more sensible archive name.

So what are those lines intended to do? They seem like they are there for liblsl, not for the applications.

[tstenner]

The COMPONENT argument for install() etc. tells CMake which component to put the file into, and if it's missing it' s put into the "Unspecified" component. Whenever I get an unspecified component I just unpack it to see which install call needs the parameter.

The bzip file is empty.

Then there's an install part that doesn't produce any output. I'll keep it in mind for the time the department Mac is back.

The culprit is this line: set(CPACK_ARCHIVE_COMPONENT_INSTALL ON). If I comment that out then I no longer get the Unspecified target and macdeployqt correctly bundles the .app bundle.

The first part is unsurprising; without the component install, everything is put into one package so the component information isn't used for anything. The second part is unexpected, but a clue. Does adding COMPONENT ${PROJECT_NAME} to the install(CODE change anything?

However, the archive it creates is called lsl-0.1-Darwin.tar.bz2.
If I also comment out the line immediately below (set(CPACK_PACKAGE_NAME lsl)) and for good measure the line above (set(CPACK_PACKAGE_INSTALL_DIRECTORY "lsl")) then it gives a more sensible archive name.

I'm currently in the home office so I can't test it right now but IIRC the CPACK_PACKAGE_NAME was needed for something on Windows. I'll check it once I get back there next week.

[cboulay]

I just encountered an admittedly very esoteric problem. libqt5gamepad5-dev is not available in Ubuntu 18.04 bionic. I can only build the Gamepad app if I link to a custom qt5 install (either from qt maintenance tool with -DQt5_DIR=/opt/Qt/5.15.0/gcc_64/lib/cmake/Qt5, or found automatically in a conda env). If I do so, then after I successfully build the executable, readelf -d GamepadLSL shows me the RUNPATH includes this Qt5 dir, and ldd GamepadLSL correctly finds the libraries: libQt5Gamepad.so.5 => /opt/Qt/5.15.0/gcc_64/lib/libQt5Gamepad.so.5. Great, for my system only.

Note, my cmake command also had -DCPACK_DEBIAN_PACKAGE_SHLIBDEPS=ON -DCPACK_DEBIAN_PACKAGE_DEPENDS=1.

So, when I finally call cmake to build for target package, I get:

CPackDeb: dpkg-shlibdeps: 'dpkg-shlibdeps: error: cannot find library
  libQt5Gamepad.so.5 needed by ./usr/GamepadLSL/GamepadLSL (ELF format:
  'elf64-x86-64' abi: '0201003e00000000'; RPATH:
  '/usr/GamepadLSL:/usr/GamepadLSL/../LSL/lib/')

The CPACK_DEBIAN_PACKAGE_SHLIBDEPS docs suggest that it can find local libs if they are on the RPATH.

---

One solution to my problem is to upgrade my system to Ubuntu 20.04 and try to get github actions to run on ubuntu focal. Edit: indeed this works.

But another potential solution that is more general might be to delay changing the RPATH until after shlibdeps is run. Is that possible?

[tstenner]

shlibdeps and RPATH is giving me flashbacks, so my answer is probably "let's avoid Qt5Gamepad". I have a prototype for an evdev backend with OS timestamps somewhere, so you could skip Linux and add a todo for me.

[cboulay]

The Ubuntu 20.04 build works. It's not like there are many Linux users clamoring for a Gamepad app and can only use Ubuntu 18.04. Even then, they could probably get it to work on 18.04 with a custom Qt installation. So I'll leave the Gamepad app as is.

But this problem will pop up again anytime someone makes an app that links a library that is neither on their system PATH or the modified RPATH (./ & ../LSL/lib). A custom Qt version is just one example, but I guess some other SDKs might be similar.