Test without building and SPM

Another day, another set of testing issues. As mentioned in my previous post, Slathering Xcode Variants, I've been making some use of Xcode's capability to build a test package and separately run that test package on a different machine, possibly with a different version of macOS or even a different CPU architecture.

In order to meet my testing needs, I've generally been building with the latest Xcode on the latest version of macOS and then running tests on both that version and the previous version of macOS. I've been running in this mode for months, without any difficulty, until this week.

Earlier this week, I added a new SPM dependency to one of the Xcode projects in my workspace. Since this project is large, the organization is a single workspace with multiple projects in it (today that number is 16) and a handful of SPM dependencies (3 across the whole set today).

After adding the dependency and running full tests on my Mac Pro (x86_64, macOS Monterey), things were looking good, so I pushed the new files to my CI server. A few minutes later, my Big Sur tests both failed. Looking at the logs, I found:

Package.resolved file is corrupted or malformed; fix or delete the file to continue: unsupported schema version 2

This only happend on the Big Sur machines, and they are running Xcode 13.2.x (last version before 13.3 came along and stopped running on Big Sur).

Not surprisingly, this is because this file was created on my Monterey Mac.

Trying without Fastlane

I'll caution that these are all still running within Fastlane, so it's possible that I'm shooting myself in the foot by not pushing all the parameters to the command line manually. I may, at some point, give that a try and see if that solves the problem.

Doing some manual testing confirmed that when not using Fastlane, I could easily skip the problematic code when running in CI by targeting the xctestrun file directly instead of using the Scheme and Workspace (thus avoiding the workspace evaluation).

For example:

xcodebuild test-without-building ⏎
    -xctestrun test_build/Build/Products/Cartographica_Cartographica-Exhaustive_macosx12.3-arm64-x86_64.xctestrun ⏎
    -destination 'platform=macOS' ⏎
    -resultBundlePath 'output/Cartographica.xcresult'

(once again using ⏎ to denote, counter-intuitively, that the line breaks here are only for readability and should be left out).

This command explicitly uses the -xctestrun option, pointing it at the specific xctestrun file instead of using

Back to Fastlane

Eventually, I decided I still wanted to use Fastlane (although I feel like I'm using few enough features that there may be a future blog post about kicking it to the curb as well).

I'm having to be more careful about how I build and run my tests (notably: needing to make sure I don't bump the test libraries too far). However, things are working and the process survived at least the initial jump to Ventura.

My build and test matrix now looks like this:

build-mac:
  stage: build
  variables:
    GIT_CLONE_PATH: ${CI_BUILDS_DIR}/${CI_PROJECT_PATH}
  before_script: *mac_build_prep
  tags: [xcode14,codesigning,build]
  needs: [check-servers]
  interruptible: true
  script:
    - bundle exec fastlane --verbose cibuild
  artifacts:
    paths:
      - test_build/Build/Products
      - test_build/Build/SourcePackages
      - test_build/Logs
      - test_build/info.plist
    expire_in: 1 day

.test_template:
  stage: test
  needs:
    - job: build-mac
      artifacts: true
  variables:
    GIT_CLONE_PATH: ${CI_BUILDS_DIR}/${CI_PROJECT_PATH}
    CTRunningUnderTest: 'YES'
  coverage: '/^CodeCoverageOverall =(\d+\.?\d*)$/'
  interruptible: true
  before_script:
    - export PATH=~/.rbenv/shims:${PATH}
    - export FL_SLATHER_ARCH=`uname -m`
    - echo $PATH
    - bundle install
    - rm -rf ${CI_PROJECT_DIR}/DerivedData/Build/ProfileData
    - rm -rf ${CI_PROJECT_DIR}/output/*
  artifacts:
    when: always
    reports:
      junit: output/report.junit
      coverage_report:
        coverage_format: cobertura
        path: output/cobertura.xml
    paths:
        - output/scan/*.log
        - output/*.xcresult

.coverage_script: &coverage_script
  - >
     grep ^\<coverage output/cobertura.xml
     | sed -n -e 's/.*line-rate=\"\([0-9.]*\)\".*/\1/p'
     | awk '{print "CodeCoverageOverall =" $1*100}'
     || true

test:
  extends: .test_template
  parallel:
    matrix:
      - PROCESSOR: [arm64]
        OS: bigsur
      - PROCESSOR: [arm64,x86_64]
        OS: monterey
      - PROCESSOR: [arm64,x86_64]
        OS: ventura
  tags:
    - codesigning
    - ${OS}
    - ${PROCESSOR}
  script:
    - bundle exec fastlane --verbose citest
    - *coverage_script

Ed. Note: Updated for ventura

And my fastfile for running the tests (without building) uses a number of different commands:

test_from_build (to run the tests based on the testplan)
build_for_tests (to create the binaries for running the tests)

Neither of these are called directly, but are called from the cibuild and citest above

Note that build_for_tests takes an argument of an array of testplans to build.

default_platform(:mac)
my_xcargs = ''

platform :mac do
  desc 'Build for testing'
  lane :build_for_tests do |options|
    final_xcargs = [my_xcargs, 'ONLY_ACTIVE_ARCH=NO'].join(' ')
    if options[:testplan]
        final_xcargs = ([my_xcargs, 'ONLY_ACTIVE_ARCH=NO']+args_with_prefix(options[:testplan],'-testPlan ')).reject(&:empty?).join(' ')
    end

    run_tests(scheme: options[:scheme],
              configuration: 'Debug',
              code_coverage: true,
              address_sanitizer: false,
              output_types: "",
              disable_slide_to_type: false,     # note: this gets around a macos bug caused by assuming ios in fastlane
              clean: options[:clean] || false,
              xcargs: final_xcargs,
              derived_data_path: "test_build",
 			  build_for_testing: true)
  end

  desc 'Runs built tests'
  lane :test_from_build do |options|
    if options[:testplan]
        final_xcargs = ([my_xcargs]+args_with_prefix(options[:testplan],'-testPlan ')).reject(&:empty?).join(' ')
    else
        final_xcargs = my_xcargs
    end

    run_tests(scheme: options[:scheme],
              configuration: 'Debug',
              code_coverage: true,
              address_sanitizer: false,
              output_types: "junit",
              disable_slide_to_type: false,     # note: this gets around a macos bug caused by assuming ios in fastlane
              clean: options[:clean] || false,
              xcargs: final_xcargs,
              derived_data_path: "test_build",
              output_directory: 'output',
 			  test_without_building: true)
  end
end