Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 21 additions & 93 deletions .github/workflows/ci.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Does it make sense to even keep this pipeline? You mentioned in the description that there is no plan to sync the fork with upstream anymore? If this is the case, I guess we could just remove this part / this workflow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We seem to need another bump (see #sig-moto) so I'd like to keep it for a while.

I'll make a separate PR in the future which cleans up this and other unused workflows from upstream.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, sounds good! As a nitpick on the side I would in this case also opt for renaming this workflow as well 😛

Original file line number Diff line number Diff line change
@@ -1,33 +1,24 @@
# LocalStack specific workflow to implement a fully-integrated continuous integration pipeline for our fork
# LocalStack specific workflow to implement a fully-integrated continuous integration pipeline for Moto-Ext
# - Rebase this fork based on the latest commit on `main` of upstream
# - Build a Python source and wheel distribution of moto-ext with deterministic versioning
# - Publish the distributions to PyPi
# - Tag the commit in this fork with the new version
# - Create a GitHub release for the new version

name: Sync / Release moto-ext
name: Sync moto-ext with upstream

on:
schedule:
- cron: 0 5 * * MON
workflow_dispatch:
inputs:
dry_run:
description: 'Dry Run?'
description: 'Dry run'
default: true
required: true
type: boolean

# limit concurrency to 1
# Limit concurrency to 1
concurrency:
group: ${{ github.workflow }}

jobs:
sync-build-release-moto-ext:
sync-moto-ext:
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/project/moto-ext/
permissions:
contents: write
id-token: write
Expand All @@ -50,35 +41,35 @@ jobs:
git config --global user.name 'LocalStack Bot'
git config --global user.email 'localstack-bot@users.noreply.github.com'
git remote set-url origin https://git:${{ secrets.PRO_ACCESS_TOKEN }}@github.com/${{ github.repository }}

# make sure to switch to the `localstack` branch (default / main branch of this fork)
git switch localstack
# add moto upstream as remote
git remote add upstream https://github.com/getmoto/moto.git
# rebase with latest changes
git pull

# Create a custom merge driver which prefers everything from upstream _BUT_ the name and the URL
mkdir -p $HOME/.local/bin
cat > $HOME/.local/bin/git-prefer-theirs-name-url << EOF
#!/bin/bash
set -e

base="\$1"
local="\$2"
remote="\$3"

echo "Executing custom merge driver for base \$base, local \$local, remote \$remote."

# Define keys to keep
KEYS=("name" "url")

# Read files into arrays
mapfile -t REMOTE_LINES < "\$remote"
mapfile -t LOCAL_LINES < "\$local"

echo "merging \$local + \$local + \$remote ..."

# Function to check if a line should be kept (matches any key)
keep_line() {
local line="\$1"
Expand All @@ -87,7 +78,7 @@ jobs:
done
return 1
}

# keep key-matched lines from local, others from remote
for i in "\${!LOCAL_LINES[@]}"; do
if keep_line "\${REMOTE_LINES[i]}"; then
Expand All @@ -96,22 +87,22 @@ jobs:
echo "\${LOCAL_LINES[i]}"
fi
done > "\$local"

exit 0
EOF

# make the script executable and add it to the PATH
chmod +x $HOME/.local/bin/git-prefer-theirs-name-url
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

# add the merge driver to the git config
cat >> .git/config << EOF

[merge "git-prefer-theirs-name-url"]
name = A driver which resolves merge conflicts on a setup.cfg such that it always takes the local name and url, and everything else from upstream
driver = git-prefer-theirs-name-url %O %A %B
EOF

# define to use the custom merge driver for the setup.cfg
cat > .gitattributes << EOF
setup.cfg merge=git-prefer-theirs-name-url
Expand All @@ -121,75 +112,12 @@ jobs:
run: |
git fetch upstream
git rebase -f upstream/master

- name: Determine new version
run: |
echo "Determining new version..."
cat > setuptools.cfg << EOF
[tool.setuptools_scm]
local_scheme = "no-local-version"
version_scheme = "post-release"
EOF
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install setuptools_scm
NEW_VERSION=$(python3 -m setuptools_scm -c setuptools.cfg)
NEW_VERSION="${NEW_VERSION//dev/post}"
echo "New version is: $NEW_VERSION"
echo "NEW_VERSION=$NEW_VERSION" >> $GITHUB_ENV

- name: Build Python distributions
# FYI: Checks in this script only work because the -e flag is enabled by default in GitHub actions
run: |
python3 -m pip install build

echo "Setting new version in setup.cfg":
# make sure setup.cfg is not dirty yet
git diff --exit-code setup.cfg
sed -i -E 's/^(version\s*=\s*)("?)[^"]+("?)/\1\2'"$NEW_VERSION"'\3/' setup.cfg
# make sure setup.cfg is dirty now
! git diff --exit-code setup.cfg

echo "Building new version and tagging commit..."
python3 -m build

- name: Tag successful build
run: |
git tag -a $NEW_VERSION -m $NEW_VERSION

- name: Clean up
run: |
git reset --hard
git clean -df

- name: Store built distributions
uses: actions/upload-artifact@v4
with:
name: moto-ext-dists
path: dist/*.*

# publish the package before pushing the tag (this might fail if the version already exists on PyPI)
- name: Publish package distributions to PyPI
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: pypa/gh-action-pypi-publish@release/v1
git log --oneline --graph --abbrev-commit --max-count 20

- name: Push
if: ${{ github.event.inputs.dry_run != 'true' }}
run: |
git push --force-with-lease
git push --atomic origin localstack $NEW_VERSION
git push --atomic origin localstack
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

# Add a retry to avoid issues where the GH CLI fails
# because it does not yet detect the pushed tag.
- name: Create Release
uses: nick-fields/retry@v3
if: ${{ github.event.inputs.dry_run != 'true' }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
max_attempts: 5
retry_wait_seconds: 120
timeout_minutes: 5
command: gh release create $NEW_VERSION --repo localstack/moto --notes "automatic rebase sync and release"
146 changes: 146 additions & 0 deletions .github/workflows/ci_release.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Might make sense to rename the workflows a bit. They don't have much to do with Continuous Integration anymore I guess 😅

Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# LocalStack specific workflow to implement a fully-integrated continuous integration pipeline for Moto-Ext
# - Guess next patch semantic version
# - Build source and wheel distributions
# - Publish the distributions to PyPi
# - Tag the commit with the new version
# - Create a GitHub release for the new version

name: Release moto-ext

on:
schedule:
- cron: 0 5 * * MON
workflow_dispatch:
inputs:
dry_run:
description: 'Dry run'
default: true
required: true
type: boolean

# Limit concurrency to 1
concurrency:
group: ${{ github.workflow }}

jobs:
analyse-commit-log:
runs-on: ubuntu-latest
outputs:
is-release-required: ${{ steps.check.outputs.new_commits_made }}
last-tag: ${{ steps.last_tag.outputs.tag }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: localstack
persist-credentials: false

- name: Get last tag
id: last_tag
uses: "WyriHaximus/github-action-get-previous-tag@v1"
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

- name: Check for new commits since the last tag
id: check
env:
LAST_TAG: ${{ steps.last_tag.outputs.tag }}
run: |
if [ $(git log --oneline $LAST_TAG..HEAD | wc -l) -eq "0" ]; then
echo "No commits since the last tag."
echo "new_commits_made=false" >> $GITHUB_OUTPUT
else
echo "new_commits_made=true" >> $GITHUB_OUTPUT
fi

build-release-moto-ext:
runs-on: ubuntu-latest
needs: analyse-commit-log
if: ${{ needs.analyse-commit-log.outputs.is-release-required == 'true' }}
environment:
name: pypi
url: https://pypi.org/project/moto-ext/
permissions:
contents: write
id-token: write
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: localstack
persist-credentials: false

- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.13'

- name: Guess next version
id: semver
uses: "WyriHaximus/github-action-next-semvers@v1"
with:
version: ${{ needs.analyse-commit-log.outputs.last-tag }}

- name: Ensure guessed version does not exist
env:
NEW_VERSION: ${{ steps.semver.outputs.patch }}
run: |
! git rev-parse $NEW_VERSION || { echo "A tag for $NEW_VERSION already exists" ; exit 1; }

- name: Build Python distributions
env:
NEW_VERSION: ${{ steps.semver.outputs.patch }}
# FYI: Checks in this script only work because the -e flag is enabled by default in GitHub actions
run: |
python3 -m pip install build

# make sure setup.cfg is not dirty yet
git diff --exit-code setup.cfg

echo "Setting new version in setup.cfg":
sed -i -E 's/^(version\s*=\s*)("?)[^"]+("?)/\1\2'"$NEW_VERSION"'\3/' setup.cfg

# make sure setup.cfg is dirty now
! git diff --exit-code setup.cfg

echo "Building distributions"
python3 -m build

- name: Create and push tag
env:
NEW_VERSION: ${{ steps.semver.outputs.patch }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
if: ${{ github.event.inputs.dry_run != 'true' }}
run: |
git tag -a $NEW_VERSION -m $NEW_VERSION
git push --tags origin localstack

- name: Clean up
run: |
git reset --hard
git clean -df

- name: Upload distributions
uses: actions/upload-artifact@v4
with:
name: moto-ext-dists
path: dist/*.*

# publish the package before pushing the tag (this might fail if the version already exists on PyPI)
- name: Publish package distributions to PyPI
if: ${{ github.event.inputs.dry_run != 'true' }}
uses: pypa/gh-action-pypi-publish@release/v1

# Add a retry to avoid issues where the GH CLI fails because it does not yet detect the pushed tag.
- name: Create Release
uses: nick-fields/retry@v3
if: ${{ github.event.inputs.dry_run != 'true' }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
max_attempts: 5
retry_wait_seconds: 120
timeout_minutes: 5
command: gh release create $NEW_VERSION --repo localstack/moto --notes "automatic release"
Loading