ci: make preview releases #1302

wjones127 · 2024-05-13T18:36:50Z

This PR changes the release process. Some parts are more complex, and other parts I've simplified.

Simplifications

Combined Create Release Commit and Create Python Release Commit into a single workflow. By default, it does a release of all packages, but you can still choose to make just a Python or just Node/Rust release through the arguments. This will make it rarer that we create a Node release but forget about Python or vice-versa.
Releases are automatically generated once a tag is pushed. This eliminates the manual step of creating the release.
Release notes are automatically generated and changes are categorized based on the PR labels.
Removed the use of LANCEDB_RELEASE_TOKEN in favor of just using GITHUB_TOKEN where it wasn't necessary. In the one place it is necessary, I left a comment as to why it is.
Reused the version in python/Cargo.toml so we don't have two different versions in Python LanceDB.

New changes

We now can create preview / beta releases. By default Create Release Commit will create a preview release, but you can select a "stable" release type and it will create a full stable release.
- For Python, pre-releases go to fury.io instead of PyPI
bump2version was deprecated, so upgraded to bump-my-version. This also seems to better support semantic versioning with pre-releases.
ci changes will now be shown in the changelog, allowing changes like this to be visible to users. chore is still hidden.

Versioning

NOTE: unlike how it is in lance repo right now, the version in main is the last one released, including beta versions.

This reverts commit fdf8440.

This reverts commit ed28e80.

This reverts commit 1391ae8.

wjones127 · 2024-05-17T04:55:37Z

python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lancedb-python"
-version = "0.4.10"


This version was completely out of sync.

westonpace

Pretty epic! Nothing major jumps out at me

westonpace · 2024-05-17T12:26:04Z

.bumpversion.toml

+ignore_missing_version = false
+ignore_missing_files = false
+tag = true
+sign_tags = false


Not right now (plenty on PR already) but should we sign tags at some point?

Yeah I just wasn't sure how the signing worked if it's a github user. Something to look into.

westonpace · 2024-05-17T12:28:29Z

.bumpversion.toml

+filename = "rust/lancedb/Cargo.toml"
+search = "version = \"{current_version}\""
+replace = "version = \"{new_version}\""


If a dependency happens to have the same version as lancedb will this bump that dependency's version as well?

i.e. is this searching for <start_of_line>version = \"{current_version}\" or is it searching for version = ... anywhere in the line?

I don't think it requires the start of the line. Reading the docs ¹ it's a pretty simple replacement. I'll see if I can switch to a regex that requires a line start.

Footnotes

https://callowayproject.github.io/bump-my-version/howtos/avoid-incorrect-replacements/ ↩

westonpace · 2024-05-17T12:29:14Z

.github/python_release_notes.json

+            "title": "## 🏆 Highlights",
+            "labels": ["python", "highlight"],
+            "exhaustive": true


This is cool, I assume this label is applied manually?

Yeah we can add this label manually and it will be added to this section. Was in the template and seemed like a useful idea we can try out.

westonpace · 2024-05-17T12:32:49Z

.github/workflows/make-release-commit.yml

          fetch-depth: 0
          lfs: true
+          token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}


Can this not be GITHUB_TOKEN? (minor nit since this workflow relies on LANCEDB_RELEASE_TOKEN at the moment anyways).

It cannot. When testing if this wasn't set to a user token it the tags we pushed would not trigger a workflow. I'll add a comment to this.

westonpace · 2024-05-17T12:35:34Z

.github/workflows/pypi-publish.yml

+        id: python_release_notes
+        uses: mikepenz/release-changelog-builder-action@v4
+        with:
+          configuration: .github/python_release_notes.json


Why can't we just use .github/release_notes.json here?

Well the python release notes have special filters that will only show the ones that are labelled python. Though TBH I'm not 100% that's the right way to do it. Maybe I should just exclude anything that says Node but not Python? What do you think about that?

Ah, good point, that process is a little messy right now. If we can get in the habit I think we should start scoping language-specific PR titles with conventional commits. In other words:

feat: some feature that is implemented in all clients feat(node): node-specific feature feat(python): python-specific feature feat(rust): rust-specific feature

I'm thinking now we'll just make them share the same template for now, and have people manually edit this for now. Later, we can come up with some way to differentiate them.

westonpace · 2024-05-17T12:38:12Z

docs/src/basic.md

+=== "Rust"
+
+    ```shell
+    cargo add lancedb --version 0.0.1-beta.1


I thought we didn't publish pre-releases for rust? Wouldn't that mean we would need to use a github sha?

Ah this is a mistake. I will remove that.

westonpace · 2024-05-17T12:40:04Z

release_process.md

+features. However, we do not guarantee that preview releases will be available
+more than 6 months after they are released. Once your application is stable, we
+recommend switching to stable releases.


Hmm, just naively reading this I wonder if people might think we are implying that stable versions will be maintained (e.g. we will make security patches) for six months. I wonder if another blurb on release maintenance (or more accurately, lack thereof) is warranted.

Yeah I think part of this is the name "stable" release itself. I should use a different name.

release_process.md

Co-authored-by: Weston Pace <weston.pace@gmail.com>

ci: script to create preview releases

4034c14

github-actions bot added the ci label May 13, 2024

wjones127 added 13 commits May 13, 2024 11:39

cleanup permissions

84c446c

temp trigger to get it going

7a59e47

small fixes

ea3940e

try linux permissions

3a226b7

checkout main

74ee83e

correct package versions

dfda90f

try different release notes system

3d30a51

fix script name

00da4d2

fix path for non-python

0bc2cc3

clean up releases

a815a07

fix tag handling

f033c66

cleanup

49318c6

increment then publish

556a8e6

github-actions bot added the Python Python SDK label May 15, 2024

wjones127 added 13 commits May 15, 2024 12:44

select correct repo for Python

04921b8

fix breakpoint

2f6c529

fix script path

bb0025a

fix python version

da4acb6

match up versions

e20830f

Bump version: 0.6.13 → 0.6.14-beta.0

fdf8440

Revert "Bump version: 0.6.13 → 0.6.14-beta.0"

4b3a835

This reverts commit fdf8440.

handle breaking changes in both

740a99b

Bump version: 0.6.13 → 0.6.13-final.1

ed28e80

Revert "Bump version: 0.6.13 → 0.6.13-final.1"

bea1cda

This reverts commit ed28e80.

Bump version: 0.6.13 → 0.6.14-beta.0

1391ae8

Revert "Bump version: 0.6.13 → 0.6.14-beta.0"

d1981c2

This reverts commit 1391ae8.

release process

b31adf2

wjones127 force-pushed the preview-releases branch from af8331b to b31adf2 Compare May 15, 2024 20:59

wjones127 added 15 commits May 15, 2024 15:34

pass down token

9b8c21f

use token to checkout

00e3500

fix fury release

88df846

use known template

a1d7788

show ci

56f7af6

fix release notes

2d6dccc

fix version bumping

80ef105

test release notes

e333795

Try again

2be382d

remove test

0ef1f1e

actually output it

258281b

mention user

6171bc3

better search for previous tag

88fc709

add debug printing

9997e51

fix empty variables

af8ad2e

wjones127 force-pushed the preview-releases branch from 5ce8787 to af8ad2e Compare May 17, 2024 04:41

fix python extraction

9068ba4

wjones127 commented May 17, 2024

View reviewed changes

wjones127 marked this pull request as ready for review May 17, 2024 04:59

wjones127 requested review from westonpace and AyushExel May 17, 2024 04:59

westonpace approved these changes May 17, 2024

View reviewed changes

wjones127 and others added 7 commits May 17, 2024 08:43

Update release_process.md

00fc929

Co-authored-by: Weston Pace <weston.pace@gmail.com>

Bump version: 0.4.20 → 0.5.0-beta.0

0dfbbd9

revert version

72a31c5

more precise matching

af9130e

more pr feedback

6ee33cf

don't have separate template for Python release notes for now

aec975d

fix merge conflict

fe7a7ce

wjones127 merged commit 5349e8b into lancedb:main May 17, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: make preview releases #1302

ci: make preview releases #1302

wjones127 commented May 13, 2024 •

edited

wjones127 May 17, 2024

westonpace left a comment

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

westonpace May 17, 2024

wjones127 May 17, 2024

ci: make preview releases #1302

ci: make preview releases #1302

Conversation

wjones127 commented May 13, 2024 • edited

Simplifications

New changes

Versioning

Choose a reason for hiding this comment

westonpace left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Footnotes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjones127 commented May 13, 2024 •

edited