chore(tracking): Test demo upgrades on nightly versions for 25.7.0

[razvan]

Part of stackabletech/issues#722

This is testing that upgrades from the stable release to the nightly release of the operators
and products do not negatively impact the products.

Note

Record the results (issues, anomalies, or success) during the process in a comment on this issue.
Eg:

:green_circle: **airflow-scheduled-job**

The CRD had been updated and I needed to change the following in the manifest:
...

The following emojis can be used to indicate the status:

• hourglass: In progress / waiting
• red_circle: In progress with errors
• orange_circle: Completed with minor issues or anomalies
• green_circle: Completed without issues or errors

List of demos

Replace the items in the lists below with the applicable Pull Requests (if any). Also put your
GitHub handle next to the demo to indicate a particular demo was picked up for testing.

• airflow-scheduled-job @adwk67
• data-lakehouse-iceberg-trino-spark We've decided to leave this until after the release
• end-to-end-security @maltesander
• hbase-hdfs-load-cycling-data @adwk67
• jupyterhub-keycloak @adwk67
• jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data @razvan
• logging @Techassi
• nifi-kafka-druid-earthquake-data @adwk67
• nifi-kafka-druid-water-level-data @adwk67
• signal-processing @Techassi
• spark-k8s-anomaly-detection-taxi-data @razvan
• trino-iceberg @dervoeti
• docs(demo/trino-taxi-data): Update stacklet list output #252 @NickLarsenNZ

Stable to Nightly Upgrade Testing Instructions

These instructions are for deploying and completing the stable demo, and then
upgrading operators, CRDs, and products to the nightly versions well as upgrading
the operators and CRDS.

Tip

Be sure to select the stable docs version on https://docs.stackable.tech/home/stable/demos/.

# Install demo (stable operators) for the stable release (OO.M).
stackablectl demo install <DEMO_NAME>

# --- IMPORTANT ---
# Run through the stable demo instructions (refer to the list above).

# Get a list of installed operators
stackablectl operator installed --output=plain

# Uninstall operators for the stable release (OO.M)
stackablectl release uninstall OO.M

# Update CRDs to nightly version (on main)
# Repeat this for every operator used by the demo (use the list from the earlier step before deleting the operators)
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/...-operator/main/deploy/helm/...-operator/crds/crds.yaml

# Install nightly version of operators (use the list from the earlier step before deleting the operators)
stackablectl operator install commons ...

# Optionally update the product versions in the CRDs (to the latest non-experimental version for the new release), e.g.:
kubectl patch hbaseclusters/hbase --type='json' -p='[{"op": "replace", "path": "/spec/image/productVersion", "value":"x.x.x"}]' # changed

[NickLarsenNZ]

🟢 trino-taxi-data

Note

It is best to run this demo on Replicated. For whatever reason it fails for me locally (and not only Trino).

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly
  • Raised a fix: fix(stack/trino-superset-s3: Update listenerClass in CRs #246
  • Raised doc updates: docs(demo/trino-taxi-data): Update stacklet list output #252
  • Had superset-> trino problems (fixed)

Note

We should make the create-ny-taxi-data-table-in-trino Job wait on conditions (it will need a ServiceAccount with permissions like in other demos). That way it won't CrashLoop (requests.exceptions.ConnectionError: HTTPSConnectionPool(host='trino-coordinator', port=8443): Max retries exceeded with url: /v1/statement) before services are ready.

Same with setup-superset (urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='superset-external', port=8088): Max retries exceeded with url: /api/v1/security/login).

And load-ny-taxi-data (Unable to initialize new alias from the provided credentials. Get "https://minio.default.svc.cluster.local:9000/probe-bsign-koq59aigrbffjr9znvsxwrevvsdb6b/?location=": dial tcp: lookup minio.default.svc.cluster.local: i/o timeout.).

[adwk67]

🟢 airflow-scheduler-job

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly (pending external dep bumps)

[adwk67]

🟢 hbase-hdfs-load-cycling-data

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly (pending external dep bumps). Stacklet list not returning all endpoints after upgrade, but OK after a clean install of dev.

In the stable version, stackablectl stacklet list only picks up the region servers:

│ hbase     ┆ hbase     ┆ default   ┆ regionserver                         172.19.0.4:31034        ┆...│
│           ┆           ┆           ┆ ui-http                              172.19.0.4:32391        ┆...│

A clean install of the dev version looks correct:

│ hbase     ┆ hbase     ┆ default   ┆ master-default-0-listener-master              172.19.0.4:32416                                ┆ Unavailable: See [1], Reconciling, Running │
│           ┆           ┆           ┆ master-default-0-listener-ui-http             172.19.0.4:30573                                ┆                                            │
│           ┆           ┆           ┆ regionserver-default-0-listener-regionserver  172.19.0.3:30577                                ┆                                            │
│           ┆           ┆           ┆ regionserver-default-0-listener-ui-http       172.19.0.3:32331                                ┆                                            │
│           ┆           ┆           ┆ regionserver-default-1-listener-regionserver  172.19.0.4:32741                                ┆                                            │
│           ┆           ┆           ┆ regionserver-default-1-listener-ui-http       172.19.0.4:32593                                ┆                                            │
│           ┆           ┆           ┆ restserver-default-0-rest-http                172.19.0.4:30733                                ┆                                            │
│           ┆           ┆           ┆ restserver-default-0-ui-http                  172.19.0.4:32351   

[razvan]

⏳ jupyterhub-pyspark-hdfs-anomaly-detection-taxi-data

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly (pending external dep bumps)

NOTE: Patching CRDs fails for secret and spark ops. Need to apply first.

[adwk67]

🟢 jupyterhub-keycloak

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly (pending external dep bumps)

[maltesander]

⏳ end-to-end-security

• 🟢 current stable (25.3.0)
• ⛔ upgrade to nightly

I had to manually restart all Statefulsets after using the 0.0.0-dev operators to get the changes done.
It started working again but access now is ClusterIP everywhere due to listener changes (works with portforwarding).

What i did:

stackablectl demo in end-to-end-security --release 25.3
# wait demo and check

stackablectl release uninstall 25.3

# Needed the creates in there for new resources (throws errors but everything gets applied)
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/main/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/main/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/main/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/main/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/main/deploy/helm/secret-operator/crds/crds.yaml
kubectl create -f https://raw.githubusercontent.com/stackabletech/secret-operator/main/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/main/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/main/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl create -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/main/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/main/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/main/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/main/deploy/helm/zookeeper-operator/crds/crds.yaml

stackablectl op in commons=0.0.0-dev secret=0.0.0-dev listener=0.0.0-dev opa=0.0.0-dev zookeeper=0.0.0-dev hdfs=0.0.0-dev hive=0.0.0-dev trino=0.0.0-dev superset=0.0.0-dev spark-k8s=0.0.0-dev

-> Restart zk, hdfs, hive, trino, superset StatefulSets

PR:
#245

[adwk67]

🟢 nifi-kafka-druid-earthquake-data

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly

The docs need to be corrected in main

• the Nifi login screenshot seems to be in the wrong place
• the earthquake scatterplot is missing the map as a backdrop (to do with our mapbox key, maybe?)

PR:

• fix: Changes made to earthquake demo and underlying stack #244

[razvan]

⏳ spark-k8s-anomaly-detection-taxi-data

• 🟢 current stable (25.3.0)
• 🟠 upgrade to nightly

Trouble

Problem: Trino, Superset and Hive op logs after upgrade:

2025-07-15T08:39:04.833802Z ERROR stackable_operator::logging::controller: Failed to reconcile object controller.name="trinocluster.trino.stackable.tech" error=reconciler for object TrinoC
luster.v1alpha1.trino.stackable.tech/trino.default failed error.sources=[failed to apply StatefulSet for role group worker/default of TrinoCluster.v1alpha1.trino.stackable.tech/trino.defau
lt, failed to apply patch, unable to patch resource "trino-worker-default", ApiError: StatefulSet.apps "trino-worker-default" is invalid: spec: Forbidden: updates to statefulset spec for f
ields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden: Invalid (ErrorRes
ponse { status: "Failure", message: "StatefulSet.apps \"trino-worker-default\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'templ
ate', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden", reason: "Invalid", code: 422 }), StatefulSet.apps "trino-worker
-default" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClai
mRetentionPolicy' and 'minReadySeconds' are forbidden: Invalid]

Solution: Deleting the STSs and allowing the ops to recreate them fixed it.

Problem: Cannot connect to Trino after upgrade:

Solution: The solution to the above is that users need to edit the existing Trino DB connection and add http_scheme: "https" to the connection arguments

PRs

• docs: spark anomaly detection, remove dead links and update ports #239
• docs: spark anomaly detection, remove dead links and update ports (#239) #240 (backport above to 25.3)
• fix(spark-anomaly-detection): Adjust Trino HTTPS connection #251
• fix: spark anomaly typo and docs #253

[Techassi]

🟢 logging

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly

During the upgrade, the secret-operator CRD needed to be applied (with --force) instead of replacing it. This is due to the new TrustStore custom resource (to be added in 25.7.0/current nightly). This might be worth pointing out in the release notes.

Everything else works without any issues.

[Techassi]

🟢 signal-processing

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly

No issues. The same remark from above applies here as well.

[dervoeti]

🟢 trino-iceberg

• 🟢 current stable (25.3.0)
• 🟢 upgrade to nightly

Upgrade requires customer to:

• Delete the Trino StatefulSets
• Adjust listenerClass
• Apply (instead of replace) the TrustStore CRD