diff --git a/demos/nifi-kafka-druid-water-level-data/create-nifi-ingestion-job.yaml b/demos/nifi-kafka-druid-water-level-data/create-nifi-ingestion-job.yaml index 51d20195..2ef923bb 100644 --- a/demos/nifi-kafka-druid-water-level-data/create-nifi-ingestion-job.yaml +++ b/demos/nifi-kafka-druid-water-level-data/create-nifi-ingestion-job.yaml @@ -63,7 +63,7 @@ data: import urllib3 # As of 2022-08-29 we cant use "https://nifi:8443" here because

The request contained an invalid host header [nifi:8443] in the request [/nifi-api]. Check for request manipulation or third-party intercept.

- ENDPOINT = f"https://nifi-node-default-0.nifi-node-default.{os.environ['NAMESPACE']}.svc.cluster.local:8443" # For local testing / developing replace it, afterwards change back to f"https://nifi-node-default-0.nifi-node-default.{os.environ['NAMESPACE']}.svc.cluster.local:8443" + ENDPOINT = f"https://nifi-node-default-0.nifi-node-default-headless.{os.environ['NAMESPACE']}.svc.cluster.local:8443" # For local testing / developing replace it, afterwards change back to f"https://nifi-node-default-0.nifi-node-default-headless.{os.environ['NAMESPACE']}.svc.cluster.local:8443" USERNAME = "admin" PASSWORD = open("/nifi-admin-credentials-secret/admin").read() diff --git a/demos/nifi-kafka-druid-water-level-data/setup-superset.yaml b/demos/nifi-kafka-druid-water-level-data/setup-superset.yaml index 6e8750ef..98a1247a 100644 --- a/demos/nifi-kafka-druid-water-level-data/setup-superset.yaml +++ b/demos/nifi-kafka-druid-water-level-data/setup-superset.yaml @@ -55,7 +55,7 @@ data: import logging import requests - base_url = "http://superset-node-default:8088" # For local testing / developing replace it, afterwards change back to http://superset-node-default:8088 + base_url = "http://superset-node:8088" # For local testing / developing replace it, afterwards change back to http://superset-node:8088 superset_username = open("/superset-credentials/adminUser.username").read() superset_password = open("/superset-credentials/adminUser.password").read() diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_1.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_1.png index 6c752eb4..2a74f977 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_1.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_1.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_2.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_2.png index 41e658f0..1a4a4047 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_2.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_2.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_3.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_3.png index bd47ae9d..3776efea 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_3.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_3.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_4.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_4.png index 0d25057f..591ad44e 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_4.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_4.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_5.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_5.png index a291f317..e5c6bc48 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_5.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_5.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_6.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_6.png index b2febe66..4e6dd50c 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_6.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_6.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_7.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_7.png index ea7362dd..3b177914 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_7.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_7.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_8.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_8.png index 5aae8cef..db523493 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_8.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/druid_8.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/minio_1.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/minio_1.png index b8dcc3d6..673f9a34 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/minio_1.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/minio_1.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_1.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_1.png index 8922aa4e..539c3de0 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_1.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_1.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_10.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_10.png index 7cb69d7a..0a0158cb 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_10.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_10.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_11.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_11.png index 10d5ea7c..ab93454e 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_11.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_11.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_12.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_12.png index af3e0090..77fc8110 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_12.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_12.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_5.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_5.png index 85457447..45590902 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_5.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_5.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_6.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_6.png index 2b35368a..9d7da70e 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_6.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_6.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_7.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_7.png index b06b25ae..44b78c5d 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_7.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_7.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_8.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_8.png index 97961e95..6e7a27e3 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_8.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_8.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_9.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_9.png index 042a90b1..6577cbfb 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_9.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/nifi_9.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_1.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_1.png index cf199323..160c4ea6 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_1.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_1.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_10.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_10.png index 3b836178..7b5fa322 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_10.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_10.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_11.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_11.png index b9ca9067..2331486c 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_11.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_11.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_12.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_12.png index 8d87c09e..2f9478d2 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_12.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_12.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_13.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_13.png index 85d776a4..9341cff2 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_13.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_13.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_2.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_2.png index 7aa4fe89..3c530847 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_2.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_2.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_3.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_3.png index 28c35fac..4099d4ee 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_3.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_3.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4a.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4a.png new file mode 100644 index 00000000..b553f349 Binary files /dev/null and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4a.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4b.png similarity index 100% rename from docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4.png rename to docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_4b.png diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_5.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_5.png index 80c4f4c9..a750dd16 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_5.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_5.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_6.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_6.png index e72eecff..71f06099 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_6.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_6.png differ diff --git a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_9.png b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_9.png index 60c1c547..ee4f8dcc 100644 Binary files a/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_9.png and b/docs/modules/demos/images/nifi-kafka-druid-water-level-data/superset_9.png differ diff --git a/docs/modules/demos/pages/nifi-kafka-druid-water-level-data.adoc b/docs/modules/demos/pages/nifi-kafka-druid-water-level-data.adoc index b7267171..c308761b 100644 --- a/docs/modules/demos/pages/nifi-kafka-druid-water-level-data.adoc +++ b/docs/modules/demos/pages/nifi-kafka-druid-water-level-data.adoc @@ -67,33 +67,27 @@ To list the installed Stackable services run the following command: ---- $ stackablectl stacklet list -┌───────────┬───────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────┐ -│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │ -╞═══════════╪═══════════════╪═══════════╪═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪═════════════════════════════════╡ -│ druid ┆ druid ┆ default ┆ broker-metrics 172.18.0.3:30800 ┆ Available, Reconciling, Running │ -│ ┆ ┆ ┆ broker-https https://172.18.0.3:32366 ┆ │ -│ ┆ ┆ ┆ coordinator-metrics 172.18.0.3:32387 ┆ │ -│ ┆ ┆ ┆ coordinator-https https://172.18.0.3:31091 ┆ │ -│ ┆ ┆ ┆ historical-metrics 172.18.0.3:30596 ┆ │ -│ ┆ ┆ ┆ historical-https https://172.18.0.3:30450 ┆ │ -│ ┆ ┆ ┆ middlemanager-metrics 172.18.0.3:30175 ┆ │ -│ ┆ ┆ ┆ middlemanager-https https://172.18.0.3:32286 ┆ │ -│ ┆ ┆ ┆ router-metrics 172.18.0.3:31588 ┆ │ -│ ┆ ┆ ┆ router-https https://172.18.0.3:31713 ┆ │ -├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ -│ kafka ┆ kafka ┆ default ┆ broker-default-0-listener-broker-kafka-tls kafka-broker-default-0-listener-broker.default.svc.cluster.local:9093 ┆ Available, Reconciling, Running │ -│ ┆ ┆ ┆ broker-default-0-listener-broker-metrics kafka-broker-default-0-listener-broker.default.svc.cluster.local:9606 ┆ │ -│ ┆ ┆ ┆ broker-default-bootstrap-kafka-tls kafka-broker-default-bootstrap.default.svc.cluster.local:9093 ┆ │ -│ ┆ ┆ ┆ broker-default-bootstrap-metrics kafka-broker-default-bootstrap.default.svc.cluster.local:9606 ┆ │ -├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ -│ nifi ┆ nifi ┆ default ┆ https https://172.18.0.3:30560 ┆ Available, Reconciling, Running │ -├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ -│ superset ┆ superset ┆ default ┆ external-http http://172.18.0.3:31366 ┆ Available, Reconciling, Running │ -├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ -│ zookeeper ┆ zookeeper ┆ default ┆ ┆ Available, Reconciling, Running │ -├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ -│ minio ┆ minio-console ┆ default ┆ http http://172.18.0.3:32376 ┆ │ -└───────────┴───────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────┘ +┌───────────┬───────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────┐ +│ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │ +╞═══════════╪═══════════════╪═══════════╪═════════════════════════════════════════════════════════════════════════════════════════════╪═════════════════════════════════╡ +│ druid ┆ druid ┆ default ┆ broker-https https://172.19.0.3:32165 ┆ Available, Reconciling, Running │ +│ ┆ ┆ ┆ coordinator-https https://172.19.0.6:31287 ┆ │ +│ ┆ ┆ ┆ router-https https://172.19.0.3:31965 ┆ │ +├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ +│ kafka ┆ kafka ┆ default ┆ broker-default-0-listener-broker-kafka-tls 172.19.0.3:31041 ┆ Available, Reconciling, Running │ +│ ┆ ┆ ┆ broker-default-0-listener-broker-metrics 172.19.0.3:31503 ┆ │ +│ ┆ ┆ ┆ broker-default-bootstrap-kafka-tls 172.19.0.3:30793 ┆ │ +│ ┆ ┆ ┆ broker-default-bootstrap-metrics 172.19.0.3:32540 ┆ │ +├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ +│ nifi ┆ nifi ┆ default ┆ node-https https://172.19.0.6:32038 ┆ Available, Reconciling, Running │ +├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ +│ superset ┆ superset ┆ default ┆ node-http http://172.19.0.3:30435 ┆ Available, Reconciling, Running │ +├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ +│ zookeeper ┆ zookeeper ┆ default ┆ server-zk zookeeper-server.default.svc.cluster.local:2282 ┆ Available, Reconciling, Running │ +├╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ +│ minio ┆ minio-console ┆ default ┆ http http://172.19.0.5:30196 ┆ │ +└───────────┴───────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────┘ + ---- include::partial$instance-hint.adoc[] @@ -123,7 +117,7 @@ You can execute a command on the Kafka broker to list the available topics as fo $ kubectl exec -it kafka-broker-default-0 -c kcat-prober -- /bin/bash -c "/stackable/kcat -b localhost:9093 -X security.protocol=SSL -X ssl.key.location=/stackable/tls-kcat/tls.key -X ssl.certificate.location=/stackable/tls-kcat/tls.crt -X ssl.ca.location=/stackable/tls-kcat/ca.crt -L" Metadata for all topics (from broker -1: ssl://localhost:9093/bootstrap): 1 brokers: - broker 1001 at 172.18.0.2:31146 (controller) + broker 1001 at 172.19.0.3:31041 (controller) 2 topics: topic "stations" with 8 partitions: partition 0, leader 1001, replicas: 1001, isrs: 1001 @@ -283,7 +277,7 @@ keeps streaming near-real-time updates for every available measuring station. === View the testdata-generation Job You can look at the ingestion job running in NiFi by opening the endpoint `https` from your `stackablectl stacklet list` -command output. You have to use the endpoint from your command output. In this case, it is https://172.18.0.2:30198. +command output. You have to use the endpoint from your command output. In this case, it is https://172.19.0.6:32038. Open it with your favourite browser. Suppose you get a warning regarding the self-signed certificate generated by the xref:home:secret-operator:index.adoc[Secret Operator] (e.g. Warning: Potential Security Risk Ahead). In that case, you must tell your browser to trust the website and continue. @@ -361,8 +355,8 @@ You can also see the number of produced records by right-clicking on `PublishKaf image::nifi-kafka-druid-water-level-data/nifi_11.png[] -After selecting `Messages Send (5 mins)` in the top right corner, you can see that ~10 million records were produced in -~5 minutes, corresponding to ~30k measurements/s. Keep in mind that the demo uses a single-node NiFi setup. The +After selecting `Messages Sent (5 mins)` in the top right corner, you can see that ~670k records were produced in +~5 minutes, corresponding to ~22k measurements/s. Keep in mind that the demo uses a single-node NiFi setup. The performance can be increased by using multiple nodes. Regarding the NiFi resources, use the hamburger menu icon on the top right corner and select `Node Status History`. @@ -381,7 +375,7 @@ deep storage. The Druid deep storage is based on the S3 store provided by MinIO. === View the Ingestion job You can have a look at the ingestion jobs running in Druid by opening the Druid endpoint `router-https` from your -`stackablectl stacklet list` command output (https://172.18.0.2:31616 in this case). +`stackablectl stacklet list` command output (https://172.19.0.3:31965 in this case). image::nifi-kafka-druid-water-level-data/druid_1.png[] @@ -395,9 +389,8 @@ number of processed records as well as the number of errors. image::nifi-kafka-druid-water-level-data/druid_3.png[] -The statistics show that Druid ingested `594` records during the last minute and has already ingested ~1,3 million records in total. All records -have been ingested successfully, indicated by having no `processWithError`, `thrownAway` or `unparseable` records in the output of the `View raw` -button at the top right. +The statistics show how many records that Druid ingested during the last minute and the total already ingested. +All records have been ingested successfully, indicated by having no `processWithError`, `thrownAway` or `unparseable` records in the output of the `View raw` button at the top right. === Query the Data Source @@ -406,12 +399,12 @@ the available data sources by clicking on `Datasources` at the top. image::nifi-kafka-druid-water-level-data/druid_4.png[] -The `Avg. row size (bytes)` column shows that a typical `measurement` record has `3` bytes, while a station record has +The `Avg. row size (bytes)` column shows that a typical `measurement` record has `4` bytes, while a station record has `222`, more than 50 times the size. So, by choosing two dedicated topics over a single topic, this demo saved 50x of storage and computation costs. -By clicking on `segments` under `Availability` for the `measurements` data source, you can see the segments of the data source. In this case, the -`measurements` data source is partitioned by the measurement day, resulting in 33 segments. +By clicking on `segments` under `Availability` for the `measurements` data source, you can see the segments of the data source. +In this case, the `measurements` data source is partitioned by the measurement day, resulting in 32 segments. image::nifi-kafka-druid-water-level-data/druid_5.png[] @@ -446,7 +439,7 @@ image::nifi-kafka-druid-water-level-data/druid_8.png[] == Superset Superset provides the ability to execute SQL queries and build dashboards. Open the Superset endpoint -`external-http` in your browser (http://172.18.0.4:32251 in this case). +`node-http` in your browser (http://172.19.0.3:30435 in this case). image::nifi-kafka-druid-water-level-data/superset_1.png[] @@ -463,7 +456,11 @@ image::nifi-kafka-druid-water-level-data/superset_3.png[] Click on the dashboard called `Water level data`. It might take some time until the dashboard renders all the included charts. -image::nifi-kafka-druid-water-level-data/superset_4.png[] +image::nifi-kafka-druid-water-level-data/superset_4a.png[] + +NOTE: The charts on the right (`Current water level deviation` and `Stations distribution`) are rendered without a background map, as this is dependent upon a mapbox API key, which cannot be hosted in a public repository. The figure below shows how this would look if the user has their own key: + +image::nifi-kafka-druid-water-level-data/superset_4b.png[] === View the charts @@ -476,21 +473,19 @@ effect. image::nifi-kafka-druid-water-level-data/superset_6.png[] -You can see that starting from `2024/06/16` some stations didn't measure or transmit their data. They started sending -measurements again at `2024/06/17`. - -=== View the Station Distribution on the World Map +// Comment out the next section as long as the mapbox api key is not active. +// === View the Station Distribution on the World Map -To look at the stations' geographical distribution, you have to click on the tab `Charts` at the top again. Afterwards, -click on the chart `Stations distribution`. +// To look at the stations' geographical distribution, you have to click on the tab `Charts` at the top again. Afterwards, +// click on the chart `Stations distribution`. -image::nifi-kafka-druid-water-level-data/superset_7.png[] +// image::nifi-kafka-druid-water-level-data/superset_7.png[] -The stations are, of course, placed alongside waterways. They are coloured by the waters they measure, so all stations -alongside a body of water have the same colour. You can move and zoom the map with your mouse to interactively explore -the map. You can, e.g. have a detailed look at the water https://en.wikipedia.org/wiki/Rhine[Rhein]. +// The stations are, of course, placed alongside waterways. They are coloured by the waters they measure, so all stations +// alongside a body of water have the same colour. You can move and zoom the map with your mouse to interactively explore +// the map. You can, e.g. have a detailed look at the water https://en.wikipedia.org/wiki/Rhine[Rhein]. -image::nifi-kafka-druid-water-level-data/superset_8.png[] +// image::nifi-kafka-druid-water-level-data/superset_8.png[] === Execute arbitrary SQL statements @@ -551,7 +546,7 @@ image::nifi-kafka-druid-water-level-data/superset_13.png[] == MinIO The S3 MinIO store provides persistent deep storage for Druid to store all the data used. Open the MinIO endpoint -`http` retrieved by `stackablectl stacklet list` in your browser (http://172.18.0.2:32345 in this case). +`http` retrieved by `stackablectl stacklet list` in your browser (http://172.19.0.5:30196 in this case). image::nifi-kafka-druid-water-level-data/minio_1.png[] @@ -577,10 +572,10 @@ If you open up a prefix for a specific day, you can see that Druid has placed a == Summary -The demo puts station records into the Kafka stream pipeline topic `station`. It also streamed ~30,000 measurements/s for -a total of ~11 million measurements into the topic `measurements`. Druid ingested the data near real-time into its data -source and enabled SQL access to it. Superset was used as a web-based frontend to execute SQL statements and build -dashboards. +The demo puts station records into the Kafka stream pipeline topic `station`. +Over time it will stream ~30,000 measurements/s for a total of ~11 million measurements into the topic `measurements`. +Druid ingested the data near real-time into its data source and enabled SQL access to it. +Superset was used as a web-based frontend to execute SQL statements and build dashboards. == Where to go from here