Skip to content

Commit 5def648

Browse files
authored
fix(demo/trino-taxi-data): Use new listener service (#254)
* fix(demo/trino-taxi-data): Use new listener service * fix(demo/trino-taxi-data): Use updated superset config - See: #251 * chore: Bump Trino worker by 2GB RAM * fix(demo/trino-taxi-data): Disable ANALYZE step - See: trinodb/trino#26214
1 parent d2103c6 commit 5def648

File tree

6 files changed

+24
-21
lines changed

6 files changed

+24
-21
lines changed

demos/trino-taxi-data/create-table-in-trino.yaml

Lines changed: 20 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -59,29 +59,32 @@ data:
5959
6060
run_query(connection, "CREATE SCHEMA IF NOT EXISTS hive.demo WITH (location = 's3a://demo/')")
6161
run_query(connection, """
62-
CREATE TABLE IF NOT EXISTS hive.demo.ny_taxi_data_raw (
63-
VendorID BIGINT,
64-
tpep_pickup_datetime TIMESTAMP,
65-
tpep_dropoff_datetime TIMESTAMP,
66-
passenger_count DOUBLE,
67-
trip_distance DOUBLE,
68-
payment_type BIGINT,
69-
Fare_amount DOUBLE,
70-
Tip_amount DOUBLE,
71-
Total_amount DOUBLE
72-
) WITH (
73-
external_location = 's3a://demo/ny-taxi-data/raw/',
74-
format = 'parquet'
75-
)
62+
CREATE TABLE IF NOT EXISTS hive.demo.ny_taxi_data_raw (
63+
VendorID BIGINT,
64+
tpep_pickup_datetime TIMESTAMP,
65+
tpep_dropoff_datetime TIMESTAMP,
66+
passenger_count DOUBLE,
67+
trip_distance DOUBLE,
68+
payment_type BIGINT,
69+
Fare_amount DOUBLE,
70+
Tip_amount DOUBLE,
71+
Total_amount DOUBLE
72+
) WITH (
73+
external_location = 's3a://demo/ny-taxi-data/raw/',
74+
format = 'parquet'
75+
)
7676
""")
7777
7878
loaded_rows = run_query(connection, "SELECT COUNT(*) FROM hive.demo.ny_taxi_data_raw")[0][0]
7979
print(f"Loaded {loaded_rows} rows")
8080
assert loaded_rows > 0
8181
82-
print("Analyzing table ny_taxi_data_raw")
83-
analyze_rows = run_query(connection, """ANALYZE hive.demo.ny_taxi_data_raw""")[0][0]
84-
assert analyze_rows == loaded_rows
82+
# Disabling ANALYZE for now, as it seems to fail when run twice (which can happen if Trino OOMs)
83+
# See issue: https://github.com/trinodb/trino/issues/26214
84+
# print("Analyzing table ny_taxi_data_raw")
85+
# analyze_rows = run_query(connection, """ANALYZE hive.demo.ny_taxi_data_raw""")[0][0]
86+
# assert analyze_rows == loaded_rows
87+
8588
stats = run_query(connection, """show stats for hive.demo.ny_taxi_data_raw""")
8689
print("Produced the following stats:")
8790
print(*stats, sep="\n")

demos/trino-taxi-data/setup-superset.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ data:
3939
import logging
4040
import requests
4141
42-
base_url = "http://superset-node-default:8088" # For local testing / developing replace it, afterwards change back to http://superset-node-default:8088
42+
base_url = "http://superset-node:8088" # You can change this while running locally, but remember not to commit the change.
4343
superset_username = open("/superset-credentials/adminUser.username").read()
4444
superset_password = open("/superset-credentials/adminUser.password").read()
4545
trino_username = "admin"
1.02 KB
Binary file not shown.

docs/modules/demos/pages/spark-k8s-anomaly-detection-taxi-data.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ This demo should not be run alongside other demos.
2323
To run this demo, your system needs at least:
2424

2525
* 10 {k8s-cpu}[cpu units] (core/hyperthread)
26-
* 32GiB memory
26+
* 34GiB memory
2727
* 35GiB disk storage
2828
2929
== Overview

docs/modules/demos/pages/trino-taxi-data.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ This demo should not be run alongside other demos.
2727
To run this demo, your system needs at least:
2828

2929
* 7 {k8s-cpu}[cpu units] (core/hyperthread)
30-
* 16GiB memory
30+
* 18GiB memory
3131
* 28GiB disk storage
3232
3333
== Overview

stacks/trino-superset-s3/trino.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ spec:
2929
min: "1"
3030
max: "4"
3131
memory:
32-
limit: '4Gi'
32+
limit: '6Gi'
3333
roleGroups:
3434
default:
3535
replicas: 1

0 commit comments

Comments
 (0)