ploosh.

Documentation

Dremio (Spark)

This connector is used to query Dremio using Spark via the Arrow Flight SQL JDBC driver.

⚠️ A Spark connector can only be used with another Spark connector. It is not possible to mix Spark and native connectors in the same test case.

See Spark mode overview for more information.

Connection configuration

NameMandatoryDefaultDescription
hostyesDremio server hostname or IP address
portno32010Arrow Flight SQL port
useencryptionnotrueEnable TLS encryption for the connection
disablecertificate_verificationnofalseDisable TLS certificate verification (not recommended in production)
usernameyesDremio username
passwordyesDremio password or PAT

Example

connections:
  dremio_connection:
    type: dremio_spark
    host: my-dremio-server.example.com
    port: 32010
    use_encryption: true
    disablecertificateverification: false
    username: my_user
    password: $var.dremio_password

Test case configuration

NameMandatoryDefaultDescription
queryyesSQL query to execute against Dremio

Example

Example Dremio Spark:
  source:
    type: dremio_spark
    connection: dremio_connection
    query: |
      SELECT *
      FROM my_schema.employees
      WHERE hire_date < '2000-01-01'
  expected:
    type: sql_spark
    query: |
      SELECT *
      FROM expected_employees
      WHERE hire_date < '2000-01-01'

Requirements

  • Apache Arrow Flight SQL JDBC driver (org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver)
  • Dremio instance accessible from the Spark cluster