🥋 Karate Connect

🇫🇷 votre partenaire de combat dans l’arène des tests API

🇬🇧 your battle partner in the API testing arena

🪄 Once upon a time…​ ✨

A great data pipeline !

data pipeline simple

$ whoami 👨‍💻

A great pipeline…​ but sometimes…​ 😥

Not so great…​ more complex

data pipeline dirty data pipeline complex

💡We need Tests 🤗 & QA 🕵️ 🐛

But also some tools…​

Let’s talk about architecture 👷

ETL vs ELT…​

kapoeira text

Kapoeira

fries factory
Feature: Fries 🍟 feature

  Background:
    Given input topic
      | topic  | alias     | key_type | value_type |
      | potato | potato-in | string   | string     |
    And output topic
      | topic       | alias           | key_type | value_type | readTimeoutInSecond |
      | side-dishes | side-dishes-out | string   | string     | 5                   |
    
  Scenario: Nominal
    When records with key and value are sent
      | topic_alias | key | value |
      | potato-in   | 😋  | 🥔    |
    Then expected records
      | topic_alias     | key | value  |
      | side-dishes-out | 😋  | result |
    And assert result $ == "🍟"

karatelabs

Karate - Example

Feature: karate 'hello world' example
  
  Scenario: create and retrieve a cat
    Given url 'http://myhost.com/v1/cats'
    And request { name: 'Billie' }
    When method post
    Then status 201
    And match response == { id: '#notnull', name: 'Billie' }
    
    Given path response.id
    When method get
    Then status 200

Karate calling Snowflake REST API

Feature: SELECT V1
  Background:
    * url "https://<SNOWFLAKE_ACCOUNT>.snowflakecomputing.com/api/v2"
    * string jwtToken = karate.exec("snow connection generate-jwt --silent --account <SNOWFLAKE_ACCOUNT> --user <SNOWFLAKE_USER> --private-key-file <SNOWFLAKE_USER_PEM>")
    * header Content-Type = "application/json"
    * header Accept = "application/json"
    * header Authorization = "Bearer " + jwtToken
    * header User-Agent = "<MY_APP_IT>"
    * header X-Snowflake-Authorization-Token-Type = "KEYPAIR_JWT"

  Scenario: Select 1 cutter
    Given path "statements"
    And text payload =
        """
        {
          "statement": "SELECT SERIAL_NUMBER, CUTTER_TYPE FROM CUTTER WHERE SERIAL_NUMBER='MY_VECTOR'",
          "timeout": 60,
          "role": "<SNOWFLAKE_ROLE>",
          "warehouse": "<SNOWFLAKE_WAREHOUSE>",
          "database": "<SNOWFLAKE_DATABASE>",
          "schema": "<SNOWFLAKE_SCHEMA>"
        }
        """
    When request payload
    And method post
    Then status 200
    And match response.resultSetMetaData.numRows == 1
    And match response.resultSetMetaData.rowType[0].name == "SERIAL_NUMBER"
    And match response.resultSetMetaData.rowType[1].name == "CUTTER_TYPE"
    And match response.data[0][0] == "MY_VECTOR"
    And match response.data[0][1] == "VECTOR"

Karate API Testing 👍 & 👎

Ok?

Comments

Snowflake

But complex API

Kafka

Closed source addon

Rabbitmq

Just code example for addon

DBT

Thanks to karate.exec(…​)

💡 Karate Connect =
Karate Core + Extensions ready to use

DSL syntax
* def result1 = <extension>.<value>
* def result2 = <extension>.<feature>.<function>(args)
Example
* json cliConfig     = snowflake.cliConfigFromEnv
* def rabbitmqClient = rabbitmq.topology.createClient({ host: "localhost", port: 5672 })

Packaging
packaging

How to run ?

java -Dextensions=<ext1,ext2...> \
    -jar karate-connect-standalone.jar \
    <features_path> <optional_params...>
🐳
docker run --rm \
  -v $(pwd)/<features_path>:/features \
  -v $(pwd)/<reports_path>:/target \
  -e KARATE_EXTENSIONS=<ext1,ext2...>  -e ... \
  karate-connect:latest <optional_params...>

How to customize ?

See karate documentation for anything related to Karate Core.

All existing extensions

  • base: common functions (uuid, time, …​)

  • rabbitmq: RabbitMQ integration

  • kafka: Kafka integration

  • snowflake: Snowflake integration

  • kubernetes: kubectl create job -from=cronjob/…​

  • dbt: dbt run …​

More? (coming soon)

  • redis

  • mongodb

  • jdbc (+ jar for JDBC driver)

  • your private extension (built by yourself) and loaded in karate-connect classpath

Snowflake extension - Example

Feature: SELECT V2
  Background:
    * json cliConfig = read('classpath:cli-config.json')
    * json snowflakeConfig = read('classpath:snowflake-config.json')
    * string jwt = snowflake.cli.generateJwt(cliConfig)
    * json restConfig = ({jwt, cliConfig, snowflakeConfig})

  Scenario: Select 1 cutter
    Given text statement =
    """
      SELECT SERIAL_NUMBER, CUTTER_TYPE
      FROM CUTTER
      WHERE SERIAL_NUMBER='MY_VECTOR'
    """
    And def response = snowflake.rest.runSql({...restConfig, statement})
    And table expectedData
      | SERIAL_NUMBER | CUTTER_TYPE |
      | "MY_VECTOR"   | "VECTOR"    |
    And match response.data == expectedData

Demo Time 🎬

meal-factory (streaming)

meal factory

burger-factory (batching)

burger factory

Next Steps 🚀

  • We need stars 🌟!

  • We need contributors 🙏!

🎉 Thank you - Questions?
🥋

Slides 🖥️

qrcode slides

Feedback🙏

qrcode bordeauxjug feedback