Getting Started with Timeplus Terraform Provider

HashiCorp Terraform is an Infrastructure-as-Code tool that lets you define both cloud and on-prem resources in human-readable configuration files. Today, we are excited to announce that Timeplus has joined the HashiCorp Technology Partner Program and published the new Timeplus Terraform Provider to HashiCorp Terraform Registry.

The new Timeplus Terraform Provider allows businesses to:

• Reduce complexity and risk with infrastructure managed as code and deployed through automated GitOps integration.
• Increase developer autonomy and productivity with consistent, version-controlled access to data streams, Apache Kafka sources, materialized views, dashboards, UDF, and more.

With this Provider, Timeplus customers at any scale can safely accelerate data streaming initiatives in the cloud (public, private, or hybrid), with infrastructure management that is fully automated through code and integrated within continuous delivery workflows.

Why not just use SQL script?

At its core, Timeplus is a SQL database that works well for both streaming and batch/OLAP. So, you may wonder why we developed this Terraform Provider for our users, instead of encouraging them to use a set of SQL statements, or SQL script, to create, update, or delete resources.

The short answer is that SQL script works well for Timeplus, while a Terraform-based workflow can work well for certain teams. They are complementary to each other.

A longer answer for why we built this Terraform Provider:

1. Terraform allows users to manage resources in a declarative way. As a data engineer or administrator, you describe the end state in the Terraform file, without handling the complexities of how to reach that end state. If using SQL or REST API instead, users have to figure out whether to create new resources, or change existing resources, or delete then recreate resources, etc.
2. Timeplus provides more than just streams and views. For example, it’s hard to imagine defining and updating dashboard definitions (which is in JSON format) with SQL/dbt scripts.
3. Regular SQL scripts don’t manage the dependencies for you. You have to write a set of CREATE.. IF NOT EXISTS, or DROP .. CREATE ... When you just want to make a small change to your SQL, you may end up recreating every resource and that can lead to data loss or inconsistency.
4. Data engineers and platform engineers are the main target users for Timeplus. In the modern software development process, it’s a common requirement to track changes in version control systems, with branching, code review, and CICD. With the Terraform-based change management process, engineers can create a branch, make code changes, and ask for code review. At the same time, Terraform can generate the deployment plan for the code maintainer and reviewer to better understand which resources will be created, updated, or deleted, in addition to reviewing the code change itself. If the reviewer is okay with the code change and deployment plan, they can approve the code and deploy the exact change to the target environment, without any guess work.

Which resources are available?

Resources are the most important element in the Terraform language. A Terraform resource describes one or more infrastructure objects, such as streams, sources, sinks, and long-running materialized views. You can manage the following Timeplus resources in Terraform:

• Streams: Timeplus streams are similar to tables in the traditional SQL databases. They can be append-only, changelog or versioned streams.
• Sources: Timeplus sources are processes run in background to collect data from specific data sources (such as Apache Kafka) and ingest them into streams.
• Views: Timeplus views are named queries. It is useful for wrapping a commonly used complex query as a view.
• Materialized Views: Timeplus materialized views are special views that persist its data. Once created, a materialized view will keep running in the background and continuously writes the query results to the underlying storage system.
• Sinks: Timeplus sinks run queries in background and continuously send query results to the target system.
• Alerts: Alerts are like sinks, used to send data to external systems. The difference is that alerts have two statuses: ‘triggered’ and ‘resolved’.
• Dashboards: A dashboard is a set of one or more panels organized and arranged in one web page. A variety of panels are supported to make it easy to construct visualization components, allowing you to create dashboards for specific monitoring and analytics needs.
• Remote UDF: Timeplus remote functions are one of the supported user defined function types. Remote functions allow users to register a HTTP webhook as a function which can be called in queries.
• JavaScript UDF: Timeplus JavaScript functions are one of the supported user defined function types. JavaScript functions allow users to implement functions with the javascript programming language, and be called in queries.

Here’s an example of a Terraform script to define a stream and a Kafka source:

resource "timeplus_stream" "example" {
  name = "example"
  column {
    name = "col_1"
    type = "int32"
  }
  column {
    name    = "col_2"
    type    = "datetime64(3)"
    default = "now()"
  }
}

resource "timeplus_source" "kafka_example" {
  name        = "kafka example"
  description = "A source example connects to a locally deployed insecure kafka cluster"
  stream      = timeplus_stream.example.name
  type        = "kafka"
  properties = jsonencode({
    brokers = "127.0.0.1:19092"
    topic   = "some-topic"
    offset  = "latest"
    tls = {
      disable = true
    }
    data_type = "json"
  })
}

Getting started

Now that you’ve learned a bit about the Timeplus Terraform provider, let’s get started and put it to work.

Prerequisites

Before you start, make sure you have the following:

1. A Timeplus Cloud account. If you don’t have a Timeplus Cloud account, create one for free now.
2. Terraform (1.0+) installed.
3. Golang (1.20.0+) installed.

Create an API Key

You need to create an API key to manage Timeplus resources without UI. To do so:

1. Click your user avatar on the top-right corner.
2. In the dropdown, choose Personal Settings.
3. Click the Create API Key button.
4. Set an optional description and choose an expiration date.
5. Save the API key securely in your computer. You are not going to retrieve the plain text key again in the console.

Set up Terraform configuration

To use the provider, simply add it to your terraform file, for example:

terraform {
  required_providers {
    timeplus = {
      source  = "timeplus-io/timeplus"
      version = ">= 0.1.2"
    }
  }
}

provider "timeplus" {
  # the workspace ID can be found in the URL https://us.timeplus.cloud/<my-workspace-id>
  workspace = "my-workspace-id"
  # API key is required to use the provider
  api_key   = "my-api-key"

Then, you can start provisioning Timeplus resources. Below is an example for stream:

resource "timeplus_stream" "example" {
    name = "example"
    description = "the example stream from the provider README file"
    column {
      name = "col_1"
      type = "string"
    }
    column {
      name = "col_2"
      type = "int64"
    }
}

Follow the Terraform documentations to build, change, and remove resources, for example:

• terraform init to download the plugin
• terraform apply to review and approve the changes
• terraform destroy to delete the resources

Give the new Timeplus Terraform Provider a try and let us know what you think! You can open tickets on https://github.com/timeplus-io/terraform-provider-timeplus or discuss with us in our community Slack.